In a project created with vite, I define a component with model such as:

<template>
  <input
    type="checkbox"
    v-bind:checked="checked"
    v-on:change="$emit('change', $event.target.checked)"
  />
</template>
<script>
export default {
  name: "BaseCheckbox"
  model: {
    prop: "checked",
    event: "change",
  },
  props: {
    checked: Boolean,
  },
};
</script>

and then I use v-model to bind checked for BaseCheckbox

<template>
  <BaseCheckbox v-model="checked" />
</template>

<script>
export default {
  data: () => ({checked: true})
}
</script>

It didn't work, and the value of checked props was false in BaseCheckbox.But v-model with native dom was running normal. Is this problem about vite, vue@next or compiler?

v-model api has changed in Vue 3 - https://github.com/vuejs/rfcs/blob/master/active-rfcs/0011-v-model-api-change.md

So this is an incompatible change? If I have a custom component base on Vue@2 and I want to use it in a project base on Vue@next, I need to refactor it at first?

I think you can make it work in both versions of Vue. You need to add this.$emit('update:modelValue, value) where you emitting change event, however you have to handle the modifiers by yourself by checking if modelModifiers prop exists

Please read RFCs for Vue core changes. This is not a vite issue.