What problem does this feature solve?

tag in <component :is="tag" /> could be custom element, even native tag, but these check step is skipped currently.

What does the proposed API look like?

Same as template compiling, ask isCustomElement first, then rest.

BTW: Do not forget to remove injectCustomElementCheck (for isRuntimeOnly case inside), because there could be runtime need.

If tag is a custom element, it will be treated as-is, what is your real intention? can you provide a repro?

resolveDynamicComponent will treat custom element tag as-is, only if the tag is not found in components.

This works right by isCustomElement API:

customElement.define('aa-bb', class extends HTMLElement {
    constructor () {
        super();
        this.innerHTML = 'element';
    }
});

const app = Vue.createApp({
    template: `<aa-bb /><AaBb />`,
    components: {
        AaBb: { template: `component` },
    },
});

app.config.isCustomElement = tag => /^[a-z]/.test(tag) && !/^(base-transition|component|keep-alive|slot|suspense|teleport|template|transition|transition-group)$/.test(tag);

app.mount('body');

But this won't:

customElement.define('aa-bb', class extends HTMLElement {
    constructor () {
        super();
        this.innerHTML = 'element';
    }
});

const app = Vue.createApp({
    template: `<component is="aa-bb" /><component is="AaBb" />`,
    components: {
        AaBb: { template: `component` },
    },
});

app.config.isCustomElement = tag => /^[a-z]/.test(tag) && !/^(base-transition|component|keep-alive|slot|suspense|teleport|template|transition|transition-group)$/.test(tag);

app.mount('body');

This is a wontfix. isCustomElement is a compiler-only concept in v3, so it should not and cannot affect resolveDynamicComponent's runtime behavior.

When using the option with build tools it is passed to vue-loader or @vitejs/plugin-vue so it cannot be even known at runtime. If we change the behavior app.config.compilerOptions.isCustomElement it makes the behavior inconsistent.

In practice, if you know what custom elements you will be using, it makes little sense to then register Vue components that have potential conflicting names.