[Enhancement] TypeScript binding generation for third-party npm packages (e.g. @couchbase/lite-js)

[wisdomaj]

Summary

I'd like to open a discussion about whether web_generator's TypeScript-input path is intended to support generating dart:js_interop bindings for arbitrary npm packages, or whether it's primarily an internal tool for producing package:web from Web IDL. The README mentions both IDL and TS modes, and I wanted to share my experience using the TS path on a real-world library and ask about its intended scope.

Based on the recent work in PR #548 (bootstrapping the generator against VS Code's TypeScript types), it looks like my motivation for contributing and generating dart:js_interop bindings for a real-world npm package's TypeScript types may already overlap with what's being pursued there.

I opened PR #525 in an attempt to fix one of the several issues I encountered (but I have been really busy recently and this PR has become pretty stale and quite behind the refactoring recently merged to main).

Context

I'm building a Flutter application that targets both mobile and web. For the mobile/desktop builds I'm using cbl-dart (FFI bindings to libcblite), which works well. For the web build, cbl-dart doesn't support the web platform, so I need to call @couchbase/lite-js, the official Couchbase Lite JavaScript SDK for the browser, from Dart via dart:js_interop.

@couchbase/lite-js is a sizable TypeScript API surface (Database, Collection, Document, MutableDocument, Query, Replicator, change listeners, etc.), so hand-writing every binding is tedious and error-prone. Generation seemed like the natural path.

Why this matters

My motivating case is my Flutter web build that needs to call @couchbase/lite-js from Dart. The question I have is, should web_generator's TS path produce dart:js_interop bindings for arbitrary npm packages? The same scope question would apply to Node-targeted or shared npm packages. A few concrete categories I can speak to from the browser side:

• Browser-based database / sync libraries that have a JS implementation but no Dart port (such as Couchbase Lite), typically used in offline-first PWAs.
• SDKs for cloud services distributed only as npm packages.
• Specialized browser libraries (charting, mapping, ML inference, audio, WebRTC helpers) where rewriting in Dart may not be practical.

To my knowledge, the practical options for generating bindings for these are:

1. Hand-write dart:js_interop declarations. This works, but doesn't scale to larger libraries and can drift over time.
2. The typings pub package.
3. web_generator's TS path.

Of those, web_generator seems like the most robust and well supported option, which is why I started here.

What I tried

I cloned dart-lang/web, ran web_generator's TS-input path against the @couchbase/lite-js type definitions, and worked through the output.

Concrete reproduction, so this can be re-run against current/future changes:

# Install the target npm package in an isolated directory
mkdir -p /tmp/cbl-js-repro && cd /tmp/cbl-js-repro
npm init -y >/dev/null
npm install @couchbase/lite-js

A custom tsconfig.json is required, because the package's .d.ts files use ES2022 standard types (Map, Set, Symbol, Promise), #private; syntax, ESNext Disposable/AsyncDisposable, and unresolved @/... path aliases:

{
  "compilerOptions": {
    "target": "es2022",
    "lib": ["es2022", "dom", "esnext.disposable"],
    "module": "esnext",
    "moduleResolution": "bundler",
    "baseUrl": ".",
    "paths": {
      "@/*": ["node_modules/@couchbase/lite-js/dist/*"]
    },
    "skipLibCheck": true,
    "declaration": true
  },
  "include": ["node_modules/@couchbase/lite-js/dist/**/*.d.ts"]
}

# Run js_interop_gen's TS path at the package's main .d.ts
dart /path/to/dart-lang-web/js_interop_gen/bin/js_interop_gen.dart \
  /tmp/cbl-js-repro/node_modules/@couchbase/lite-js/dist/couchbase-lite.d.ts \
  --output=/tmp/cbl-bindings.dart \
  --ts-config=/tmp/cbl-js-repro/tsconfig.json

(Adjust paths to your checkout. The exact .d.ts entry point may vary by package version. package.json's "exports"."types" or "types" field is the source of truth.)

Issues encountered

Issue 1: Infinite recursion during generation

The most blocking issue I hit was infinite recursion during generation on certain TypeScript constructs in the lite-js typings. I opened #525 with an attempted fix.

Since I've come back to work on this PR, I noticed that PR #548 (@kevmoo's VS Code interop bootstrap) bundles fixes for several recursion / resolution bugs that I also hit, including this one. Some of the issues below may be partially or fully addressed there.

Some history worth surfacing here: my initial commit on PR #525 used essentially the same approach as #548's recursion fix at commit 9d831f9 ("prevent infinite recursion crashes on mutually recursive types"), adding a single nodeMap.add(outputType); that registers the in-progress class in the global nodeMap directly. The Gemini automated reviewer flagged that first commit for "global symbol pollution... types resolved to underlying class names instead of intended typedefs," which motivated #525's current redesign (which was force-pushed afterwards): a scoped _pendingTypes map populated around the class member loop and cleared in finally, with _searchForDeclRecursive consulting it before re-entering transformAndReturn. Since #548's fix is structurally the same shape as that initial commit, the Gemini reviewer concern may apply there too.

Status against couchbase-lite + PR #548: running #548's branch against @couchbase/lite-js (using the reproduction and tsconfig.json above), the stack overflow no longer reproduces, so the recursion fix appears sufficient for couchbase-lite's cycle patterns. However, generation still fails end-to-end on a different bug (see Issue 4 below). No output file is produced.

Issue 2: Recursive generic constraints

Types like Database<S extends SchemaLike<S>>, where SchemaLike transitively references itself, cause a different stack overflow than #525, this time inside TypeScript's own type parameter resolution (via getFullyQualifiedName then symbolToString then infinite type expansion) rather than in the generator's declaration lookup. Workable around by walking the symbol's parent chain locally instead of calling into TS's qualified-name machinery.

Status against couchbase-lite + PR #548: not surfaced in this run. Couchbase-lite either doesn't exercise this constraint pattern, or generation crashes earlier (on the typedef merger) before reaching it. The bug surface still exists, just not on this input.

Issue 3: JS undefined leaking through dart:js_interop extension type getters

In lib/src/js/typescript.types.dart, an external T? get foo declaration on an extension type returns a JSUndefined sentinel for absent JS properties rather than Dart null. As a result, == null returns false, pattern matches like case final t? bind to the undefined sentinel, and the next cast to JSObject crashes with JSNull is not a subtype of JSObject. The visible trigger is #private; class field syntax (the one .d.ts shape that legally produces a PropertyDeclaration with no .type annotation), but every nullable getter in the bindings has the same latent leak. The robust fix is to normalize JS undefined to Dart null once at the binding boundary rather than at each call site.

Status against couchbase-lite + PR #548: not surfaced as a crash in this run, even though couchbase-lite uses #private; heavily. With target: es2022 in the tsconfig and #548's other fixes, the generator processes those declarations without hitting the JSNull is not a subtype of JSObject path. The latent leak almost certainly still exists for any nullable getter in the bindings, but is not blocking couchbase-lite generation today.

Issue 4: Cross-file declaration merging

How TypeScript stitches together interface Array<T> across lib.es5.d.ts and successive lib.es2015.*.d.ts files looks like a separate adjacent class of recursion/identity problem from #525, as @nikeokoronkwo flagged in the PR comments. Supporting the TS stdlib as input would presumably require handling it.

Status against couchbase-lite + PR #548: this is what currently blocks generation end-to-end. With #548 applied, the generator now crashes on Assertion failed: "Typedefs in TS do not allow other decls" at js_interop_gen/lib/src/ast/merger.dart:123. Couchbase-lite combines a typedef plus an interface (or namespace) declaration for several names (such as CBLDocument, CBLArray, CBLDictionary, CBLValue) via TypeScript declaration merging, and the current merger refuses to combine them. No output file is produced. Same class of issue as the TS stdlib case @nikeokoronkwo flagged, just surfaced from a real npm package.

Questions

I'm not sure whether supporting arbitrary npm packages is in scope for this repo and if this repo is meant to be used by other developers or primarily as a internal tool for the flutter team.

If the answer is "in scope" I'm happy to contribute further if needed. I'm also wondering what you would like to do with the existing PR #525 PR since it looks like you may have tackled some of the same issues in PR #548 and my PR is pretty stale.

Thanks for the work on this project and pushing flutter forward!

[kevmoo]

@wisdomaj – we're totally on this. We need to work slowly through a LOT of missing features, but that's the PR I shared - #549

We have limited engineering bandwidth in the Dart team to crank through here.

My current thought is to get #549 landed and then to start picking off the other bits!

[wisdomaj]

Awesome! I dind't realize that you had picked up this work before I wrote this issue. You guys read my mind 😃 Excited to see where this goes.

[nikeokoronkwo]

Hey @wisdomaj.

I'd like to open a discussion about whether web_generator's TypeScript-input path is intended to support generating dart:js_interop bindings for arbitrary npm packages, or whether it's primarily an internal tool for producing package:web from Web IDL.

The package js_interop_generator is meant to be able to do both, but it was initially designed to generate dart:js_interop bindings from Web IDL files. In order to be able to (in the future) make use of the same compilation output generation, we also extended the package to support running with arbitrary .d.ts files and subsequently NPM packages as well.

I'm not sure whether supporting arbitrary npm packages is in scope for this repo and if this repo is meant to be used by other developers or primarily as a internal tool for the flutter team.

• Having said the above, supporting NPM packages is in scope and was part of the inspiration that led to the development of this package.
• We are discussing publishing this package in the future once it reaches a stable format and point in its development (see web_generator: Publish a pre-release on pub.dev? #475) so hopefully others can use it too (similar to packages like package:ffigen maybe)

I'm hoping to continue to work on this package at the moment, starting with the issue @kevmoo referenced.

[wisdomaj]

@nikeokoronkwo thanks for that context. This is great. I think being able to easily interop with NPM packages on web would be huge for the flutter community since there isn't always a mature or well supported alternative package in dart.

I'll keep following this work and will try to contribute as time allows (although things are pretty chaotic and busy for me at the moment). Thanks!