Repository navigation
feat: Add a MySQL table prefix and fix PlanetScale's findings - #60
Merged
Merged
Conversation
…status read-only Four findings from PlanetScale's testing of the sync against Vitess, all in the command line and engine every destination shares. reconcile exited 0 when a stream's diff threw. The stream was reported with `complete: false`, but only drift changed the exit code, so a run in which every stream was refused by the database passed as a scheduled check. A failed stream now carries `failed: true` and the command exits 1. A stream a page budget left for a later slice is still exit 0. Output through a pipe was cut off at 64 KB. `exitWith` called `process.exit` straight after writing, and a pipe is written asynchronously on macOS: `init --print-sql | cat` wrote 65,536 of 154,588 bytes and exited 0. The exit now waits for stdout and stderr to drain. A CA variable that was set but empty was read as unset, which is what a mistyped `$(cat path)` leaves behind. It is now refused by name. status applied the schema before reading, so a status check on an empty database created every table and view. A destination can now implement `ready()`, and status asks it instead: when the tables are not there it reports `initialized: false` and creates nothing. Destinations without `ready()` behave as before.
…tion before listing
Two findings from PlanetScale's testing against Vitess.
With a CA set, the certificate chain was verified but the host name was
not. mysql2 checks it only when `ssl.verifyIdentity` is on, and the
destination passed `{ ca, rejectUnauthorized: true }`. With the operating
system's bundle as the CA, any publicly trusted certificate passed for any
host. `sslCa` now turns the check on, and so does an `ssl={...}` object in
the connection string unless it turns verification off or sets
`verifyIdentity` itself. The mysql2 floor moves to 3.24.6, the version this
was verified against.
reconcile held one transaction open for a stream's whole listing. PlanetScale
ends a transaction after 20 seconds, so a stream that lists for longer could
never be reconciled there. Only the capture of the live ids needs the
snapshot: the diff now runs on a pinned connection, commits after the
capture, and lists with no transaction open. A pinned connection's
`transaction()` begins its own instead of taking a savepoint outside one.
…re it reads `init --repair` did nothing once the schema fingerprint was recorded: the fingerprint check returned before any view or column was looked at, so a dropped `_live` view or generated column stayed missing while the command printed `schema ready`. A repair now forces the whole pass on the destination's first `init()`, and a forced pass that finds drift it cannot repair takes the fingerprint back. Plain `init` keeps the one-query path. Both SQL destinations implement `ready()`, so `status` no longer creates the mirror on an empty database.
The mirror's tables have plain names: `users`, `organizations`, `sessions`, `roles`, `events`. On plain MySQL the mirror has a database to itself and that is fine. A PlanetScale Vitess branch is one database, the application's tables are in it, and most applications already have a `users`: PlanetScale found that `init` takes that table for its own, tries to add generated columns to it, and fails with `Unknown column '_raw'`. `tablePrefix` (`--table-prefix`, `WORKOS_SYNC_TABLE_PREFIX`) puts a prefix on every object the engine names in the database: stream tables, `_live` views, the bookkeeping tables, the event archive and the temporary tables. With `workos_`, `users` becomes `workos_users`. Lowercase letters, digits and underscores, up to 24 characters. The prefix travels with the database name as one `Target`, in the place the name alone was passed, so no statement can name the database and forget the prefix. Without one, every name, every statement and the schema fingerprint are what they were, so an existing mirror runs no DDL.
PlanetScale ran the guide against Postgres and Vitess and reported where it did not match what they saw. The guide now says: - Vitess TLS works with the `ssl` parameter PlanetScale's connection string ends with; the CA variable is the alternative, not a requirement. - With safe migrations on, the schema goes in from a development branch as two deploy requests, because one holding all 17 views fails the lint. - Install the package before `npx`, set `WORKOS_ENVIRONMENT_NAME`, and use the host shown with the password rather than a host shape we guessed. - Which table names the mirror takes in a Vitess database, how to prefix them, and that the keyspace has to be unsharded. - What a new branch and a restored branch each start with. - How `reconcile` behaves under PlanetScale's 20-second transaction limit, on 0.1.0 and after. The package READMEs follow the behavior changes in this branch: the reconcile exit code, read-only `status`, `init --repair`, host name verification, and the table prefix. The MySQL README no longer describes a single deploy request, or an `init` that names a missing column.
[High impact] The PR appears safe to merge. SummaryAdds MySQL table prefixes and improves TLS checks, command exits, schema repair, and PlanetScale guidance.
No new actionable issues found in the changes since the previous review. Reviews (5) · Last reviewed commit: "fix(mysql): refuse verified TLS to IP ho..." · Reviewed by Greptile |
…alled Every doc ran the command as `npx workos-sync-postgres`. That is the name of the command a package installs, not of a package. With nothing installed, npx asks the registry for a package by that name, and there is none, so the first command in each guide failed with a 404. The unscoped names are also unowned, so a package published under one of them would run in place of ours. `npx @workos/sync-postgres` names the package, which has one command, so npx runs it. Installed, it runs the version in the project. The same change is made for the MySQL, MongoDB and ClickHouse commands.
…uilds a column A repair forgot every other fingerprint before its first DDL statement, because MySQL commits each statement on its own and a repair that dies between dropping a column and adding it back has already changed the table. That was enough while a pass only ran with the current fingerprint unrecorded. `init --repair` now forces the pass with it recorded. A process that lost its connection after the drop left that fingerprint behind, so the next plain `init` trusted it and the column stayed missing. Every fingerprint is forgotten now, and the current one is recorded again when the pass comes back clean.
…s one The shared help said `status` reads only, for every destination's command. That holds for Postgres and MySQL, which implement `ready()`. MongoDB and ClickHouse do not, so their `status` still runs `init` first, and their help told a user it would not touch the database. The line now follows the spec. A destination that answers `ready()` sets `statusReadsOnly`, and the others say that the mirror is set up first.
The repo ships four destinations now, so the title no longer names only Postgres.
mysql2 verifies localhost for IP-address URLs, rejecting a valid IP certificate while accepting one for the wrong identity. Require a DNS hostname until the driver can verify the actual address, so credentials cannot be sent under a false identity-verification guarantee.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@workos/sync-mysqlverified the certificate chain but not the host name: mysql2 checks it only whenverifyIdentityis on. With the operating system's bundle as the CA, any publicly trusted certificate passed for any host.sslCaand anssl={...}object in the connection string now both turn the check on. The mysql2 floor moves to 3.24.6, the version this was verified against.reconcileexits 1 when a stream could not be compared, where it exited 0. Output is flushed before the process exits, soinit --print-sqlthrough a pipe is no longer cut at 64 KB.statusonly reads, and reportsinitialized: falseon an empty database instead of creating the mirror. A CA variable that is set but empty is refused.init --repair. It stopped at the recorded fingerprint like plaininit, so a dropped view or column stayed missing while it printedschema ready. A repair now forces the whole pass, in both SQL packages.tablePrefix,--table-prefixandWORKOS_SYNC_TABLE_PREFIXfor@workos/sync-mysql, so the mirror can share a Vitess database with an application that already has auserstable. Without a prefix, every name, statement and fingerprint is unchanged.The PlanetScale guide is corrected for D1 to D9 and L2, and the package READMEs follow the behavior changes.
Behavior changes
reconcileexits 1 when any stream failed, in every destination's command. A stream that a page budget left for a later slice is still exit 0.statusno longer applies the schema on Postgres and MySQL, andEnvironmentStatusgainsinitialized. MongoDB and ClickHouse do not implementready()yet, so theirstatusbehaves as before.localhostinstead of the target IP. The URL opt-out is"verifyIdentity":falsein itssslobject, only whenWORKOS_SYNC_MYSQL_SSL_CAis unset; an explicit CA always requires verification. A pre-built pool carries its own TLS settings.init --repairruns the full schema pass even when the fingerprint is recorded. A forced pass that finds drift it cannot repair removes the recorded fingerprint.