Member-only story
A DEBUGGING JOURNEY OF A TIMEOUT BUG
How MongoDB’s CSOT Kills KMS Requests in AWS Lambda
A debugging story that starts with “the network is fine” and ends deep inside the MongoDB driver’s timeout propagation.
We shipped Client-Side Field Level Encryption (CSFLE) to our staging backend on a Friday. The change was small: a few fields marked as encrypted, a ClientEncryption instance wired up against AWS KMS, and a Data Encryption Key created on first boot. It worked fine on every developer laptop. It worked fine in the integration test suite.
The moment it hit our Lambda in the private VPC, every cold start started logging the same thing:
CSFLE initialisation failed, running without field encryption.
MongoOperationTimeoutError: KMS request timed outThe fix turned out to be a single line: remove timeoutMS from the Mongoose connection options.
That is a weird fix, and it is an even weirder bug. This article is the story of how we got there, because it took the better part of a day and two wrong theories to figure out that a perfectly reasonable-looking timeout setting was silently starving our KMS calls of the time they needed to complete.
