OpenTelemetry in Lambda environment doesn't export trace data #5923
Description
Apollo Package Details
Expected Behavior
- Running Apollo Server on Lambda should generate and allow instrumentation of OpenTelemetry trace data
- Using the AWS Distro for OpenTelemetry Lambda layer extension should auto-instrument Apollo Server Lambda
Actual Behavior
- Running the Apollo Server Lambda handler in a Lambda environment with the AWS Distro for OTel Lambda results in no trace data being exported
- Running the Apollo Server Lambda handler in a Lambda environment with the Apollo-provided code for instrumenting the Apollo Server also results in no trace data being exported
- Running the Apollo Server Lambda handler locally with Express does result in trace data being exported
- Running a Gateway on Apollo Server Lambda also does result in Gateway trace data being exported, but not any of the subgraphs
More Details
I've been attempting to use the OpenTelemetry trace data generated by Apollo Server in a deployed Lambda environment and I can't seem to get any trace data to be exported. My end goal is to use the AWS Distro for OpenTelemetry on Lambda to instrument and export traces from Apollo Server to AWS X-Ray tracing service. I chatted with one of the X-Ray engineers who worked on the Lambda extension, and he said that in theory, adding the extension should be all that is needed to auto-instrument the Lambda, as the HttpInstrumentation, ExpressInstrumentation, and GraphQLInstrumentation required by Apollo were all included in the extension layer.
Since that wasn't working, I instead attempted to use the open-telemetry.js file from the Apollo OTel docs. Running this locally on an Express server emulating API Gateway worked as expected, displaying traces in the console, but deploying it to AWS resulted in no traces being logged. Assuming this might be due to the Lambda returning the handler response and freezing execution before traces could be processed, I attempted to use the forceFlush functions on the providers and processors before returning the handler response, but that did nothing to change the result. Here's the code for that:
export const handler = async (event: any, context: any, callback: any) => {
const handler = server.createHandler();
const response = await handler(event, context, callback);
console.log("forcing provider flush");
await provider.forceFlush();
console.log("finished forcing provider flush");
console.log("forcing console processor flush");
await consoleProcessor.forceFlush();
console.log("finished console processor flush");
return response;
};One interesting caveat to this is that when I was trying OTel out with a federated graph running on Lambda (the gateway in one function and each subgraph in their own functions), I was actually able to see the gateway's traces in the CloudWatch logs as expected, but not any of the subgraphs. That simply confused me more, but it is something worth noting.
I've uploaded the sample code I was using for a basic Lambda-based Apollo Server with OpenTelemetry here: https://github.com/mkossoris/apollo-server-lambda-otel-issue. It includes the local Express server I mentioned as well as CDK code for deploying it to AWS.