GitHub Demo Project

This blog post references this demo project. You can deploy this project to your AWS environment to see everything up and running.

After running the above project you should see a trace like the above.

The Problem

Why can I not just import and use @opentelemetry/api?

You can. Though, the issue with this package is that it buffers traces in memory and then periodically pushes them to the telemetry service. This buffering creates issues with AWS lambda. This problem is because once the handler is run, the Lambda Service will freeze the container. Because the Lambda container is frozen, the traces may never end up at the telemetry service.

The solution

AWS Attempted to solve this problem with Lambda Extensions. These extensions run in the background as a sort of 'sidecar'. This sidecar can continue to run and push metrics to an OTEL service even after the hander has finished. 🎉

The AWS maintained extension is this aws-observability/aws-otel-lambda. This extension embeds the work done here open-telemetry/opentelemetry-lambda. Contained with the aws-otel-lambda package is a version of @opentelemetry/api, we will reference this package from within our lambda function.

If only it were that simple

Getting the above solution working with the TypeScript CDK is difficult, to say the least. It requires a bunch of changes to the default CDK project.

1. Setting up the layer

Below is the baseline configuration you will need to get the layer working/deployed. The lambda extension uses the collector.yaml to determine how it should be set up. The commandHooks instructs ESBuild to include the file lambda zip.

File: lib/hello-world.ts

1// From https://github.com/aws-observability/aws-otel-lambda
2layers: [
3  lambda.LayerVersion.fromLayerVersionArn(
4    this,
5    'otel-layer',
6    'arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-arm64-ver-1-7-0:2' // Replace the region based on where the lambda is deployed
7  ),
8]

File: lib/hello-world.ts

1environment: {
2  AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
3  OPENTELEMETRY_COLLECTOR_CONFIG_FILE: '/var/task/collector.yaml',
4  OTEL_PROPAGATORS: 'tracecontext',
5  OTEL_SERVICE_NAME: 'demo-example-service',
6}

File: lib/hello-world.ts

1commandHooks: {
2  beforeBundling(inputDir: string, outputDir: string): string[] {
3    return [`cp ${inputDir}/collector.yaml ${outputDir}`]
4  },
5  afterBundling(): string[] {
6    return []
7  },
8  beforeInstall() {
9    return []
10  },
11}

File: collector.yaml

1receivers:
2  otlp:
3    protocols:
4      grpc:
5      http: 
6
7exporters:
8  otlp/traces:
9    endpoint: api.honeycomb.io:443
10    headers:
11      'x-honeycomb-team': xxx
12
13  logging:
14    loglevel: debug
15
16service:
17  extensions: []
18  pipelines:
19    traces:
20      receivers: [otlp]
21      processors: []
22      exporters: [otlp/traces]

2. Lack of support for ESM Modules for handler exports

Normally you would export a lambda function with const export handler = () => {}. This is not possible due to open-telemetry/opentelemetry-js/issues/1946. This appears to be an issue with the way that the auto instrumentation wants to monkeypatch the handler. The way we can get around this is to simply export it via module exports.

File: lib/hello-world.function.ts

1module.exports = { handler }

3. Referencing the NodeJS OTEL library in the layer / Lack of support for ESM Modules for @opentelemtry imports

Normally you would be able to just import '@opentelemetry/api' by running import { trace, SpanStatusCode } from '@opentelemetry/api'. This appears to be an issue with the lambda layer. The solution to this is just using require, then import the types.

File: lib/hello-world.function.ts

1const { trace, SpanStatusCode } = require('@opentelemetry/api')
2import type { Span, SpanOptions } from '@opentelemetry/api'

Also, because we want to reference the @opentelemetry/api from our layer we need to exclude it from the bundling step.

File: lib/hello-world.ts

1externalModules: [
2   '@opentelemetry/api',
3   '@opentelemetry/sdk-node',
4   '@opentelemetry/auto-instrumentations-node',
5]

4. Root Span & XRay Tracing

AWS has native XRay integration with Lambda. This means that when your function gets invoked, it has X-Ray information from the Lambda service that is passed into it. This means that when we work with OTEL, it will already have a parent span (the lambda service). The problem is that span will not go to Honeycomb, and Honeycomb will think there is a missing span. To fix this issue we just say that our first span is our root span.

To fix this we need both of these options:

File: lib/hello-world.ts

1tracing: lambda.Tracing.PASS_THROUGH

File: lib/hello-world.function.ts

1tracer.startActiveSpan('handler', { root: true }, async (span: any) => {
2   // your code here
3}

5. ESBuild Bundling, Auto instrumentation

The TypeScript CDK uses ESBuild to compile and build your lambda function. ESBuild will attempt to pull all of your node module dependencies and source dependencies into a single file. This helps to improve lambda cold starts.

The problem with this bundling is that it messes with OTEL's auto-instrumation. To get the auto-instrumentation working again, you need to specify each package in the nodeModules section of ESBuild. This option stops ESBuild from trying to bundle them together and just keeps those dependencies in the node_modules folder.

lib/hello-world.ts

1nodeModules: [
2  '@aws-sdk/client-dynamodb',
3],

Summary

AWS Lambda, Honeycomb, and OTel are great services. They are even better together. It took me a long time to put this together so let me know if it was helpful