# GitHub Demo Project

This blog post references this [demo project](https://github.com/Compulsed/cdk-lambda-typescript-otel-honeycomb). You can deploy this project to your AWS environment to see everything up and running.

![Example Trace](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/c5839fab-5ab1-43b0-9389-d4519bb9cde1-Screenshot%202022-12-11%20at%204.56.56%20PM.png)

After running the above project you should see a trace like the above.

# The Problem

**Why can I not just import and use @opentelemetry/api?**

You can. Though, the issue with this package is that it buffers traces in memory and then periodically pushes them to the telemetry service. This buffering creates issues with AWS lambda. This problem is because once the handler is run, the Lambda Service will [freeze](https://www.youtube.com/watch?v=yDplzXEdBTI) the container. Because the Lambda container is frozen, the traces may never end up at the telemetry service.

# The solution

AWS Attempted to solve this problem with Lambda [Extensions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html). These extensions run in the background as a sort of 'sidecar'. This sidecar can continue to run and push metrics to an OTEL service even after the hander has finished. 🎉

The AWS maintained extension is this [aws-observability/aws-otel-lambda](https://github.com/aws-observability/aws-otel-lambda). This extension embeds the work done here [open-telemetry/opentelemetry-lambda](https://github.com/open-telemetry/opentelemetry-lambda). Contained with the `aws-otel-lambda` package is a version of `@opentelemetry/api`, we will reference this package from within our lambda function.

## If only it were that simple

Getting the above solution working with the TypeScript CDK is difficult, to say the least. It requires a bunch of changes to the default CDK project.

### 1. Setting up the layer

Below is the baseline configuration you will need to get the layer working/deployed. The lambda extension uses the `collector.yaml` to determine how it should be set up. The `commandHooks` instructs ESBuild to include the file lambda zip.

*File: lib/hello-world.ts*
~~~ts
// From https://github.com/aws-observability/aws-otel-lambda
layers: [
  lambda.LayerVersion.fromLayerVersionArn(
    this,
    'otel-layer',
    'arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-arm64-ver-1-7-0:2' // Replace the region based on where the lambda is deployed
  ),
]
~~~

*File: lib/hello-world.ts*
~~~ts
environment: {
  AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
  OPENTELEMETRY_COLLECTOR_CONFIG_FILE: '/var/task/collector.yaml',
  OTEL_PROPAGATORS: 'tracecontext',
  OTEL_SERVICE_NAME: 'demo-example-service',
}
~~~

*File: lib/hello-world.ts*
~~~ts
commandHooks: {
  beforeBundling(inputDir: string, outputDir: string): string[] {
    return [`cp ${inputDir}/collector.yaml ${outputDir}`]
  },
  afterBundling(): string[] {
    return []
  },
  beforeInstall() {
    return []
  },
}
~~~

*File: collector.yaml*
~~~yaml
receivers:
  otlp:
    protocols:
      grpc:
      http: 

exporters:
  otlp/traces:
    endpoint: api.honeycomb.io:443
    headers:
      'x-honeycomb-team': xxx

  logging:
    loglevel: debug

service:
  extensions: []
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlp/traces]
~~~

### 2. Lack of support for ESM Modules for handler exports

Normally you would export a lambda function with `const export handler = () => {}`. This is not possible due to [open-telemetry/opentelemetry-js/issues/1946](https://github.com/open-telemetry/opentelemetry-js/issues/1946). This appears to be an issue with the way that the auto instrumentation wants to monkeypatch the handler. The way we can get around this is to simply export it via module exports.

*File: lib/hello-world.function.ts*
~~~ts
module.exports = { handler }
~~~ 

### 3. Referencing the NodeJS OTEL library in the layer / Lack of support for ESM Modules for @opentelemtry imports

Normally you would be able to just import `'@opentelemetry/api'` by running `import { trace, SpanStatusCode }  from '@opentelemetry/api'`. This appears to be an issue with the lambda layer. The solution to this is just using `require`, then import the types.

*File: lib/hello-world.function.ts*
~~~ts
const { trace, SpanStatusCode } = require('@opentelemetry/api')
import type { Span, SpanOptions } from '@opentelemetry/api'
~~~

Also, because we want to reference the `@opentelemetry/api` from our layer we need to exclude it from the bundling step.

*File: lib/hello-world.ts*
~~~ts
externalModules: [
   '@opentelemetry/api',
   '@opentelemetry/sdk-node',
   '@opentelemetry/auto-instrumentations-node',
]
~~~

### 4. Root Span & XRay Tracing

AWS has native XRay integration with Lambda. This means that when your function gets invoked, it has X-Ray information from the Lambda service that is passed into it. This means that when we work with OTEL, it will already have a parent span (the lambda service). The problem is that span will not go to Honeycomb, and Honeycomb will think there is a missing span. To fix this issue we just say that our first span is our root span.

To fix this we need both of these options:

*File: lib/hello-world.ts*
~~~ts
tracing: lambda.Tracing.PASS_THROUGH
~~~

*File: lib/hello-world.function.ts*
~~~ts
tracer.startActiveSpan('handler', { root: true }, async (span: any) => {
   // your code here
}
~~~

### 5. ESBuild Bundling, Auto instrumentation

The TypeScript CDK uses ESBuild to compile and build your lambda function. ESBuild will attempt to pull all of your node module dependencies and source dependencies into a single file. This helps to improve lambda cold starts.

The problem with this bundling is that it messes with OTEL's [auto-instrumation](https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node). To get the auto-instrumentation working again, you need to specify each package in the `nodeModules` section of ESBuild. This option stops ESBuild from trying to bundle them together and just keeps those dependencies in the node_modules folder.

*lib/hello-world.ts*
~~~ts
nodeModules: [
  '@aws-sdk/client-dynamodb',
],
~~~

# Summary

AWS Lambda, Honeycomb, and OTel are great services. They are even better together. It took me a long time to put this together so let me know if it was helpful

Instrumenting a Typescript Lambda function with Honeycomb / CDK is hard. This guide will run you through how you configure your Lambda function so that all your traces / spans end up at Honecyomb.

Instrumenting a Typescript Lambda function with Honeycomb / CDK

Dale Salter

Software Engineering, Leadership, DevOps

🐝 Typescript CDK Lambda Function with OTEL / Honeycomb

Instrumenting a Typescript Lambda function with Honeycomb / CDK