Sentry in AWS Lambda (.NET)

A bit more “going beyond the published documentation of Sentry” — this time centred on using Sentry to monitor AWS Lambda functions written in .NET. This was one of those things which seemed like it would be easy, and turned out to be harder, so I’ve jotted some notes.

The Problem

As part of a CI / IAC environment, I want to run some provisioning and maintenance routines in AWS Lambda. As I hinted at in my previous article, I’m a C# developer by experience and inclination, so using the .NET platform seemed like the way to go.

Because this fits into a wider CI context, I wanted to link up with Sentry. We already have Sentry monitoring for the EC2 provisioning scripts (described here) so extending Sentry to cover the Lambda provisioning and maintenance functions seemed an obvious choice.

Going Beyond the Documentation

Initial signs were promising. Sentry’s documentation covers a number of .NET platforms, including AWS Lambda.

But, the documentation requires the Sentry.AspNetCore package. The tailored-for-lambda stuff all seems to hook into the AspNetCore IWebHostBuilder. That may be fine for running a .NET serverless web app, but not so much for a utility function.

So, we’re going to need to go down the route of vanilla sentry and fix any issues as we go.

Round 1 —First Attempt

We start out with a fairly basic lambda function — after all, all we’re trying to do is trigger a Sentry event to prove the concept. Taken straight from the documentation:

// The function handler that will be called for each Lambda event
var handler = (string input, ILambdaContext context) =>
{
    SentrySdk.Init(options =>
    {
        options.Dsn = "<YOUR_DSN_HERE>";
        // When debug is enabled, the Sentry client will emit detailed debugging information to the console.
        // This might be helpful, or might interfere with the normal operation of your application.
        // We enable it here for demonstration purposes when first trying Sentry.
        // You shouldn't do this in your applications unless you're troubleshooting issues with Sentry.
        options.Debug = true;
        // This option is recommended. It enables Sentry's "Release Health" feature.
        options.AutoSessionTracking = true;
        // This option will enable Sentry's tracing features. You still need to start transactions and spans.
        options.EnableTracing = true;
    });
    SentrySdk.CaptureMessage("Test");
    return input.ToUpper();
};

After deploying and running, we eagerly check the CloudWatch logging, and get the following:

Debug: Attempting to recover persisted session from file.
Debug: Persistence directory is not set, returning.
Error: Failed to resolve persistent installation ID.
System.IO.IOException: Read-only file system : '/var/task/Sentry/8DC035E8255874016E7D5C9F59EE228FE33F810C'
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at Sentry.GlobalSessionManager.TryGetPersistentInstallationId()

Hmm, ok. Plausible if Sentry is trying to persist stuff to the file system.

Round 2 —Removing Session Persistence

There’s no obvious documentation on what it’s trying to persist and under what circumstances. So let’s disable some stuff and see what happens — AutoSessionTracking sounds like a potential culprit…

// The function handler that will be called for each Lambda event
var handler = (string input, ILambdaContext context) =>
{
    SentrySdk.Init(options =>
    {
        options.Dsn = "<YOUR_DSN_HERE>";
        // When debug is enabled, the Sentry client will emit detailed debugging information to the console.
        // This might be helpful, or might interfere with the normal operation of your application.
        // We enable it here for demonstration purposes when first trying Sentry.
        // You shouldn't do this in your applications unless you're troubleshooting issues with Sentry.
        options.Debug = true;
        // Disable auto session tracking to remove dependency on file system
        options.AutoSessionTracking = false;
        // This option will enable Sentry's tracing features. You still need to start transactions and spans.
        options.EnableTracing = true;
    });
    SentrySdk.CaptureMessage("Test");
    return input.ToUpper();
};

Back to Cloudwatch, and see what we’ve got.

Info: Envelope queued up: '51839d12b74043c9be33d9f2dcca6b47'
Task timed out after 3.05 seconds

Well, Envelope queued up sounds positive. But still no message in Sentry.

Round 3 — Fixing the Lambda Timeout

Task timed out rather sounds like our Lambda function isn’t waiting long enough for Sentry to finish doing its thing. So let’s up the Lambda timeout to something higher than 3s. Let’s go for 1 minute.

No timeout this time, but no Sentry message either. Let’s dig a bit deeper into what the Sentry.AspNetCore / Lambda documentation seems to do differently.

Round 4 — Flushing the Queue

Our clue comes from this line in the Sentry Lambda documentation:

// Required in Serverless environments
o.FlushOnCompletedRequest = true;

This makes sense — in a serverless environment, the execution context doesn’t hang around for Sentry to finish off. So we’re going to need to do something to force Sentry to flush its event queue before the Lambda shuts down.

Again, no obvious documentation here, so time for a little poke around in the source code (which you can find here thanks to Sentry’s permissive licensing).

This configuration option just appears to force Sentry to call a Flush method on the IHub before the application terminates. So, we can probably do that manually. Let’s try calling SentrySdk.Flush();

Debug: Timeout when trying to flush queue.

A new timeout!

Round 4 — Sentry Timeout

So, it turns out the Flush() call isn’t blocking and doesn’t hold the application open. So, we can set a timeout, but have to do so by await-ing the FlushAsync(Timespan timeout) function.

Of course, this will need more recoding to make our Lambda function async.

SentrySdk.Init(options =>
{
    options.Dsn = "<YOUR_DSN_HERE";
    options.Debug = true;
    options.AutoSessionTracking = false;
    options.EnableTracing = true;
});
// The function handler that will be called for each Lambda event, modified to be async
var handler = async (string input, ILambdaContext context) =>
{
    try
    {
        SentrySdk.CaptureMessage("Test");
        // Simulate some asynchronous operation, if necessary
        // e.g., await Task.Delay(1000);
        // Your logic here...
    }
    catch (Exception ex)
    {
        SentrySdk.CaptureException(ex);
        // Handle exception or rethrow as needed
    }
    finally
    {
        // Ensure all Sentry events are flushed before the Lambda function ends
        await SentrySdk.FlushAsync(TimeSpan.FromSeconds(5));
    }
    return input.ToUpper();
};

We’re now setting a timeout of 5 seconds.

Success!!

Our Sentry event arrives!!

Next Steps

One crucial aspect of the proof of concept so far is, our Lambda function isn’t running in a VPC. The downside of this is obvious — our Lambda function has no access to private resources such as RDS databases.

The issue arising when connecting a Lambda function into a VPC is that it will (only) run in a private subnet with no internet access, and therefore no talking to Sentry. This will likely be the next challenge, and one I’ll probably write about.

The other question is how this Lambda function fits into our wider CI/CD environment. In reality, I was developing the CI/CD in parallel through a combination of Azure Pipelines to build the code and deploy to S3, and an OpenTofu tofu applyprocess orchestrated by Azure Pipelines to deploy successive versions of the Lambda function. Again, I may write more on this later.

Conclusion

So, to recap, to get Sentry running in a non-web-server (non-AspNetCore) .NET Lambda function we need the following:

• Disable AutoSessionTracking
• Flush the buffer asynchronously using SentrySdk.FlushAsync(Timespan); including modifying the Lambda to be an async function
• Set the timeouts on the Lambda function and Sentry Flush operations to give sufficient time for the message queue to be dispatched to Sentry.

This isn’t sufficient to get Sentry working in a VPC, but more on that later!