Handling Server Initialisation Errors in CI/CD & Terraform with Sentry

I should probably set the scene here. If you held a gun to my head and told me to write code, I’d reach for Visual Studio and write code in C#. But, as a software architect, I have to step out of my comfort zone so that I can continue to meaningfully input into the work my wider team’s doing. So, the current excursion out of my comfort zone is using Terraform / OpenTofu to provision infrastructure in AWS.

The Problem

Terraform allows me to create an AWS EC2 instance fairly easily. I can then set the user_data property to initialise the instance with a bash script. So far so good. But, how do I know it succeeded?

The answer is, I don’t (or at least, terraform doesn’t).

Whilst this lack of visibility in dev is irritating, I can log into the AWS Console and take a look at the instance system log. However, in a CI/CD environment, it’s far more critical — if my initialisation script fails on a live deployment, I’m risking my customers finding out before I do.

So, I need to close the loop and have a reliable way to know if my server initialisation failed.

We already use Sentry for application observability, so I wondered if I could incorporate Sentry into this workflow.

Sentry CLI

My starting point was this article from Sentry’s Armin Ronacher. To quote:

One feature that is not nearly as well known as the rest: sentry-cli can be used to send events to Sentry.

Setting this up is fairly straightforward:

1. Install sentry-cli using an installation script via curl to bash. Note that you don’t need (and shouldn’t use) elevated privileges here because the script obtained from get-cli will sudoas required.

curl -sL https://sentry.io/get-cli/ | bash

2. Export an environment variable to configure the SENTRY_DSN

export SENTRY_DSN=<your-dsn-goes-here>

Sentry is now configured and ready to go.

Automating Error Reporting

Sentry’s biggest asset is automating the error you didn’t anticipate as well as the errors you’ve handled, and whilst we can send manual events and breadcrumbs from bash, being able to automate is by far and away the most useful feature.

We can do this by adding the following line:

eval "$(sentry-cli bash-hook)"

This hooks error handling from the script to report an issue / exception to Sentry, and terminates the script using set -e . Whilst there are limitations, this is massively useful in giving visibility on what deployment scripts failed and why.

Putting it Together

One of my areas of research interest is around automation using CI/CD tooling. I’ve been a huge fan of (and early adopter of) what’s now Azure DevOps since its early days as a hosted service. So automation in Azure Pipelines is somewhere I spend a fair bit of time implementing application build, test, deployment, vulnerability scanning and more. (And yes, I’m aware of the irony of using Azure to deploy to AWS!)

So the overall workflow looks something like this (with steps around code review and approval omitted for brevity):

1. Change to the infrastructure-as-code (terraform) pushed to Azure Repositories.
2. Release pipeline triggered to run terraform commands (actually, OpenTofu, but who’s counting?!) tofu init and tofu apply
3. Errors running the pipeline reported back in Azure DevOps Boards as a new bug for a failing deployment.
4. EC2 instances initialised by pushing an initialisation script in the user_data variable.
5. Errors in the initialisation script trigger a Sentry issue.
6. Sentry issue creates a corresponding Azure DevOps Boards bug.

One More Thing

Whilst this works successfully, most applications will have multiple environments. So it would be good to distinguish between these within a single project.

There’s no obvious configuration for this, but the good people at Sentry support have alerted me to an environment variable missing from the docs:

export SENTRY_ENVIRONMENT=your_env_name

This may be hardcoded in your script, substituted in from a terraform variable or even passed through from your CI tool, allowing filtering by environment within Sentry.

The Final Script

The final script looks a bit like this (using terraform locals for the dsn and environment:

#!/bin/bash

# Set up Sentry for error handling
curl -sL https://sentry.io/get-cli/ | bash
export SENTRY_DSN=${local.saas_manager_deploy_sentry_dsn}
export SENTRY_ENVIRONMENT=${local.sentry_environment}
eval "$(sentry-cli bash-hook)"

So that’s it! Sentry reports our deployment errors.