DevOps | Dilex Networks

Creating and AWS CI/CD Environment with GitHub Actions

Sun, 29 Sep 2024 00:00:00 +0000

Overview

Getting a development environment set up on AWS for the first time can be a bit overwhelming. Working in the cloud has a different mentality than doing everything on-prem like the idea that you need to “treat your servers like cattle, not like pets”¹ which basically means you want to be able to build (and rebuild) your entire architecture without being able to access the servers. Cloud native concepts and services can take your architecture to a new level but it does require a lot of diligence and testing (and retesting) to get it all right. Infrastructure as Code(IaC) concepts are not just your friend but are absolutely required to fully realize the power of the cloud but IaC is often not as quite straight forward as it initially seems. This article runs through some of the common approaches used to build a continuous integration and deployment environment (often called a CI/CD pipeline) on AWS and gives examples of how to create a seamless, automated system to build production, testing/review(Q.A.), and development environments on AWS. In other words, we are going to build a system, using IaC that can be used to build/test/review and deploy production systems that are build with IaC.

CI/CD Pipelines

Using GitHub Actions is a great way to create a CI/CD pipeline to manage your AWS environment. Not only can you use this setup to manage your AWS applications and workloads, you can also use it to manage your AWS infrastructure using any number of Infrastructure as Code (IaC) applications such as Terraform, AWS CloudFormation or AWS CDK. The idea here is that you can write your IaC code and manage it on GitHub, which means that you can use pull requests, code review sessions and also track how your infrastructure has changed over time by reviewing your pull request notes, who made changes and maybe more importantly, who approved changes. Granted, while you CAN (and probably will) run IaC jobs from your local computer at times, the advantage of using GitHub Actions (or another build system) is not only the idea that you can automate your build process, run test suites and automatically route deployments to the correct environments (dev/test/prod, etc.) but it also makes sure that these deployments are built in a consistent and repeatable way. GitHub Actions relies on docker containers to provide a consistent environment for building and deploying. This takes all of your local system dependencies out of your workflow - you don’t need to worry about what operating system (and versions of that os) the engineering team is using. Again, the whole point is to make deployments as simple as possible.

AWS Organizations

Exciting stuff right? Before you just dive in and start building stuff, you should think about what your development environment is going to look like. How will the engineering team make changes without affecting the production environment? How will code be tested and reviewed? A really nice way to handle this on AWS is to enable AWS organizations and then create separate accounts for each environment. Here’s a simple layout with three engineering accounts, a couple of Q.A. accounts and a single production account:

---
markmap:
  zoom: false
  pan: false
---
# AWS Org Root Account
  - Development
    - Software Engineer 1
    - Software Engineer 2
    - Designer
  - Q.A.
    - Functionality Testing
    - Product Team Review
  - Production
    - Production Server

Using AWS Organizations to manage your environment is considered an AWS best practice for a few reasons:

Security

This setup is more secure as we can control who has access to each account. Ideally, access to production and testing accounts should be fairly limited and since we will be deploying into these accounts with a CI/CD pipeline, there really is not any reason for anyone to actually log into these accounts other than to look at logs. This setup also simplifies the process of suspending the account when a staff member changes roles or leaves the company.

Billing

This set up can also make billing easier as you can get a sense of how much your production environment is costing without having to worry about the resources your engineering team is using. You can also put billing alerts on your non production accounts so that someone does not forget to suspend or delete a resource they are not using or to catch a runaway process that is dumping tons of data into an S3 bucket. Billing alerts are also important for giving engineers some freedom to explore AWS.

Innovation

As mentioned above, another great reason to give engineers their own AWS accounts is innovation. Giving an engineer their own account (without them having to worry about messing up anyone else) gives them the freedom and opportunity to explore AWS offerings, and build new systems in a controlled way.

Overall though, using a CI/CD pipeline to deploy into all of these different AWS accounts ensures that all the accounts are provisioned the same way and ensures that everything will work as expected when we deploy to production.

Git Workflows

The main goal in this example is to keep development paths as simple as possible, I mean, we could have an endless debate on what the best git workflow is - I like this one, mostly because I usually work on my own or with a very small team. The main branch will contain the production version of the code. So after the code is merged into main, we will typically cut a new release and push it to production. This example shows a team working on a feature and another team or individual contributor working on a hotfix.

Looking at is a git workflow, we may see something like this:

--- config: theme: default themeVariables: git0: "#f7cd70" git1: '#bd64f7' git2: '#80a2f8' --- gitGraph LR: commit id: "Normal" tag: "v1.0.0" commit id: "1" branch feature checkout feature commit id: "Add IaC for New S3 bucket" commit id: "Add New bucket permissions" checkout main commit branch hotfix checkout hotfix commit id: "Adjust IaC Params for CloudFront" checkout main merge hotfix commit id: "Patch Release" tag: "v1.0.1" checkout feature commit id: "Add Route53 IaC" commit id: "Add Cloudfront IaC" checkout main merge feature checkout main commit id: "Minor Release" tag: "v1.1.0"

We start on the main branch at v1.0.0. The tags on the main branch represent what was deployed to production. The goal is to keep the main branch as clean as possible at all times as we really don’t want people creating new feature branches with non-production code. In our example, engineers create branches for features or hotfixes (or whatever), work on the code and then create pull requests to merge the code back into the main branch. Before the code can be merged back into the main branch, we need to run our automated test suite, do code review with another member of the team and potentially get a review with the product team or the system architecture team (more than likely want to get the CEO to take a look and the janitor may have some input as well). If everything looks good, we merge the code into the main branch and cut a release by creating a new tag. In the example above we create a patch release called v1.0.1 after the hotfix branch is merged back in and a minor release when the feature branch is merged in called v1.1.0.

GitHub Actions

In the above example, the v1.1.0 tag represents the code that is currently in production but how do we see what’s going on in the other branches so that we can build, test and review the code before we move it into the main branch? This is where GitHub Actions comes in. GitHub Actions have rules which trigger custom processes which gives you an opportunity to take “Action” everytime the GitHub Repo is interacted with.

There are challenges though. How do we set up CI/CD processes so that new code goes to the correct account in a seamless/automated way?

Here are the rules and actions we care about in this example:

Rule	Action
the main branch is tagged	run action to push a new release into production
pull request created for hotfix branch against the main branch	run action to push a release to a testing/review environment
commit created on feature or hotfix branch	deploy to a development environment

You could set this up in separate GitHub actions so that each action has a single rule at the top and a specific set of steps. The problem with this approach is that you will need to make sure that all the action workflow files are kept synced up otherwise a deployment may work differently in the different environments. So, it might look great on the dev version of the site but it breaks in testing or, even worse, production!

NOTE: It’s probably worth mentioning that you can create any rules that make sense in your environment. For example, if you are maintaining a blog site you might just want to push the deployment out when you merge the pull request into the main branch and not bother creating a release.

Multi Destination Actions

As mentioned earlier, to combat having to keep a number of action files all synced up, we can use a single action workflow and add some logic that decides what action needs to be taken.

The action file will look something like this:

name: Example Multiple AWS Account Workflow

on:
  push:
    branches:
      - main    # Push or merge to main branch
      - '**'    # Push to any branch (for development/testing)
  pull_request:
    branches:
      - main    # Pull request targeting main
  release:
    types: [published] # Trigger deployment when a new release is published

jobs:
  setup_environment:
    runs-on: ubuntu-latest

    steps:
      - name: Set environment based on event and user
        id: set-env
        run: |
          if [[ "${{ github.event_name }}" == "pull_request" && "${{ github.base_ref }}" == "main" ]]; then
            echo "environment=staging" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref_type }}" == "tag" ]]; then
            echo "environment=production" >> $GITHUB_OUTPUT
          else
            echo "environment=development" >> $GITHUB_OUTPUT
          fi          
    outputs:
      environment: ${{ steps.set-env.outputs.environment }}

The first bit sets the triggers. In this case, our action will trigger when there is a push to the main branch or to any other branch. Yes, there is nothing special about the main branch and yes, it is included in ‘**’ but there is something to be said about being explicit and that this action will run on the main release branch. This action will also trigger if there is a pull request against the main branch. Finally, it can be manually triggered from the web interface in GitHub - this can be useful for debugging in which case the web interface is going to ask you what branch to run on and whet the trigger is i.e., push or pull_request in this case.

In the jobs section we will create a job to figure out what environment we are running in. The options for environment will be ‘development, staging’ or ‘production’. We can figure out what the environment is by looking at the trigger and then checking what branch we are on. So, if the trigger is a pull_request and we are running on the main branch we are going to run our actions on the staging environment. If github.ref is refs/heads/main it means we are pushing to main. If neither of those conditions are true, it must be a trigger to build in a development account.

In this next section we will modify the setup-environment job so that instead of just setting the environment to development, we can set the environment so that the deployment goes to a specific development AWS account by getting the username of the person doing the deployment. This is kind of cool if you have a number of engineers on the team but might be overkill depending on the type of work you are doing.

To do this, the only change we need to make is to replace developmet with github.actor. Now the set_provision_environment job looks like this:

jobs:
  set_provision_environment:
    runs-on: ubuntu-latest

    steps:
      - name: Set environment based on event and user
        id: set-env
        run: |
          if [[ "${{ github.event_name }}" == "pull_request" && "${{ github.base_ref }}" == "main" ]]; then
            echo "environment=staging" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref_type }}" == "tag" ]]; then
            echo "environment=production" >> $GITHUB_OUTPUT
          else
            echo "environment=${{ github.actor }}" >> $GITHUB_OUTPUT
          fi          
    outputs:
      environment: ${{ steps.set-env.outputs.environment }}

The outputs section create a variable that we can reference later on in the workflow. In this case it will contain our environment variable so that we know what account we will be authenticating and deploying to.

Secure Connection to AWS

Before you can start building your infrastructure, you’ll need to configure a connection between GitHub and your AWS account. Of course, we want this to be a secure as possible and also flexible so that we can have ways to deploy to different environments like development, testing and production. We also want it to be as foolproof as possible so that we don’t have dev code accidentally being deployed to production.

You should NEVER put passwords, access keys, tokens or anything else that could allow someone to gain unauthorized access to another account in code being checked in to GitHub or any other source control system!

There are a couple of ways to set up the secure connection. The easiest way to get up and running is to use the configure-aws-credentials action which is maintained by AWS.

GitHub Environments and Secrets

GitHub allows you to store credentials for third party accounts in a secure manner using GitHub Secrets. Think of it kind of like a password manager. One thing to keep in mind with GitHub Secrets is that you cannot see the ‘secrets’ after they have been set and the only way to edit them is to enter new values for the secret. Generally speaking this is not a problem but some people add values to GitHub secrets that are not really secret data, just configuration parameters like AWS_REGION which does not really need to be kept secret. You can also use GitHub’s environment variables to store this type of information which may come in handy when you are debugging a deployment, check the region and realize that you are looking at the wrong region on AWS (yes, this happens to the best of us).

Above, we went to a lot of trouble to set an output variable called environment. The purpose for this is that GitHub has environments that can be created for a specific repo, and these environments can be used to store variables and secrets. This means that we could have different AWS credentials set up depending on which environment we are running in. This is very handy as it keeps our jobs very clean and guarantees that all of our deployment scripts run exactly the same way regardless of which environment we are running in. This helps us avoid the “it looks fine on my machine” type scenario where our different environments get out of sync.

Next we can look at a couple of different ways to establish a secure connection to AWS.

Using OpenId Connect

OpenID Connect² is the preferred way to connect GitHub to your AWS account. This works by creating an OpenID provider and an IAM role on AWS and then telling GitHub which role to use to connect to the account. Since we have different environments defined in our GitHub action, we can provide a role for each account we would like to be able to deploy to.

Here’s the CDK code to set this up:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';

export class OpenIdStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Get the GitHub organization and repo from CDK context which can be 
    //     set on the command line when we deploy the template
    const githubOrg = this.node.tryGetContext('githubOrg') || 'default-org';  // Default to 'default-org'
    const githubRepo = this.node.tryGetContext('githubRepo') || 'default-repo';  // Default to 'default-repo'

    // Create the OIDC provider
    const oidcProvider = new iam.OpenIdConnectProvider(this, 'GitHubOidcProvider', {
      url: 'https://token.actions.githubusercontent.com',
      clientIds: ['sts.amazonaws.com'],
    });

    // Create the Oidc Role
    const githubOidcRole = new iam.Role(this, 'GitHubOidcRole', {
      assumedBy: new iam.FederatedPrincipal(
        oidcProvider.openIdConnectProviderArn,
        {
          'StringLike': {
            'token.actions.githubusercontent.com:sub': `repo:${githubOrg}/${githubRepo}:*`
          }
        },
        'sts:AssumeRoleWithWebIdentity'
      ),
      description: 'Role assumed by GitHub Actions using OIDC',
    });

    new cdk.CfnOutput(this, 'GitHubOidcRoleArn', {
      value: githubOidcRole.roleArn,
      description: 'The ARN of the role that GitHub Actions can assume via OIDC',
    });
  }
}

Ok, that’s a lot, lets break it down a bit and see what’s going on under the hood.

This bit:

const githubOrg = this.node.tryGetContext('githubOrg') || 'default-org';  // Default to 'default-org'
const githubRepo = this.node.tryGetContext('githubRepo') || 'default-repo';  // Default to 'default-repo'

is going to allow us to supply the GitHub org and repo on the command line when we deploy the template. So, the command will look something like this when we deploy:

$ cdk deploy --context YOUR-GitHub-ORG_NAME --context githubRepo=YOUR-GitHub-REPO_NAME

Now we can create the OIDC provider and establish that AWS will trust tokens issued by the GitHub Actions identity provider.

// Create the OIDC provider
const oidcProvider = new iam.OpenIdConnectProvider(this, 'GitHubOidcProvider', {
    url: 'https://token.actions.githubusercontent.com',
    clientIds: ['sts.amazonaws.com'],
});

This part defines the role that GitHub will be assigned when it connects to AWS. The StringLike part tells AWS what the token from GitHub is going to look like. It’s basically saying that if the token being sent by GitHub matches this string, that AWS will allow access.

// Create the Oidc Role
const githubOidcRole = new iam.Role(this, 'GitHubOidcRole', {
    assumedBy: new iam.FederatedPrincipal(
        oidcProvider.openIdConnectProviderArn,
        {
            'StringLike': {
                'token.actions.githubusercontent.com:sub': `repo:${githubOrg}/${githubRepo}:*`
            }
        },
        'sts:AssumeRoleWithWebIdentity'
    ),
    description: 'Role assumed by GitHub Actions using OIDC',
});

Did you notice the * on the end of the token? This is because GitHub is going to send a token that has some additional information on the end. The critical part is making sure the GitHub Org and repo are correct. We can make it more granular by adding more information on the end of the token. There is more information about this the GitHub Documentation³ and here⁴ and the AWS documentation here⁵.

The last section will output the ARN for the role. We will create a secret on GitHub with the role name so that when we try to connect to AWS, GitHub will know which role to use.

new cdk.CfnOutput(this, 'GitHubOidcRoleArn', {
    value: githubOidcRole.roleArn,
    description: 'The ARN of the role that GitHub Actions can assume via OIDC',
});

The cool part here is that we will make roles in each AWS account we want to connect to i.e., we will create separate roles and install the in our production, staging, and developer AWS accounts. Once we have created all the roles, we can create secrets in GitHub to hold the names of the different roles. You can do this in the GitHub web interface by going to the settings section of the repo you are connecting and finding either Environments or Secrets and Variables in the menu on the left side. We need to create environments for production and staging and then, depending on how you set it up, either an environment for development or individual environments for each developer using their GitHub login name! Pretty cool right?

When we save the secret in GitHub we can call it something like OPENID_GITHUB_ROLE, it will contain the Arn for the role we created. The important part is that we use the same name for each environment. So create the role in the Production AWS account and then got to your GitHub repo and create a secret in the production environment called OPENID_GITHUB_ROLE, then do the staging environment, and then the devs.

Next Level it!! Everyone knows that web interfaces are for whimps. I mean, nothing (necessarily) wrong with using a web interface but when we are doing some serious DevOps work, we don’t want to have to worry about some web interface, that at best is bound to change at some point or that we may not even have access to when we need it the most and, at worst, someone goes in there and toggles some button somewhere and does not document what they did, or tell anyone, or gets hit by a bus. So, to take this to the next level, we can create the secret in the GitHub repo’s environment from the command line using the gh cli tool from GitHub. A quick shell script (of which, ChatGPT was a major contributor) is available in the companion GitHub repo for this article. The nice thing about this approach is that you can run the scripts multiple times and make small changes as you go along. It’s a lot better than clicking through ton of webpages, but that’s my bias and your mileage may vary. Ultimately, you know that once your script is established, everything will be configured the same way. Of course, you can also give this script to your devs and have them set up their own connections as well which is a lot easier than documenting how to do this in the web interfaces.

The GitHub CLI tool - The gh CLI utility is pretty handy for other things like creating repos and managing GitHub Actions! You can get more info https://cli.github.com/ .

Using AWS credentials

This is a very popular and fairly secure way to set up a connection. The main issue with this approach is that you need to create access keys and store them on GitHub. Not the end of the world but it just means that you have one more set of credentials that you need to manage and share with your team. We all know that no one would ever share credentials though email, text messages, or Slack but yet, it still happens. You also need to consider the fact that if a team member leaves the company, and they had access to those keys, they would still have access to the AWS account until you (remember to) rotate the credentials…and then you have to update all the places that those credentials are used by distributing them to the team in a secure way, etc. Kind of a mess, no?

It’s still an option though. The process here is to create a new user in the IAM service on AWS, generate access keys for the user and then copy those into GitHub secrets in much the same way that was described in the OpenID section above. It’s a good idea not to create login credentials for this user, you really don’t need them.

GitHub Action Workflow

Now that we have established how we will connect to AWS, we can use the configure-aws-credentials action from AWS to get the connection configured. This small snippet of code shows how this might be set up to do something simple on AWS, like get the account_id of the account we are connected to.

jobs: ## continued
  get_aws_account_id:
    runs-on: ubuntu-latest
    needs: setup_environment
    
    environment:
      name ${{ needs.setup_environment.outputs.environment }}  # Get the environment we are connecting to

    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.OPENID_GITHUB_ROLE }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Get AWS Account ID
        id: aws-account-id
        run: |
          ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
          echo "AWS_ACCOUNT_ID=$ACCOUNT_ID" >> $GITHUB_ENV          

      - name: Echo AWS Account ID
        run: |
          echo "Connected to AWS account: ${{ env.AWS_ACCOUNT_ID }}"

Breaking this apart, we need to include the environment that was identified in the first part of the workflow and then set the environment. This tells GitHub actions which secrets to use when connecting to AWS.

  get_aws_account_id:
    runs-on: ubuntu-latest
    needs: setup_environment

    environment:
      name ${{ needs.setup_environment.outputs.environment }}

This section connects GitHub Actions to AWS. This is what it looks like if you are using the OpenID method to authenticate. Also notice that you can include the region here by creating a variable in the environment on GitHub. In GitHub environments, variables are similar to secrets (they work the same way) but you can actually see what the variable is set to while secrets are, well, secret.

- name: Configure AWS credentials using OpenID
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.OPENID_GITHUB_ROLE }}
    aws-region: ${{ vars.AWS_REGION }}

and this is what it looks like if you are using the access keys:

- name: Configure AWS credentials using AWS User Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ vars.AWS_REGION }}

Just remember that best practice would be to use OpenID Connect for this although, it is very common to see people using the access keys

The last bit is just a sanity check that we have connectivity and to make sure we are connecting to the correct account.

- name: Get AWS Account ID
  id: aws-account-id
  run: |
    ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    echo "AWS_ACCOUNT_ID=$ACCOUNT_ID" >> $GITHUB_ENV    

- name: Echo AWS Account ID
  run: |
    echo "Connected to AWS account: ${{ env.AWS_ACCOUNT_ID }}"

Putting it all together

Ok great! We have a secure way to connect to AWS, we have a GitHub action workflow that knows how to deploy to different accounts based on what is going on in the repo.

Adding a new developer account

In the examples above we created production and staging environments. If all your developers have their own AWS accounts you can have them configure their own environment in the GitHub repo they want to work on using their GitHub login name for the name of the environment. Then they would add their OpenID Role to their environment as a GitHub secret called OPENID_GITHUB_ROLE. Now when they commit code to the repo, the action will run against their dev account!

Installing the Files

As you are building the repo you need to connect to AWS, you can put the GitHub actions files in the <your-repo>/.github/workflows directory. I would create another repo for the OpenID connection CDK apps and shell scripts. This will make it easier to keep your scripts consistent across all of your repos.

TroubleShooting

One common issue with getting the OpenID connection to work is that the token being sent by GitHub to AWS does not match what AWS is expecting. You can print out the token that GitHub is sending with the following code section in your GitHub action:

      - name: Get Account ID
        run: |
          echo "ENVIRONMENT:  ${{ needs.set_provision_environment.outputs.environment }}"
          echo "ACCOUNT ID: ${{ secrets.AWS_ACCOUNT_ID }}"
          echo "OPENID ROLE: ${{ secrets.OPENID_GITHUB_ROLE }}"
          echo "AWS REGION: ${{ vars.AWS_REGION }}"          

      - name: Install jq
        run: |
          curl -L -o /usr/local/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
          chmod +x /usr/local/bin/jq          

      - name: Retrieve OIDC token
        id: get-oidc-token
        run: |
          # Function to add base64 padding
          add_padding() {
            case $((${#1} % 4)) in
              2) echo "$1==";;
              3) echo "$1=";;
              *) echo "$1";;
            esac
          }

          # Request the OIDC token from the GitHub OIDC provider
          OIDC_TOKEN=$(curl -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
                              -H "Accept: application/json" \
                              "${ACTIONS_ID_TOKEN_REQUEST_URL}" \
                              | jq -r '.value')

          # Split and decode the OIDC token's header and payload
          DECODED_HEADER=$(echo "$(add_padding $(echo "$OIDC_TOKEN" | cut -d '.' -f1))" | base64 -d)
          DECODED_PAYLOAD=$(echo "$(add_padding $(echo "$OIDC_TOKEN" | cut -d '.' -f2))" | base64 -d)

          echo "OIDC Token Payload: $DECODED_PAYLOAD"

          # Extract and print the 'sub' claim from the payload
          SUB_CLAIM=$(echo "$DECODED_PAYLOAD" | jq -r '.sub')
          echo "OIDC Token Subject (sub): $SUB_CLAIM"

Conclusion

This may seem like a lot of work to get everything connected but there is honestly not a ton of code in here and it can scale up to however many developers/AWS accounts you have. It’s also worth pointing out that once you have established this workflow with one GitHub repo it’s very straight forward to set it up for other repos as well.

I hope you found this article interesting and maybe even useful.

Thanks for reading!

Randy Bias - The History of Pets vs Cattle and How to Use the Analogy Properly https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ ↩︎
OpenID Connect https://openid.net/developers/how-connect-works/ ↩︎
GitHub - Security Hardening with OpenID Connect https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect ↩︎
Github - Configuring OpenID Connect in Amazon Web Services https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services ↩︎
AWS - Create an OpenId Connect (OIDC) identity provider in IAM https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html ↩︎

AWS Serverless CI/CD with GitHub Actions

Wed, 22 Apr 2020 00:00:00 +0000

AWS and GitHub are great services for managing and deploying cloud applications but sometimes getting the code from GitHub to AWS is not as straight forward as we would like it to be. Sure, there are some tools out there that bridge the gap (CircleCi, Jenkins, many others) but that means learning another framework, and hoping that everything is tightly integrated and that you remember how it all works when something breaks half a year after you set it all up. GitHub Actions can help simplify this though. Actions are easily configured workflows that are triggered when you do something with git. For example, you can set up an action that automatically test your code every time you push it to github. You can then extend that workflow so that it builds your code in a test environment when you create a pull request. This allows another member of the team to easily access the running code while reviewing the source code, simplifying the code review process as you get that warm, cozy feeling knowing that the code you are reviewing does what you expect it to do.

In this post, I’m going to walk through the steps of setting up a CI/CD pipeline inside GitHub to manage Lambda Layers builds.

Coding Workflows

YAML is used to configure the workflows and complete reference of the workflow syntax use by GitHub Actions can be found here. Fortunately, both YAML and the workflow syntax are pretty easy to work with. Start out by creating a new branch in your repo. The workflows are going to go in the .github/workflows directory in the root of your repo. You can have as many workflows as you like in here as long as each file has a .yml or .yaml extension. I’m going to focus on creating a workflow for building a distribution for AWS Lambda Layers, a full description of how to build workflows for other things can be found here. In the .github/workflows directory, create a file called awsLayers.yaml - you can give it any name you like as long as you have the .yml or .yaml extension. Here’s my file:

# This workflow will install dependencies and create a build suitable
# to be used in an AWS Lambda Layer.  The build will then be uploaded
# to S3 and then can be accessed from any lambda that uses the layer.
#
# This build is only for dev builds.  Releases will be built from a
# seperate action.
#
# A new version of the layer will be created for every branch when a
# pull request is intitiated.  This allows us to test the layer in a
# dev environment on AWS BEFORE the code is merged into main.


name: Build Lambda Layer

on:
  pull_request:
    branches: [ main ]

jobs:
  
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # For more info: https://github.com/aws-actions/configure-aws-credentials
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

      - name: Zip it all up and upload to S3
        env:
          release_bucket: 
          release_bucket_uri: 
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python
          
          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

Let’s walk through all of this…

Here’s the name of the workflow:

name: Build Lambda Layer

This next bit is telling github that the workflow should be run on pull_requests on the main branch (i.e., code you are merging into main).

on:
   pull_request:
     branches[main]

Now, if you have a more complex git flow going on, you may not be merging into main - so don’t just blindly cut and paste this code.

Define the jobs that will be run. According to the GitHub Actions docs, you can have multiple jobs and they will run in parallel by default unless you specifically indicate that you want the jobs to run in sequence.

I only have one job in this case called deploy. This job has a name Upload Layer to AWS Lambda. We are also going to specify a ubuntu virtual machine to run on. We are not really doing anything exciting here so the distribution and version of linux is really not that important.

jobs:
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

The next section defines the steps that will be run. The first step is to checkout the code. The workflow is running in a virtualized environment so we need to grab our code from the repo before we can do anything useful.

steps:
  - name: Checkout
    uses: actions/checkout@v2

The next bit is to configure an AWS user that we can use to upload the files safely to S3.

It’s important that the user that is created only has minimal permissions on aws! The user should only be used for github access!

It’s probably a good idea to create a separate bucket on S3 to receive the files that will be uploaded to AWS layers. To lockdown the user, I created two policy files, One to allow uploads to a specific bucket on S3, and a second one that allows layers to be installed on Lambda.

The upload policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutBucketNotification",
                "s3:AbortMultipartUpload",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my_bucket_name/*",
                "arn:aws:s3:::my_bucket_name"
            ]
        }
    ]
}

The policy to add layers to lambda only needs one action!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:PublishLayerVersion",
            ],
            "Resource": "*"
        }
    ]
}

You could certainly combine the two policies if you wanted to but I prefer making smaller policies that I can attach to different groups of users if I need to.

We will load the configure_aws_credentials action which can be found in the aws-actions repo on github.

You will need to store the AWS credentials for the user you created earlier in a secret vault on GitHub. DO NOT PUT YOUR CREDENTIALS IN YOUR REPO - of course, you already know this, right? Instructions for stashing your credentials securely on github can be found here

Something to take note of here is that the secrets are stored on a per repo basis. This means that you could create a separate AWS user for each github repo - a pretty good idea if you are remotely paranoid about such things. Even better would be to set up a test account using AWS Organizations to keep your test code separate from other environments - but that’s beyond the scope of this article.

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

Load up python - I’m using python 3.8

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

Finally, the interesting part. This is where we build the distribution, copy it to S3 and then load it into lambda layers. In order for the code to be installed correctly in the layer, it needs to be in a directory called python. The run section below, creates the directory, installs all of the dependencies and then install the code. For the release_id, you can really use anything you want. The format command below is going to create something like MyRepo-BranchName-dev.zip. If you look at the github docs you can find a bunch of other things you can stick on there.

Side Note: Instead of using a standard requirements file, i.e., requirements.txt, I have created a requirements directory with a dex.txt, test.txt and a prod.txt. This allows me to control what is being installed in the production environment which is important when developing for lambda. For example, you don’t need to load boto as lambda already has it, you also don’t want to load up any dev or test tools to the production environment. For that matter, you also do not want to load up all of your tests and documentation files to production, so you should consider writing filters to remove (or not include) this stuff when building your layer code. A simple way to do this would be to use the —exclude flag in zip (check the man page for more info on how that would work).

      - name: Zip it all up and upload to S3
        env:
          release_bucket: my_bucket_name
          release_bucket_uri: s3://my_bucket_name
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python

The code is then compressed with zip and the file is copied to S3. Once the file has been uploaded, the aws command line tool is used to publish the layer in AWS Lambda so that it can be used from other lambdas.

          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

When you are done building your workflow, push it up to github and try it out by creating a new pull request. Don’t be discouraged when it does not work on the first try. In fact, a solid strategy for this would be to build up the file section by section so you can see how it all works. You can see the actions running in the Actions tab in you repo. You should see a history of all your prior runs in this tab too.

GitHub Repo Configuration

The next step is to get your GitHub repo configured - this is going to be a bit different for each team/repo depending on what workflow your team is choosing to use. I personally use a very simple main/feature branch workflow since most of my repos and teams are fairly small so we rarely have to worry about clobbering each other’s changes. This simple workflow is also very fast which allows us to move changes into production multiple times a day. With all that said, go to the Settings tab in your repo and select Branches. We want to create a a new rule in Branch Protection Rules - so click Add Rule. There’s a full description of the options in the GitHub Documentation.

There are a bunch of interesting options in here that can make your development process more secure and less prone to accidents. The option we are particularly interested in is Require status checks to pass before merging. This allows us to hook in our GitHub Actions and will require the Actions to run cleanly before the code can be merged back into your main branch.

GitHub Action Workflows

With the branch rules configured you will now find that the pull request cannot be merged unless the workflow completes successfully. Now that the layer is installed in your test environment, a code reviewer can look at code and then go to the environment to make sure everything is working they way they would expect it to.

Of course, you can do all kinds of testing once you get a couple of workflows running. A few ideas:

Blocking merges until all the unit tests pass
making sure that static testing passes
integration tests - like running api checks on the test environment that the layer was just installed in

Let your imagination run wild!