Serverless | Dilex Networks

Lambda Extensions with Python

Tue, 20 Oct 2020 00:00:00 +0000

Keeping up with AWS these days is like drinking from a firehose and while a lot of their newer products are designed to make existing products more accessible, every once in a while you get something that helps solve a nagging issue, or opens up a whole new world of possibilities on one of their core services.

AWS Lambda extensions, currently in preview and announced on Oct 8, 2020, is one of these types of products. Think of an extension as a wrapper or “sidecar” that runs in the same execution environment as the lambda’s code. According to the press release, some of the uses for these extensions include:

capturing diagnostic information before during and after the lambda function executes
instrumenting code without changing the structure of the code
fetching secrets before the function is invoked
detecting and alerting on function activity through security agents

That’s a lot to think about but initializing code and fetching secrets before the function runs seems like it could be a big deal in reducing function runtimes. For example, I often need to load parameters and secrets from SSM Parameter store. I usually try to do this once and then cache everything so it’s available the next time the function runs. As you can see in the diagram below, the extension runs in the same container as the function. This means that the extension has access to everything in the lambda’s environment including all of the environment variables, the actual lambda code, code in layers and the 500MB in /tmp.

A quick overview of how the extensions API fits into the lambda environment

The goal will be to write an extension that grabs a few parameters from SSM and caches them to see how fast it runs and then compare the results with a similar lambda that does not use the extension. The team at AWS (as always) has supplied us with some great sample code (https://github.com/aws-samples/aws-lambda-extensions) to get everything going. I highly recommend working through a few of their simple samples to get a feel for what is going on.

Set Up

The first challenge is that the extension needs to be installed in a layer in a directory called /extensions . Setting up a lambda layer is actually pretty straight forward and if you really want to get fancy, I wrote a post about using GitHub Actions to create a CI/CD pipeline for installing lambda layers (available here: https://www.dilex.net/data-blog/aws-serverless-cicd-with-github-actions) although if you just want to get something running, follow the examples in the aws-samples example code referenced above.

My approach was to write an extensions class (basically borrowed from the example code from the aws-samples repo) which looks like this:

import json
import os
import signal
import sys
import requests

'''
This is the base class for lambda extensions.  The actual extensions can be found in the top
level /extensions directory.  Extensions can be used to initialize objects and data before
the lambda starts running as well as handing logging or other types of error reporting without
affecting the code in the lambda itself
'''


class Extension:
    def __init__(self):
        self.name = os.path.basename(sys.argv[0])
        self.extension_id = None
        self.events = ['INVOKE', 'SHUTDOWN']
        self.lambda_api = os.environ['AWS_LAMBDA_RUNTIME_API']

        # catch and handle signals
        signal.signal(signal.SIGINT, self.handle_signal)
        signal.signal(signal.SIGTERM, self.handle_signal)

    def register_extension(self):
        'the extension needs to be registered or lambda will not run it'
        print(f"[{self.name}] Registering extension", flush=True)

        url = f"http://{self.lambda_api}/2020-01-01/extension/register"
        headers = {
            'Lambda-Extension-Name': self.name
        }
        payload = {
            'events': self.events
        }
        response = requests.post(
            url=url,
            headers=headers,
            json=payload
        )
        self.extension_id = response.headers['Lambda-Extension-Identifier']
        print(f"[{self.name}] Registered with ID: {self.extension_id}", flush=True)

    def process_events(self):
        'this listens for events from the lambda service'
        url = f"http://{self.lambda_api}/2020-01-01/extension/event/next"
        headers = {
            'Lambda-Extension-Identifier': self.extension_id
        }

        # set up a loop and wait for incoming requests
        while True:
            print(f"[{self.name}] Waiting for event...", flush=True)
            response = requests.get(
                url=url,
                headers=headers,
                timeout=None
            )
            event = json.loads(response.text)
            if event['eventType'] == 'SHUTDOWN':
                self.exit_processing()
            else:
                self.event_processing(event)

    def event_processing(self, event):
        'runs on every lambda invoke event - override function to do something more interesting'
        print(f'[{self.name}] Received event: {json.dumps(event)}', flush=True)

    def exit_processing(self):
        'do something when the function exits - override this to do something more interesting'
        print(f'[{self.name}] Received SHUTDOWN event. Exiting!', flush=True)
        sys.exit(0)

    def handle_signal(self, sig, frame):
        'if needed, pass this signal down to child process'
        print(f"[{self.name}] Received signal={sig}. Exiting", flush=True)
        sys.exit(0)

Admittedly, this is overkill for this simple test but the idea was to create something I could use to build a few different extensions.

Let’s take a look at a couple of sections of the above code. The instantiation method is going to grab the filename the function is being run from to establish self.name and, by default, this class will run the extension on INVOKE and SHUTDOWN events. The register_extension method is going to connect to the lambda api to establish an extension_id. The process_events method will set up a listener and wait for events from the lambda service.

The event_processing and exit_processing functions are basically placeholders that can be overridden from the extension code. This class was installed in a layer under /opt/python.

Now, let’s take a look at the extension code:

#!/usr/bin/env python3

import os
import sys

# These need to be imported from /opt/python
sys.path.append('/opt/python')
from AwAws.Lambda.extensions import Extension
from AwAws.SharedResources.parameters import Parameters
from AwAws.Utils.env import Env


class SSMParamsExtension(Extension):
    def __init__(self):
        self.service = None
        super().__init__()

    def store_params(self):
        try:
            Parameters(self.service).create_tmp_dict()
            print(f'[{self.name}] SSM params found for {self.service}')
        except Exception as e:
            print(f'[{self.name}] No SSM params found for {self.service} - check SSM', e)
            pass

    # override event_processing in the base class
    def event_processing(self, event):
        pass
        # print(f'[MY {self.name}] Received event: {json.dumps(event)}', flush=True)


ext = SSMParamsExtension()
ext.events = ['SHUTDOWN']
ext.register_extension()

# This extension only runs in the init and shutdown stage
ext.service = Env().get_env('AWAWS_SSM_SERVICE')
if ext.service is not None:
    ext.store_params()
else:
    print('AWAWS_SSM_SERVICE is required for SSM caching - ssm parameter caching turned off')

ext.process_events()

`` NOTE: make sure to make the function executable (chmod 755 your_extension_name)

This really just loads a couple of things from my AwAws boto3 wrapper code . The Extension class has been installed in AwAws.Lambda.extensions and I’m pulling in a module that grabs parameters from SSM as well as a utility to deal with environment variables.

Ignoring the class for a moment, this section at the bottom of the extension code:

ext = SSMParamsExtension()
ext.events = ['SHUTDOWN']
ext.register_extension()

# This extension only runs in the init stage
ext.service = Env().get_env('AWAWS_SSM_SERVICE')
if ext.service is not None:
    ext.store_params()
else:
    print(f'AWAWS_SSM_SERVICE is required for SSM caching - ssm parameter caching turned off')

ext.process_events()

is going to instantiate the class, set the extension up so that it will only run on the SHUTDOWN event and then register the extension with the lambda service. Since we are only going to load the parameters once, we don’t need to listen for INVOKE events. Next, we look for an environment variable that is set on the lambda which contains the root of our SSM parameters. For example, if the SSM parameters are organized something like /ServiceName/Param1, /ServiceName/Param2, etc. the env variable would be set to ServiceName.

Now we call the store_params method in the class which establishes a botocore client to connect to SSM, connects to SSM and grabs the parameters we asked for and then uses pickle to put the parameters in a file in /tmp.

Now we can push the layer up to AWS Lambda and start working on the lambda code to run the test.

Testing the Extension With the layer loaded we create a very simple Hello World lambda using the SAM toolkit.

import json

from AwAws.Lambda.base import Lambda
from AwAws.SharedResources.parameters import Parameters

class LambdaTestClass(Lambda):
	def handle(self, event, context):
        print('FUNCTION: started')
        
        ssm = Parameters('Announce')
        try:
            params = ssm.read_tmp_dict()
        except Exception as e:
            params = ssm.get_param_dictionary()
            
        for name in params.keys():
            print('SSM PARAM:', name, 'Value:', params[name])

        print('FUNCTION:', json.dumps(event))
        print('FUNCTION: done')

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "hello world",
                # "location": ip.text.replace("\n", "")
            }),
        }

lambda_class = LambdaTestClass()

# this does not get inoked when initializing
# but only when the lambda service invokes it
lambda_handler = lambda_class.get_handler()

A bit of a sidebar here. I use a base class for all of my lambda functions. All this really means is that my function code is wrapped in the get_handler() method which allows me to do stuff like send an event to the function that keeps the function warm - a bit of legacy code to reduce cold start times.

So, ignoring the class, this function is going to read the parameters from the file in /tmp that was created by the extension. If you recall earlier, we set up an environment variable to specify which parameters we want from SSM. If we don’t set the environment, the extension will not get any parameters from SSM in which case we fall back to setting up the SSM client and getting the parameters in the lambda function itself. By testing it both ways we can get an idea of what impact the extension is having on our start up times.

With the Extension

REPORT RequestId: 6b9d5fbb-cb31-4bf1-b961-698b6f654182 Duration: 18.95 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 96 MB Init Duration: 675.52 ms

Without the Extension

REPORT RequestId: bdc9dfe3-c962-4afc-896c-37c2389a9679 Duration: 904.77 ms Billed Duration: 1000 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 669.66 ms

As expected, running the lambda without running the extension first resulted in a much slower execution time - in fact, it’s off by an order of magnitude while the Init Duration for both function is about the same. Now, the astute reader may have noticed that there are a few more things we can do with the non extension lambda to tune it up a bit. For example, in both runs the extension is running, just in the second case, the extension does not fetch the data from SSM and cache it. It’s also worth pointing out that we should be establishing the botocore client connection before the function runs.

Let’s overhaul the function by tuning it up a bit and see what happens. Here’s the new function:

import json

from AwAws.Lambda.base import Lambda
from AwAws.SharedResources.parameters import Parameters

# set up the botocore client connection
ssm = Parameters('Announce')
ssm.get_ssm()

class LambdaTestClass(Lambda):
    def handle(self, event, context):
        print('FUNCTION: started')
        
        # try:
        #    params = ssm.read_tmp_dict()
        # except Exception as e:
        params = ssm.get_param_dictionary()
            
        for name in params.keys():
            print('SSM PARAM:', name, 'Value:', params[name])
        
        print('FUNCTION:', json.dumps(event))
        print('FUNCTION: done')

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "hello world",
                # "location": ip.text.replace("\n", "")
            }),
        }


lambda_class = LambdaTestClass()

# this does not get inoked when initializing
# but only when the lambda service invokes it
lambda_handler = lambda_class.get_handler()

Which gives us:

REPORT RequestId: c76f03f9-7292-46bc-9d4e-92c044a24a27 Duration: 240.84 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 725.15 ms

This is definitely better but still not as fast as the version with the extension. It’s also bumped up the initiation time a little bit which is not that surprising.

Wrapping Up

Obviously, to really test this properly we should set up a framework that reloads and runs the function a few hundred times but even this quick and dirty analysis gives us some insights on how the lambda extensions can be implemented. Extensions have a lot of other nice features too. The incoming events are passed into the extension if you register it with the INVOKE event. This allows you to do things like send events to cloudwatch or eventbridge without modifying the function code, even do some analysis on the incoming event to see if someone is trying something malicious. Definitely a cool feature and worth spending some time leveraging how to use it.

Did I miss something? Leave a note in the comments below!

Envelope Encryption on AWS

Sat, 19 Sep 2020 00:00:00 +0000

Over the past few years, there have been a number of data breaches of companies using AWS but not securing the data correctly. Leaving Personally Identifiable Information (PII) in publicly accessible buckets on S3 seems to be a common problem. While AWS is a very secure environment, AWS’s shared responsibility clearly shows that their job with security ends at “Security of the Cloud” which leaves “Security in the Cloud” up to the customer (https://aws.amazon.com/compliance/shared-responsibility-model/). The customer’s security responsibility includes securing operating systems on EC2/ECS, network configurations and firewall settings, Identity and Access Management (IAM) and data encryption which is what the rest of this article is about.

One strategy for data encryption is called Envelope Encryption which can be used to encrypt data in transit or at rest. The envelope encryption service used in AwAws was built around AWS Key Management Service (KMS) which is a dedicated Hardware Security Module (HSM) to secure and encrypt data at rest and by leveraging the AWS Encryption SDK (available for python) following their best practices.

How it works KMS allows us to build and store encryption keys in a very secure way using FIPS 140-2 validated cryptographic modules which do not allow the managed keys to leave the service. The idea here is that you can’t get access to the the actual key - you can send data to be encrypted/decrypted by the service but you can’t get the key yourself and use it from encryption/decryption. Items to be encrypted are sent to the service over a TLS connection, encrypts the data and then returns the encrypted data. The amount of data that can be sent to the service is limited to 4kb (that’s 4 kilobytes) which is not very much. This is great for passwords or API secrets but not very useful for large files. This kind of makes sense though, you don’t necessarily want to be moving large amounts of sensitive unencrypted data around. It also makes sense from the service’s standpoint as the KMS service could easily become a bottleneck if people start trying to send huge files to it.

This is where envelope encryption comes in. Envelope encryption is a two-layer encryption system which involves creating a custom data encryption key to encrypt a specific set of data and then using the KMS master key to encrypt the custom data key. The encrypted key is then stored along side the data that was encrypted. The benefit of this technique is that all of the data objects will have their own unique encryption key which can only be opened with the master key.

Envelope Encryption Steps Let’s say we want to create a lambda service that encrypts files coming in through API Gateway and then stores the encrypted files in S3. Our encryption service would work as follows:

receive data object from API Gateway

locally create a unique key specifically for this data object

use this key to encrypt the data

So, at this point, we have a data object that has been encrypted with some random key that we generated on the fly. Now we need to do something with the key so that we can decrypt the data later on. In theory you could store the key in a database, but if someone got access to the database they would be able to decrypt all of your data! Besides, managing all the data objects locations in S3 and the keys for those objects would be a bit of a nightmare (like say someone renames the files). This is where the envelope part comes in. We take the unique key we generated and:

send the unique key to KMS to be encrypted with our master key

get back the encrypted key

attach the now encrypted unique key to the encrypted data

now the data can be safely moved or stored (encryption in transit, encryption at rest)

We will also attach an encryption context to the object. This is a set of plain text key/value pairs (tags) that identifies the encrypted data. These tags not only help identify and track the data, the tags are also cryptographically bound to the encrypted data - meaning it is also required for encrypting and decrypting the data.

Ultimately, we wind up with a new data object that contains:

an encrypted version of the original data object

a set of plain text key/value pairs that identify the original data which cannot be altered

the encrypted key which (when decrypted by KMS) can be used to decrypt the data

Kind of cool right. Since we attach the key to the data we don’t have to worry about losing track of which keys belong to which data, or someone coming in and stealing all our keys and data. The only way to get at the original data object is via access to the master key.

In Transit This could be set up to move data securely across the internet. The idea would be that we can encrypt data before it leaves the sender site, send it over a secure TLS connection, and the immediately store the data in object storage in our cloud production environments. In this case, we would encrypt the data encryption key with a shared RSA key (similar to the way ssh connections work), send that encrypted key over TLS to the KMS service which would then decrypt RSA and encrypt the key with AES-GCM.

Encryption Algorithms (The Gory Details) AwAws was set up to use a 256 bit AES-GCM (Advanced Encryption Standard algorithm in Galois/Counter Mode) with an Elliptic Curve Digital Signature Algorithm (ECDSA with P-384 and SHA-384) and a HMAC-based extract-and-expand key derivation function (HKDF - SHA-384) which helps prevent accidental reuse of a data encryption keys.

Service Based Architecture with AWS Organizations

Sat, 16 May 2020 00:00:00 +0000

I was having a beer with a good friend of mine about a year ago trying to get my head around a problem I was running into while designing a new system for a job at a healthcare startup. I knew that we would only have a limited number of clients (initially in the tens) and each client would be interacting with our system on a daily basis. We would be receiving a lot of sensitive data from our clients which had to be encrypted on the wire as well as at rest. Our clients could vary greatly in size and hence would be very different in the amount of resources they would consume. I also wanted to keep things somewhat separate from each other, especially client data, to try to reduce the risk associated with data getting associated with the wrong client.

I had this idea that I could build an event driven, service based architecture across multiple AWS accounts and it was my friend’s job to try to talk me out of it. The idea was to break up all of the services I would need to build so that they could run independently from its own AWS account and then providing an AWS account for each client that would provide access to all of the services they subscribe to as well as be the location for all of their data.

Here are some of the benefits of this architecture in no particular order:

client data is kept separate from other clients which good from a security perspective - no way for other clients to access each other’s data
billing is easier as each account would have its own AWS bill - makes it easier to see who is using resources
don’t need to have all of our clients in the same region
can have some clients multi region while others are not - this is a big deal if you have to mirror a lot of data as you only have to mirror the data for a particular client as opposed to all of them
assuming the accounts are built and managed with cloudformation, it should be trivial to create a client environment
don’t have to write database code to manage clients, just set up an instance of cognito for each client and configure it to meet the needs of each client
outages due to software issues would only affect one client, outages in a region would only affect clients in that region (unless they have a multi-region configuration)
removing a client from our system and deleting all of their data (a requirement under HIPAA) is trivial, just ask AWS to delete the account.

That’s a lot of pluses except for one big minus:

how do you support/manage/maintain this beast?

Fair question. It may not be as hard as you think though, especially if you are starting from scratch.

CI/CD Pipelines

There is no way anyone is going to want to sit around deploying code to each server account. I like my team to work with a very fast cadence which can mean multiple deployments a day. There is no possible way we can manage a bunch of separate client accounts if we don’t have CI/CD pipeline set up to run automated tests, but more importantly, deploy changes to all of the client accounts. Of course, since each client has their own account, it should be easy for use to set up test accounts with fully automated integration testing which can be performed before the release is actually pushed out.

Consistency

Possibly one of the trickiest tasks is build the system so that each client has the exact same software. There is really no easy way to build in customizations for a particular client unless they are purely cosmetic type things which should be easy enough to handle. As soon as you start trying to install different software on each client account though you are going to wind up with a bunch of systems to support that do not all work quite the same way.

Building solid APIs and SDKs makes it easier to keep things consistent

just make sure you never change the behavior of your API or SDK. You can add features but if your API starts returning stuff in a different way you’ll wind up with trying to simultaneous deploys so that nothing breaks.

AWS Organization Accounts

Unfortunately, one bigger problem with using accounts this way is that there is no easy way to just delete an account and start over again. So if you want to rebuild an account from scratch you either have to completely clean it out or have a bunch of deleted accounts sitting around in your AWS Organization dashboard - maybe this is not that big of a deal, just keep in mind that you cannot reuse email addresses when you delete and create a new account. You can use a + sign in the email address though (if your email provider supports it) so your account names might look something like qa+20200516@mydomain.com - not the end of the world.

AWS Serverless CI/CD with GitHub Actions

Wed, 22 Apr 2020 00:00:00 +0000

AWS and GitHub are great services for managing and deploying cloud applications but sometimes getting the code from GitHub to AWS is not as straight forward as we would like it to be. Sure, there are some tools out there that bridge the gap (CircleCi, Jenkins, many others) but that means learning another framework, and hoping that everything is tightly integrated and that you remember how it all works when something breaks half a year after you set it all up. GitHub Actions can help simplify this though. Actions are easily configured workflows that are triggered when you do something with git. For example, you can set up an action that automatically test your code every time you push it to github. You can then extend that workflow so that it builds your code in a test environment when you create a pull request. This allows another member of the team to easily access the running code while reviewing the source code, simplifying the code review process as you get that warm, cozy feeling knowing that the code you are reviewing does what you expect it to do.

In this post, I’m going to walk through the steps of setting up a CI/CD pipeline inside GitHub to manage Lambda Layers builds.

Coding Workflows

YAML is used to configure the workflows and complete reference of the workflow syntax use by GitHub Actions can be found here. Fortunately, both YAML and the workflow syntax are pretty easy to work with. Start out by creating a new branch in your repo. The workflows are going to go in the .github/workflows directory in the root of your repo. You can have as many workflows as you like in here as long as each file has a .yml or .yaml extension. I’m going to focus on creating a workflow for building a distribution for AWS Lambda Layers, a full description of how to build workflows for other things can be found here. In the .github/workflows directory, create a file called awsLayers.yaml - you can give it any name you like as long as you have the .yml or .yaml extension. Here’s my file:

# This workflow will install dependencies and create a build suitable
# to be used in an AWS Lambda Layer.  The build will then be uploaded
# to S3 and then can be accessed from any lambda that uses the layer.
#
# This build is only for dev builds.  Releases will be built from a
# seperate action.
#
# A new version of the layer will be created for every branch when a
# pull request is intitiated.  This allows us to test the layer in a
# dev environment on AWS BEFORE the code is merged into main.


name: Build Lambda Layer

on:
  pull_request:
    branches: [ main ]

jobs:
  
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # For more info: https://github.com/aws-actions/configure-aws-credentials
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

      - name: Zip it all up and upload to S3
        env:
          release_bucket: 
          release_bucket_uri: 
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python
          
          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

Let’s walk through all of this…

Here’s the name of the workflow:

name: Build Lambda Layer

This next bit is telling github that the workflow should be run on pull_requests on the main branch (i.e., code you are merging into main).

on:
   pull_request:
     branches[main]

Now, if you have a more complex git flow going on, you may not be merging into main - so don’t just blindly cut and paste this code.

Define the jobs that will be run. According to the GitHub Actions docs, you can have multiple jobs and they will run in parallel by default unless you specifically indicate that you want the jobs to run in sequence.

I only have one job in this case called deploy. This job has a name Upload Layer to AWS Lambda. We are also going to specify a ubuntu virtual machine to run on. We are not really doing anything exciting here so the distribution and version of linux is really not that important.

jobs:
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

The next section defines the steps that will be run. The first step is to checkout the code. The workflow is running in a virtualized environment so we need to grab our code from the repo before we can do anything useful.

steps:
  - name: Checkout
    uses: actions/checkout@v2

The next bit is to configure an AWS user that we can use to upload the files safely to S3.

It’s important that the user that is created only has minimal permissions on aws! The user should only be used for github access!

It’s probably a good idea to create a separate bucket on S3 to receive the files that will be uploaded to AWS layers. To lockdown the user, I created two policy files, One to allow uploads to a specific bucket on S3, and a second one that allows layers to be installed on Lambda.

The upload policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutBucketNotification",
                "s3:AbortMultipartUpload",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my_bucket_name/*",
                "arn:aws:s3:::my_bucket_name"
            ]
        }
    ]
}

The policy to add layers to lambda only needs one action!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:PublishLayerVersion",
            ],
            "Resource": "*"
        }
    ]
}

You could certainly combine the two policies if you wanted to but I prefer making smaller policies that I can attach to different groups of users if I need to.

We will load the configure_aws_credentials action which can be found in the aws-actions repo on github.

You will need to store the AWS credentials for the user you created earlier in a secret vault on GitHub. DO NOT PUT YOUR CREDENTIALS IN YOUR REPO - of course, you already know this, right? Instructions for stashing your credentials securely on github can be found here

Something to take note of here is that the secrets are stored on a per repo basis. This means that you could create a separate AWS user for each github repo - a pretty good idea if you are remotely paranoid about such things. Even better would be to set up a test account using AWS Organizations to keep your test code separate from other environments - but that’s beyond the scope of this article.

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

Load up python - I’m using python 3.8

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

Finally, the interesting part. This is where we build the distribution, copy it to S3 and then load it into lambda layers. In order for the code to be installed correctly in the layer, it needs to be in a directory called python. The run section below, creates the directory, installs all of the dependencies and then install the code. For the release_id, you can really use anything you want. The format command below is going to create something like MyRepo-BranchName-dev.zip. If you look at the github docs you can find a bunch of other things you can stick on there.

Side Note: Instead of using a standard requirements file, i.e., requirements.txt, I have created a requirements directory with a dex.txt, test.txt and a prod.txt. This allows me to control what is being installed in the production environment which is important when developing for lambda. For example, you don’t need to load boto as lambda already has it, you also don’t want to load up any dev or test tools to the production environment. For that matter, you also do not want to load up all of your tests and documentation files to production, so you should consider writing filters to remove (or not include) this stuff when building your layer code. A simple way to do this would be to use the —exclude flag in zip (check the man page for more info on how that would work).

      - name: Zip it all up and upload to S3
        env:
          release_bucket: my_bucket_name
          release_bucket_uri: s3://my_bucket_name
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python

The code is then compressed with zip and the file is copied to S3. Once the file has been uploaded, the aws command line tool is used to publish the layer in AWS Lambda so that it can be used from other lambdas.

          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

When you are done building your workflow, push it up to github and try it out by creating a new pull request. Don’t be discouraged when it does not work on the first try. In fact, a solid strategy for this would be to build up the file section by section so you can see how it all works. You can see the actions running in the Actions tab in you repo. You should see a history of all your prior runs in this tab too.

GitHub Repo Configuration

The next step is to get your GitHub repo configured - this is going to be a bit different for each team/repo depending on what workflow your team is choosing to use. I personally use a very simple main/feature branch workflow since most of my repos and teams are fairly small so we rarely have to worry about clobbering each other’s changes. This simple workflow is also very fast which allows us to move changes into production multiple times a day. With all that said, go to the Settings tab in your repo and select Branches. We want to create a a new rule in Branch Protection Rules - so click Add Rule. There’s a full description of the options in the GitHub Documentation.

There are a bunch of interesting options in here that can make your development process more secure and less prone to accidents. The option we are particularly interested in is Require status checks to pass before merging. This allows us to hook in our GitHub Actions and will require the Actions to run cleanly before the code can be merged back into your main branch.

GitHub Action Workflows

With the branch rules configured you will now find that the pull request cannot be merged unless the workflow completes successfully. Now that the layer is installed in your test environment, a code reviewer can look at code and then go to the environment to make sure everything is working they way they would expect it to.

Of course, you can do all kinds of testing once you get a couple of workflows running. A few ideas:

Blocking merges until all the unit tests pass
making sure that static testing passes
integration tests - like running api checks on the test environment that the layer was just installed in

Let your imagination run wild!