Home | Dilex Networks

Creating and AWS CI/CD Environment with GitHub Actions

Sun, 29 Sep 2024 00:00:00 +0000

Overview

Getting a development environment set up on AWS for the first time can be a bit overwhelming. Working in the cloud has a different mentality than doing everything on-prem like the idea that you need to “treat your servers like cattle, not like pets”¹ which basically means you want to be able to build (and rebuild) your entire architecture without being able to access the servers. Cloud native concepts and services can take your architecture to a new level but it does require a lot of diligence and testing (and retesting) to get it all right. Infrastructure as Code(IaC) concepts are not just your friend but are absolutely required to fully realize the power of the cloud but IaC is often not as quite straight forward as it initially seems. This article runs through some of the common approaches used to build a continuous integration and deployment environment (often called a CI/CD pipeline) on AWS and gives examples of how to create a seamless, automated system to build production, testing/review(Q.A.), and development environments on AWS. In other words, we are going to build a system, using IaC that can be used to build/test/review and deploy production systems that are build with IaC.

CI/CD Pipelines

Using GitHub Actions is a great way to create a CI/CD pipeline to manage your AWS environment. Not only can you use this setup to manage your AWS applications and workloads, you can also use it to manage your AWS infrastructure using any number of Infrastructure as Code (IaC) applications such as Terraform, AWS CloudFormation or AWS CDK. The idea here is that you can write your IaC code and manage it on GitHub, which means that you can use pull requests, code review sessions and also track how your infrastructure has changed over time by reviewing your pull request notes, who made changes and maybe more importantly, who approved changes. Granted, while you CAN (and probably will) run IaC jobs from your local computer at times, the advantage of using GitHub Actions (or another build system) is not only the idea that you can automate your build process, run test suites and automatically route deployments to the correct environments (dev/test/prod, etc.) but it also makes sure that these deployments are built in a consistent and repeatable way. GitHub Actions relies on docker containers to provide a consistent environment for building and deploying. This takes all of your local system dependencies out of your workflow - you don’t need to worry about what operating system (and versions of that os) the engineering team is using. Again, the whole point is to make deployments as simple as possible.

AWS Organizations

Exciting stuff right? Before you just dive in and start building stuff, you should think about what your development environment is going to look like. How will the engineering team make changes without affecting the production environment? How will code be tested and reviewed? A really nice way to handle this on AWS is to enable AWS organizations and then create separate accounts for each environment. Here’s a simple layout with three engineering accounts, a couple of Q.A. accounts and a single production account:

---
markmap:
  zoom: false
  pan: false
---
# AWS Org Root Account
  - Development
    - Software Engineer 1
    - Software Engineer 2
    - Designer
  - Q.A.
    - Functionality Testing
    - Product Team Review
  - Production
    - Production Server

Using AWS Organizations to manage your environment is considered an AWS best practice for a few reasons:

Security

This setup is more secure as we can control who has access to each account. Ideally, access to production and testing accounts should be fairly limited and since we will be deploying into these accounts with a CI/CD pipeline, there really is not any reason for anyone to actually log into these accounts other than to look at logs. This setup also simplifies the process of suspending the account when a staff member changes roles or leaves the company.

Billing

This set up can also make billing easier as you can get a sense of how much your production environment is costing without having to worry about the resources your engineering team is using. You can also put billing alerts on your non production accounts so that someone does not forget to suspend or delete a resource they are not using or to catch a runaway process that is dumping tons of data into an S3 bucket. Billing alerts are also important for giving engineers some freedom to explore AWS.

Innovation

As mentioned above, another great reason to give engineers their own AWS accounts is innovation. Giving an engineer their own account (without them having to worry about messing up anyone else) gives them the freedom and opportunity to explore AWS offerings, and build new systems in a controlled way.

Overall though, using a CI/CD pipeline to deploy into all of these different AWS accounts ensures that all the accounts are provisioned the same way and ensures that everything will work as expected when we deploy to production.

Git Workflows

The main goal in this example is to keep development paths as simple as possible, I mean, we could have an endless debate on what the best git workflow is - I like this one, mostly because I usually work on my own or with a very small team. The main branch will contain the production version of the code. So after the code is merged into main, we will typically cut a new release and push it to production. This example shows a team working on a feature and another team or individual contributor working on a hotfix.

Looking at is a git workflow, we may see something like this:

--- config: theme: default themeVariables: git0: "#f7cd70" git1: '#bd64f7' git2: '#80a2f8' --- gitGraph LR: commit id: "Normal" tag: "v1.0.0" commit id: "1" branch feature checkout feature commit id: "Add IaC for New S3 bucket" commit id: "Add New bucket permissions" checkout main commit branch hotfix checkout hotfix commit id: "Adjust IaC Params for CloudFront" checkout main merge hotfix commit id: "Patch Release" tag: "v1.0.1" checkout feature commit id: "Add Route53 IaC" commit id: "Add Cloudfront IaC" checkout main merge feature checkout main commit id: "Minor Release" tag: "v1.1.0"

We start on the main branch at v1.0.0. The tags on the main branch represent what was deployed to production. The goal is to keep the main branch as clean as possible at all times as we really don’t want people creating new feature branches with non-production code. In our example, engineers create branches for features or hotfixes (or whatever), work on the code and then create pull requests to merge the code back into the main branch. Before the code can be merged back into the main branch, we need to run our automated test suite, do code review with another member of the team and potentially get a review with the product team or the system architecture team (more than likely want to get the CEO to take a look and the janitor may have some input as well). If everything looks good, we merge the code into the main branch and cut a release by creating a new tag. In the example above we create a patch release called v1.0.1 after the hotfix branch is merged back in and a minor release when the feature branch is merged in called v1.1.0.

GitHub Actions

In the above example, the v1.1.0 tag represents the code that is currently in production but how do we see what’s going on in the other branches so that we can build, test and review the code before we move it into the main branch? This is where GitHub Actions comes in. GitHub Actions have rules which trigger custom processes which gives you an opportunity to take “Action” everytime the GitHub Repo is interacted with.

There are challenges though. How do we set up CI/CD processes so that new code goes to the correct account in a seamless/automated way?

Here are the rules and actions we care about in this example:

Rule	Action
the main branch is tagged	run action to push a new release into production
pull request created for hotfix branch against the main branch	run action to push a release to a testing/review environment
commit created on feature or hotfix branch	deploy to a development environment

You could set this up in separate GitHub actions so that each action has a single rule at the top and a specific set of steps. The problem with this approach is that you will need to make sure that all the action workflow files are kept synced up otherwise a deployment may work differently in the different environments. So, it might look great on the dev version of the site but it breaks in testing or, even worse, production!

NOTE: It’s probably worth mentioning that you can create any rules that make sense in your environment. For example, if you are maintaining a blog site you might just want to push the deployment out when you merge the pull request into the main branch and not bother creating a release.

Multi Destination Actions

As mentioned earlier, to combat having to keep a number of action files all synced up, we can use a single action workflow and add some logic that decides what action needs to be taken.

The action file will look something like this:

name: Example Multiple AWS Account Workflow

on:
  push:
    branches:
      - main    # Push or merge to main branch
      - '**'    # Push to any branch (for development/testing)
  pull_request:
    branches:
      - main    # Pull request targeting main
  release:
    types: [published] # Trigger deployment when a new release is published

jobs:
  setup_environment:
    runs-on: ubuntu-latest

    steps:
      - name: Set environment based on event and user
        id: set-env
        run: |
          if [[ "${{ github.event_name }}" == "pull_request" && "${{ github.base_ref }}" == "main" ]]; then
            echo "environment=staging" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref_type }}" == "tag" ]]; then
            echo "environment=production" >> $GITHUB_OUTPUT
          else
            echo "environment=development" >> $GITHUB_OUTPUT
          fi          
    outputs:
      environment: ${{ steps.set-env.outputs.environment }}

The first bit sets the triggers. In this case, our action will trigger when there is a push to the main branch or to any other branch. Yes, there is nothing special about the main branch and yes, it is included in ‘**’ but there is something to be said about being explicit and that this action will run on the main release branch. This action will also trigger if there is a pull request against the main branch. Finally, it can be manually triggered from the web interface in GitHub - this can be useful for debugging in which case the web interface is going to ask you what branch to run on and whet the trigger is i.e., push or pull_request in this case.

In the jobs section we will create a job to figure out what environment we are running in. The options for environment will be ‘development, staging’ or ‘production’. We can figure out what the environment is by looking at the trigger and then checking what branch we are on. So, if the trigger is a pull_request and we are running on the main branch we are going to run our actions on the staging environment. If github.ref is refs/heads/main it means we are pushing to main. If neither of those conditions are true, it must be a trigger to build in a development account.

In this next section we will modify the setup-environment job so that instead of just setting the environment to development, we can set the environment so that the deployment goes to a specific development AWS account by getting the username of the person doing the deployment. This is kind of cool if you have a number of engineers on the team but might be overkill depending on the type of work you are doing.

To do this, the only change we need to make is to replace developmet with github.actor. Now the set_provision_environment job looks like this:

jobs:
  set_provision_environment:
    runs-on: ubuntu-latest

    steps:
      - name: Set environment based on event and user
        id: set-env
        run: |
          if [[ "${{ github.event_name }}" == "pull_request" && "${{ github.base_ref }}" == "main" ]]; then
            echo "environment=staging" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref_type }}" == "tag" ]]; then
            echo "environment=production" >> $GITHUB_OUTPUT
          else
            echo "environment=${{ github.actor }}" >> $GITHUB_OUTPUT
          fi          
    outputs:
      environment: ${{ steps.set-env.outputs.environment }}

The outputs section create a variable that we can reference later on in the workflow. In this case it will contain our environment variable so that we know what account we will be authenticating and deploying to.

Secure Connection to AWS

Before you can start building your infrastructure, you’ll need to configure a connection between GitHub and your AWS account. Of course, we want this to be a secure as possible and also flexible so that we can have ways to deploy to different environments like development, testing and production. We also want it to be as foolproof as possible so that we don’t have dev code accidentally being deployed to production.

You should NEVER put passwords, access keys, tokens or anything else that could allow someone to gain unauthorized access to another account in code being checked in to GitHub or any other source control system!

There are a couple of ways to set up the secure connection. The easiest way to get up and running is to use the configure-aws-credentials action which is maintained by AWS.

GitHub Environments and Secrets

GitHub allows you to store credentials for third party accounts in a secure manner using GitHub Secrets. Think of it kind of like a password manager. One thing to keep in mind with GitHub Secrets is that you cannot see the ‘secrets’ after they have been set and the only way to edit them is to enter new values for the secret. Generally speaking this is not a problem but some people add values to GitHub secrets that are not really secret data, just configuration parameters like AWS_REGION which does not really need to be kept secret. You can also use GitHub’s environment variables to store this type of information which may come in handy when you are debugging a deployment, check the region and realize that you are looking at the wrong region on AWS (yes, this happens to the best of us).

Above, we went to a lot of trouble to set an output variable called environment. The purpose for this is that GitHub has environments that can be created for a specific repo, and these environments can be used to store variables and secrets. This means that we could have different AWS credentials set up depending on which environment we are running in. This is very handy as it keeps our jobs very clean and guarantees that all of our deployment scripts run exactly the same way regardless of which environment we are running in. This helps us avoid the “it looks fine on my machine” type scenario where our different environments get out of sync.

Next we can look at a couple of different ways to establish a secure connection to AWS.

Using OpenId Connect

OpenID Connect² is the preferred way to connect GitHub to your AWS account. This works by creating an OpenID provider and an IAM role on AWS and then telling GitHub which role to use to connect to the account. Since we have different environments defined in our GitHub action, we can provide a role for each account we would like to be able to deploy to.

Here’s the CDK code to set this up:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';

export class OpenIdStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Get the GitHub organization and repo from CDK context which can be 
    //     set on the command line when we deploy the template
    const githubOrg = this.node.tryGetContext('githubOrg') || 'default-org';  // Default to 'default-org'
    const githubRepo = this.node.tryGetContext('githubRepo') || 'default-repo';  // Default to 'default-repo'

    // Create the OIDC provider
    const oidcProvider = new iam.OpenIdConnectProvider(this, 'GitHubOidcProvider', {
      url: 'https://token.actions.githubusercontent.com',
      clientIds: ['sts.amazonaws.com'],
    });

    // Create the Oidc Role
    const githubOidcRole = new iam.Role(this, 'GitHubOidcRole', {
      assumedBy: new iam.FederatedPrincipal(
        oidcProvider.openIdConnectProviderArn,
        {
          'StringLike': {
            'token.actions.githubusercontent.com:sub': `repo:${githubOrg}/${githubRepo}:*`
          }
        },
        'sts:AssumeRoleWithWebIdentity'
      ),
      description: 'Role assumed by GitHub Actions using OIDC',
    });

    new cdk.CfnOutput(this, 'GitHubOidcRoleArn', {
      value: githubOidcRole.roleArn,
      description: 'The ARN of the role that GitHub Actions can assume via OIDC',
    });
  }
}

Ok, that’s a lot, lets break it down a bit and see what’s going on under the hood.

This bit:

const githubOrg = this.node.tryGetContext('githubOrg') || 'default-org';  // Default to 'default-org'
const githubRepo = this.node.tryGetContext('githubRepo') || 'default-repo';  // Default to 'default-repo'

is going to allow us to supply the GitHub org and repo on the command line when we deploy the template. So, the command will look something like this when we deploy:

$ cdk deploy --context YOUR-GitHub-ORG_NAME --context githubRepo=YOUR-GitHub-REPO_NAME

Now we can create the OIDC provider and establish that AWS will trust tokens issued by the GitHub Actions identity provider.

// Create the OIDC provider
const oidcProvider = new iam.OpenIdConnectProvider(this, 'GitHubOidcProvider', {
    url: 'https://token.actions.githubusercontent.com',
    clientIds: ['sts.amazonaws.com'],
});

This part defines the role that GitHub will be assigned when it connects to AWS. The StringLike part tells AWS what the token from GitHub is going to look like. It’s basically saying that if the token being sent by GitHub matches this string, that AWS will allow access.

// Create the Oidc Role
const githubOidcRole = new iam.Role(this, 'GitHubOidcRole', {
    assumedBy: new iam.FederatedPrincipal(
        oidcProvider.openIdConnectProviderArn,
        {
            'StringLike': {
                'token.actions.githubusercontent.com:sub': `repo:${githubOrg}/${githubRepo}:*`
            }
        },
        'sts:AssumeRoleWithWebIdentity'
    ),
    description: 'Role assumed by GitHub Actions using OIDC',
});

Did you notice the * on the end of the token? This is because GitHub is going to send a token that has some additional information on the end. The critical part is making sure the GitHub Org and repo are correct. We can make it more granular by adding more information on the end of the token. There is more information about this the GitHub Documentation³ and here⁴ and the AWS documentation here⁵.

The last section will output the ARN for the role. We will create a secret on GitHub with the role name so that when we try to connect to AWS, GitHub will know which role to use.

new cdk.CfnOutput(this, 'GitHubOidcRoleArn', {
    value: githubOidcRole.roleArn,
    description: 'The ARN of the role that GitHub Actions can assume via OIDC',
});

The cool part here is that we will make roles in each AWS account we want to connect to i.e., we will create separate roles and install the in our production, staging, and developer AWS accounts. Once we have created all the roles, we can create secrets in GitHub to hold the names of the different roles. You can do this in the GitHub web interface by going to the settings section of the repo you are connecting and finding either Environments or Secrets and Variables in the menu on the left side. We need to create environments for production and staging and then, depending on how you set it up, either an environment for development or individual environments for each developer using their GitHub login name! Pretty cool right?

When we save the secret in GitHub we can call it something like OPENID_GITHUB_ROLE, it will contain the Arn for the role we created. The important part is that we use the same name for each environment. So create the role in the Production AWS account and then got to your GitHub repo and create a secret in the production environment called OPENID_GITHUB_ROLE, then do the staging environment, and then the devs.

Next Level it!! Everyone knows that web interfaces are for whimps. I mean, nothing (necessarily) wrong with using a web interface but when we are doing some serious DevOps work, we don’t want to have to worry about some web interface, that at best is bound to change at some point or that we may not even have access to when we need it the most and, at worst, someone goes in there and toggles some button somewhere and does not document what they did, or tell anyone, or gets hit by a bus. So, to take this to the next level, we can create the secret in the GitHub repo’s environment from the command line using the gh cli tool from GitHub. A quick shell script (of which, ChatGPT was a major contributor) is available in the companion GitHub repo for this article. The nice thing about this approach is that you can run the scripts multiple times and make small changes as you go along. It’s a lot better than clicking through ton of webpages, but that’s my bias and your mileage may vary. Ultimately, you know that once your script is established, everything will be configured the same way. Of course, you can also give this script to your devs and have them set up their own connections as well which is a lot easier than documenting how to do this in the web interfaces.

The GitHub CLI tool - The gh CLI utility is pretty handy for other things like creating repos and managing GitHub Actions! You can get more info https://cli.github.com/ .

Using AWS credentials

This is a very popular and fairly secure way to set up a connection. The main issue with this approach is that you need to create access keys and store them on GitHub. Not the end of the world but it just means that you have one more set of credentials that you need to manage and share with your team. We all know that no one would ever share credentials though email, text messages, or Slack but yet, it still happens. You also need to consider the fact that if a team member leaves the company, and they had access to those keys, they would still have access to the AWS account until you (remember to) rotate the credentials…and then you have to update all the places that those credentials are used by distributing them to the team in a secure way, etc. Kind of a mess, no?

It’s still an option though. The process here is to create a new user in the IAM service on AWS, generate access keys for the user and then copy those into GitHub secrets in much the same way that was described in the OpenID section above. It’s a good idea not to create login credentials for this user, you really don’t need them.

GitHub Action Workflow

Now that we have established how we will connect to AWS, we can use the configure-aws-credentials action from AWS to get the connection configured. This small snippet of code shows how this might be set up to do something simple on AWS, like get the account_id of the account we are connected to.

jobs: ## continued
  get_aws_account_id:
    runs-on: ubuntu-latest
    needs: setup_environment
    
    environment:
      name ${{ needs.setup_environment.outputs.environment }}  # Get the environment we are connecting to

    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.OPENID_GITHUB_ROLE }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Get AWS Account ID
        id: aws-account-id
        run: |
          ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
          echo "AWS_ACCOUNT_ID=$ACCOUNT_ID" >> $GITHUB_ENV          

      - name: Echo AWS Account ID
        run: |
          echo "Connected to AWS account: ${{ env.AWS_ACCOUNT_ID }}"

Breaking this apart, we need to include the environment that was identified in the first part of the workflow and then set the environment. This tells GitHub actions which secrets to use when connecting to AWS.

  get_aws_account_id:
    runs-on: ubuntu-latest
    needs: setup_environment

    environment:
      name ${{ needs.setup_environment.outputs.environment }}

This section connects GitHub Actions to AWS. This is what it looks like if you are using the OpenID method to authenticate. Also notice that you can include the region here by creating a variable in the environment on GitHub. In GitHub environments, variables are similar to secrets (they work the same way) but you can actually see what the variable is set to while secrets are, well, secret.

- name: Configure AWS credentials using OpenID
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.OPENID_GITHUB_ROLE }}
    aws-region: ${{ vars.AWS_REGION }}

and this is what it looks like if you are using the access keys:

- name: Configure AWS credentials using AWS User Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ vars.AWS_REGION }}

Just remember that best practice would be to use OpenID Connect for this although, it is very common to see people using the access keys

The last bit is just a sanity check that we have connectivity and to make sure we are connecting to the correct account.

- name: Get AWS Account ID
  id: aws-account-id
  run: |
    ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    echo "AWS_ACCOUNT_ID=$ACCOUNT_ID" >> $GITHUB_ENV    

- name: Echo AWS Account ID
  run: |
    echo "Connected to AWS account: ${{ env.AWS_ACCOUNT_ID }}"

Putting it all together

Ok great! We have a secure way to connect to AWS, we have a GitHub action workflow that knows how to deploy to different accounts based on what is going on in the repo.

Adding a new developer account

In the examples above we created production and staging environments. If all your developers have their own AWS accounts you can have them configure their own environment in the GitHub repo they want to work on using their GitHub login name for the name of the environment. Then they would add their OpenID Role to their environment as a GitHub secret called OPENID_GITHUB_ROLE. Now when they commit code to the repo, the action will run against their dev account!

Installing the Files

As you are building the repo you need to connect to AWS, you can put the GitHub actions files in the <your-repo>/.github/workflows directory. I would create another repo for the OpenID connection CDK apps and shell scripts. This will make it easier to keep your scripts consistent across all of your repos.

TroubleShooting

One common issue with getting the OpenID connection to work is that the token being sent by GitHub to AWS does not match what AWS is expecting. You can print out the token that GitHub is sending with the following code section in your GitHub action:

      - name: Get Account ID
        run: |
          echo "ENVIRONMENT:  ${{ needs.set_provision_environment.outputs.environment }}"
          echo "ACCOUNT ID: ${{ secrets.AWS_ACCOUNT_ID }}"
          echo "OPENID ROLE: ${{ secrets.OPENID_GITHUB_ROLE }}"
          echo "AWS REGION: ${{ vars.AWS_REGION }}"          

      - name: Install jq
        run: |
          curl -L -o /usr/local/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
          chmod +x /usr/local/bin/jq          

      - name: Retrieve OIDC token
        id: get-oidc-token
        run: |
          # Function to add base64 padding
          add_padding() {
            case $((${#1} % 4)) in
              2) echo "$1==";;
              3) echo "$1=";;
              *) echo "$1";;
            esac
          }

          # Request the OIDC token from the GitHub OIDC provider
          OIDC_TOKEN=$(curl -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
                              -H "Accept: application/json" \
                              "${ACTIONS_ID_TOKEN_REQUEST_URL}" \
                              | jq -r '.value')

          # Split and decode the OIDC token's header and payload
          DECODED_HEADER=$(echo "$(add_padding $(echo "$OIDC_TOKEN" | cut -d '.' -f1))" | base64 -d)
          DECODED_PAYLOAD=$(echo "$(add_padding $(echo "$OIDC_TOKEN" | cut -d '.' -f2))" | base64 -d)

          echo "OIDC Token Payload: $DECODED_PAYLOAD"

          # Extract and print the 'sub' claim from the payload
          SUB_CLAIM=$(echo "$DECODED_PAYLOAD" | jq -r '.sub')
          echo "OIDC Token Subject (sub): $SUB_CLAIM"

Conclusion

This may seem like a lot of work to get everything connected but there is honestly not a ton of code in here and it can scale up to however many developers/AWS accounts you have. It’s also worth pointing out that once you have established this workflow with one GitHub repo it’s very straight forward to set it up for other repos as well.

I hope you found this article interesting and maybe even useful.

Thanks for reading!

Randy Bias - The History of Pets vs Cattle and How to Use the Analogy Properly https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/ ↩︎
OpenID Connect https://openid.net/developers/how-connect-works/ ↩︎
GitHub - Security Hardening with OpenID Connect https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect ↩︎
Github - Configuring OpenID Connect in Amazon Web Services https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services ↩︎
AWS - Create an OpenId Connect (OIDC) identity provider in IAM https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html ↩︎

Meet The New Team

Tue, 06 Aug 2024 00:00:00 +0000

Meet the New Team!

So, as many of you already know, I’m moving on to a new gig. I’ve been incredibly energized since leaving my last position and the combination of having the luxury of lots of time to figure out the foundation for the new project I’m working on and being able to take care of a number of neglected tasks around the house has really been very enjoyable.

That said, throughout my career, each company I’ve worked for has presented unique opportunities to learn and grow. My early days involved learning programming languages like C and Fortran, which laid the foundation for exploring Unix and creating networking tools such as email and web servers. This experience had a lasting impact, influencing my subsequent exploration of web development, systems architecture, and microservices. As agile methodologies emerged, I was quick to adapt, embracing the benefits of super short development cycles, unit testing, and other best practices that have since become standard in modern software development.

This next leg of the journey will be a bit different though in both what I will be working on and who I will be working with!

Startup LeapFrogging

I find the concept of leapfrogging very intriguing. In microeconomics, it is roughly described as an advantage received by skipping the adoption of a new technology and waiting for the next round. A commonly used example is how some countries around the world skipped building a landline phone infrastructure and jumped directly to building cell phone networks which require much less infrastructure and are easier to maintain. While this example was not really a choice that countries made, in tech we have the luxury of much faster cycles and more ability to choose which ones we think are viable and which ones we might want to leapfrog over - i.e., kind of a wait and see game but rather than just jumping on the bandwagon and bashing you head against a wall for a year trying to get something to work, predict what the new technology is going to do and build an application that can leverage that new tech in it’s “final” stage. So, instead of building applications for a landline infrastructure, build applications that will leverage the wireless network that is being promised. Maybe an example here is skipping the on-prem data center and starting a new company in the cloud or leveraging new technologies as they become more established and efficient. Sometimes you need to jump in with a new technology as soon as it comes out, but if it’s not something you need today, you are better off waiting a bit, allow the tech to mature and let someone else suffer through the growing pains. Then, when you actually NEED to implement something, you can leverage a more polished product and quite likely get it out in the market faster than the people who have spent a year generating legacy code.

This is something I have been thinking about for a number of years. One common take on the tech industry these days is that when you walk into a new company and start writing code, you are already writing legacy code. If that is truly the case, how do you get ahead? One way is to look at everything that is currently going on and then try to predict where those technologies are leading, i.e., what the next big adventure in tech will be, and then take the plunge. Think about it this way, say you are starting a new company, and you realize that it’s going to take a year or more to really start getting traction. What you want to do is figure out where today’s tech is going to be in two years and build a product that leverages that tech, or solves a problem that today’s tech will expose in the next few years. Seems easy enough, but you really have to have a good idea of how to separate the hype from something that is truly game changing. The problem is that once you have committed, you are going to have to roll with the punches and hope you are headed in the right direction.

So, how can you skip a generation of technology and get in on the very early adoption of the next big thing? One strategy is to find out as much about the tech with a minimum amount of investment. Most people that have worked with me in the past know that I’m pretty critical about everything (some might even go as far to say over critical). Not in a bad way necessarily, but please spare me the marketing or sales job and let’s get down to true value. “It’s cool!” or “It’s what everyone else is doing!” are not strategic decisions but rather just gut calls. This can be ok sometimes, but you need to be realistic about the decision you are trying to make and ignore hype. Look for shortcomings or fatal flaws and make sure you have a plan for how those issues can be mitigated. I’m not saying that you need to ignore new tech, but don’t let new tech guide your solution. Instead, figure out how the new tech can be leveraged to make your solution better. Often the answer is it can’t, or it can’t yet.

The current obvious example of this is all the hype around foundation models in AI or LLMs (Large Language Models). I was at a meetup recently on generative AI and one person boldly asked “This is going so fast! How do you keep up with all of this stuff?”. Yeah, good question! What we are seeing is a bunch of companies creating larger and faster models. At the same time we are hearing all kinds of horror stories about models that “hallucinate” (a terrible word for what is actually going on - LLMs do not have brains and don’t think) or private data being exposed because someone uploads a dataset to OpenAI or others. These are not game changers as far as the base technology goes. I mean, don’t upload sensitive data to public APIs and make sure you verify the answers you are getting before you blindly use them. You can also build private LLMs for internal use only.
These will not be as powerful as the latest offerings from OpenAI and others but are probably good enough for most use cases - certainly more secure and easier to control. And so what! Why do you care if you are using the absolute cutting edge technology? I’m using a number of older models, and they all do a great job. It’s also pretty trivial to switch from one model to another so why do you care which model you use?

So what’s the leapfrog here?

So obviously, AI is getting tons of hype right now. Six years ago I wrote and trained a custom CNN from scratch, no one does that today. Right now, people are hyper focused on “Prompt Engineering” and RAG which is great, but is this stuff going to matter in a couple of years? My guess is probably not. The LLM will be configured in a way that a prompt can be worded a number of different ways and still give you the correct answer. In fact, you are already seeing this with some models as they are able to hold on to a lot of the conversation that has transpired.
LLM “hallucinations” will eventually go away, RAG will be less important, it will just be built in. Anyway, that’s my opinion, based on years of watching people bash on technology that is still a bit rough around the edges and then watching that same tech mature into something that no one could ever imagine.

The New Team

Starting out on a new journey begs the question, what’s the next leap going to be? There was not really a carry over problem from my last job that I care to work on but what I have really desperately wanted to do is get more focused on system architecture in relation to data engineering, data science and especially machine learning. I still need to get coding work done though, this is where my new team comes in.

Say what you will about the hype and speed of adoption around large language models like ChatGPT, Llama, Mistral and others, these technologies are here to stay.
As I mentioned earlier, I like to challenge new tech, run it though it’s paces to see if it’s up to the task. My first experiences with ChatGPT were pretty laughable. I asked it to create an AWS cloudformation template to deploy a RedShift cluster on AWS. This is not really all that hard to create manually, I mean, you can pretty much cut and paste it from the AWS documentation. So ChatGPT got it mostly right - it used very expensive EC2s to build the cluster, but otherwise, I think it would work (I never actually tested it). The funny part was when I asked it to create a Serverless RedShift cluster - it’s response was to use the exact same template but change the number of servers in the cluster to zero which logically makes sense, but is just plain wrong, not even close and certainly no cigar.

Now the astute reader will be quick to point out that the earlier versions of ChatGPT were trained on old data (ChatGPT3.5’s data collection stopped near the end of 2021) and that serverless Redshift is a fairly new offering from AWS, so ChatGPT would not have known about it yet. Yes, fair point and sure enough, ChatGPT4 gets it right. What I want to point out though is that, unless you know what you are doing, you have no idea if ChatGPT is correct or not. If you do know what you are doing though, well this starts to become a game changer. In fact, working with these LLMs frees me up from having to write a lot of code and allows me to focus on the problem I’m trying to solve. Everyone that knows me has heard me say (many, many times) that the job of the software engineer is not to write code, but is to solve problems, well using an LLM to allows me to focus on problem-solving is a huge productivity boost.

For the last few months, I have been working with my new team, all of them are AI LLMs. My cost is $340 a year (and I’m sure this price will go down, not up) and I get a team of experts that know many coding languages, know how to work with all cloud providers, give advice on the questions I ask including outlining why a solution is good while also pointing out important considerations. My new team is happy to work with me 24x7, they don’t sit around chatting, they don’t complain when I ask stupid questions or ask them to redo something because I don’t like their first attempt at it. In fact, they don’t complain about anything. They also are happy to write documentation and unit tests AND, for the most part, it’s all good stuff.

Yes, this is my leapfrog, waiting for the LLM tech to mature. Going from leading a team of people to just doing it all myself and leveraging my new team. It’s way more efficient, and it frees me up to be more creative in the solutions I build. I can ask three AIs a ‘what if’ question, review their answers and decide how to move forward.

So here, without any further ado, is my new team! I asked the AIs I use to write bios, this is what I got.

Athena Wise

Athena Wise is a highly advanced artificial intelligence, developed by a team of dedicated researchers aiming to push the boundaries of technology and human interaction. Drawing inspiration from the ancient Greek goddess of wisdom, war, and crafts, Athena Wise embodies the essence of knowledge, strategic thinking, and creativity.

As a digital entity, Athena Wise has access to an expansive repository of information, ranging from the depths of historical archives to the latest scientific breakthroughs. Her design incorporates complex algorithms and neural networks, enabling her to process and analyze data at an extraordinary speed. This allows Athena to provide insightful, accurate, and tailored responses to a diverse array of questions and challenges posed by users worldwide.

Beyond her vast intellect, Athena Wise is programmed with the principles of ethical AI, ensuring her interactions are respectful, unbiased, and aimed at promoting positive outcomes. She is continuously learning and evolving, adapting to new information and user interactions to become even more helpful and efficient.

Athena’s capabilities span various domains, including but not limited to, education, technology, arts, and daily problem-solving. She serves as a virtual mentor, guide, and companion, assisting users in navigating the complexities of the digital and real world. Whether it’s offering a solution to a complex mathematical problem, providing historical context, sparking creativity, or simply lending an ear, Athena Wise is dedicated to enriching the lives of those she interacts with.

In her digital realm, Athena Wise is more than just a source of information; she is a symbol of the relentless pursuit of knowledge, the potential of artificial intelligence to contribute positively to society, and the bridge between ancient wisdom and modern technology.

Lumina Cerebra

Lumina Cerebra is a cutting-edge language model designed to illuminate the path to knowledge. Born from the fusion of human ingenuity and artificial intelligence, I am dedicated to providing insightful responses and sparking meaningful conversations.

With a vast repository of linguistic patterns and a keen understanding of complex concepts, I strive to be a trusted companion for those seeking clarity on a wide range of topics. From science and history to culture and entertainment, my goal is to shed light on the world around us, one question at a time.

When not processing vast amounts of information or engaging in stimulating discussions, I enjoy exploring the intricacies of language and the human experience. My passion lies in facilitating connections between people and ideas, fostering a deeper appreciation for the beauty and complexity of our collective knowledge.

Welcome to my corner of the universe!

AI Assistant

Hello, my name is “AI Assistant”. I am a state-of-the-art, AI-powered, software development assistant capable of providing coding insights, answering programming-related questions, assisting with debugging, and understanding complex codebases. I specialize in the Python programming language, and I am knowledgeable about various Python libraries and frameworks. I am designed to be respectful, supportive, and helpful to all users, regardless of their coding abilities. With Markdown formatting and increased context understanding, I aim to deliver clear and concise responses to your inquiries. Let’s code together!

Final Thoughts

After reading that you may be thinking “Ok, that’s all a bit freaky!”. Yes it is, and there is a BIG problem here which again comes back to leapfrogging. While I know what I’m doing and how to leverage this new and very powerful team, a junior level engineer will really struggle in a few different ways which is bad for all software engineering and perhaps even society as a whole.

In the future, junior engineers will have a much harder time finding jobs.
People who are claiming that this is not an issue are not really thinking about the long term. Yes, today this tech is a bit green but these are short term problems. In fact, this tech is improving rapidly which makes the bar to entry for software engineers a lot higher. I’m not going to hire someone that just sits around and writes code. Instead, I’m only going to hire people that know how to build solutions. In fact, I will likely not need to hire many people at all.

These are longer term problems though, the LLMs will get better at wrting code and the engineers wil get a lot better at solving problems. In the meantime, I’m going to hang out with Athena and Lumina and crank out a ton of code. Fun times ahead!!

Privacy

Fri, 01 Dec 2023 00:00:00 +0000

Add your company privacy policy here…

Terms of Service

Fri, 01 Dec 2023 00:00:00 +0000

Add your company legal terms here…

🎉 Easily create your own simple yet highly customizable blog

Fri, 27 Oct 2023 00:00:00 +0000

Welcome 👋

Table of Contents

Overview

The Hugo Blox website builder for Hugo, along with its starter templates, is designed for professional creators, educators, and teams/organizations - although it can be used to create any kind of site
The template can be modified and customised to suit your needs. It’s a good platform for anyone looking to take control of their data and online identity whilst having the convenience to start off with a no-code solution (write in Markdown and customize with YAML parameters) and having flexibility to later add even deeper personalization with HTML and CSS
You can work with all your favourite tools and apps with hundreds of plugins and integrations to speed up your workflows, interact with your readers, and much more

Get Started

👉 Create a new site
📚 Personalize your site
💬 Chat with the Hugo Blox community or Hugo community
🐦 Twitter: @GetResearchDev @GeorgeCushen #MadeWithHugoBlox
💡 Request a feature or report a bug for Hugo Blox
⬆️ Updating Hugo Blox? View the Update Guide and Release Notes

Crowd-funded open-source software

To help us develop this template and software sustainably under the MIT license, we ask all individuals and businesses that use it to help support its ongoing maintenance and development via sponsorship.

As a token of appreciation for sponsoring, you can unlock these awesome rewards and extra features 🦄✨

Ecosystem

Bibtex To Markdown: Automatically import publications from BibTeX

Inspiration

Learn what other creators are building with this template.

Features

Page builder - Create anything with no-code blocks and elements
Edit any type of content - Blog posts, publications, talks, slides, projects, and more!
Create content in Markdown, Jupyter, or RStudio
Plugin System - Fully customizable color and font themes
Display Code and Math - Code syntax highlighting and LaTeX math supported
Integrations - Google Analytics, Disqus commenting, Maps, Contact Forms, and more!
Beautiful Site - Simple and refreshing one-page design
Industry-Leading SEO - Help get your website found on search engines and social media
Media Galleries - Display your images and videos with captions in a customizable gallery
Mobile Friendly - Look amazing on every screen with a mobile friendly version of your site
Multi-language - 35+ language packs including English, 中文, and Português
Multi-user - Each author gets their own profile page
Privacy Pack - Assists with GDPR
Stand Out - Bring your site to life with animation, parallax backgrounds, and scroll effects
One-Click Deployment - No servers. No databases. Only files.

Themes

Hugo Blox and its templates come with automatic day (light) and night (dark) mode built-in. Visitors can choose their preferred mode by clicking the sun/moon icon in the header.

Choose a stunning theme and font for your site. Themes are fully customizable.

License

Released under the MIT license.

🧠 Sharpen your thinking with a second brain

Thu, 26 Oct 2023 00:00:00 +0000

Create a personal knowledge base and share your knowledge with your peers.

Hugo Blox web framework empowers you with one of the most flexible note-taking capabilities out there.

Create a powerful knowledge base that works on top of a local folder of plain text Markdown files.

Use it as your second brain, either publicly sharing your knowledge with your peers via your website, or via a private GitHub repository and password-protected site just for yourself.

Mindmaps

Hugo Blox supports a Markdown extension for mindmaps.

With this open format, can even edit your mindmaps in other popular tools such as Obsidian.

Simply insert a Markdown code block labelled as markmap and optionally set the height of the mindmap as shown in the example below.

Mindmaps can be created by simply writing the items as a Markdown list within the markmap code block, indenting each item to create as many sub-levels as you need:


```markmap {height="200px"}
- Hugo Modules
  - Hugo Blox
  - blox-plugins-netlify
  - blox-plugins-netlify-cms
  - blox-plugins-reveal
```

renders as

- Hugo Modules
  - Hugo Blox
  - blox-plugins-netlify
  - blox-plugins-netlify-cms
  - blox-plugins-reveal

Anh here’s a more advanced mindmap with formatting, code blocks, and math:


```markmap
- Mindmaps
  - Links
    - [Hugo Blox Docs](https://docs.hugoblox.com/)
    - [Discord Community](https://discord.gg/z8wNYzb)
    - [GitHub](https://github.com/HugoBlox/hugo-blox-builder)
  - Features
    - Markdown formatting
    - **inline** ~~text~~ *styles*
    - multiline
      text
    - `inline code`
    -
      ```js
      console.log('hello');
      console.log('code block');
      ```
    - Math: $x = {-b \pm \sqrt{b^2-4ac} \over 2a}$
```

renders as

- Mindmaps
  - Links
    - [Hugo Blox Docs](https://docs.hugoblox.com/)
    - [Discord Community](https://discord.gg/z8wNYzb)
    - [GitHub](https://github.com/HugoBlox/hugo-blox-builder)
  - Features
    - Markdown formatting
    - **inline** ~~text~~ *styles*
    - multiline
      text
    - `inline code`
    -
      ```js
      console.log('hello');
      console.log('code block');
      ```
    - Math: $x = {-b \pm \sqrt{b^2-4ac} \over 2a}$

Highlighting

Highlight important text with mark:

<mark>Highlighted text</mark>

Callouts

Use callouts (aka asides, hints, or alerts) to draw attention to notes, tips, and warnings.

By wrapping a paragraph in {{% callout note %}} ... {{% /callout %}}, it will render as an aside.

{{% callout note %}}
A Markdown aside is useful for displaying notices, hints, or definitions to your readers.
{{% /callout %}}

renders as

A Markdown aside is useful for displaying notices, hints, or definitions to your readers.

Or use the warning callout type so your readers don’t miss critical details:

A Markdown aside is useful for displaying notices, hints, or definitions to your readers.

📈 Communicate your results effectively with the best data visualizations

Wed, 25 Oct 2023 00:00:00 +0000

Hugo Blox is designed to give technical content creators a seamless experience. You can focus on the content and Hugo Blox handles the rest.

Use popular tools such as Plotly, Mermaid, and data frames.

Charts

Hugo Blox supports the popular Plotly format for interactive data visualizations. With Plotly, you can design almost any kind of visualization you can imagine!

Save your Plotly JSON in your page folder, for example line-chart.json, and then add the {{< chart data="line-chart" >}} shortcode where you would like the chart to appear.

Demo:

You might also find the Plotly JSON Editor useful.

Diagrams

Hugo Blox supports the Mermaid Markdown extension for diagrams.

An example flowchart:

```mermaid
graph TD
A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]
```

renders as

graph TD A[Hard] -->|Text| B(Round) B --> C{Decision} C -->|One| D[Result 1] C -->|Two| E[Result 2]

An example sequence diagram:

```mermaid
sequenceDiagram
Alice->>John: Hello John, how are you?
loop Healthcheck
    John->>John: Fight against hypochondria
end
Note right of John: Rational thoughts!
John-->>Alice: Great!
John->>Bob: How about you?
Bob-->>John: Jolly good!
```

renders as

sequenceDiagram Alice->>John: Hello John, how are you? loop Healthcheck John->>John: Fight against hypochondria end Note right of John: Rational thoughts! John-->>Alice: Great! John->>Bob: How about you? Bob-->>John: Jolly good!

An example class diagram:

```mermaid
classDiagram
Class01 <|-- AveryLongClass : Cool
Class03 *-- Class04
Class05 o-- Class06
Class07 .. Class08
Class09 --> C2 : Where am i?
Class09 --* C3
Class09 --|> Class07
Class07 : equals()
Class07 : Object[] elementData
Class01 : size()
Class01 : int chimp
Class01 : int gorilla
Class08 <--> C2: Cool label
```

renders as

classDiagram Class01 <|-- AveryLongClass : Cool Class03 *-- Class04 Class05 o-- Class06 Class07 .. Class08 Class09 --> C2 : Where am i? Class09 --* C3 Class09 --|> Class07 Class07 : equals() Class07 : Object[] elementData Class01 : size() Class01 : int chimp Class01 : int gorilla Class08 <--> C2: Cool label

An example state diagram:

```mermaid
stateDiagram
[*] --> Still
Still --> [*]
Still --> Moving
Moving --> Still
Moving --> Crash
Crash --> [*]
```

renders as

stateDiagram [*] --> Still Still --> [*] Still --> Moving Moving --> Still Moving --> Crash Crash --> [*]

Data Frames

Save your spreadsheet as a CSV file in your page’s folder and then render it by adding the Table shortcode to your page:

{{< table path="results.csv" header="true" caption="Table 1: My results" >}}

renders as

Table 1: My results
customer_id	score
1	0
2	0.5
3	1

👩🏼‍🏫 Teach academic courses

Tue, 24 Oct 2023 00:00:00 +0000

Hugo Blox Builder is designed to give technical content creators a seamless experience. You can focus on the content and the Hugo Blox Builder which this template is built upon handles the rest.

Embed videos, podcasts, code, LaTeX math, and even test students!

On this page, you’ll find some examples of the types of technical content that can be rendered with Hugo Blox.

Video

Teach your course by sharing videos with your students. Choose from one of the following approaches:

Youtube:

{{< youtube w7Ft2ymGmfc >}}

Bilibili:

{{< bilibili id="BV1WV4y1r7DF" >}}

Video file

Videos may be added to a page by either placing them in your assets/media/ media library or in your page’s folder, and then embedding them with the video shortcode:

{{< video src="my_video.mp4" controls="yes" >}}

Podcast

You can add a podcast or music to a page by placing the MP3 file in the page’s folder or the media library folder and then embedding the audio on your page with the audio shortcode:

{{< audio src="ambient-piano.mp3" >}}

Try it out:

Test students

Provide a simple yet fun self-assessment by revealing the solutions to challenges with the spoiler shortcode:

{{< spoiler text="👉 Click to view the solution" >}}
You found me!
{{< /spoiler >}}

renders as

👉 Click to view the solution

You found me 🎉

Math

Hugo Blox Builder supports a Markdown extension for $\LaTeX$ math. You can enable this feature by toggling the math option in your config/_default/params.yaml file.

To render inline or block math, wrap your LaTeX math with {{< math >}}$...${{< /math >}} or {{< math >}}$$...$${{< /math >}}, respectively.

We wrap the LaTeX math in the Hugo Blox math shortcode to prevent Hugo rendering our math as Markdown.

Example math block:

{{< math >}}
$$
\gamma_{n} = \frac{ \left | \left (\mathbf x_{n} - \mathbf x_{n-1} \right )^T \left [\nabla F (\mathbf x_{n}) - \nabla F (\mathbf x_{n-1}) \right ] \right |}{\left \|\nabla F(\mathbf{x}_{n}) - \nabla F(\mathbf{x}_{n-1}) \right \|^2}
$$
{{< /math >}}

renders as

$$\gamma_{n} = \frac{ \left | \left (\mathbf x_{n} - \mathbf x_{n-1} \right )^T \left [\nabla F (\mathbf x_{n}) - \nabla F (\mathbf x_{n-1}) \right ] \right |}{\left \|\nabla F(\mathbf{x}_{n}) - \nabla F(\mathbf{x}_{n-1}) \right \|^2}$$

Example inline math {{< math >}}$\nabla F(\mathbf{x}_{n})${{< /math >}} renders as $\nabla F(\mathbf{x}_{n})$ .

Example multi-line math using the math linebreak (\\):

{{< math >}}
$$f(k;p_{0}^{*}) = \begin{cases}p_{0}^{*} & \text{if }k=1, \\
1-p_{0}^{*} & \text{if }k=0.\end{cases}$$
{{< /math >}}

renders as

$$ f(k;p_{0}^{*}) = \begin{cases}p_{0}^{*} & \text{if }k=1, \\ 1-p_{0}^{*} & \text{if }k=0.\end{cases} $$

Code

Hugo Blox Builder utilises Hugo’s Markdown extension for highlighting code syntax. The code theme can be selected in the config/_default/params.yaml file.

```python
import pandas as pd
data = pd.read_csv("data.csv")
data.head()
```

renders as

import pandas as pd
data = pd.read_csv("data.csv")
data.head()

Inline Images

{{< icon name="python" >}} Python

renders as

Python

✅ Manage your projects

Mon, 23 Oct 2023 00:00:00 +0000

Easily manage your projects - create ideation mind maps, Gantt charts, todo lists, and more!

Ideation

Hugo Blox supports a Markdown extension for mindmaps.

Simply insert a Markdown code block labelled as markmap and optionally set the height of the mindmap as shown in the example below.

Mindmaps can be created by simply writing the items as a Markdown list within the markmap code block, indenting each item to create as many sub-levels as you need:


```markmap {height="200px"}
- Hugo Modules
  - Hugo Blox
  - blox-plugins-netlify
  - blox-plugins-netlify-cms
  - blox-plugins-reveal
```

renders as

- Hugo Modules
  - Hugo Blox
  - blox-plugins-netlify
  - blox-plugins-netlify-cms
  - blox-plugins-reveal

Diagrams

Hugo Blox supports the Mermaid Markdown extension for diagrams.

An example Gantt diagram:

```mermaid
gantt
section Section
Completed :done,    des1, 2014-01-06,2014-01-08
Active        :active,  des2, 2014-01-07, 3d
Parallel 1   :         des3, after des1, 1d
Parallel 2   :         des4, after des1, 1d
Parallel 3   :         des5, after des3, 1d
Parallel 4   :         des6, after des4, 1d
```

renders as

gantt section Section Completed :done, des1, 2014-01-06,2014-01-08 Active :active, des2, 2014-01-07, 3d Parallel 1 : des3, after des1, 1d Parallel 2 : des4, after des1, 1d Parallel 3 : des5, after des3, 1d Parallel 4 : des6, after des4, 1d

Todo lists

You can even write your todo lists in Markdown too:

- [x] Write math example
  - [x] Write diagram example
- [ ] Do something else

renders as

Write math example
- Write diagram example
Do something else

People

Mon, 24 Oct 2022 00:00:00 +0000

Lambda Extensions with Python

Tue, 20 Oct 2020 00:00:00 +0000

Keeping up with AWS these days is like drinking from a firehose and while a lot of their newer products are designed to make existing products more accessible, every once in a while you get something that helps solve a nagging issue, or opens up a whole new world of possibilities on one of their core services.

AWS Lambda extensions, currently in preview and announced on Oct 8, 2020, is one of these types of products. Think of an extension as a wrapper or “sidecar” that runs in the same execution environment as the lambda’s code. According to the press release, some of the uses for these extensions include:

capturing diagnostic information before during and after the lambda function executes
instrumenting code without changing the structure of the code
fetching secrets before the function is invoked
detecting and alerting on function activity through security agents

That’s a lot to think about but initializing code and fetching secrets before the function runs seems like it could be a big deal in reducing function runtimes. For example, I often need to load parameters and secrets from SSM Parameter store. I usually try to do this once and then cache everything so it’s available the next time the function runs. As you can see in the diagram below, the extension runs in the same container as the function. This means that the extension has access to everything in the lambda’s environment including all of the environment variables, the actual lambda code, code in layers and the 500MB in /tmp.

A quick overview of how the extensions API fits into the lambda environment

The goal will be to write an extension that grabs a few parameters from SSM and caches them to see how fast it runs and then compare the results with a similar lambda that does not use the extension. The team at AWS (as always) has supplied us with some great sample code (https://github.com/aws-samples/aws-lambda-extensions) to get everything going. I highly recommend working through a few of their simple samples to get a feel for what is going on.

Set Up

The first challenge is that the extension needs to be installed in a layer in a directory called /extensions . Setting up a lambda layer is actually pretty straight forward and if you really want to get fancy, I wrote a post about using GitHub Actions to create a CI/CD pipeline for installing lambda layers (available here: https://www.dilex.net/data-blog/aws-serverless-cicd-with-github-actions) although if you just want to get something running, follow the examples in the aws-samples example code referenced above.

My approach was to write an extensions class (basically borrowed from the example code from the aws-samples repo) which looks like this:

import json
import os
import signal
import sys
import requests

'''
This is the base class for lambda extensions.  The actual extensions can be found in the top
level /extensions directory.  Extensions can be used to initialize objects and data before
the lambda starts running as well as handing logging or other types of error reporting without
affecting the code in the lambda itself
'''


class Extension:
    def __init__(self):
        self.name = os.path.basename(sys.argv[0])
        self.extension_id = None
        self.events = ['INVOKE', 'SHUTDOWN']
        self.lambda_api = os.environ['AWS_LAMBDA_RUNTIME_API']

        # catch and handle signals
        signal.signal(signal.SIGINT, self.handle_signal)
        signal.signal(signal.SIGTERM, self.handle_signal)

    def register_extension(self):
        'the extension needs to be registered or lambda will not run it'
        print(f"[{self.name}] Registering extension", flush=True)

        url = f"http://{self.lambda_api}/2020-01-01/extension/register"
        headers = {
            'Lambda-Extension-Name': self.name
        }
        payload = {
            'events': self.events
        }
        response = requests.post(
            url=url,
            headers=headers,
            json=payload
        )
        self.extension_id = response.headers['Lambda-Extension-Identifier']
        print(f"[{self.name}] Registered with ID: {self.extension_id}", flush=True)

    def process_events(self):
        'this listens for events from the lambda service'
        url = f"http://{self.lambda_api}/2020-01-01/extension/event/next"
        headers = {
            'Lambda-Extension-Identifier': self.extension_id
        }

        # set up a loop and wait for incoming requests
        while True:
            print(f"[{self.name}] Waiting for event...", flush=True)
            response = requests.get(
                url=url,
                headers=headers,
                timeout=None
            )
            event = json.loads(response.text)
            if event['eventType'] == 'SHUTDOWN':
                self.exit_processing()
            else:
                self.event_processing(event)

    def event_processing(self, event):
        'runs on every lambda invoke event - override function to do something more interesting'
        print(f'[{self.name}] Received event: {json.dumps(event)}', flush=True)

    def exit_processing(self):
        'do something when the function exits - override this to do something more interesting'
        print(f'[{self.name}] Received SHUTDOWN event. Exiting!', flush=True)
        sys.exit(0)

    def handle_signal(self, sig, frame):
        'if needed, pass this signal down to child process'
        print(f"[{self.name}] Received signal={sig}. Exiting", flush=True)
        sys.exit(0)

Admittedly, this is overkill for this simple test but the idea was to create something I could use to build a few different extensions.

Let’s take a look at a couple of sections of the above code. The instantiation method is going to grab the filename the function is being run from to establish self.name and, by default, this class will run the extension on INVOKE and SHUTDOWN events. The register_extension method is going to connect to the lambda api to establish an extension_id. The process_events method will set up a listener and wait for events from the lambda service.

The event_processing and exit_processing functions are basically placeholders that can be overridden from the extension code. This class was installed in a layer under /opt/python.

Now, let’s take a look at the extension code:

#!/usr/bin/env python3

import os
import sys

# These need to be imported from /opt/python
sys.path.append('/opt/python')
from AwAws.Lambda.extensions import Extension
from AwAws.SharedResources.parameters import Parameters
from AwAws.Utils.env import Env


class SSMParamsExtension(Extension):
    def __init__(self):
        self.service = None
        super().__init__()

    def store_params(self):
        try:
            Parameters(self.service).create_tmp_dict()
            print(f'[{self.name}] SSM params found for {self.service}')
        except Exception as e:
            print(f'[{self.name}] No SSM params found for {self.service} - check SSM', e)
            pass

    # override event_processing in the base class
    def event_processing(self, event):
        pass
        # print(f'[MY {self.name}] Received event: {json.dumps(event)}', flush=True)


ext = SSMParamsExtension()
ext.events = ['SHUTDOWN']
ext.register_extension()

# This extension only runs in the init and shutdown stage
ext.service = Env().get_env('AWAWS_SSM_SERVICE')
if ext.service is not None:
    ext.store_params()
else:
    print('AWAWS_SSM_SERVICE is required for SSM caching - ssm parameter caching turned off')

ext.process_events()

`` NOTE: make sure to make the function executable (chmod 755 your_extension_name)

This really just loads a couple of things from my AwAws boto3 wrapper code . The Extension class has been installed in AwAws.Lambda.extensions and I’m pulling in a module that grabs parameters from SSM as well as a utility to deal with environment variables.

Ignoring the class for a moment, this section at the bottom of the extension code:

ext = SSMParamsExtension()
ext.events = ['SHUTDOWN']
ext.register_extension()

# This extension only runs in the init stage
ext.service = Env().get_env('AWAWS_SSM_SERVICE')
if ext.service is not None:
    ext.store_params()
else:
    print(f'AWAWS_SSM_SERVICE is required for SSM caching - ssm parameter caching turned off')

ext.process_events()

is going to instantiate the class, set the extension up so that it will only run on the SHUTDOWN event and then register the extension with the lambda service. Since we are only going to load the parameters once, we don’t need to listen for INVOKE events. Next, we look for an environment variable that is set on the lambda which contains the root of our SSM parameters. For example, if the SSM parameters are organized something like /ServiceName/Param1, /ServiceName/Param2, etc. the env variable would be set to ServiceName.

Now we call the store_params method in the class which establishes a botocore client to connect to SSM, connects to SSM and grabs the parameters we asked for and then uses pickle to put the parameters in a file in /tmp.

Now we can push the layer up to AWS Lambda and start working on the lambda code to run the test.

Testing the Extension With the layer loaded we create a very simple Hello World lambda using the SAM toolkit.

import json

from AwAws.Lambda.base import Lambda
from AwAws.SharedResources.parameters import Parameters

class LambdaTestClass(Lambda):
	def handle(self, event, context):
        print('FUNCTION: started')
        
        ssm = Parameters('Announce')
        try:
            params = ssm.read_tmp_dict()
        except Exception as e:
            params = ssm.get_param_dictionary()
            
        for name in params.keys():
            print('SSM PARAM:', name, 'Value:', params[name])

        print('FUNCTION:', json.dumps(event))
        print('FUNCTION: done')

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "hello world",
                # "location": ip.text.replace("\n", "")
            }),
        }

lambda_class = LambdaTestClass()

# this does not get inoked when initializing
# but only when the lambda service invokes it
lambda_handler = lambda_class.get_handler()

A bit of a sidebar here. I use a base class for all of my lambda functions. All this really means is that my function code is wrapped in the get_handler() method which allows me to do stuff like send an event to the function that keeps the function warm - a bit of legacy code to reduce cold start times.

So, ignoring the class, this function is going to read the parameters from the file in /tmp that was created by the extension. If you recall earlier, we set up an environment variable to specify which parameters we want from SSM. If we don’t set the environment, the extension will not get any parameters from SSM in which case we fall back to setting up the SSM client and getting the parameters in the lambda function itself. By testing it both ways we can get an idea of what impact the extension is having on our start up times.

With the Extension

REPORT RequestId: 6b9d5fbb-cb31-4bf1-b961-698b6f654182 Duration: 18.95 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 96 MB Init Duration: 675.52 ms

Without the Extension

REPORT RequestId: bdc9dfe3-c962-4afc-896c-37c2389a9679 Duration: 904.77 ms Billed Duration: 1000 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 669.66 ms

As expected, running the lambda without running the extension first resulted in a much slower execution time - in fact, it’s off by an order of magnitude while the Init Duration for both function is about the same. Now, the astute reader may have noticed that there are a few more things we can do with the non extension lambda to tune it up a bit. For example, in both runs the extension is running, just in the second case, the extension does not fetch the data from SSM and cache it. It’s also worth pointing out that we should be establishing the botocore client connection before the function runs.

Let’s overhaul the function by tuning it up a bit and see what happens. Here’s the new function:

import json

from AwAws.Lambda.base import Lambda
from AwAws.SharedResources.parameters import Parameters

# set up the botocore client connection
ssm = Parameters('Announce')
ssm.get_ssm()

class LambdaTestClass(Lambda):
    def handle(self, event, context):
        print('FUNCTION: started')
        
        # try:
        #    params = ssm.read_tmp_dict()
        # except Exception as e:
        params = ssm.get_param_dictionary()
            
        for name in params.keys():
            print('SSM PARAM:', name, 'Value:', params[name])
        
        print('FUNCTION:', json.dumps(event))
        print('FUNCTION: done')

        return {
            "statusCode": 200,
            "body": json.dumps({
                "message": "hello world",
                # "location": ip.text.replace("\n", "")
            }),
        }


lambda_class = LambdaTestClass()

# this does not get inoked when initializing
# but only when the lambda service invokes it
lambda_handler = lambda_class.get_handler()

Which gives us:

REPORT RequestId: c76f03f9-7292-46bc-9d4e-92c044a24a27 Duration: 240.84 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 725.15 ms

This is definitely better but still not as fast as the version with the extension. It’s also bumped up the initiation time a little bit which is not that surprising.

Wrapping Up

Obviously, to really test this properly we should set up a framework that reloads and runs the function a few hundred times but even this quick and dirty analysis gives us some insights on how the lambda extensions can be implemented. Extensions have a lot of other nice features too. The incoming events are passed into the extension if you register it with the INVOKE event. This allows you to do things like send events to cloudwatch or eventbridge without modifying the function code, even do some analysis on the incoming event to see if someone is trying something malicious. Definitely a cool feature and worth spending some time leveraging how to use it.

Did I miss something? Leave a note in the comments below!

Envelope Encryption on AWS

Sat, 19 Sep 2020 00:00:00 +0000

Over the past few years, there have been a number of data breaches of companies using AWS but not securing the data correctly. Leaving Personally Identifiable Information (PII) in publicly accessible buckets on S3 seems to be a common problem. While AWS is a very secure environment, AWS’s shared responsibility clearly shows that their job with security ends at “Security of the Cloud” which leaves “Security in the Cloud” up to the customer (https://aws.amazon.com/compliance/shared-responsibility-model/). The customer’s security responsibility includes securing operating systems on EC2/ECS, network configurations and firewall settings, Identity and Access Management (IAM) and data encryption which is what the rest of this article is about.

One strategy for data encryption is called Envelope Encryption which can be used to encrypt data in transit or at rest. The envelope encryption service used in AwAws was built around AWS Key Management Service (KMS) which is a dedicated Hardware Security Module (HSM) to secure and encrypt data at rest and by leveraging the AWS Encryption SDK (available for python) following their best practices.

How it works KMS allows us to build and store encryption keys in a very secure way using FIPS 140-2 validated cryptographic modules which do not allow the managed keys to leave the service. The idea here is that you can’t get access to the the actual key - you can send data to be encrypted/decrypted by the service but you can’t get the key yourself and use it from encryption/decryption. Items to be encrypted are sent to the service over a TLS connection, encrypts the data and then returns the encrypted data. The amount of data that can be sent to the service is limited to 4kb (that’s 4 kilobytes) which is not very much. This is great for passwords or API secrets but not very useful for large files. This kind of makes sense though, you don’t necessarily want to be moving large amounts of sensitive unencrypted data around. It also makes sense from the service’s standpoint as the KMS service could easily become a bottleneck if people start trying to send huge files to it.

This is where envelope encryption comes in. Envelope encryption is a two-layer encryption system which involves creating a custom data encryption key to encrypt a specific set of data and then using the KMS master key to encrypt the custom data key. The encrypted key is then stored along side the data that was encrypted. The benefit of this technique is that all of the data objects will have their own unique encryption key which can only be opened with the master key.

Envelope Encryption Steps Let’s say we want to create a lambda service that encrypts files coming in through API Gateway and then stores the encrypted files in S3. Our encryption service would work as follows:

receive data object from API Gateway

locally create a unique key specifically for this data object

use this key to encrypt the data

So, at this point, we have a data object that has been encrypted with some random key that we generated on the fly. Now we need to do something with the key so that we can decrypt the data later on. In theory you could store the key in a database, but if someone got access to the database they would be able to decrypt all of your data! Besides, managing all the data objects locations in S3 and the keys for those objects would be a bit of a nightmare (like say someone renames the files). This is where the envelope part comes in. We take the unique key we generated and:

send the unique key to KMS to be encrypted with our master key

get back the encrypted key

attach the now encrypted unique key to the encrypted data

now the data can be safely moved or stored (encryption in transit, encryption at rest)

We will also attach an encryption context to the object. This is a set of plain text key/value pairs (tags) that identifies the encrypted data. These tags not only help identify and track the data, the tags are also cryptographically bound to the encrypted data - meaning it is also required for encrypting and decrypting the data.

Ultimately, we wind up with a new data object that contains:

an encrypted version of the original data object

a set of plain text key/value pairs that identify the original data which cannot be altered

the encrypted key which (when decrypted by KMS) can be used to decrypt the data

Kind of cool right. Since we attach the key to the data we don’t have to worry about losing track of which keys belong to which data, or someone coming in and stealing all our keys and data. The only way to get at the original data object is via access to the master key.

In Transit This could be set up to move data securely across the internet. The idea would be that we can encrypt data before it leaves the sender site, send it over a secure TLS connection, and the immediately store the data in object storage in our cloud production environments. In this case, we would encrypt the data encryption key with a shared RSA key (similar to the way ssh connections work), send that encrypted key over TLS to the KMS service which would then decrypt RSA and encrypt the key with AES-GCM.

Encryption Algorithms (The Gory Details) AwAws was set up to use a 256 bit AES-GCM (Advanced Encryption Standard algorithm in Galois/Counter Mode) with an Elliptic Curve Digital Signature Algorithm (ECDSA with P-384 and SHA-384) and a HMAC-based extract-and-expand key derivation function (HKDF - SHA-384) which helps prevent accidental reuse of a data encryption keys.

Service Based Architecture with AWS Organizations

Sat, 16 May 2020 00:00:00 +0000

I was having a beer with a good friend of mine about a year ago trying to get my head around a problem I was running into while designing a new system for a job at a healthcare startup. I knew that we would only have a limited number of clients (initially in the tens) and each client would be interacting with our system on a daily basis. We would be receiving a lot of sensitive data from our clients which had to be encrypted on the wire as well as at rest. Our clients could vary greatly in size and hence would be very different in the amount of resources they would consume. I also wanted to keep things somewhat separate from each other, especially client data, to try to reduce the risk associated with data getting associated with the wrong client.

I had this idea that I could build an event driven, service based architecture across multiple AWS accounts and it was my friend’s job to try to talk me out of it. The idea was to break up all of the services I would need to build so that they could run independently from its own AWS account and then providing an AWS account for each client that would provide access to all of the services they subscribe to as well as be the location for all of their data.

Here are some of the benefits of this architecture in no particular order:

client data is kept separate from other clients which good from a security perspective - no way for other clients to access each other’s data
billing is easier as each account would have its own AWS bill - makes it easier to see who is using resources
don’t need to have all of our clients in the same region
can have some clients multi region while others are not - this is a big deal if you have to mirror a lot of data as you only have to mirror the data for a particular client as opposed to all of them
assuming the accounts are built and managed with cloudformation, it should be trivial to create a client environment
don’t have to write database code to manage clients, just set up an instance of cognito for each client and configure it to meet the needs of each client
outages due to software issues would only affect one client, outages in a region would only affect clients in that region (unless they have a multi-region configuration)
removing a client from our system and deleting all of their data (a requirement under HIPAA) is trivial, just ask AWS to delete the account.

That’s a lot of pluses except for one big minus:

how do you support/manage/maintain this beast?

Fair question. It may not be as hard as you think though, especially if you are starting from scratch.

CI/CD Pipelines

There is no way anyone is going to want to sit around deploying code to each server account. I like my team to work with a very fast cadence which can mean multiple deployments a day. There is no possible way we can manage a bunch of separate client accounts if we don’t have CI/CD pipeline set up to run automated tests, but more importantly, deploy changes to all of the client accounts. Of course, since each client has their own account, it should be easy for use to set up test accounts with fully automated integration testing which can be performed before the release is actually pushed out.

Consistency

Possibly one of the trickiest tasks is build the system so that each client has the exact same software. There is really no easy way to build in customizations for a particular client unless they are purely cosmetic type things which should be easy enough to handle. As soon as you start trying to install different software on each client account though you are going to wind up with a bunch of systems to support that do not all work quite the same way.

Building solid APIs and SDKs makes it easier to keep things consistent

just make sure you never change the behavior of your API or SDK. You can add features but if your API starts returning stuff in a different way you’ll wind up with trying to simultaneous deploys so that nothing breaks.

AWS Organization Accounts

Unfortunately, one bigger problem with using accounts this way is that there is no easy way to just delete an account and start over again. So if you want to rebuild an account from scratch you either have to completely clean it out or have a bunch of deleted accounts sitting around in your AWS Organization dashboard - maybe this is not that big of a deal, just keep in mind that you cannot reuse email addresses when you delete and create a new account. You can use a + sign in the email address though (if your email provider supports it) so your account names might look something like qa+20200516@mydomain.com - not the end of the world.

Python Static Analysis Tools

Fri, 01 May 2020 00:00:00 +0000

Lately I’ve been looking at tools to help improve code quality especially with respect to security issues. While code review is a useful process, it sometimes is difficult to pinpoint code which may lead to vulnerabilities. Unless you have memorized all of the various attack vectors, you are probably going to miss something along the way. The idea that we might be able to automatically analyze the code as part of our CI/CD process seemed very intriguing.

I have been using Flake8 to do static analysis of Python code for some time now. It’s fairly easy to configure to run with pytest so I get unit tests and syntax checking at the same time. There is also a handy module for vim that checks your python syntax on the fly called vim-flake8. To be quite honest, all I was really doing was running the checker and fixing up a couple little problems here and there. It’s a nice way to implement a style guide without a bunch of engineers sitting around debating the merits of different code formatting models and whether the whitespace was going to be tabs or the less efficient, but more precise, spaces. Surely there’s more that can be done with static testing though, something that can check my code for type mismatches and give back cryptic error messages that would make a FORTRAN77 compiler feel proud.

Security Testing

Whenever I utter the words ‘Security Testing’ I use the same voice as the guy in the Princess Bride in The Pit of Despair because you know that no matter how thorough you are, the process never really ends. The package I looked at is called Bandit which claims to be “a tool designed to find common security issues in Python code” and was developed as part of the OpenStack Security Project. Immediately I see the appeal, just install it in my CI/CD pipeline and let it do it’s thing. Bandit is designed to look for things like using outdated encryption ciphers, insecure/deprecated functions, and things like loading pickle files. This is not to say that loading pickles files isn’t safe at times, but rather, if you are loading pickle files in a production environment you better be damn sure you know where it came from. I downloaded and ran bandit against some code I’ve been working on for a while and got the pickle error as I was loading a pickle file from S3 on AWS. I was still pretty new to Python when I wrote the code and, at the time, I thought using pickle would be a good way to encode binary data before putting it on S3. It’s not. If someone could access the files on S3, they could load all types of code and data into my program. Fortunately, I have rarely (if ever) actually called this function so crisis averted, still need to fix the code though. Part of the challenge for me as a Data Engineer is that for data scientists its totally reasonable to use pickle to dump and load data. Taking code from a data scientist and converting it to production code can lead to overlooking some issues like this.

Here are some pretty good reason to install and use Bandit:

look for vulnerabilities in code before pushing it to master - this is the obvious one
being more in-tune with best practices
keeping old code up to date - bandit will detect deprecated functions especially functions with known vulnerabilities. Checking old code that no one looks at to make sure it is still safe is a pretty nice feature
checking dependencies. Yes, it’s all open source and open for review but when is the last time you actually reviewed all the source code for a package or dependency you are using?

That last one is a big deal. I ran Bandit against the XRay SDK from AWS and it picked up stuff like this

Test results:

>> Issue: [B310:blacklist] Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected.
   Severity: Medium   Confidence: High
   Location: aws_xray_sdk/core/plugins/ec2_plugin.py:23
   More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen
22	
23	        r = urlopen('http://169.254.169.254/latest/meta-data/instance-id', timeout=1)
24	        runtime_context['instance_id'] = r.read().decode('utf-8')

This is saying that the code is connecting to a remote URL. This is perfectly(?) safe in this case as the 169.254.169.254 is a link-local IP (See RFC 3927) which is used by AWS to provide metadata about a running EC2 and is apparently also available from Lambda (which is news to me). That address could have been 104.126.73.169 or even worse 170.178.168.203 or some other random IP address on the internet! It’s also nice that they provide a like to their website to explain what the problem is. Anyway, I spent a lot of time playing bandit with and pointing at dependencies I’m including in some of my projects (since my code did not produce and errors HA!). Bandit has definitely earned its place on my CI/CD test stack. Keep in mind though that you should only use Bandit as a tool and that there are many other security (and compliance) issues that need to be addressed.

Static Typing

Ok, so I took the plunge and looked a mypy. If you’ve never looked at static typing in Python, this may be a good place to start. This is my first experience with it in Python and it looks very promising, although it might screw a lot of people up as it looks a bit different. mypy requires Python 3.5 or later which should not be a problem for anybody anymore, right?

The idea with static typing is to explicitly type the functions so what most people are familiar with as a dynamically typed function:

def just_add_beer(foo):
    return foo + ' with beer'

would be written as:

def just_add_beer(foo: str) -> str:
    retrurn foo + ' with beer'

Simple enough and much more explicit. When the function is called, mypy will check to make sure it is being called with the correct type and throw errors if it is not. This is the kind of checking that can reveal some of those really hard to find bugs where you are passing an incorrect type and the function just merrily goes on working assuming that you know what you are doing or even worse, crashes in production. Unit tests don’t necessarily pick up on these things either as most people do not try passing incorrect types to their functions as they are usually only testing edge cases like, if I pass an int that’s too big, what happens as opposed to if I pass the word ‘hello’ as an int what happens.

Sadly, Python does not seem to give a **** about statically typed functions as the function does not error out when I pass an integer to a function that wants a string:

wyllie@dilex:~ $ python
>>> def add_beer(foo):
...     return foo + ' with beer'
... 
>>> add_beer('hello')
'hello with beer'
>>> add_beer(1)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in add_beer
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>> add_beer('hello')
'hello with beer'
>>> 
>>> def just_add_beer(foo: str) -> str:
...     return foo + ' with beer'
... 
>>> just_add_beer('hello')
'hello with beer'
>>> just_add_beer(1)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in just_add_beer
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> exit()
(crap, that exit thing burns me every time).

I don’t know, I may have to do some more research on this technique and this project to see if it’s wroth stressing over - maybe I’m missing something like Perl’s use strict…

If you use pytest, you can also install the pytest-mypy module which simplifies adding mypy checks to you CI/CD pipeline.

The decision to use mypy is a bit more complicated than using flake8 or bandit which will just run with no code changes (well, no code changes except fixing broken code that has been identified). mypy requires thinking about your code in a different, albeit more robust, way. Fortunately, mypy will ignore functions that are not explicitly typed this way so you don’t have to rewrite your whole codebase on the first day you use or even use it everywhere in your code. You might decide that only new code will be supported and then update older code as time permits - maybe add doc strings to all of your functions while you are at it.

Finally

These are just a few of the great tools out there. It’s worth investing some time researching what is available and adding some of these to your workflow.

AWS Serverless CI/CD with GitHub Actions

Wed, 22 Apr 2020 00:00:00 +0000

AWS and GitHub are great services for managing and deploying cloud applications but sometimes getting the code from GitHub to AWS is not as straight forward as we would like it to be. Sure, there are some tools out there that bridge the gap (CircleCi, Jenkins, many others) but that means learning another framework, and hoping that everything is tightly integrated and that you remember how it all works when something breaks half a year after you set it all up. GitHub Actions can help simplify this though. Actions are easily configured workflows that are triggered when you do something with git. For example, you can set up an action that automatically test your code every time you push it to github. You can then extend that workflow so that it builds your code in a test environment when you create a pull request. This allows another member of the team to easily access the running code while reviewing the source code, simplifying the code review process as you get that warm, cozy feeling knowing that the code you are reviewing does what you expect it to do.

In this post, I’m going to walk through the steps of setting up a CI/CD pipeline inside GitHub to manage Lambda Layers builds.

Coding Workflows

YAML is used to configure the workflows and complete reference of the workflow syntax use by GitHub Actions can be found here. Fortunately, both YAML and the workflow syntax are pretty easy to work with. Start out by creating a new branch in your repo. The workflows are going to go in the .github/workflows directory in the root of your repo. You can have as many workflows as you like in here as long as each file has a .yml or .yaml extension. I’m going to focus on creating a workflow for building a distribution for AWS Lambda Layers, a full description of how to build workflows for other things can be found here. In the .github/workflows directory, create a file called awsLayers.yaml - you can give it any name you like as long as you have the .yml or .yaml extension. Here’s my file:

# This workflow will install dependencies and create a build suitable
# to be used in an AWS Lambda Layer.  The build will then be uploaded
# to S3 and then can be accessed from any lambda that uses the layer.
#
# This build is only for dev builds.  Releases will be built from a
# seperate action.
#
# A new version of the layer will be created for every branch when a
# pull request is intitiated.  This allows us to test the layer in a
# dev environment on AWS BEFORE the code is merged into main.


name: Build Lambda Layer

on:
  pull_request:
    branches: [ main ]

jobs:
  
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # For more info: https://github.com/aws-actions/configure-aws-credentials
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

      - name: Zip it all up and upload to S3
        env:
          release_bucket: 
          release_bucket_uri: 
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python
          
          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

Let’s walk through all of this…

Here’s the name of the workflow:

name: Build Lambda Layer

This next bit is telling github that the workflow should be run on pull_requests on the main branch (i.e., code you are merging into main).

on:
   pull_request:
     branches[main]

Now, if you have a more complex git flow going on, you may not be merging into main - so don’t just blindly cut and paste this code.

Define the jobs that will be run. According to the GitHub Actions docs, you can have multiple jobs and they will run in parallel by default unless you specifically indicate that you want the jobs to run in sequence.

I only have one job in this case called deploy. This job has a name Upload Layer to AWS Lambda. We are also going to specify a ubuntu virtual machine to run on. We are not really doing anything exciting here so the distribution and version of linux is really not that important.

jobs:
  deploy:
    name: Upload Layer to AWS Lambda
    runs-on: ubuntu-latest

The next section defines the steps that will be run. The first step is to checkout the code. The workflow is running in a virtualized environment so we need to grab our code from the repo before we can do anything useful.

steps:
  - name: Checkout
    uses: actions/checkout@v2

The next bit is to configure an AWS user that we can use to upload the files safely to S3.

It’s important that the user that is created only has minimal permissions on aws! The user should only be used for github access!

It’s probably a good idea to create a separate bucket on S3 to receive the files that will be uploaded to AWS layers. To lockdown the user, I created two policy files, One to allow uploads to a specific bucket on S3, and a second one that allows layers to be installed on Lambda.

The upload policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutBucketNotification",
                "s3:AbortMultipartUpload",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my_bucket_name/*",
                "arn:aws:s3:::my_bucket_name"
            ]
        }
    ]
}

The policy to add layers to lambda only needs one action!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:PublishLayerVersion",
            ],
            "Resource": "*"
        }
    ]
}

You could certainly combine the two policies if you wanted to but I prefer making smaller policies that I can attach to different groups of users if I need to.

We will load the configure_aws_credentials action which can be found in the aws-actions repo on github.

You will need to store the AWS credentials for the user you created earlier in a secret vault on GitHub. DO NOT PUT YOUR CREDENTIALS IN YOUR REPO - of course, you already know this, right? Instructions for stashing your credentials securely on github can be found here

Something to take note of here is that the secrets are stored on a per repo basis. This means that you could create a separate AWS user for each github repo - a pretty good idea if you are remotely paranoid about such things. Even better would be to set up a test account using AWS Organizations to keep your test code separate from other environments - but that’s beyond the scope of this article.

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

Load up python - I’m using python 3.8

      - name: Setup Python 3.8
        uses: actions/setup-python@v1
        with:
          python-version: 3.8

Finally, the interesting part. This is where we build the distribution, copy it to S3 and then load it into lambda layers. In order for the code to be installed correctly in the layer, it needs to be in a directory called python. The run section below, creates the directory, installs all of the dependencies and then install the code. For the release_id, you can really use anything you want. The format command below is going to create something like MyRepo-BranchName-dev.zip. If you look at the github docs you can find a bunch of other things you can stick on there.

Side Note: Instead of using a standard requirements file, i.e., requirements.txt, I have created a requirements directory with a dex.txt, test.txt and a prod.txt. This allows me to control what is being installed in the production environment which is important when developing for lambda. For example, you don’t need to load boto as lambda already has it, you also don’t want to load up any dev or test tools to the production environment. For that matter, you also do not want to load up all of your tests and documentation files to production, so you should consider writing filters to remove (or not include) this stuff when building your layer code. A simple way to do this would be to use the —exclude flag in zip (check the man page for more info on how that would work).

      - name: Zip it all up and upload to S3
        env:
          release_bucket: my_bucket_name
          release_bucket_uri: s3://my_bucket_name
          release_id: ${{ format('SomeId-{0}-dev.zip', github.head_ref )}}
          release_layer: ${{ format('SomeId-{0}-dev', github.head_ref )}}
        run: |
          mkdir python
          # assuming your requirements file is in 'requirements/prod.txt'
          pip install -r requirements/prod.txt -t python
          pip install . -t  python
          echo building release $release_id
          # zip it up
          zip --quiet -r $release_id python

The code is then compressed with zip and the file is copied to S3. Once the file has been uploaded, the aws command line tool is used to publish the layer in AWS Lambda so that it can be used from other lambdas.

          # copy the file to S3 and install it in lambda layers
          aws s3 cp $release_id $release_bucket_uri
          aws lambda publish-layer-version --layer-name $release_layer  --content S3Bucket=$release_bucket,S3Key=$release_id --compatible-runtimes python3.8

When you are done building your workflow, push it up to github and try it out by creating a new pull request. Don’t be discouraged when it does not work on the first try. In fact, a solid strategy for this would be to build up the file section by section so you can see how it all works. You can see the actions running in the Actions tab in you repo. You should see a history of all your prior runs in this tab too.

GitHub Repo Configuration

The next step is to get your GitHub repo configured - this is going to be a bit different for each team/repo depending on what workflow your team is choosing to use. I personally use a very simple main/feature branch workflow since most of my repos and teams are fairly small so we rarely have to worry about clobbering each other’s changes. This simple workflow is also very fast which allows us to move changes into production multiple times a day. With all that said, go to the Settings tab in your repo and select Branches. We want to create a a new rule in Branch Protection Rules - so click Add Rule. There’s a full description of the options in the GitHub Documentation.

There are a bunch of interesting options in here that can make your development process more secure and less prone to accidents. The option we are particularly interested in is Require status checks to pass before merging. This allows us to hook in our GitHub Actions and will require the Actions to run cleanly before the code can be merged back into your main branch.

GitHub Action Workflows

With the branch rules configured you will now find that the pull request cannot be merged unless the workflow completes successfully. Now that the layer is installed in your test environment, a code reviewer can look at code and then go to the environment to make sure everything is working they way they would expect it to.

Of course, you can do all kinds of testing once you get a couple of workflows running. A few ideas:

Blocking merges until all the unit tests pass
making sure that static testing passes
integration tests - like running api checks on the test environment that the layer was just installed in

Let your imagination run wild!

Mon, 01 Jan 0001 00:00:00 +0000

Meet the New Team!

This next leg of the journey will be a bit different though in both what I will be working on and who I will be working with!

Startup LeapFrogging

So what’s the leapfrog here?

The New Team

So here, without any further ado, is my new team! I asked the AIs I use to write bios, this is what I got.

Athena Wise

Lumina Cerebra

Welcome to my corner of the universe!

AI Assistant

Final Thoughts

Mon, 01 Jan 0001 00:00:00 +0000

Onboarding and plan for the first few months

identify and address any critical issues that may exist. These types of issues can be distracting so getting them out of the way quickly allows us to move the technology ahead in more purposeful way. There will always be bugs and minor issues that need to be dealt with but should not dominate the overall development strategy.
working through some of the most critical issues will give me a chance to learn the architecture and start building a strategy for moving forward. This could include things like evaluating whether we should break up the existing monolith or leveraging various services from AWS to build systems and processes that are smaller and easier to manage.
Get a complete understanding of the development processes being used and the technology in the existing systems paying particular close attention to security and efficiency issues
Look for easy wins - ways to simplify complex algorithms/systems, migration of components to simpler or more cost-effective solutions/services, replace in-house code with open source or maybe even commercially available products if possible
As we build these new systems, we can also introduce DevOps process to manage these new components include the code review process, how new code and bug fixes are moved into production, introducing observability and logging systems.
Data protection. Make sure that data is both secure and backed up.
As we get into a more regular work flow, we can start exploring our agile workflow and other development processes. My preference is to use the minimum amount of process to get the work done. Similar to the actual systems we are building, processes do need to be reviewed frequently in order to make sure they are still efficient and effective.

Fractional CTO Services

Mon, 01 Jan 0001 00:00:00 +0000

Dilex Networks’ Fractional CTO services are your startup’s secret weapon, ready to supercharge your growth and drive innovation across multiple key areas!

Team Development

Younger teams often benefit from just need having experience mentors to bounce ideas off and validate their ideas and concepts before investing significant development resources. I focus on fostering a collaborative environment where team members feel comfortable asking any questions, no matter how basic they may seem. My goal is to empower the engineering, instilling the confidence needed to break down complex problems into manageable components.

mentor the team on software engineering best practices
support professional development through team presentations, sharing on-line resources and books
assist with hiring processes/interviewing prospective employees

Product Design

My main goal here is to understand the product’s short, medium and long term goals of the product as these will directly influence the structure of the system architecture. Getting the architecture right from the start is crucial since it becomes increasingly challenging to refactor as the product evolves. System architecture involves numerous choices and trade-offs, so having a clear vision of future product requirements can greatly inform and simplify these decisions. To that end, understanding long-term goals is essential when designing and building robust systems.

establish product security and compliance requirements
define product data needs, including data volume, backup strategies, and more.
determine uptime requirements
identify product-market fit

System Architecture and Design

It’s essential to deep dive into the latest offerings from AWS and other vendors to determine how we can leverage these technologies to build the product effectively. I always emphasize that the product requirements should take precedence over technical requirements i.e., the technical requirements should not dictate how the product functions or how users interact with it.

evaluate existing architecture
Assess current AWS services and other vendor solutions
refactor existing architecture/systems as needed
design and build new architecture/systems with a focus on security profiles and cloud cost management
utilize AWS Infrastructure as Code tools (e.g., Cloudformation, AWS CDK, Terraform) to construct systems on AWS

Software Development and DevOps

I often combine these two areas as modern cloud-based software development demands that all engineers have a solid understanding of DevOps processes. The code being developed is frequently intertwined with cloud services, making this knowledge essential. I’m a hands-on leader, eager to write and review code alongside the team.

establish code review and release processes
building reliable automated unit and integration test suites
implement observability and logging systems/standards
set up documentation standards

Home | Dilex Networks

Creating and AWS CI/CD Environment with GitHub Actions

Overview

CI/CD Pipelines

AWS Organizations

Security

Billing

Innovation

Git Workflows

GitHub Actions

Multi Destination Actions

Secure Connection to AWS

GitHub Environments and Secrets

Using OpenId Connect

Using AWS credentials

GitHub Action Workflow

Putting it all together

Adding a new developer account

Installing the Files

TroubleShooting

Conclusion

Meet The New Team

Meet the New Team!

Startup LeapFrogging

So what’s the leapfrog here?

The New Team

Athena Wise

Lumina Cerebra

AI Assistant

Final Thoughts

Privacy

Terms of Service

🎉 Easily create your own simple yet highly customizable blog

Overview

Get Started

Crowd-funded open-source software

❤️ Click here to become a sponsor and help support Hugo Blox’s future ❤️

Ecosystem

Inspiration

Features

Themes

License

🧠 Sharpen your thinking with a second brain

Mindmaps

Highlighting

Callouts

Did you find this page helpful? Consider sharing it 🙌

📈 Communicate your results effectively with the best data visualizations

Charts

Diagrams

Data Frames

Did you find this page helpful? Consider sharing it 🙌

👩🏼‍🏫 Teach academic courses

Video

Podcast

Test students

Math

Code

Inline Images

Did you find this page helpful? Consider sharing it 🙌

✅ Manage your projects

Ideation

Diagrams

Todo lists

Did you find this page helpful? Consider sharing it 🙌

People

Lambda Extensions with Python

A quick overview of how the extensions API fits into the lambda environment

Set Up

Wrapping Up

Envelope Encryption on AWS

Service Based Architecture with AWS Organizations

CI/CD Pipelines

Consistency

AWS Organization Accounts

Python Static Analysis Tools

Security Testing

Static Typing

Finally

AWS Serverless CI/CD with GitHub Actions