Deploy AWS Lambda Functions to Local/Cloud Environments with Jenkins and Terraform
In a previous post, I documented learnings from setting up a local Devops environment.
This article aims to apply a potential use-case for this environment by including support for Terraform. The following overview of the goals to be achieved should help you in deciding whether to read on:
Tweak/expand the local Devops environment, mentioned above, to include support for Terraform and Localstack.
Use the environment to create a Continuous Delivery Jenkins pipeline to deploy a Python-based Lambda function to Localstack and AWS Cloud.
The Lambda function is to be triggered by an S3 event, in response to a csv file being uploaded to an input bucket. Once triggered, the Lambda processes the file and exports results to an output bucket.
The key topics and learning objectives are:
- Use docker-compose to bring up a stack comprising of 3 networked containers running the following services
➀ Gogs — locally hosted Git SCM
➁ Localstack — mock AWS environment
➂ Jenkins node — CI/CD - Using the stack, create a Jenkins Continuous Delivery pipeline to package and deploy a sample Python-based Lambda function using Terraform
- Configure the pipeline to deploy the Lambda to
◽ Localstack
◽ AWS Cloud environment, deployed only after interactive prompt confirmation - The Lambda’s primary purpose is to load a csv file from an input bucket and transpose the contents, using a Pandas Dataframe, into a given output bucket
- Configure the Lambda function to be triggered by an
s3:ObjectCreated
event, attached to the input bucket configuration - Set up a Gogs Webhook, to automatically trigger the Jenkins job when changes are pushed to locally hosted SCM
- Test the deployed Lambda function by uploading sample file to the input bucket
All components required to achieve the final framework are included in a git repo, which forms the basis for establishing the set up.
Preparing Local Environment with docker-compose
Clone Git Repository
Clone the repository mentioned above from github. This repository will eventually make its way onto our local, self-hosted Git SCM, aka Gogs.
$ cd $HOME
$ git clone \
https://github.com/tonys-code-base/jenkins_deploy_lambda_terraform.git
$ cd jenkins-deploy-lambda-terraform
$ rm -fr .git
Create Host Directories
Create host directories to be used for persisting data from container mounts.
$ mkdir -p /tmp/gogs/data
$ mkdir -p /tmp/jenkins_home
$ mkdir -p /tmp/localstack
Build Custom Jenkins Docker Image
Build the custom Jenkins docker image, which includes the AWS CLI, Terraform and Python distribution.
$ cd $HOME/jenkins-deploy-lambda-terraform
$ docker build -t jenkins/jenkins:master .
Bring up the Stack
Using docker-compose, bring up the stack using:
$ AWS_CRED=$HOME/.aws TMPDIR=/tmp/localstack docker-compose up -d
This brings up 3 containers, accessible from the host via the following endpoints:
Gogs: http://localhost:10080
Jenkins: http://localhost:8085
Localstack: http://localhost:4566
or, if accessing from internal docker-compose network:
Gogs: http://github_mock:3000
Jenkins: http://jenkins_master:8080
Localstack: http://localstack:4566
Note on AWS Credentials
In the command used to bring up the stack, AWS_CRED
is set to host location of AWS config/credentials folder ($HOME/.aws
), which is mounted into the home directory (/var/jenkins_home
) of the user running within the Jenkins container.
version: '3.7'
services:
...
jenkins_master:
...
volumes:
...
- ${AWS_CRED}:/var/jenkins_home/.aws
Two profiles should be added to the host’s $HOME/.aws/config
file to support deployment to each target environment, i.e Localstack (aws_local
), and AWS Cloud (aws_cloud
).
$HOME/.aws/config:
[profile aws_local]
region = us-east-1
output = json
aws_access_key_id = test
aws_secret_access_key = test
[profile aws_cloud]
region = us-east-1
output = json
aws_access_key_id = <your AWS key>
aws_secret_access_key = <your AWS secret>
Configure Gogs SCM and Upload Code Repository
Configure Gogs — Locally Hosted Git
To configure Gogs as a local Git SCM host, visit http://localhost:10080
and complete the configuration. Refer to Appendix A for config values. The remainder of the article uses these values.
Once Gogs has been configured, we can create an empty repository to host our source.
Create Git Repository on Gogs Host and Push Code
From http://localhost:10080
, create an empty repository named jenkins-deploy-lambda-terraform
, leaving "Initialise this repository with selected files and template" unticked.
Initialise and push the local repo we cloned earlier on, to our self hosted remote:
The repository should be accessible via:
http://localhost:10080/git-user/jenkins-deploy-lambda-terraform
Overview of Repository Components
The following is a listing of the repository source.
.
├── Dockerfile
├── Jenkinsfile
├── Pipfile
├── Pipfile.lock
├── my-lambda
│ ├── iam
│ │ ├── lambda_iam_policy.json
│ │ └── lambda_iam_trust_policy.json
│ └── src
│ └── my-lambda.py
├── lambda-input-test-file.csv
├── aws_local
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── aws_cloud
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── docker-compose.yml
The main components are:my-lambda/*
: Lambda source code and IAM policy definitionslambda-input-test-file.csv
: Test input file for Lambda functionaws_local/*
: Terraform components for deployment to Localstackaws_cloud/*
: Terraform components for deployment to AWS Cloud accountJenkinsfile
: Contains the deployment pipeline
Walkthrough of Terraform Code for Lambda Deployment
The folder structures hosting Terraform code, aws_local
and aws_cloud
, are almost identical, therefore a walkthrough of only one of these is covered (aws_local
). The differences are mainly related to provider configuration, i.e. Localstack vs AWS Cloud.
aws_local/variables.tf
This file contains the Terraform variables along with comments describing their purpose.
# AWS Regionvariable "aws_region" {
type = string
default = "us-east-1"
}
# Application name to include in names of AWS resources
variable "app_name" {
type = string
default = "transposer"
}
# AWS Account (for Localstack, value is zeroes)
variable "aws_account" {
type = string
default = "000000000000"
}
# AWS profile to source credentials
variable "aws_profile" {
type = string
default = "aws_local"
}
# Source name and location containing Lambda zip.
# Zip is created during the Jenkins pipeline.
variable "lambda_zip" {
type = string
default = "../dist/src.zip"
}
# Deployment target - AWS Cloud (aws_cloud)
# or Localstack (aws_local)
variable "env" {
description = "Env - localstack or cloud"
type = string
default = "aws_local"
}
aws_local/main.tf
Contains Terraform resources required for deploying the Lambda. Comments/descriptions are included for each resource.
# Localstack provider configuration
provider "aws" {
region = var.aws_region
profile = var.aws_profile
s3_force_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
insecure = true
endpoints {
acm = "https://localstack:4566"
apigateway = "https://localstack:4566"
cloudformation = "https://localstack:4566"
cloudwatch = "https://localstack:4566"
dynamodb = "https://localstack:4566"
ec2 = "https://localstack:4566"
es = "https://localstack:4566"
firehose = "https://localstack:4566"
iam = "https://localstack:4566"
kinesis = "https://localstack:4566"
kms = "https://localstack:4566"
lambda = "https://localstack:4566"
rds = "https://localstack:4566"
route53 = "https://localstack:4566"
s3 = "https://localstack:4566"
secretsmanager = "https://localstack:4566"
ses = "https://localstack:4566"
sns = "https://localstack:4566"
sqs = "https://localstack:4566"
ssm = "https://localstack:4566"
stepfunctions = "https://localstack:4566"
sts = "https://localstack:4566"
}
}
# Create IAM role with trust relationship for lambda service
resource "aws_iam_role" "iam_role_lambda" {
name = var.app_name
assume_role_policy = file("${path.module}/../my-lambda/iam/lambda_iam_trust_policy.json")
}
# IAM policy template
data "template_file" "lambda_iam_policy" {
template = file("${path.module}/../my-lambda/iam/lambda_iam_policy.json")
vars = {
app_name = var.app_name
aws_region = var.aws_region
aws_account = var.aws_account
}
}
# Create the policy
resource "aws_iam_policy" "iam_policy" {
name = var.app_name
path = "/"
description = "IAM policy for lambda"
policy = data.template_file.lambda_iam_policy.rendered
}
# Attach policy to IAM role
resource "aws_iam_role_policy_attachment" "policy_for_lambda" {
role = aws_iam_role.iam_role_lambda.name
policy_arn = aws_iam_policy.iam_policy.arn
}
# Input bucket used by Lambda
resource "aws_s3_bucket" "in_bucket" {
bucket = "${var.app_name}-input"
}
# Output bucket used by lambda
resource "aws_s3_bucket" "out_bucket" {
bucket = "${var.app_name}-output"
}
# Create Lambda function
resource "aws_lambda_function" "func" {
filename = "../dist/${var.lambda_zip}"
function_name = var.app_name
role = aws_iam_role.iam_role_lambda.arn
handler = "my-lambda.lambda_handler"
depends_on = [
aws_iam_role_policy_attachment.policy_for_lambda
]
source_code_hash = filebase64sha256("../dist/${var.lambda_zip}")
runtime = "python3.8"
# Lambda function environment variables
environment {
variables = {
OUTPUT_BUCKET = "${var.app_name}-output"
DEPLOYMENT_TARGET = var.env
}
}
}
# Add permissions to allow s3 to trigger lambda function
resource "aws_lambda_permission" "allow_s3_trigger" {
statement_id = "AllowTriggerFromS3"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.func.arn
principal = "s3.amazonaws.com"
source_arn = aws_s3_bucket.in_bucket.arn
}
# s3 event config:
# Configure bucket notification for triggering lambda
# when s3 csv file uploaded to input bucket
resource "aws_s3_bucket_notification" "bucket_notification" {
bucket = aws_s3_bucket.in_bucket.id
lambda_function {
lambda_function_arn = aws_lambda_function.func.arn
id = "s3_trigger"
events = ["s3:ObjectCreated:*"]
filter_suffix = "csv"
}
depends_on = [aws_lambda_permission.allow_s3_trigger]
}
Configuring Terraform Variables for AWS Cloud Deployment
Before deployment to AWS Cloud, aws_cloud/variables.tf
needs to be updated to reflect details of the target AWS Account.
aws_cloud/variables.tf
Modify this file, replacing xxxxxxxxxxxx with the target AWS Cloud account:
variable "aws_account" {
type = string
default = "xxxxxxxxxxxx"
}
Jenkins Configuration and Deployment Pipeline
Configure Jenkins and Install Gogs Plugin
Visit http://localhost:8085
and complete the Jenkins setup.
Once complete, add the Gogs plugin
via Jenkin's Plugin manager
. This plugin is required for establishing Webhooks between Gogs and Jenkins.
Create New Jenkins Job
Create a new Jenkins job:
- New Item → Pipeline
- Enter an item name → jenkins-deploy-lambda-terraform
- Choose Pipeline for job type
On the job configuration screen, select/enter the following values
Gogs Webhook
- Enable option → Use Gogs secret
- Secret → <choose a password> (example, mysecretlambda)
This will be required in the following section, when setting up the Webhook from within Gogs
Build Triggers
- Enable option → Build when a change is pushed to Gogs
Pipeline
- Definition → Pipeline script from SCM
- SCM → Git
- Repository URL →
http://github_mock:3000/git-user/jenkins-deploy-lambda-terraform.git
Note: we use the container’s internal host and port in the repo URL
- Credentials → Add → Jenkins → kind → username with password
username → git-user
password → <your password as chosen during Gogs config>
ID → git-creds
Note:
username
andpassword
should correspond to your Gogs git username and password, as created during the configuration of Gogs (if you've used the values mentioned in the Appendix, thenusername
isgit-user
)
- Branches to build → */master
- Script Path → Jenkinsfile
Configure Gogs Webhook
To configure the Gogs Webhook for the new job,jenkins-deploy-lambda-terraform
:
- Head back to
http://localhost:10080
- Go to the repository’s Settings and choose Webhooks from the left panel
- For Payload URL, the format is as follows:
http://<jenkins_host>/gogs-webhook/?job=<jenkins_job_name>
which resolves to,
http://jenkins_master:8080/gogs-webhook/?job=jenkins-deploy-lambda-terraform
Again, we use the container’s internal network host and port in the URL.
- Application Content → application/json
- Secret → <secret for the webhook>
The secret must match what was entered during the Jenkin’s job setup for Gog’s Webhook, e.g.: mysecretlambda) - When should this webhook be triggered? → Just the push event
- Add Webhook
Once complete, any changes pushed to the repository should automatically trigger the Jenkins job.
We can perform a test from Gogs to ensure the Webhook has been correctly configured. Gogs offers a Test Delivery
option which attempts to trigger the job on-request.
Test Gogs->Jenkins Webhook
To test the Webhook,
- Go to Gogs →
http://localhost:10080
- Go to
Settings
for repo,jenkins-deploy-lambda-terraform
- Under
Webhooks
, choose the Webhook we just configured - Click
Test Delivery
- This should trigger the Jenkins job
- Go back to Jenkins
http://localhost:8085
and view the build history for our job
- The job should automatically deploy the Lambda to Localstack (i.e, the first 4 stages automatically executed)
- The very last stage (stage 5), deploys the Lambda to configured AWS Cloud account only when
Proceed
is selected in response to an interactive prompt. The stage automatically aborts after timeout duration (60sec) if a response has not been received
Test Deployed Lambda using Localstack
With the Lambda successfully deployed to Localstack, we run a test using the supplied input file (contains 3 sample rows with 5 columns). To trigger the Lambda, copy the file to the S3 input bucket:
$ cd $HOME/jenkins-deploy-lambda-terraform$ aws --profile localstack \
--endpoint-url http://localhost:4566 \
s3 cp lambda-input-test-file.csv \
s3://transposer-input/lambda-input-test-file.csv
This should trigger the function in a separate container. The container is not removed after execution completes, this allows for troubleshooting errors by examining the container logs.
If the Lambda executes successfully, a transposed version of the input file should be located at s3://transposer-output/lambda-input-test-file_transposed.csv
.
A quick look at the contents should show 5 rows, 3 columns:
$ aws --profile localstack \
--endpoint-url http://localhost:4566 \
s3 cp \
s3://transposer-output/lambda-input-test-file_transposed.csv -
Output:
column1,1,6
column2,2,7
column3,3,8
column4,4,9
column5,5,10
From here onwards, Lambda code changes applied to local repo, and pushed to the self-hosted Git instance, would automatically trigger the Jenkins pipeline for redeployment.
Deploying to AWS Cloud
Once the Lambda is tested successfully on Localstack, it can be deployed to an AWS Cloud account for further testing by responding with “Proceed” to the interactive prompt for the last stage of the Jenkins pipeline. The additional testing would be targeted at covering off on aspects that cannot be practically tested in a local environment.
Conclusion
So what do we gain from all this?
My personal take-home points from having gone through the set up:
- An understanding/appreciation of technologies used by Devops
- An insight into Terraform
- Having my own local environment which allows for more destructive testing
- The typical AWS resources associated with creating and implementing a Lambda
- Working with Lambda S3 event triggers
- Packaging Python-based Lambdas containing non-standard libraries
- Reducing AWS costs by using Localstack for the majority of testing and keeping Cloud usage to a minimum
- Minimising manual tasks by incorporating automation tweaks — such as the Gogs Webhook
Why did you use Terraform and not Ansible?
Some may argue that Ansible is a more suitable option for the task. I’m sure there are several reasons for and against the use of Ansible vs Terraform. This article was focused more on the learning “how to”, by using Terraform, instead of the “why nots”.
Appendix
Configure Gogs Git Service
Visit http://localhost:10080/
and complete the initial setup. Below are sample config values.
After selecting install Gogs
, a redirect to http://localhost:3000/user/login
occurs, however, since we have the container's internal port of 3000
mapped to our external host port 10080
, we need to change the port in the URL to 10080
.
Sign up
The first user to go through the signup process is allocated as Administrator. The below shows sign-up of initial user git-user
.