Deploy AWS Lambda Functions to Local/Cloud Environments with Jenkins and Terraform

Tweak/expand the local Devops environment, mentioned above, to include support for Terraform and Localstack.

Use the environment to create a Continuous Delivery Jenkins pipeline to deploy a Python-based Lambda function to Localstack and AWS Cloud.

The Lambda function is to be triggered by an S3 event, in response to a csv file being uploaded to an input bucket. Once triggered, the Lambda processes the file and exports results to an output bucket.

Preparing Local Environment with docker-compose

$ cd $HOME
$ git clone \
https://github.com/tonys-code-base/jenkins_deploy_lambda_terraform.git
$ cd jenkins-deploy-lambda-terraform
$ rm -fr .git
$ mkdir -p /tmp/gogs/data
$ mkdir -p /tmp/jenkins_home
$ mkdir -p /tmp/localstack
$ cd $HOME/jenkins-deploy-lambda-terraform
$ docker build -t jenkins/jenkins:master .
$ AWS_CRED=$HOME/.aws TMPDIR=/tmp/localstack docker-compose up -d
version: '3.7'
services:
...
jenkins_master:
...
volumes:
...
- ${AWS_CRED}:/var/jenkins_home/.aws
[profile aws_local]
region = us-east-1
output = json
aws_access_key_id = test
aws_secret_access_key = test

[profile aws_cloud]
region = us-east-1
output = json
aws_access_key_id = <your AWS key>
aws_secret_access_key = <your AWS secret>

Configure Gogs SCM and Upload Code Repository

Create Git Repository on Gogs Host and Push Code

Overview of Repository Components

.
├── Dockerfile
├── Jenkinsfile
├── Pipfile
├── Pipfile.lock
├── my-lambda
│ ├── iam
│ │ ├── lambda_iam_policy.json
│ │ └── lambda_iam_trust_policy.json
│ └── src
│ └── my-lambda.py
├── lambda-input-test-file.csv
├── aws_local
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── aws_cloud
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── docker-compose.yml

Walkthrough of Terraform Code for Lambda Deployment

# AWS Regionvariable "aws_region" {
type = string
default = "us-east-1"
}

# Application name to include in names of AWS resources

variable "app_name" {
type = string
default = "transposer"
}

# AWS Account (for Localstack, value is zeroes)

variable "aws_account" {
type = string
default = "000000000000"
}

# AWS profile to source credentials

variable "aws_profile" {
type = string
default = "aws_local"
}

# Source name and location containing Lambda zip.
# Zip is created during the Jenkins pipeline.


variable "lambda_zip" {
type = string
default = "../dist/src.zip"
}

# Deployment target - AWS Cloud (aws_cloud)
# or Localstack (aws_local)


variable "env" {
description = "Env - localstack or cloud"
type = string
default = "aws_local"
}
# Localstack provider configuration

provider "aws" {
region = var.aws_region
profile = var.aws_profile
s3_force_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
insecure = true

endpoints {
acm = "https://localstack:4566"
apigateway = "https://localstack:4566"
cloudformation = "https://localstack:4566"
cloudwatch = "https://localstack:4566"
dynamodb = "https://localstack:4566"
ec2 = "https://localstack:4566"
es = "https://localstack:4566"
firehose = "https://localstack:4566"
iam = "https://localstack:4566"
kinesis = "https://localstack:4566"
kms = "https://localstack:4566"
lambda = "https://localstack:4566"
rds = "https://localstack:4566"
route53 = "https://localstack:4566"
s3 = "https://localstack:4566"
secretsmanager = "https://localstack:4566"
ses = "https://localstack:4566"
sns = "https://localstack:4566"
sqs = "https://localstack:4566"
ssm = "https://localstack:4566"
stepfunctions = "https://localstack:4566"
sts = "https://localstack:4566"
}
}

# Create IAM role with trust relationship for lambda service

resource "aws_iam_role" "iam_role_lambda" {
name = var.app_name

assume_role_policy = file("${path.module}/../my-lambda/iam/lambda_iam_trust_policy.json")
}

# IAM policy template

data "template_file" "lambda_iam_policy" {
template = file("${path.module}/../my-lambda/iam/lambda_iam_policy.json")
vars = {
app_name = var.app_name
aws_region = var.aws_region
aws_account = var.aws_account
}
}

# Create the policy

resource "aws_iam_policy" "iam_policy" {
name = var.app_name
path = "/"
description = "IAM policy for lambda"
policy = data.template_file.lambda_iam_policy.rendered
}

# Attach policy to IAM role

resource "aws_iam_role_policy_attachment" "policy_for_lambda" {
role = aws_iam_role.iam_role_lambda.name
policy_arn = aws_iam_policy.iam_policy.arn
}

# Input bucket used by Lambda

resource "aws_s3_bucket" "in_bucket" {
bucket = "${var.app_name}-input"
}

# Output bucket used by lambda

resource "aws_s3_bucket" "out_bucket" {
bucket = "${var.app_name}-output"
}

# Create Lambda function

resource "aws_lambda_function" "func" {
filename = "../dist/${var.lambda_zip}"
function_name = var.app_name
role = aws_iam_role.iam_role_lambda.arn
handler = "my-lambda.lambda_handler"

depends_on = [
aws_iam_role_policy_attachment.policy_for_lambda
]

source_code_hash = filebase64sha256("../dist/${var.lambda_zip}")

runtime = "python3.8"

# Lambda function environment variables

environment {
variables = {
OUTPUT_BUCKET = "${var.app_name}-output"
DEPLOYMENT_TARGET = var.env
}
}
}

# Add permissions to allow s3 to trigger lambda function

resource "aws_lambda_permission" "allow_s3_trigger" {
statement_id = "AllowTriggerFromS3"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.func.arn
principal = "s3.amazonaws.com"
source_arn = aws_s3_bucket.in_bucket.arn
}

# s3 event config:
# Configure bucket notification for triggering lambda
# when s3 csv file uploaded to input bucket


resource "aws_s3_bucket_notification" "bucket_notification" {
bucket = aws_s3_bucket.in_bucket.id

lambda_function {
lambda_function_arn = aws_lambda_function.func.arn
id = "s3_trigger"
events = ["s3:ObjectCreated:*"]
filter_suffix = "csv"
}

depends_on = [aws_lambda_permission.allow_s3_trigger]
}

Configuring Terraform Variables for AWS Cloud Deployment

variable "aws_account" {
type = string
default = "xxxxxxxxxxxx"
}

Jenkins Configuration and Deployment Pipeline

http://github_mock:3000/git-user/jenkins-deploy-lambda-terraform.git

Note: username and password should correspond to your Gogs git username and password, as created during the configuration of Gogs (if you've used the values mentioned in the Appendix, then username is git-user)

Configure Gogs Webhook

http://jenkins_master:8080/gogs-webhook/?job=jenkins-deploy-lambda-terraform

Test Deployed Lambda using Localstack

$ cd $HOME/jenkins-deploy-lambda-terraform$ aws --profile localstack \
--endpoint-url http://localhost:4566 \
s3 cp lambda-input-test-file.csv \
s3://transposer-input/lambda-input-test-file.csv
$ aws --profile localstack \
--endpoint-url http://localhost:4566 \
s3 cp \
s3://transposer-output/lambda-input-test-file_transposed.csv -
column1,1,6
column2,2,7
column3,3,8
column4,4,9
column5,5,10

Deploying to AWS Cloud

Conclusion

Appendix

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store