Exporting Data from RDS Postgres to S3 with NodeJs & Lambda

Streaming Data from RDS/Postgres to S3 CSV using Lambda and NodeJs
Streaming Data from RDS/Postgres to S3 CSV using Lambda and NodeJs

Alternatives Considered

“Streamless” Approach using Lambda Filesystem/Memory

AWS RDS PostgresQL Extension (aws_s3)

Proposed Solution — Limitations

Sample Lambda Function

  • Postgres Export Query: select * from public.pg_sample_tbl
  • RDS Endpoint: sample.xxxxxxxxxxxx.us-east-1.RDS.amazonaws.com
  • Database Name: postgres
  • Database Credentials: Note, these should be stored in secretsmanager and retrieved at runtime
    Username:
    postgres
    Password:
    sampleonly
  • S3 Bucket for CSV Export: s3://pg-rds-csv-exports
  • CSV Filename: pg-sample-tbl.csv

Lambda Source Code

Lambda Function for Streaming Data from RDS/Postgres to S3 CSV file

Deployment and Testing — AWS Cloud

Typical Deployment Overview

  • Create S3 bucket (location for CSV exports)
  • Create Lambda zip deployment package
  • Create new IAM Lambda Execution Role, with trust relationship allowing Lambda service to assume this role
  • Create new IAM policy with permissions to resources accessed by the function
  • Attach this policy to the Execution Role
  • During function creation, specify the Execution Role and path to the zip package

Sandpit/Performance Testing

Closing Points

--

--

--

Learner. Interests include Cloud and Devops technologies.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Before I start with my own app I’ll try to connect my IP camera to the demo app.

Clojure — Slowly but surely

Meet the Winners: Gelato + ETHGlobal Web3 Jam Hackathon 2021

Advantages of Using Micro Services over Monolithic Services

The decade of fierce database debate

TryHackme | Rootme | Full Writeup

7 Easy Ways To Be Productive As A Developer

Turn your RaspberryPi into a VPN with Tor as Access Point

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tony Tannous

Tony Tannous

Learner. Interests include Cloud and Devops technologies.

More from Medium

AWS Simple Queue Service(SQS)

AWS Lambda Hands-On| Part-3

Migrating a WSGI Python application to Serverless on AWS

Use Python logging module in AWS Lambda to send messages to CloudWatch and SNS