This article provides an approach to streaming query output from RDS Postgres to an S3 CSV file via a NodeJs Lambda function.

The process uses pg-copy-streams and the query results to create a data stream which is piped to S3 using the aws-sdk.

Streaming Data from RDS/Postgres to S3 CSV using Lambda and NodeJs

Alternatives to removing the function’s dependency on pg-copy-streams were considered. These options and associated limitations will be briefly discussed.

A sample Lambda function is also included to assist in assessing the validity of the solution for a given use case.

Alternatives Considered

The following alternatives to using pg-copy-streams were considered but were disqualified for a variety of reasons.

“Streamless” Approach using Lambda Filesystem/Memory


Testing of playbooks for AWS deployments can be a challenge for an application developer. Within controlled environments, playbooks are generally executed via workflows/pipelines managed by DevOps teams.

The combination of Localstack and Ansible provides an app developer the capability for preliminary local/mock testing of playbooks without incurring AWS billing costs.

With the above context in mind, the following describes a procedure for local testing of AWS-based playbooks with Localstack.



The examples and code used throughout the article have been tested using Ansible version 2.10.3. Some revisions may be required if you intend on using an alternate version.

$ ansible --version

What follows is a walkthrough outlining the steps involved in implementing AWS cross-account access to an encrypted S3 bucket.

The following is a summary which describes the scenario used for the walkthrough:

  • An S3 bucket, s3://account-a-bucket, is to be created in account-a and made accessible to an external AWS account, account-b
  • A new KMS-CMK key needs to be created, with bucket encryption enabled using this key
  • IAM user Ann, in account-a, requires access to the bucket
  • IAM user Dilip, in account-b, also requires the same level of access to the bucket
  • Level of access required includes, ListBucket/GetObject/PutObject

Overview of Activities Required for each Account

Below diagram shows…

In a previous post, I documented learnings from setting up a local Devops environment.

This article aims to apply a potential use-case for this environment by including support for Terraform. The following overview of the goals to be achieved should help you in deciding whether to read on:

Tweak/expand the local Devops environment, mentioned above, to include support for Terraform and Localstack.

Use the environment to create a Continuous Delivery Jenkins pipeline to deploy a Python-based Lambda function to Localstack and AWS Cloud.

The Lambda function is to be triggered by an S3 event, in response to a csv file…

If you’ve worked with Localstack for mock AWS testing, then chances are most of the interactions with the stack have been via the AWS CLI or an SDK. Localstack’s Community Edition does not currently offer an official web-based UI, however, this feature is offered in Pro and Enterprise versions.

At a basic level, Commandeer provides a neat UI for interaction with AWS cloud and Localstack services.

The context of this article is limited to providing a basic overview of setting up Commandeer (release version 1.6.0) with Localstack.

Installing Commandeer

Ensure you have Snap and Docker installed on your Linux distro.

Install Commandeer…

Localstack has become popular in the DevOps and AWS local testing space.

I was looking to gain some basic experience with Terraform Cloud and using Localstack as the AWS provider with minimal or no firewall/router configuration changes. To achieve this would require localhost be accessible from the Internet.

This article is a guide on using tunnelling service to achieve the set up by enabling access to localhost via a public URL solely for the purposes of sandpit testing.

Signup for a Terraform Cloud Account

Hashicorp Terraform Cloud offers access to a free account with feature limitations. …

If you’ve interfaced with Devops teams, chances are that you’re well aware of tools used to manage infrastructure and application releases. Serverless holds a place on this long list.

Having worked with AWS, I became interested in exploring how Serverless fits into this space. This article outlines the learnings from setting up Serverless to work with Localstack for the purposes of testing application provisioning.

The aim is to create and deploy a service helloservice comprising of a basic hello Lambda function.

Installing Serverless

The following command installs Serverless onto a Linux host:

$ curl -o- -L | bash

As part of…

As of now, Google has yet to release an official backup/sync client for Linux, and as a frequent WSL2 user, I was after a solution that would allow automated backups to Google drive from the WSL2 Linux file system/distro.

I’m a big fan of both Windows & Linux, and the evolution of WSL2 (with an actual Linux kernel) is making it easier to work in an almost “real” Linux environment without the need for third-party virtualization software. …

The inspiration for writing this article came after reading the following on Airflow extensibility using CWL-Airflow:

The latest documentation, published by the above developer/s, can be found at,

With the documentation link above and concepts learned from, what follows is an outline of setting up a cwl-airflow Docker Compose stack.

A git repo containing the stack components be found at this link.

This stack is not intended for use on a public network. Windows…

In the space of Data Science/Analytics, we rely heavily on automation to drive insights from several data sources. Docker, Python, node, JSON and YAML are a amongst a wide range of technologies used as part of the automation.

I recently came across Rabix and Common Workflow Language (CWL) in the space of biomedical research. As a developer, and not having come from a biomedical background, I wanted to explore opportunities for using Rabix/CWL.

After looking into Cancer Genomics Cloud and signing up, I started to get a feel for Rabix/CWL. I decided to setup a local Rabix/CWL playground on my…

Tony Tannous

Learner. Interests include Cloud and Devops technologies.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store