Ansible: Testing AWS Deployments using Localstack

Prerequisites

Ansible

The examples and code used throughout the article have been tested using Ansible version 2.10.3. Some revisions may be required if you intend on using an alternate version.

$ ansible --version
ansible 2.10.3

Ansible Collections for Amazon Cloud

Modules/collections for AWS can be installed using

$ ansible-galaxy collection install \
community.aws \
amazon.aws

Localstack

The following sample docker-compose.yml can be used to launch a Localstack instance exposing AWS services at endpoint http://localhost:4566.

version: '3.7'
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack:latest
ports:
- "4566:4566"
- "4571:4571"
networks:
- dev_ops
environment:
- SERVICES=${SERVICES- }
- DEBUG=${DEBUG-}
- LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-docker}
- KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY-}
- DOCKER_HOST=unix:///var/run/docker.sock
- DATA_DIR=${DATA_DIR-/tmp/localstack/data}
- HOST_TMP_FOLDER=${TMPDIR:-/tmp/localstack}
- LAMBDA_REMOTE_DOCKER=false
- LAMBDA_REMOVE_CONTAINERS=false
- LAMBDA_DOCKER_NETWORK=dev_ops
- DEFAULT_REGION=us-east-1
volumes:
- "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
$ mkdir -p /tmp/localstack/data
$ docker-compose up -d

AWS Modules and endpoint_url

Under the covers, Ansible utilises boto3 to establish a client/connection to an AWS endpoint (endpoint_url). Depending on the AWS module being used, this address may be set via module parameters, or OS environment variables.

Overriding the AWS Endpoint for Supported Modules

Before attempting to run an Ansible playbook against Localstack, it’s best to check the respective module’s documentation reference to determine whether custom endpoints are supported. Not all the modules for Amazon Cloud support this feature.

  • Parameter: ec2_url aliases aws_endpoint_url, endpoint_url
  • Environment variable: EC2_URL alias AWS_URL

What about S3?

S3 modules aws_s3 and s3_bucket, provide support for custom endpoints via parameter s3_url, or environment variable S3_URL. Again, to use these modules with Localstack, set either of s3_url or S3_URL to http://localhost:4566.

Sample Playbook for Creation of IAM User on Localstack

The module community.aws.iam_user, described above, will be used to illustrate a basic example for creation of an IAM user on Localstack using a playbook.

Option 1: Run Playbook with Endpoint set via Environment variable (EC2_URL)

The following playbook shows the module being used to create IAM user, testuser.

---
- name: Playbook to Create IAM User on Localstack
hosts: 127.0.0.1

tasks:
- name: Create user and attach managed policy
community.aws.iam_user:
name: testuser
managed_policies:
- arn:aws:iam::aws:policy/PowerUserAccess
state: present
register: new_user

- name: print all values returned
debug:
msg: "{{ new_user.iam_user.user }}"
$ EC2_URL=http://localhost:4566 ansible-playbook \
--connection=local create_iam_user_noparam.yml
TASK [print all returned values] ***************************************************************************
ok: [127.0.0.1] => {
"msg": {
"arn": "arn:aws:iam::000000000000:user/testuser",
"create_date": "2021-05-02T20:16:14.681000+00:00",
"path": "/",
"tags": [],
"user_id": "k6kp1mn0vs5td8mz20f4",
"user_name": "testuser"
}
}

Option 2 : Run Playbook with Endpoint set via Parameter ec2_url

The following is the same playbook, however, in this case the endpoint is set using module parameter ec2_url: http://localhost:4566.

---
- name: Playbook to Create IAM User on Localstack
hosts: 127.0.0.1

tasks:
- name: Create user and attach managed policy
community.aws.iam_user:
name: testuser
ec2_url: http://localhost:4566
managed_policies:
- arn:aws:iam::aws:policy/PowerUserAccess
state: present
register: new_user

- name: print all values returned
debug:
msg: "{{ new_user.iam_user.user }}"
$ ansible-playbook --connection=local create_iam_user.yml

Conclusion

In summary, an Ansible AWS module which accepts parameter ec2_url or OS environment variable EC2_URL will provide the capability to connect to Localstack.

  • Parameter: ec2_url aliases aws_endpoint_url, endpoint_url
  • Environment variable: EC2_URL alias AWS_URL

Appendix: boto3.client() endpoint_url and ec2_url/EC2_URL

The following is an overview, showing how Ansible AWS modules use the value supplied for ec2_url/EC2_URL as the endpoint_url when creating the boto3 connection.

<collections_src_parent>/community/aws/plugins/modules/iam_user.py
$ ansible-galaxy collection list
# /home/<username>/.ansible/collections/ansible_collections
  1. <collections_src_parent>/community/aws/plugins/modules/iam_user.py
from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
...
module = AnsibleAWSModule(
argument_spec=argument_spec,
supports_check_mode=True
)

connection = module.client('iam')

...
from .ec2 import get_aws_connection_info
from .ec2 import boto3_conn
...
class AnsibleAWSModule(object):
...
def client(self, service, retry_decorator=None):
..., ec2_url, ...= get_aws_connection_info(self, boto3=True)
...
...
def get_aws_connection_info(module, boto3=False):

ec2_url = module.params.get('ec2_url')
...
...
if not ec2_url:
if 'AWS_URL' in os.environ:
ec2_url = os.environ['AWS_URL']
elif 'EC2_URL' in os.environ:
ec2_url = os.environ['EC2_URL']
...
...
return region, ec2_url, boto_params
...
from .ec2 import get_aws_connection_info
from .ec2 import boto3_conn
...
class AnsibleAWSModule(object):
...
def client(self, service, retry_decorator=None):
...
conn = boto3_conn(self, conn_type='client', endpoint=ec2_url,...)
...
...
try:
import boto3
import botocore
...

def boto3_conn(module, conn_type=None, resource=None, region=None, endpoint=None, **params):
try:
return _boto3_conn(conn_type=conn_type, resource=resource, region=region, endpoint=endpoint, **params)
...
...
try:
import boto3
import botocore
...
...
def _boto3_conn(conn_type=None, resource=None, region=None, endpoint=None, **params):
session = boto3.session.Session(
profile_name=profile,
)

...
elif conn_type == 'client':
return session.client(resource,..., endpoint_url=endpoint
...
...
class AnsibleAWSModule(object):
...
def client(self, service, retry_decorator=None):
...
return conn if retry_decorator is None else _RetryingBotoClientWrapper(conn, retry_decorator)
...

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tony Tannous

Tony Tannous

62 Followers

Learner. Interests include Cloud and Devops technologies.