Understanding Lambda Layers

When a Lambda function is invoked, AWS launches a containerized Linux environment known as an execution environment to run your code. One of the key configurations you can set during the creation of a Lambda function is the runtime, which essentially refers to the programming language you intend to use. For example, the current available runtimes include nodejs18.x and python3.11.

AWS provides base images tailored for each runtime, including only default libraries to keep the environments lightweight. This means that if you install Python 3.11 on your EC2 instance, it will come equipped only with its default libraries.

However, many developers rely on additional libraries for their projects. This is where AWS Lambda Layers come into play, allowing users to package libraries and attach them to their functions for easy access.

Assumptions

I assume that you are already somewhat familiar with Lambda, comfortable using a terminal, and have experience with the AWS CLI. While familiarity with Python is beneficial, it isn’t strictly necessary.

Important Note

Since Lambda functions operate on Amazon Linux, any libraries you package must be compatible with a Linux environment. For instance, if you compile packages on a Windows or Mac machine, the resulting binaries may not function correctly when uploaded to Lambda. For Windows users, the Windows Subsystem for Linux (WSL) is a viable option, while Mac users may need to experiment or consider using Amazon Cloud9 or similar Linux-based IDEs.

For this tutorial, I will demonstrate how to create a layer for Python that contains the Splunk SDK, but this process can be adapted for various libraries.

Creating a Layer Environment

First, let’s establish a basic folder structure for your layers. Depending on your setup, you might choose a working directory like /home/{user}/:

mkdir -p /home/{user}/code/lambda/layers/

cd /home/{user}/code/lambda/layers/

Once in your working directory, set up a folder for your layer. It’s often best to create a separate layer for each library to maintain flexibility and avoid bloat in your functions.

Option 1: This approach creates Python version-specific site packages, ensuring that the folder structure matches the runtime of your function.

mkdir layer_splunk_sdk

mkdir -p layer_splunk_sdk/python/lib/python3./site-packages/

Option 2: This method is more generic and should theoretically work with multiple Python versions, though I’ve encountered issues in the past.

mkdir layer_splunk_sdk

mkdir -p layer_splunk_sdk/python/

Next, you will install the required Python libraries into the designated folder:

# For Option 1

pip3 install --target layer_splunk_sdk/python/lib/python3./site-packages/ splunk_sdk

# For Option 2

pip3 install --target layer_splunk_sdk/python/ splunk_sdk

Afterward, you’ll need to zip your layer for upload to AWS Lambda:

cd layer_splunk_sdk

zip -r ./layer_splunk_sdk.zip .

Make sure that the root of your zip file is the Python folder.

Uploading Your Zip File to AWS

Option 1: Use the AWS CLI:

aws lambda publish-layer-version --layer-name splunk_sdk

--description "This is for the Splunk Python SDK"

—zip-file fileb://layer_splunk_sdk.zip

--compatible-runtimes python3.X

Ensure that the specified compatible runtimes align with the runtime of your Lambda functions.

Option 2: Use the AWS Console:

1. Log into your AWS Console.
2. Navigate to the Lambda layers page.
3. Select “Create layer.”
4. Enter a name and description for your layer.
5. Upload the .zip file.
6. Specify compatible architectures and runtimes if necessary.
7. Click "Create."

And just like that, your layer is now ready for use with your Python Lambda functions! Keep in mind that compatible runtimes can affect availability. Although this guide focuses on Python, the process can be adapted for other runtimes like NodeJS.

For further reference, you may find the AWS Lambda Layers Documentation helpful.

Explore how to create and utilize Lambda layers with this detailed video tutorial.

This quick, two-minute video demonstrates the process of creating a Lambda layer in AWS.