AWS Lambda functions are a great way to execute short-running processes without worrying about what hardware they’re going to run on. Sometimes though, we have a requirement to execute a longer-lived process, but unfortunately AWS imposes a 15 minute execution limit.
Fear not though, because in this article you’ll learn how to write recursive Node.js JavaScript Lambda functions which call themselves, bypassing the execution time limit.
1. Overview
In October 2018 AWS increased the Lambda execution time limit to 15 minutes:
You can now configure your AWS Lambda functions to run up to 15 minutes per execution. Previously, the maximum execution time (timeout) for a Lambda function was 5 minutes. Now, it is easier than ever to perform big data analysis, bulk data transformation, batch event processing, and statistical computations…
Whilst this is good news, we still have some examples of tasks that we want to run as Lambas, but take longer than 15 minutes. For example:
- CloudFormation calls a Lambda which waits for an SSL certificate to be created. This process could take up to several hours.
- Processing a large S3 file
The best way to get around this limitation is to have a Lambda function do some work, check if the work is finished, and if not call itself again.
Recursion considerations
If you’re from a coding background you’ll probably be familiar with recursion, and some of it’s potential pitfalls. There’s nothing to worry about, as long as we’re conscious of the following:
a recursive Lambda must have a stop condition, otherwise it’s at risk to run forever
Lambda invocation types
When you run a Lambda function, there are 2 main ways to execute it:
- Synchronous execution is where the lambda is executed immediately when the Lambda Invoke API call is made. The function executes and you receive the response from the Lambda.
- Asynchronous execution is where the invoke request is placed on a queue, and it’s executed when it reaches the front of the queue. Whatever called the Lambda Invoke API doesn’t receive the response from the Lambda, because it’s executed at a later time.
For the recursive execution use case, we’ll be using asynchronous execution. This is because when we call the same Lambda again, we want the old execution to finish after the new execution starts.
2. A quick JavaScript recap
To understand the JavaScript code that we’ll define in the next section, you’ll need to know at a high-level the following JavaScript concepts. If you’re already familiar with promises, async, and await, skip to the next section.
Promises
Since JavaScript is single threaded it’s important not to block execution of code with long running processes (e.g. network calls). Promises allow us to execute asynchronous-like blocks of code, whilst providing an easy way to handle success and failure states.
Here’s a simple example:
function getPromise() {
return new Promise((resolve, reject) => {
console.log("Hello\n");
resolve("Success");
});
}
function callPromise() {
getPromise().then(result => console.log(result + '\n'))
}
callPromise()
Which prints:
Hello
Success
Info: the code within the promise is executed immediately. This is a simple example, but if you imagine that the code within the promise were to make an HTTP call, we would benefit from the ability to easily manage the response using the then syntax.
Async & await keywords
The async keyword improves promises by forcing the execution to wait until the promise has been resolved. This way, we don’t need to provide the callback using the then method (as above) but can instead just get the result of the promise directly by using the await keyword.
The example above could be rewritten like this:
function getPromise() {
return new Promise((resolve, reject) => {
console.log("Hello\n");
resolve("Success");
});
}
async function callPromise() {
const result = await getPromise()
console.log(result + '\n')
}
callPromise()
The benefits of using async and await become more apparent when you have multiple promises chained together. It makes the code a lot easier to understand.
Go deeper: to learn more about promises and async functions, check out this JavaScript promises for dummies article.
3. Writing a Lambda function that recurses
We’re going to write a JavaScript Node.js Lambda function that exhibits the following behaviours:
- Sleeps for 2 seconds using the
async&awaitkeywords discussed above. This will prevent our Lambdas from firing too quickly. - Checks whether the number of iterations of the Lambda has reached a pre-defined recursion limit. This is the Lambda’s stop condition.
- If there are more iterations to go, calls the AWS Lambda API
invokemethod on itself, to recurse.
3.1. Navigating the AWS console to create the Lambda
Login to the AWS console, click Services, then select Lambda:
]
Click the orange Create function button:
]
Keep all the defaults to create a Node.js Lambda, but enter a function name of RecursiveLambda. Click the orange Create function button:
]
Now we’re on the edit page for our Lambda function, you can enter code directly in the Function code area into the file index.js:
]
3.2. Writing the recursive Lambda code
Let’s build up the Lambda code within index.js.
Initialisation variables
First up, we’re going to define some variables at the top:
const AWS = require('aws-sdk')
const lambda = new AWS.Lambda()
const sleep = (milliseconds) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
const recursionLimit = 5
- a constant lambda which will provide us access to the AWS JavaScript Lambda SDK. It’s derived from the constant AWS which is a reference to the JavaScript library aws-sdk provided by AWS
- a sleep function which pauses execution for the given number of milliseconds
- a recursionLimit which will be used by our stop function later on to decide when to stop recursing
Lambda handler
Next we’ll define the handler for this Lambda. AWS provides an empty skeleton of this by default, which follows the same format. Replace the contents with the following code:
exports.handler = async(event) => {
await sleep(2000)
const iteration = event.iteration || 1
console.log('Iteration: ' + iteration + '\n')
if (iteration <= recursionLimit) {
// call lambda again
}
return 'Finished executing on iteration ' + iteration
};
So, what are we doing here?
- sleeping for 2 seconds
- grabbing the iteration from the event object which was passed to the Lambda. If it’s undefined, we set it to 1. We also log the value.
- checking if the iteration we’re on is less than the recursion limit. If it is, then execute the contents of the
ifstatement (to be defined later). - returning a string value
Invoking the Lambda recursively
Finally, we just need to add the contents of the if statement, to call the Lambda recursively:
// code before
if (iteration <= recursionLimit) {
var params = {
FunctionName: 'RecursiveLambda',
InvocationType: 'Event',
Payload: JSON.stringify({ "iteration": iteration + 1 })
};
return new Promise(function(resolve, reject) {
lambda.invoke(params, function(err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
});
}
// code after
What’s the deal here then?
- we’re creating the parameters variable params to invoke the Lambda:
- FunctionName is as we set earlier when we created the Lambda
- InvocationType is Event, meaning the Lambda will get executed asynchronously
- Payload contains the iteration number, which we increment by 1
- we’re returning a promise, which Lambda handles by ensuring that the value passed to resolve or reject gets returned to the invoker
- inside the promise, we’re invoking a call to the same Lambda using the params
The whole Lambda
And here’s all the Lambda code in one place.
const AWS = require('aws-sdk')
const lambda = new AWS.Lambda()
const sleep = (milliseconds) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
const recursionLimit = 5
exports.handler = async(event) => {
await sleep(2000)
const iteration = event.iteration || 1
console.log('Iteration: ' + iteration + '\n')
if (iteration <= recursionLimit) {
var params = {
FunctionName: 'RecursiveLambda',
InvocationType: 'Event',
Payload: JSON.stringify({ "iteration": iteration + 1 })
};
return new Promise(function(resolve, reject) {
lambda.invoke(params, function(err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
});
}
return 'Finished executing on iteration ' + iteration
};
3.3. Executing the Lambda
Having written or pasted in the Lambda code, click the orange Save button, then the Test button:
]
You’ll get a popup asking you to create a test event. Accept the default options but type in an event name TestEvent. Click the orange Create button:
]
Click the Test button again. Oh no! You got a big fat error:
]
not authorized to perform: lambda:InvokeFunction
This means that we don’t have permissions set up for our Lambda to be able to call itself. No problem. Make a note of your Lambda’s ARN from the error message, which in the screenshot above is arn:aws:lambda:eu-west-1:299404798587:function:RecursiveLambda.
Adding permissions for our Lambda to call itself
Scroll down to the Execution role section, and click View the RecusiveLambda-role- role:
]
You’re shown the role summary page in a new window. We’re going to add a new policy to this role, so click the blue Attach policies button:
]
On the next page you can select which policies to attach. Since we haven’t created the policy yet, click Create policy:
]
A new window should open on the Create policy page. Select the JSON tab as we’ll manually enter JSON:
]
Now we’re in the JSON editor:
]
Enter the following JSON, inserting the Lambda ARN copied from the error message earlier:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "<lambda-arn>"
}
]
}
The JSON policy should now look as below. Click the blue Review policy button:
]create-policy-json-editor-filled-out.png)
On the review policy page, enter the name InvokeLambda. Click the blue Create policy button:
]
Go back to the attach policies window, and search for the InvokeLambda policy we just created (you may need to refresh the page). Select the check-box, then click the blue Attach policy button.
]png)
Back on the role summary page, you can see the new InvokeLambda policy is attached:
]
Go back to the Lambda page and click Test again. You may have to wait a moment for the role changes to propagate. This time, success!
]
You can see in the logs that we have output from the first iteration:
INFO Iteration: 1
To see future iterations though, we’ll need to look in CloudWatch.
3.4. Viewing the Lambda logs in CloudWatch
Click on Services and this time choose CloudWatch:
]
From the CloudWatch home page, click Log groups from the left hand navigation:
]
Select the /aws/lambda/RecursiveLambda log group:
]
Select the top, most recent, log stream:
]
Now you’re presented with the log output:
]
You can clearly see 6 iterations here. The 1st iteration, plus 5 more up to the defined recursionLimit set in the code. We definitely have some recursion going on here!
Tip: you might not necessarily see all the log output under the same log stream. If you’re missing something, go back and view another log stream.
4. Cleanup
As good citizens, we always clean up after ourselves, right? :lol:
Delete the Lambda
Go to the Lambda home page and select your Lamda. Choose Actions > Delete.
]
You’ll get a confirmation popup. Note that it says it doesn’t delete roles, so we’ll have to do that manually. Hit the orange Delete button:
]
Delete the role
Go back to the Roles page we were on earlier. You can find it by navigating to Services > IAM > Roles and searching for RecursiveLambda. Click on the role that appears in the search results:
]
On the role page, click the Delete role button:
]
On the confirmation popup, click Yes, delete:
]
Delete the policy
On the left hand navigation of IAM select Policies. Search for the InvokeLambda policy. Select the Lambda, then go to Policy actions > Delete:
]
On the confirmation popup, hit the red Delete button:
]
5. Conclusion
Now you’ve seen how to write a recursive Node.js Lambda in JavaScript, maybe you can use this technique to apply to your own project.
Why not see if you can write the same type of function in Python, Java, or your favourite Lambda language?
6. Resources
DOCUMENTATION Lambda JavaScript SDK for invoke docs AWS Lambda function handler docs JavaScript Promises tutorial
VIDEO If you prefer to learn in video format, check out this accompanying video to this post on my YouTube channel.