For many of us moving to serverless from traditional applications, it may be puzzling to find an easy way to generate sequence numbers. Traditionally, relational database systems (RDBMS) such as Oracle provided this out of the box or custom applications written in favourable languages supplied them in abundance. So it was never an issue that made us worry.
However, when we land on the serverless shores, things aren’t quite that visible unless we know the right approach and the tools. In serverless, the use of RDBMS is rare for this purpose, especially when the functions are outside of the VPC (virtual private cloud) boundaries. In addition, we have the statelessness of the functions to deal with.
In this brief writeup, we will explore a common approach to generate sequence numbers while working with serverless applications on AWS.
Use of sequence numbers
In case you are somebody new to sequence numbering, then let us start with a quick introduction. If you have seen all this before then please skip to the sections further below.
In an ecommerce application we generate order numbers. These are mostly in a sequence. In a college candidate registration system, there could be application numbers arranged in an ascending order. Another very common use case is the visitor counter of a website. There are many more similar domains where we can find some form of sequence numbering in use.
There are a couple of important criteria in such a number generation. Every number must be unique in the context where it is generated. Though over-counting (or under-counting) is tolerated in most cases, it may not be the case with financial applications where missing an increment can cause unacceptable consequences.
Examples:
Order numbers: 200 000 230, 200 000 231, 200 000 232, 200 000 234, …
Candidate IDs: 1010, 1011, 1012, 1013, 1014, …
User visits to a site: 100 123, 100 124, 100 125, …
RDBMS way of generating sequence numbers
Those who have worked with Oracle databases must be familiar with ‘sequence’ objects. With a sequence object we can set a starting number, a maximum limit, increment value, generation in batches, rotation policy, etc.
Here is a sequence object named as D_REQUEST_CATALOG_S as shown in SQL Developer tool.
This is the SQL script for the creation of the above sequence D_REQUEST_CATALOG_S in Oracle.
CREATE SEQUENCE D_REQUEST_CATALOG_S
INCREMENT BY 1
MAXVALUE 900000000
MINVALUE 600000000
CACHE 10
DynamoDB to the rescue
As we know, DynamoDB is a NoSQL data store that is fully managed, highly available and auto-scalable. If you haven’t yet worked with DynamoDB and if you are in the serverless space, then I am pretty sure sooner rather than later you will get your hands on DynamoDB. You simply can’t avoid this big elephant in the serverless room!
Atomic counters
An atomic counter is a number attribute in a DynamoDB table that can be used to update the value atomically. Atomic counters are just like any normal numeric attribute, but the UpdateItem operation on that attribute is what makes it to behave as an atomic field. DynamoDB applies the updates to this attribute in the order of receiving the requests.
One word of caution though. With the atomic counter and UpdateItem operation, over-counting (or under-counting) is possible if the operation had to be retried due to failure. Due to this, it may not be suitable for certain applications as mentioned previously. In such situations, the recommendation is to use conditional writes.
API for generating sequence numbers
We will now walk through the different steps involved in setting up an API in order to generate a sequence number.
Step 1: Setup a sequence number table in DynamoDB
Here is a simple table order-number-sequence that has a partition key (id) and a sequence number attribute (sequence_number). The names of these attributes could be any valid names and need not be as shown below.
{“id”: “100”,
“sequence_number”: 250000
}
I set id as type string with a value “100”. This could be set as a number or any allowed types for the partition key field. The sequence_number is a numeric attribute that will act as the atomic number and in this example I set 250000 as its initial value.
Step 2: Create an API
We will create a very minimal API with a POST method that when invoked will return the next sequence number from the above table.
Note: We are not going to implement a Lambda function to interact with DynamoDB. Instead we are going to use the API Gateway’s built-in integration for DynamoDB. After all, why write a function when it is not needed?
For this demo, I have created a sequence-generator API with a number resource and a POST method on it.
The highlighted settings are needed for configuring the DynamoDB integration with API Gateway and to perform the UpdateItem action on DynamoDB. The instructions on how to perform the update action and on what table and attribute the update should be performed are going to be part of the request mapping template section which we will soon explore.
Step 3: Setup the execution role
As with other security permissions, policies and roles in AWS, this API also requires permissions to perform the update on the DynamoDB table. For this we need to set up a role with the relevant policies and assign that new role to API Gateway.
In the example below I have setup a role ‘sheen-api-role’ that has DynamoDB access policies attached to it.
If you do not have a pre-existing role then you can create a new one from the IAM console. In the above example I am using an AWS-managed policy that allows full access to DynamoDB, but in real life this could be different with custom policies with more granular level permissions.
Step 4: Configure request mapping template
After creating the API with the above details, access the Integration Request section of the POST method, as below.
In order to set up the mapping template, scroll all the way down to the Mapping Templates section and then create a new mapping template for content type application/json, as shown.
This is the most important part of this setup and also the complicated part - especially if you are not familiar with the Velocity template scripts. We don’t need any complex scripting for this, so hopefully it should be simple enough to understand.
This template contains the parameters and the payload for the UpdateItem action that we specified as part of the DynamoDB integration in Step 2.
{“TableName”: “order-sequence-number”,
“Key”: { “id”: {“S”: “100”
}
},
“ExpressionAttributeValues”: { “:val”: {“N”: “1”
}
},
“UpdateExpression”: “SET sequence_number = sequence_number + :val”,
“ReturnValues”: “UPDATED_NEW”
}
“TableName” section specifies the DynamoDB table name and the partition key of the item that we are updating.
“ExpressionAttributeValues” is where we set up the increment value — in our case it is incrementing by 1.
“UpdateExpression” is the SQL element of the NoSQL datastore! Well, kind of. This instruction is used to change the attribute by the increment value supplied. In the above expression the sequence_number attribute is being incremented by 1.
Finally, we return the updated value — and that’s all!
With the above script added, the request body mapping template looks like this:
And that’s about it. We are almost done and the above should just work. If we now test the API via the gateway console or deploy the API and then invoke via curl or Postman we should get a successful response.
Step 5: Testing the API
For simplicity, I am going to test it from the console.
Voila!!
Let’s look at the response body in the above picture. It appears that the response contains items that we do not need. The attributes and the numeric data type N of the sequence_number are things probably we can avoid and just return the sequence_number with the new value. For this we need to transform the response.
Step 6: Transform the response
Now that we’ve been through the request body template mapping and the associated template script, what we now need here is something similar but on the response integration. For this demo, the mapping is going to be set on status 200.
We need to take just the sequence_number and its value from the response which is returned from DynamoDB’s UpdateItem.
{“sequence_number”: “$input.path(‘$.Attributes.sequence_number.N’)”
}
With the above settings, if we now test the API again, we should see the response as below:
{“sequence_number”: “250006”
}
To verify that everything worked as expected, go to the DynamoDB table and refresh the table values and you will see that the sequence_number attribute value has gone up as many times as the API invocation.
What we have achieved above is a good starting point but there are number of ways we can improve the solution to make it robust, secure and production ready. These are beyond the scope of this introductory article but I would like to leave you with one enhancement that you can try.
Enhance it yourself
Imagine a scenario where you have a few different sequence number generation scenarios — like we said earlier, order numbers, website visitor counter, student numbers, etc. For these you may have different DynamoDB tables with the necessary attributes and data setup. How about if you need an API for cross team access? Wouldn’t it be better to have just one API rather than setting up one per table or sequence?
Try it yourself by adding a path parameter to the API and pass this parameter as the table name in the request mapping template. Your resource path will be /number/{table_name} and you need to pass the value of this table_name in the template script. Good luck!
Conclusion
Hope this article helped to understand the need for sequence numbering and the common and easy way to generate in serverless. Though it is difficult to cover every aspect of this and all possible constructs required for a production system, the example and the thoughts shared here can be extended to tailor to fit those needs.
This write-up also throws light on another aspect of serverless and that is, we don’t always need Lambda functions for many things that we do with API Gateway. I do an entire talk - “Don’t wait for Functionless. Write less Functions” - dedicated to this subject and other areas in AWS where we can reduce the Lambda footprint and still achieve the required functionality.
Sheen Brisals is a Senior Application Engineer at The LEGO Group.
Sequence Numbering in Serverless via API Gateway was originally published in LEGO Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.