 |
Serverless applications are event-driven, using ephemeral compute functions ranging from web APIs, mobile backends, and streaming analytics to data processing stages in machine learning (ML) and high-performance applications. While AWS Lambda includes a 512 MB temporary file system (/tmp) for your code, this is an ephemeral scratch resource not intended for durable storage such as Amazon Elastic File System (Amazon EFS).
However, extract, transform, and load (ETL) jobs and content generation workflows such as creating PDF files or media transcoding require fast, scalable local storage to process large amounts of data quickly. Data-intensive applications require large amounts of temporary data specific to the invocation or cached data that can be reused for all invocation in the same execution environment in a highly performant manner. With the previous limit of 512 MB, customers had to selectively load data from Amazon Simple Storage Service (Amazon S3) and Amazon EFS, or increase the allocated function memory and thus increase their cost, just to handle large objects downloaded from Amazon S3. Since customers could not cache larger data locally in the Lambda execution environment, every function invoke had to read data in parallel, which made scaling out harder for customers.
Today, we are announcing that AWS Lambda now allows you to configure ephemeral storage (/tmp) between 512 MB and 10,240 MB. You can now control the amount of ephemeral storage a function gets for reading or writing data, allowing you to use AWS Lambda for ETL jobs, ML inference, or other data-intensive workloads.
With increased AWS Lambda ephemeral storage, you get access to a secure, low-latency ephemeral file system up to 10 GB. You can continue to use up to 512 MB for free and are charged for the amount of storage you configure over the free limit for the duration of invokes.
Setting Larger Ephemeral Storage for Your Lambda Function
To configure your Lambda function with larger ephemeral storage, choose the Configuration tab under the General Configuration section in the AWS Lambda Console. You will see a new configuration for Ephemeral storage setting at 512MB by default.
When you click the Edit button, you can configure the ephemeral storage from 512 MB to 10,240 MB in 1 MB increments for your Lambda functions.
With AWS Command Line Interface (AWS CLI), you can update your desired size of ephemeral storage using theupdate-function-configuration command.
$ aws lambda update-function-configuration –function-name PDFGenerator –ephemeral-storage ‘{“Size”: 10240}’
You can configure ephemeral storage using Lambda API via AWS SDK and AWS CloudFormation. To learn more, see Configuring function options in the AWS Documentation.
As a review, AWS Lambda provides a comprehensive range of storage options. To learn more, see a great blog post, Choosing between AWS Lambda data storage options in web apps, written by my colleague James Beswick. I want to quote the table to show the differences between these options and common use-cases to help you choose the right one for your own applications.
| Features | Ephemeral Storage (/tmp) | Lambda Layers | Amazon EFS | Amazon S3 |
| Maximum size | 10,240 MB | 50 MB (direct upload) | Elastic | Elastic |
| Persistence | Ephemeral | Durable | Durable | Durable |
| Content | Dynamic | Static | Dynamic | Dynamic |
| Storage type | File system | Archive | File system | Object |
| Lambda event source integration | N/A | N/A | N/A | Native |
| Operations supported | Any file system operation | Immutable | Any file system operation | Atomic with versioning |
| Object tagging and metadata | N | N | N | Y |
| Pricing model | Included in Lambda (Charged over 512MB) |
Included in Lambda | Storage + data transfer + throughput | Storage + requests + data transfer |
| Shared across all invocations | N | Y | Y | Y |
| Sharing/permissions model | Function-only | IAM | IAM + NFS | IAM |
| Source for AWS Glue and Amazon Quicksight | N | N | N | Y |
| Relative data access speed from Lambda | Fastest | Fastest | Very fast | Fast |
Available Now
You can now configure up to 10 GB of ephemeral storage per Lambda function instance in all Regions where AWS Lambda is available. With 10 GB container image support, 10 GB function memory, and now 10 GB of ephemeral function storage, you can support workloads such as using large temporal files, data and media processing, machine learning inference, and financial analysis.
Support is also available through many AWS Lambda Partners such as Datadog, HashiCorp (Terraform), Lumigo, Thundra, Slalom, and Contino.
For this feature, you are charged for the storage you configure over the 512 MB free limit for the duration of your function invokes. To learn more, visit AWS Lambda product and pricing page and send feedback through the AWS re:Post for AWS Lambda or your usual AWS Support contacts.
– Channy
