Page History

...

tmpfs for sequences and compositions 

This seems to be a long standing demand in the community as well:

• Why can't you mount EFS to Lambda ?
• Can EFS be mounted from Lambda environment ?
• AWS Developer Forum: Mounting EFS File Systems

Problem

Serverless application decomposition encourages the split of tasks into more granular functions. This is the source of 2 categories of problems: 

Problem 1 - Transferring large assets between actions

It's hard to pipe data between actions, when the its size of the response is above over the response size limits. As a result, each action in a sequence must read the asset independently, and upload any new content to a temporary storage, for the next action in the sequence to read it. This creates 2 main problems: 

• Poor developer experience
• Poor execution performance

system limits.

Problem 2 - Multiple actions processing the same asset

In case multiple actions are processing the same asset, each action needs to independently re-download the same asset, resulting in unnecessary network traffic.

Workarounds

Send Asset by Reference

Developers can use a 3rd party blob storage such as AWS S3 to upload the asset, and pass it to the next action using the URL to of the asset, as "the reference" to it.

The problem with this workaround is that the size of the asset influences the performance. The . Having each action uploading and downloading to/from a blob storage for each activation increases the execution time. The bigger the asset, the bigger the impact. If the asset is 2GB, then uploading and downloading to a blob storage on each activation may add up to a minute to the execution time.

Developer experience is also impacted by the fact that developers must manage the integration with a 3rd party blob storage. This implies that:

• developers are responsible to perform regular credential rotation for the blob storage, 
• developers must manage multi-region buckets, corresponding to the regions the actions may be invoked in, to optimize the latencies

Combine multiple actions in one action

Developers can combine the operations that act on the same asset into a single action. This workaround makes it harder to reuse actions in sequences or compositions.

It also restricts developers from using a polyglot implementation with action written in multiple languages; for instance the AI actions could be written in python, and combined with JS actions that may download the asset at the beginning of the sequence, and upload it at the end, by the last action in the composition.

Compress Asset and include it in the payload

If the asset format can be further compressed, it could be sent as Base64 into the JSON payload for the next action. 

This workaround it still constrained by the system limit, and it makes the action code more complex that it needs to be.

Measure the impact

The problem described above results in:

• Poor developer experience
• Poor execution performance

In order to asses the impact we need a methodology to quantify how big this problem is: 

Proposal to measure the impact

Measuring execution performance 

Questions:

1. How much time is spent in moving the content around vs processing it ?
2. What's the impact of using a Blob Storage when compared to an NFS/EFS-like systems ?

Measuring developer impact

Questions:

1. How much is the maintenance overhead ?
2. How is the implementation time affected ?
3. How much complexity is added to the action ?

Proposal

The proposal is to transparently provide developers with a temporary way to store large assets in the OW cluster. This is probably the hardest problem to solve when compared to the other ones b/c it involves persistence, state, and possibly handling large amounts of data. Bellow are listed a few possible options:

...