Lambda as a Unit of Compute
We refer to Lambda as a unit of compute instead of AWS Lambda, the service that allows you to run lambdas. In this section we explain how HTC-Grid integrates the environment to execute units of compute that look to all effects like a lambda, but that allow to select which compute plane you want to execute them on, with the current implementation done for EKS.
One of the main Tenets as we described earlier on is making the system modular. When looking at how the worker side of the architecture would invoke the execution we had to consider:
- The worker could invoke the execution of different programs, in different programming languages
- The interface for execution of the library on the agent should be simple, and provide a level of familiarity to Grid users.
- The deployment mechanism should be able to run in multiple Compute Planes, as different users may want different compute planes such as: Amazon Elastic Kubernetes Service (EKS), Amazon Elastic Container Servie (ECS), AWS Lambda, Amazon EC2 Instances (EC2), AWS Batch, etc.)
We selected the Lambda interface as a main unit of compute. The release of AWS Lambda Container Image Support allows us then to use the minimum deployable unit of a lambda within a container, and thus meet the tenets for the project.
To simplify:
- The programming interface of the HTC-Grid worker code looks like a lambda.
- The deployment of the code follows all the best practices for containers
For example for python code the entry-point for the worker functionality looks like the following code snippet where the event will be a copy of the data for the task and the context provides contextual information about the session and task. Other programming languages follow a similar approach (you can read more about those here):
def handler_name(event, context):
...
return some_value
Within the EKS implementation that the project uses by default, the agent provides a connectivity layer between the HTC-Grid and the Lambda container. The Agent pulls new tasks from the task queues in the Control Plane, once a new task is acquired the agent invokes the Lambda container and passes the task definition and payload. The Lambda container contains custom executable that performs the work. The following diagram depicts how the HTC-Agent integrates in EKS with the Lambda container. HTC-Agent container runs in the same pod as the Lambda container. The lambda container exposes an HTTP interface on the localhost so that only the the HTC-Agent container can access the endpoint.
At time of writing this, the project is undergoing changes to move from lambci to the aws-lambda-runtime-interface-emulator. aws-lambda-runtime-interface-emulator is an AWS project that simplifies the integration, reduce the code needed and allows us to provide the lambda simulation integrated with other architecture, enabling us in the future to support Graviton instances.