Intro

When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).

Advantage

• FasS advantage
  • scaling benefit - cost, ops, etc
  • Resiliency
  • faster development
• No need to maintain Broker (e.g elastic search redis)
• Smaller side effect on code change
• Fully-managed

Disadvantage

• FasS disadvantage
  • limited state
• Hard to reuse app code
  • Django ORM or settings module
  • You can still use them with django package installed but inefficient
  • can't reuse code from django app
  • Django package is too heavy to use on lambda (still usable but too complex to setup
• Latency on startup
• Json formatted Invocation parameter
  • If you need to pass binary or big size paras, need to consider S3 or other medium

Limitation

Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

AWS Lambda quotas

• timeout - 15min
  • If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
• Invocation payload
  • asynchronous - 256KB

Consideration

• VPC, Subnet, Security Group
  • If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
  • If you are lambda needs to talk to outside world, private subnets needs NAT gateway
• Maintenance

Setup

AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.