Scaling up lesson

My previous company is an on-demand delivery services which operates in 2 big cities: Hà Nội, Hồ Chính Minh.

With the high demand from the market, we need to move really fast including operation and technology with roughly ~ 50k MAU (at the time this article published).

Old architecture

AhaMove old architecture

Old issues:

  • without dockerization
  • only 1 ec2 c5.4xlarge (1 nginx)

Pros:

  • moving super fast
  • features rollout fucking fast

Cons:

  • Waste of resource (only utilize 70% peak time)
  • Can’t scale

On the way to stabilize and scale up API system

1 main api, and 1 backup api, nginx proxy on main api

1 main instance (c5.2xlarge) + 1 back up instance (c5.2xlarge)

Cons:

  • Fixed resources thus no utilize resource when low traffic


AhaMove old arch - CPU utilization

Avg rps: 500

Then, we move the backend to AWS EKS

We want to resolve:

  • Auto scale up when traffic increase
  • Resource utilization when low traffic
  • And keep the features roll out speed

New architecture

AhaMove eks architecture

Dashboard and Logs

EKS Production Cloud Watch Dashboard
Cloud watch Logs Insights

Up until now, the systems is very stable and scalable, we still need to improve many things. Moving up the systems to K8s help us a lot to reach 99.99% uptime target.

p/s: Even I left my previous company, I still miss a lot good memories with this startup, learn a lot of things:

  • KISS: keep it simple, very simple and fucking simple, don’t make color and think too much.
  • Need to balance between “do things right” and “do right things”
  • LEAN team: every team member is an expert in what they took responsibility
  • Straight forward in communication
  • Good planning (we accomplished this by using OKR)
  • And, have fun … and take it easy

Image source: AhaMove