As of March 2026, ingress-nginx will no longer receive new releases, bugfixes, or updates to resolve any security vulnerabilities that may be discovered.

So you have a Kubernetes cluster and are using (or considering using) the NGINX ingress controller to forward outside traffic to in-cluster services. That’s awesome!

The first time I looked at it, everything looked so easy; installing the NGINX ingress controller was one helm install away, so I did it. Then, after hooking up the DNS to the load balancer and creating a few Ingress resources, I was in business.

Fast-forward a few months, all external traffic for all environments (dev, staging, production) was going through the ingress servers. Everything was good. Until it wasn’t.

We all know how it happens. First, you get excited about that shiny new thing. You start using it. Then, eventually, some shit happens.

My First Ingress Outage

Let me start by saying that if you are not alerting on accept queue overflows, well, you should.

What happened was that one of the applications being proxied through NGINX started taking too long to respond, causing connections to completely fill the NGINX listen backlog, which caused NGINX to quickly start dropping connections, including the ones being made by Kubernetes' liveness/readiness probes.

What happens when some pod fails to respond to the liveness probes? Kubernetes thinks there’s something wrong with the pod and restarts it. The problem is that this is one of those situations where restarting a pod will actually make more harm than good; the accept queue will overflow, again and again, causing Kubernetes to keep restarting the NGINX pods until they all started to crash-loop.

What are the lessons learned from this incident?

• Know every bit of your NGINX configuration. Look for anything that should (or should not) be there, and don’t blindingly trust any default values.
• Most Linux distributions do not provide an optimal configuration for running high load web servers out-of-the-box; double-check the values for each kernel param via sysctl -a.
• Make sure to measure the latency across your services and set the various timeouts based on the expected upper bound + some slack to accommodate slight variations.
• Change your applications to drop requests or degrade gracefully when overloaded. For instance, in NodeJS applications, latency increases in the event loop might indicate the server is in trouble keeping up with the current traffic.
• Do not use just one NGINX ingress controller deployment for balancing across all types of workloads/environments.

The Importance of Observability

Before detailing each of the previous points, my 0th advice is to never run a production Kubernetes cluster (or anything else for that matter) without proper monitoring; by itself, monitoring won’t prevent bad things from happening, but collecting telemetry data during such incidents will give you means to root-cause and fix most issues you’ll find along the way.

If you choose to jump on the Prometheus bandwagon, you can leverage node_exporter in order to collect node-level metrics that could help you detect situations like the one I’ve just described.

Also, the NGINX ingress controller itself exposes Prometheus metrics; make sure to collect those as well.

Know Your Config

The beauty of ingress controllers is that you delegate the task of generating and reloading the proxy configuration to this fine piece of software and never worry about it; you don’t even have to be familiar with the underlying technology (NGINX in this case). Right? Wrong!

If you haven’t done that already, I urge you to take a look at the configuration your ingress controller generated for you. For the NGINX ingress controller, all you need to do is grab the contents of /etc/nginx/nginx.conf via kubectl.

1
2

kubectl -n <namespace> exec <nginx-ingress-controller-pod-name> -- /
   cat /etc/nginx/nginx.conf > ./nginx.conf

Now look for anything that’s not compatible with your setup. Want an example? Let’s start with worker_processes auto;

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# $ cat ./nginx.conf
daemon off;

worker_processes auto;
pid /run/nginx.pid;

worker_rlimit_nofile 1047552;
worker_shutdown_timeout 10s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;
    # ...
}

# ...

The optimal value depends on many factors including (but not limited to) the number of CPU cores, the number of hard disk drives that store data, and load pattern. When one is in doubt, setting it to the number of available CPU cores would be a good start (the value “auto” will try to autodetect it).

Here’s the first gotcha: as of now (will it ever be?), NGINX is not Cgroups-aware, which means the auto value will use the number of physical CPU cores on the host machine, not the number of “virtual” CPUs as defined by the Kubernetes resource requests/limits.

Let’s run a little experiment. What happens when you try to load the following NGINX configuration file from a container limited to only one CPU in a dual-core server? Will it spawn one or two worker processes?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# $ cat ./minimal-nginx.conf

worker_processes auto;

events {
  worker_connections 1024;
}

http {
  server {
    listen 80;
    server_name localhost;

    location / {
      root  html;
      index index.html index.htm;
    }
  }
}

Thus, if you intend to restrict the NGINX ingress CPU share, it might not make sense to spawn a large number of workers per container. If that’s the case, make sure to explicitly set the desired number in the worker_processes directive.

1
2
3
4
5
6
7

$ docker run --rm --cpus="1" -v `pwd`/minimal-nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
fc7d98c412a9b90a217388a094de4c4810241be62c4f7501e59cc1c968434d4c

$ docker exec fc7 ps -ef | grep nginx
root         1     0  0 21:49 pts/0    00:00:00 nginx: master process nginx -g daemon off;
nginx        6     1  0 21:49 pts/0    00:00:00 nginx: worker process
nginx        7     1  0 21:49 pts/0    00:00:00 nginx: worker process

Now take the listen directive; it does not specify the backlog parameter (which is 511 by default on Linux). If your kernel’s net.core.somaxconn is set to, say, 1024, you should also specify the backlog=X parameter accordingly. In other words, make sure your config is in tune with your kernel.

And please, don’t stop there. Do this thought exercise to every line of the generated config. Hell, take at look at all the things the ingress controller will let you change, and don’t hesitate to change anything that does not fit your use case. Most NGINX directives can be customized via ConfigMap entries and/or annotations.

Kernel Params

Using ingress or not, make sure to always review and tune the kernel params of your nodes according to the expected workloads.

This is a rather complex subject on its own, so I have no intention of covering everything in this post; take a look at the References section for more pointers in this area.

Kube-Proxy: Conntrack Table

If you are using Kubernetes, I don’t need to explain to you what Services are and what they are used for. However, I think it’s important to understand in more detail how they work.

Every node in a Kubernetes cluster runs a kube-proxy, which is responsible for implementing a form of virtual IP for Services of type other than ExternalName. In Kubernetes v1.0 the proxy was purely in userspace. In Kubernetes v1.1 an iptables proxy was added, but was not the default operating mode. Since Kubernetes v1.2, the iptables proxy is the default.

In other words, all packets sent to a Service IP are forwarded/load-balanced to the corresponding Endpoints (address:port tuples for all pods that match the Service label selector) via iptables rules managed by kube-proxy; connections to Service IPs are tracked by the kernel via the nf_conntrack module, and, as you might have guessed, this connection tracking information is stored in RAM.

As the values of different conntrack params need to be set in conformance with each other (ie. nf_conntrack_max and nf_conntrack_buckets), kube-proxy configures sane defaults for those as part of its bootstrapping procedure.

$ kubectl -n kube-system logs <some-kube-proxy-pod>
I0829 22:23:43.455969       1 server.go:478] Using iptables Proxier.
I0829 22:23:43.473356       1 server.go:513] Tearing down userspace rules.
I0829 22:23:43.498529       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
I0829 22:23:43.498696       1 conntrack.go:52] Setting nf_conntrack_max to 524288
I0829 22:23:43.499167       1 conntrack.go:83] Setting conntrack hashsize to 131072
I0829 22:23:43.503607       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0829 22:23:43.503718       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0829 22:23:43.504052       1 config.go:102] Starting endpoints config controller
...

These are good defaults, but you might want to increase those if your monitoring data shows you’re running out of conntrack space. However, bear in mind that increasing these params will result in increased memory usage, so be gentle.

Sharing Is (Not) Caring

We used to have just a single NGINX ingress deployment responsible for proxying requests to all applications in all environments (dev, staging, production) until recently. I can say from experience this is bad practice; don’t put all your eggs in one basket.

I guess the same could be said about sharing one cluster for all environments, but we found that, by doing this, we get better resource utilization by allowing dev/staging pods to run on a best-effort QoS tier, taking up resources not used by production applications.

The trade-off is that this limits the things we can do to our cluster. For instance, if we decide to run a load test on a staging service, we need to be really careful or we risk affecting production services running in the same cluster.

Even though the level of isolation provided by containers is generally good, they still rely on shared kernel resources that are subject to abuse.

Split Ingress Deployments Per Environment

That being said, there’s no reason not to use dedicated ingresses per environment. This will give you an extra layer of protection in case your dev/staging services get misused.

Some other benefits of doing so:

• You get the chance to use different settings for each environment if needed
• Allow testing ingress upgrades in a more forgiving environment before rolling out to production
• Avoid bloating the NGINX configuration with lots of upstreams and servers associated with ephemeral and/or unstable environments
• As a consequence, your configuration reloads will be faster, and you’ll have fewer configuration reload events during the day (we’ll discuss later why you should strive to keep the number of reloads to a minimum)

Ingress Classes To The Rescue

One way to make different ingress controllers manage different Ingress resources in the same cluster is by using a different ingress class name per ingress deployment, and then annotate your Ingress resources to specify which one is responsible for controlling it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Ingress controller 1
apiVersion: extensions/v1beta1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - args:
            - /nginx-ingress-controller
            - --ingress-class=class-1
            - ...

# Ingress controller 2
apiVersion: extensions/v1beta1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - args:
            - /nginx-ingress-controller
            - --ingress-class=class-2
            - ...

# This Ingress resource will be managed by controller 1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: class-1
spec:
  rules: ...

# This Ingress resource will be managed by controller 2
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: class-2
spec:
  rules: ...

Ingress Reloads Gone Wrong

At this point, we were already running a dedicated ingress controller for the production environment. Everything was running pretty smoothly until we decided to migrate a WebSocket application to Kubernetes + ingress.

Shortly after the migration, I started noticing a strange trend in memory usage for the production ingress pods.

Why was the memory consumption skyrocketing like this? After I kubectl exec’d into one of the ingress containers, what I found was a bunch of worker processes stuck in shutting down state for several minutes.

root     17755 17739  0 19:47 ?        00:00:00 /usr/bin/dumb-init /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17765 17755  0 19:47 ?        00:00:08 /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17776 17765  0 19:47 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody   18866 17776  0 19:49 ?        00:00:05 nginx: worker process is shutting down
nobody   19466 17776  0 19:51 ?        00:00:01 nginx: worker process is shutting down
nobody   19698 17776  0 19:51 ?        00:00:05 nginx: worker process is shutting down
nobody   20331 17776  0 19:53 ?        00:00:05 nginx: worker process is shutting down
nobody   20947 17776  0 19:54 ?        00:00:03 nginx: worker process is shutting down
nobody   21390 17776  1 19:55 ?        00:00:05 nginx: worker process is shutting down
nobody   22139 17776  0 19:57 ?        00:00:00 nginx: worker process is shutting down
nobody   22251 17776  0 19:57 ?        00:00:01 nginx: worker process is shutting down
nobody   22510 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   22759 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   23038 17776  1 19:59 ?        00:00:03 nginx: worker process is shutting down
nobody   23476 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   23738 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   24026 17776  2 20:01 ?        00:00:02 nginx: worker process is shutting down
nobody   24408 17776  4 20:01 ?        00:00:01 nginx: worker process

In order to understand why this happened, we must take a step back and look at how configuration reloads is implemented in NGINX.

Once the master process receives the signal to reload configuration, it checks the syntax validity of the new configuration file and tries to apply the configuration provided in it. If this is a success, the master process starts new worker processes and sends messages to old worker processes, requesting them to shut down. Otherwise, the master process rolls back the changes and continues to work with the old configuration. Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit.

Remember we are proxying WebSocket connections, which are long-running by nature; a WebSocket connection might take hours, or even days to close depending on the application. The NGINX server cannot know if it’s okay to break up a connection during a reload, so it’s up to you to make things easier for it. (One thing you can do is to have a strategy in place to actively close connections that are idle for far too long, both at the client and server-side; don’t leave this as an afterthought)

Now back to our problem. If we have that many workers in that state, this means the ingress configuration got reloaded many times, and workers were unable to terminate due to the long-running connections.

That’s indeed what happened. After some debugging, we found that the NGINX ingress controller was repeatedly generating a different configuration file due to changes in the ordering of upstreams and server IPs.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145

I0810 23:14:47.866939       5 nginx.go:300] NGINX configuration diff
I0810 23:14:47.866963       5 nginx.go:301] --- /tmp/a072836772	2017-08-10 23:14:47.000000000 +0000
+++ /tmp/b304986035	2017-08-10 23:14:47.000000000 +0000
@@ -163,32 +163,26 @@
 
     proxy_ssl_session_reuse on;
 
-    upstream production-app-1-80 {
+    upstream upstream-default-backend {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.71.14:3000 max_fails=0 fail_timeout=0;
-        server 10.2.32.22:3000 max_fails=0 fail_timeout=0;
+        server 10.2.157.13:8080 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-2-80 {
+    upstream production-app-3-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.110.13:3000 max_fails=0 fail_timeout=0;
-        server 10.2.109.195:3000 max_fails=0 fail_timeout=0;
+        server 10.2.82.66:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.124:3000 max_fails=0 fail_timeout=0;
+        server 10.2.59.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.45.219:3000 max_fails=0 fail_timeout=0;
     }
 
     upstream production-app-4-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.109.177:3000 max_fails=0 fail_timeout=0;
         server 10.2.12.161:3000 max_fails=0 fail_timeout=0;
-    }
-
-    upstream production-app-5-80 {
-        # Load balance algorithm; empty for round robin, which is the default
-        least_conn;
-        server 10.2.21.37:9292 max_fails=0 fail_timeout=0;
-        server 10.2.65.105:9292 max_fails=0 fail_timeout=0;
+        server 10.2.109.177:3000 max_fails=0 fail_timeout=0;
     }
 
     upstream production-app-6-80 {
@@ -201,61 +195,67 @@
     upstream production-lap-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.45.223:8000 max_fails=0 fail_timeout=0;
+        server 10.2.21.36:8000 max_fails=0 fail_timeout=0;
         server 10.2.78.36:8000 max_fails=0 fail_timeout=0;
+        server 10.2.45.223:8000 max_fails=0 fail_timeout=0;
         server 10.2.99.151:8000 max_fails=0 fail_timeout=0;
-        server 10.2.21.36:8000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-7-80{
+    upstream production-app-1-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.79.126:3000 max_fails=0 fail_timeout=0;
-        server 10.2.35.105:3000 max_fails=0 fail_timeout=0;
-        server 10.2.114.143:3000 max_fails=0 fail_timeout=0;
-        server 10.2.50.44:3000 max_fails=0 fail_timeout=0;
-        server 10.2.149.135:3000 max_fails=0 fail_timeout=0;
-        server 10.2.45.155:3000 max_fails=0 fail_timeout=0;
+        server 10.2.71.14:3000 max_fails=0 fail_timeout=0;
+        server 10.2.32.22:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-8-80 {
+    upstream production-app-2-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.53.23:5000 max_fails=0 fail_timeout=0;
-        server 10.2.110.22:5000 max_fails=0 fail_timeout=0;
-        server 10.2.35.91:5000 max_fails=0 fail_timeout=0;
-        server 10.2.45.221:5000 max_fails=0 fail_timeout=0;
+        server 10.2.110.13:3000 max_fails=0 fail_timeout=0;
+        server 10.2.109.195:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream upstream-default-backend {
+    upstream production-app-9-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.157.13:8080 max_fails=0 fail_timeout=0;
+        server 10.2.78.26:3000 max_fails=0 fail_timeout=0;
+        server 10.2.59.22:3000 max_fails=0 fail_timeout=0;
+        server 10.2.96.249:3000 max_fails=0 fail_timeout=0;
+        server 10.2.32.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.114.177:3000 max_fails=0 fail_timeout=0;
+        server 10.2.83.20:3000 max_fails=0 fail_timeout=0;
+        server 10.2.118.111:3000 max_fails=0 fail_timeout=0;
+        server 10.2.26.23:3000 max_fails=0 fail_timeout=0;
+        server 10.2.35.150:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.125:3000 max_fails=0 fail_timeout=0;
+        server 10.2.157.165:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-3-80 {
+    upstream production-app-5-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.79.124:3000 max_fails=0 fail_timeout=0;
-        server 10.2.82.66:3000 max_fails=0 fail_timeout=0;
-        server 10.2.45.219:3000 max_fails=0 fail_timeout=0;
-        server 10.2.59.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.21.37:9292 max_fails=0 fail_timeout=0;
+        server 10.2.65.105:9292 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-9-80 {
+    upstream production-app-7-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.96.249:3000 max_fails=0 fail_timeout=0;
-        server 10.2.157.165:3000 max_fails=0 fail_timeout=0;
-        server 10.2.114.177:3000 max_fails=0 fail_timeout=0;
-        server 10.2.118.111:3000 max_fails=0 fail_timeout=0;
-        server 10.2.79.125:3000 max_fails=0 fail_timeout=0;
-        server 10.2.78.26:3000 max_fails=0 fail_timeout=0;
-        server 10.2.59.22:3000 max_fails=0 fail_timeout=0;
-        server 10.2.35.150:3000 max_fails=0 fail_timeout=0;
-        server 10.2.32.21:3000 max_fails=0 fail_timeout=0;
-        server 10.2.83.20:3000 max_fails=0 fail_timeout=0;
-        server 10.2.26.23:3000 max_fails=0 fail_timeout=0;
+        server 10.2.114.143:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.126:3000 max_fails=0 fail_timeout=0;
+        server 10.2.45.155:3000 max_fails=0 fail_timeout=0;
+        server 10.2.35.105:3000 max_fails=0 fail_timeout=0;
+        server 10.2.50.44:3000 max_fails=0 fail_timeout=0;
+        server 10.2.149.135:3000 max_fails=0 fail_timeout=0;
+    }
+
+    upstream production-app-8-80 {
+        # Load balance algorithm; empty for round robin, which is the default
+        least_conn;
+        server 10.2.53.23:5000 max_fails=0 fail_timeout=0;
+        server 10.2.45.221:5000 max_fails=0 fail_timeout=0;
+        server 10.2.35.91:5000 max_fails=0 fail_timeout=0;
+        server 10.2.110.22:5000 max_fails=0 fail_timeout=0;
     }
 
     server {

This caused the NGINX ingress controller to reload its configuration several times per minute, making these shutting down workers pile up until the pod got OOMKilled.

Things got a lot better once I upgraded the NGINX ingress controller to a fixed version and specified the --sort-backends=true command line flag.

Thanks to @aledbf for his assistance in finding and fixing this bug!

Further Minizing Config Reloads

The lesson here is to keep in mind that configuration reloads are expensive operations and it’s a good idea to avoid those especially when proxying WebSocket connections. This is why we decided to create a specific ingress controller deployment just for proxying these long-running connections.

In our case, changes to WebSocket applications happen much less frequently than other applications; by using a separate ingress controller, we avoid reloading the configuration for the WebSocket ingress whenever there are changes (or scaling events/restarts) to other applications.

Separating the deployment also gave us the ability to use a different ingress configuration that’s more suited to long-running connections.

Fine-Tune Pod Autoscalers

Since NGINX ingress uses pod IPs as upstream servers, every time the list of endpoints for a given Service changes, the ingress configuration must be regenerated and reloaded. Thus, if you are observing frequent autoscaling events for your applications during normal load, it might be a sign that your HorizontalPodAutoscalers need adjustment.

Another thing that most people don’t realize is that the horizontal pod autoscaler have a back-off timer that prevents the same target to be scaled several times in a short period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

Name:                                                   <app>
Namespace:                                              production
Labels:                                                 <none>
Annotations:                                            <none>
CreationTimestamp:                                      Fri, 23 Jun 2017 11:41:59 -0300
Reference:                                              Deployment/<app>
Metrics:                                                ( current / target )
  resource cpu on pods  (as a percentage of request):   46% (369m) / 60%
Min replicas:                                           8
Max replicas:                                           20
Conditions:
  Type                  Status  Reason                  Message
  ----                  ------  ------                  -------
  AbleToScale           False   BackoffBoth             the time since the previous scale is still within both the downscale and upscale forbidden windows
  ScalingActive         True    ValidMetricFound        the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited        True    TooFewReplicas          the desired replica count was less than the minimum replica count
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  14d           10m             39      horizontal-pod-autoscaler                       Normal          SuccessfulRescale       New size: 10; reason: cpu resource utilization (percentage of request) above target
  14d           3m              69      horizontal-pod-autoscaler                       Normal          SuccessfulRescale       New size: 8; reason: All metrics below target

According to the default value for the --horizontal-pod-autoscaler-upscale-delay flag in kube-controller-manager, if your application scaled up, it won’t be able to scale up again for 3 minutes.

Thus, in case your application really experiences an increased load, it might take ~4 minutes (3m from the autoscaler back-off + ~1m from the metrics sync) for the autoscaler to react to the increased load, which might be just enough time for your service to degrade.

References

• Tuning NGINX for Performance
• How TCP backlog works in Linux
• How TCP Sockets Work
• Netfilter Conntrack Memory Usage
• Optimizing web servers for high throuhgput and low latency
• Container isolation gone wrong

It was a good decision at the time. In general, Elastic Beanstalk works fine and has a very gentle learning curve; it didn’t take long for all teams to start using it for their projects.

Fast-forward a few months, everything was nice and good. Our old problems were solved, but - as you might have guessed - we had new ones to worry about.

Cost Issues

In Elastic Beanstalk, each EC2 instance runs exactly one application container.¹ This means that, if you follow reliability best practices, you’ll have two or more instances (spread across multiple availability zones) for each application. You might need even more instances if you have other environments besides the production one, i.e. staging.

Anyway, you’ll end up having multiple dedicated instances per service which, depending on your workload, will sit there doing nothing most of the time.

We needed to find a way to use our available compute resources more wisely.

The Winner

After looking around for alternatives to ECS, Kubernetes seemed to be the right one for us.

Kubernetes is a container orchestration tool that builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

Although Kubernetes is a feature-rich project, a few key features caught our attention: namespaces, automated rollouts and rollbacks, service discovery via DNS, automated container scaling based on resource usage, and of course, the promise of a self-healing system.

Kubernetes is somewhat opinionated around how containers are supposed to be organized and networked, but this should not be a problem if your service follows the Twelve-Factor practices.

Our Path to Production

In order to ensure Kubernetes was a viable option for us, the first thing we did was perform some reliability tests to make sure it could handle failure modes such as dying nodes, killed Kubelet/Proxy/Docker daemons, and availability zone outages.

It’s impossible to anticipate all the ways things can go wrong, but in the end, we were very impressed by how Kubernetes managed to handle these failures.

At that time, we used kube-up to bootstrap our test clusters. This tool, although it served its purpose, not always worked as expected; it suffered from a number of issues, such as poorly chosen defaults, random timeouts that left the stack only half-created, and inconsistent behavior when destroying the cluster causing orphan resources to be left behind.²

Once we agreed that Kubernetes was the way to go, we needed a more reliable way to create and destroy our Kubernetes clusters.

Enter kube-aws

kube-aws is a tool created by some good guys from CoreOS. The cool thing about it is that it uses CloudFormation under the hoods, which gives us some neat advantages.

The first obvious advantage is that it’s very easy to create and destroy clusters without leaving anything silently hanging around.

Another feature is that, unlike kube-up, you can create a cluster in an existing VPC so all services running in Kubernetes have access to your existing AWS resources - such as relational databases - right off the bat.

In fact, you can run multiple clusters at the same time in the same VPC. This has a nice side-effect in which you can treat each cluster as an immutable piece of infrastructure; instead of modifying a running cluster - and risking break something - you simply create a new cluster and gradually shift traffic from the old one to the new in a way that any incidents has limited impact.

The final and probably the most useful feature is that you can easily customize nearly every aspect of the cluster provisioning configuration to make it fit your own needs. In our case, we added cluster level logging that ingests application logs to Sumologic, cluster monitoring with InfluxDB and Grafana, ABAC-based authorization, among other things.

The First Environment

After solving the problem of reliably creating and destroying clusters, we felt confident to start migrating our staging environment over to Kubernetes.

It was easy enough to manually create the yaml manifests for the first deployments, but we needed an automated way to deploy new application images as soon as they were built by our continuous integration system.

Just as a proof of concept, we quickly hacked together a small function in AWS Lambda (based on this article) that automatically updated the corresponding deployment object whenever it received a merge notification in which the tests passed.

This small Lambda function has now evolved into a major component in our delivery pipeline, orchestrating deployments to other environments as well, including production.

With this done, migrating staging services from Beanstalk to Kubernetes was pretty straightforward. First, we created one DNS record for each service (each initially pointing to the legacy deployment in Elastic Beanstalk) and made sure that all services referenced each other via this DNS. Then, it was just a matter of changing those DNS records to point the corresponding Kubernetes-managed load balancers.

To ensure every part of the pipeline was working as expected, we spent some time monitoring all staging deployments looking for bugs and polishing things up as we could.

More Tests, More Learning

Before deploying our first production service to Kubernetes, we did some load testing to find out the optimal configuration for the resource requirements needed by each service and out how many pods we needed to handle the current traffic.

Observing how your services behave under load and how much compute they need is essential.

Also take some time to understand how QoS classes work in Kubernetes so you have a more fine control over which pods gets killed in the case of memory pressure. This is particularly important if you, like us, share the same cluster for all environments.

Tip: Enable Cross-Zone Load Balancing (AWS)

This is already fixed in Kubernetes 1.4, but for now, if you expose your services via the LoadBalancer type, don’t forget to manually enable cross-zone load balancing for the corresponding ELB; if you don’t, you might notice uneven balancing across your application pods if they are spread in nodes from different availability zones.

Tip: Give the kube-system Namespace Some Love

If you ever tried Kubernetes, you probably noticed there’s a kube-system namespace there with a bunch of stuff in it; do yourself a favor and take some time to understand the role of each of those things.

For instance, take the DNS add-on; it’s rather common to see people having issues because they forgot to add more DNS pods to handle their ever-increasing workload.

Going Live

Instead of shifting all traffic at once, like we did in staging, we thought we needed to take a more careful approach and used weighted routing policy to gradually shift traffic to the Kubernetes cluster.

Once we noticed no more requests were reaching the legacy Beanstalk environments, we went ahead and killed them.

Update (Sep 21, 2016): All major services were migrated to our new platform! These are the final numbers:³

• ~53-63% decrease in monthly costs
• ~72-82% decrease in # of instances

Beyond Production

Kubernetes gave us the power to almost effortlessly mold our delivery pipeline in a way we never thought possible. One example of such improvement is what we call here development environments.

Whenever someone opens a Pull Request to one of our projects, the AWS Lambda function I mentioned earlier creates a temporary environment running the modifications introduced by the PR.

Also, whenever new code is pushed, this environment gets automatically updated as long as they pass the tests. Finally, when the PR is merged (or closed), the environment is deleted.

This feature made our code reviews more thorough because the developers can actually see the changes running. This is even more useful for UX changes in front-end services; artists and product owners get the chance to validate the changes and share their inputs before the PR is merged.

To send the GitHub Status notifications you see in these pictures, we implemented a small daemon in Go that monitors deployments to our development namespace and reconciles the deployment status for each revision.

Conclusion

Kubernetes is a very complex piece of software that aims to solve a very complex problem, so expect to spend some time learning how its many pieces fit together before using it in your projects.

Kubernetes is production-ready, but avoid the temptation of trying to run everything on it. In our experience, Kubernetes does not offer a clean solution for a number of problems you might face, such as stateful applications.

The documentation is not great as well, but initiatives like the Kubernetes Bootcamp and Kelsey Hightower’s Kubernetes The Hard Way gives me hope that this will no longer be a problem in the near future.

Without Kubernetes, I don’t know how - or if - we could have accomplished all the things we did in such a small period of time with such a small engineering team.⁴

We hope to continue building on Kubernetes to make our delivery platform even more dynamic and awesome!