Only in 2025, there were at least half a dozen campaigns exploiting various pieces of modern supply chain software to exfiltrate sensitive data at scale. Some of these events even leveraged AI agents installed in people’s machines to help locate and extract sensitive information like tokens, crypto wallets and service account keys.

Even if you are careful with these things, it’s possible – likely even – that you have at least a couple of tokens or secrets laying around in plain text in your system.¹

Unfortunately, many CLI tools still rely on – or at least encourage in “Getting Started”-style guides – insecure practices, such as storing tokens and secrets in plain text configuration files and passing them as command line arguments.

Is the OS Keyring a Viable Option?

If you use Linux with a fully-featured Desktop Environment such as GNOME, chances are you already have a Secret Service implementation installed, that you can interact with via the secret-tool utility:

1
2
3
4
5
6
7
8

# Storing a secret in the system keyring:
secret-tool store --label="Secret Description" [[attribute] [value] ...]

# Retrieving the secret:
secret-tool lookup [attribute] [value]

# Deleting the secret:
secret-tool clear [attribute] [value]

While this can fulfill some basic use-cases, the data persistence options for the OS keyring are limited, so you should only use it for non-critical short-lived secrets.

Keyrings also usually stay unsealed for the duration of the user session, making them not suitable for protecting more sensitive information.

Password Managers

For critical and/or long-lived secrets, it’s recommended the usage of a proper password manager solution.

Solutions such as 1Password or Bitwarden would work fine (they both provide CLI tools for interacting with their vaults), but I’ve grown to prefer more lightweight options such as GNU Pass, at least for tokens and secrets that are mainly accessed via the terminal.

Dealing With Environment Variables

Some people just export their tokens in their ~/.bashrc and get on with it. This is not a recommended approach for sensitive information as these variables would be exposed indiscriminately to all user processes.

The same applies for other secrets stored in plain text files in your hard drive. Any process you start could just scan your files, discover sensitive information, and compromise them. This is more and more of a concern these days with the advent of AI tools that still lack proper sandboxing and security controls.

The GoPass Way

GoPass is a GNU Pass reimplementation in Go with a few extra goodies.

One of those goodies is the gopass env subcommand. Suppose you have the following secrets:

You can run a command with these vars set with gopass env keys/aws -- aws s3 ls.

This feature alone greatly reduces the need for secrets stored in plain-text configuration files, since most tools allow you to pass credentials via environment variables as an alternative.

However, this implementation doesn’t cover all my needs for consuming secrets in the terminal. For instance, it’s not possible to run a command with secrets from multiple different prefixes (say a GitHub token + AWS credentials), unless you lay out your passwords in a specific way, possibly by duplicating files or symlinking files around.

My New Way (With GNU Pass)

After reading the GNU Pass main page recently I came across a section Extensions for pass. I used GNU Pass for many years and didn’t know that it could be extended like that!

So I decided to ask my new best friend Claude Code to come up with an extension that worked just like the gopass env subcommand, but with more options for composing secrets.

The full up-to-date version for this script can be found here, but the API looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/env bash
cmd_env_usage() {
    cat <<-_EOF
Usage:
    $PROGRAM env secret-path [secret-path2 ...] -- [command] [args...]
        Decrypt the secrets at the specified paths, export their contents as
        environment variables, and execute the command + args with those variables set.
        If multiple secrets define the same variable, later secrets take precedence.

        Secret paths can be either:
        - Individual secret files (containing KEY=VALUE pairs)
        - Directories ending with '/' (will recursively decrypt all secrets within,
          using the secret filename as the variable name)

Examples:
    $PROGRAM env /env/infra-keys -- terraform init
    $PROGRAM env /env/common /env/production -- aws s3 ls
    $PROGRAM env secret/app/ -- node server.js
    $PROGRAM env secret/app/ /env/common -- ./run.sh
_EOF
}

You can use this by putting this file in /usr/lib/password-store/extensions or whatever that directory is for your distro.²

Security Considerations

This approach may have some security considerations of their own. In other words, do not blindly follow me, I may not know what I’m doing. 😄

A couple of things I can think of:

1. As with any solution based on environment variables, the env vars for the launched processes can still be accessed via /proc/<pid>/environ and the ps e command.
2. Exported variables are available to children processes, not only the original process launched by the script.

Alternative Approaches

Linus Heckemann’s Blog provide an alternative and more secure approach by not relying on environment variables at all, which of course can be more or less applicable depending on your workflow and the tools involved.

One particularly nice trick is the usage of process substitution capabilities supported by many of the modern shells, including bash, which suits tools like curl and vault:

1
2
3
4
5
6

# Fetch my-secret from GNU Pass and store it into Hashicorp Vault:
vault write secret/my-secret value=@<(pass my-secret)

# Fetch key from GNU Pass and use it as a Bearer token in a POST request:
curl -X POST -H @<(echo "Authorization: Bearer $(pass api-keys/my-app)") \
    https://api.my-app.xyz/data

The secrets extracted from GNU pass in the previous commands would not show up in usual places such as /proc/<PID>/environ and /proc/<PID>/cmdline, making this a very strong option for more sensitive tasks.

Of course you can also leverage Bash Aliases or Functions for abstracting away the details about how secrets are injected onto your most frequent commands.

SSH Keys

Other common type of secret to have laying around are SSH keys.

A recommendation is to use a hardware token such as the YubiKey for this, keeping the private key material secure, meaning that you don’t need to keep your SSH keys in your filesystem anymore. You can also configure the token to require “touch” confirmation for different operations, such as signing, encryption, or authentication, depending on your security and convenience requirements.

After configuring gpg-agent to also act as the SSH agent, you could use ssh-add -L to display the public key (which you can then add to ~/.ssh/authorized_keys on your servers), and ssh should forward authentication requests to the GPG agent.

If you choose this path, I’d strongly recommend checking out drduh’s Yubikey Guide.

Other option would be to leverage your own password manager’s features. Bitwarden (here) and 1Password (here), for instance, allow you to store your public and private keys in the password manager itself and then allow access to them via custom SSH agent implementations that you can plug into your system.

Closing Note

The idea for this post is just to show you how I do it. It may or may not work for you, and it’s definitely not the only way to do it, so feel free to experiment and choose whatever works best for you considering your security requirements!

Recently, my house was hit by a lightning that fried the Shield, and all of a sudden I had no way of streaming my stuff.

At some point I may buy a new box, or try some new thing for the same purpose, but I don’t want to spend money right now, especially because these boxes are very expensive here in Brazil due to high shipping costs and taxes.

As many of you, I also have a few Raspberry Pi boards sitting idle in my junk drawer. The last one I bought, many many years ago, is a Raspberry Pi 3 Model B, that I once used as a cron job runner (i.e. for running Ansible playbooks), and I decided to use that for replacing the Shield. It isn’t capable of streaming 4k media, can’t hardware decode HEVC or other more modern media formats, but it’s better than nothing.

Was it Still Working?

After plugging it in, it booted, but it was running an old Raspbian version (based on Buster). So I manually upgraded the system to Bullseye and then to Trixie.

I should have installed a clean version, but I lost my SD card adapter, so I wasn’t able to plug the SD card to my computer for a clean install.

Everything went seemingly okay, but after booting into Trixie, I got an error regarding a missing signing key. For the most part, everything seemed to work despite of the error, but when I tried to install Kodi with sudo apt install kodi, the installation failed due to unmet dependencies, so I had to fix it if I wanted to watch my stuff.

I tried Googling about it but the fix didn’t emerge too easily (which key is missing? Where to find it? How to configure it?), so I decided to fire up Claude Code and let it fix it for me. I provided it with my SSH key, the SSH command for it to sign into the Raspberry Pi, and asked it to fix my shit.

The Experiment

First, I asked Claude Code to resolve the signing key issue in apt and install Kodi, which it managed to do in a single shot in a couple of minutes:

I have a raspberry pi hosted in my local network, at the address pi.local. You can ssh into it with the following command:

ssh pi@pi.local

This pi was updated from Debian 11 to 12, and then from 12 to 13, but I think something went wrong with the the last upgrade, as commands such as sudo apt update show missing key errors. Not sure how to fix those.

What I ultimately want is to install Kodi, but the command sudo apt install kodi fails due to unmet dependencies.

I want you to fix this installation so we could install kodi.

Then, I asked it to auto-start Kodi when the Pi was booted, which it, again, implemented correctly in the first try. It even rebooted the Pi to ensure that the service was started after the boot, by checking the Systemd service status. Neat.

Now update the pi installation so that kodi starts up automatically once the system is powered up.

After Kodi started, the resolution was all messed up, instead of the expected Full HD resolution. Again, it made a few changes to /boot/config.txt, and after a few retries, the issue was resolved.

The next issue is that I cannot seem to configure kodi to run at full HD resolution (1920x1080); the maximum I could get was to 1024x768. How can I configure kodi to run at the maximum output resolution?

This is what it added (don’t ask what those mean, after all I was just vibin’):

1
2
3
4
5
6
7

# Force HDMI output at 1080p
hdmi_force_hotplug=1
hdmi_group=1
hdmi_mode=31
hdmi_drive=2
disable_overscan=1
config_hdmi_boost=4

Then I noticed that the dphys-swapfile.service was failing, and asked it to fix it, and it gladly reconfigured the swapfile, restarted the service, and everything was good.

Okay, got image back. Let’s leave it like this for now.

Now I found one of the systemd services to report as failed during boot, the dphys-swapfile.service. Can you fix it as well?

Now, the time zone is incorrect. How to configure this anyway? I don’t remember, but apparently he did; checked the NTP daemon was running, asked for the correct time zone to set, and configured it once I provided the information it needed.

The clock is incorrect on my raspberry pi. Configure it to fetch correct date and time from the internet.

What Didn’t Work

I could not use the TV remote control for navigating through the Kodi menus. I remember that it worked at some point, so I asked Claude Code to fix this as well.

Man, this is where things got crazy. It went in a reboot loop, changing random stuff in /boot/config.txt to see if it would work, and eventually messed up all the things it fixed previously. Sometimes it booted to a black screen, sometimes with the wrong resolution, sometimes the sound stopped working.

Then I eventually asked it to stop and revert to the previously working configuration, which to my surprise it did, so I didn’t have to start from scratch again.

For now, I ended up using an iPhone app as the remote control. I may try to fix this at some point.

Wrapping Up

I wouldn’t recommend vibe-patching servers like this without a proper backup first, unless you are just doing it for fun like me.

Or if you just don’t care.

The widely accepted “healthy” ratio, particularly at companies with mature SRE practices like Google, is one SRE for every 10 Software Engineers (SWEs) (a 1:10 ratio). However, this ratio can vary significantly based on an organization’s specific needs, size, and the maturity of its automation and tooling.

– Gemini

For the past 10+ years working as SRE, most of it while in short-staffed squads in late-stage start-ups, the SRE:SWE ratio was nowhere near the 1:10 “gold standard”. A more accurate ratio in my experience is 1:30 - 1:40, and in this setting, saying things can get pretty busy is a bit of an understatement.

Working in highly constrained environments such as these does not come without its challenges, both cultural and technical. For every task you choose to work on, there are like ten others you need to put on hold, all while doing your best not to become a bottleneck for the entire organization.

But this is not the only constraint I’m used to. The other one is cost efficiency; here, offloading everything to expensive SaaS offerings is not an option, so we usually resort to self-hosting, tuning and stitching open source tools for our needs. Deploying a new tool is also something we do not take lightly, as each new tool could drag the team down into an operational hell, so we tend to bet on boring/flexible tools as opposed to over-specialized ones.

The Problem

While working on improving the operational and security maturity for the organization, we wanted to review and reduce direct permissions to the production environment, and one possibility was to use Terraform, which the SRE team already used and was familiar with. But how could we provide other teams the same tools we used¹ in a way that was safe, within the established best practices, and most importantly, did not require the SRE team to review each and every Terraform PR?

Here’s a non-exhaustive list of things we were after:

• Enforcing consistent naming conventions²
• Ensuring resources had the correct labels for proper ownership and cost tracking
• Blocking dangerous operations (i.e. database/bucket deletion)
• Ensuring durability best practices (i.e. enforcing backups for production databases)
• Forcing security best practices (i.e. requiring TLS for DB instances)

At the same time, we didn’t want to always block things that went off the standard path. Some of these requirements were created after critical infrastructure was already in place, so these needed to be supported.

Other reason is that sometimes the team might have a good reason to create a publicly exposed bucket or disabling authentication for a Redis instance, etc, so a workflow for approving these changes was also needed.

The simplest (and cheapest) solution at the time was to codify these practices as OPA policies and enable conftest in our Atlantis pipelines. The workflow is not perfect, but it gets the job done.

With these policies in place, teams could own their Terraform workflows and apply changes to production without waiting on the SRE for changes that were considered harmless, deeply reducing the PR review load on the team.³

Other options we considered:

• GCP Organization Policies: We use Google Cloud for most things, but use specific services from other public clouds as well (AWS, Azure), so a cloud-specific solution was not an option; higher than desired friction in case of policy violations / break glass scenarios.
• Terraform Cloud + Sentinel: Recent history of hostile decisions against the community; pricing model based on resources under management.
• Spacelift and Env0: Too expensive for the number of users and features (single sign-on, RBAC, etc) we required.

A Simple Example

We had great success in using GenAI for generating both policies and tests for various cases, but to get you started, here’s a policy I got from our library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

package gcp

import rego.v1

import data.utils

delete_buckets_with_force_destroy := [res |
	some res in utils.resource_op({"delete"}, [`^google_storage_bucket$`])
	res.change.after.force_destroy == true
]

deny contains msg if {
	some res in delete_buckets_with_force_destroy

	msg := sprintf(
		"The resource '%s' will be force-deleted, which might cause data loss",
		[res.address],
	)
}

As our collection of policies grew, we ended up with a few extra packages for reusing some common behavior, such as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

package utils

import rego.v1

resource_op(operations, resource_types) := [res |
	some res in input.resource_changes
	some action in res.change.actions

	operations[action]

	some type in resource_types
	regex.match(type, res.type)
]

Linting and Testing

I recommend using Regal to avoid common mistakes and help you write more idiomatic Rego code:

% regal lint .
3 files linted. No violations found.

It’s also important to write tests for your policies to accelerate the feedback loop when introducing new policies while catching regressions early.

Here’s an example test suite for that policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

package gcp_test

import rego.v1

import data.gcp

test_deny_delete_bucket_with_force_destroy if {
	result := gcp.deny with input as {"resource_changes": [{
		"address": "google_storage_bucket.my_bucket",
		"type": "google_storage_bucket",
		"change": {
			"actions": ["delete"],
			"after": {"force_destroy": true},
		},
	}]}

	result == {"The resource 'google_storage_bucket.my_bucket' will be force-deleted, which might cause data loss"}
}

test_allow_delete_bucket_without_force_destroy if {
	result := gcp.deny with input as {"resource_changes": [{
		"address": "google_storage_bucket.my_bucket",
		"type": "google_storage_bucket",
		"change": {
			"actions": ["delete"],
			"after": {"force_destroy": false},
		},
	}]}

	count(result) == 0
}

# TODO: Add tests for other corner cases...

Running the tests:

% conftest verify --report full
policy/gcp/bucket_force_destroy_tests.rego:
data.gcp_test.test_allow_create_bucket_with_force_destroy: PASS (926.401µs)
data.gcp_test.test_allow_empty_resource_changes: PASS (1.026078ms)
data.gcp_test.test_allow_update_bucket_with_force_destroy: PASS (857.508µs)
data.gcp_test.test_deny_delete_bucket_with_force_destroy: PASS (1.506513ms)
data.gcp_test.test_allow_delete_bucket_without_force_destroy: PASS (1.319212ms)
data.gcp_test.test_allow_delete_other_resource_type: PASS (1.329778ms)
--------------------------------------------------------------------------------
PASS: 6/6

What Doesn’t Work so Well

The following points are specific to the built-in OPA integration for Atlantis, which is what we currently use.

UX for Policy Violations

If you have a single policy set with a couple dozen policies and a single approver, the UX is good enough, but as your library of policies grow and you start separating them into different policy sets with different owners/approvers, the policy violation messages can get too long and confusing.

By default, this notification is also sent even when no policy violations are detected, increasing the comment noise in your PRs. To mitigate this, you can experiment with the server flag --quiet-policy-checks.

Workflow for Managing Policy Sets

Policy sets are configured via the Atlantis server-side repo configuration, which is a static configuration file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

policies:
  owners:
    teams:
      - sre
  policy_sets:
    - name: gcp.common
      path: <CODE_DIRECTORY>/policies/gcp/common/
      source: local
    - name: gcp.iam
      path: <CODE_DIRECTORY>/policies/gpc/iam/
      source: local
      approve_count: 2
      owners:
        teams:
          - sre
          - sec
    - ...

It’s okay to start creating policies in a single set, but as you grow, you’ll likely need to assign different owners for different policy sets.

Also, depending on how frequently you update your policies, you might want to decouple the policy update workflow from the Atlantis server lifecycle. Conftest (and thus Atlantis) can fetch policies from a GitHub repository, OCI repository, etc, so use this to your advantage – but you need to be careful to sync disruptive changes (i.e. directory structure changes) to the Atlantis server-side repo configuration.

No Built-In Support for Policy Notifications

When a PR triggers one or more policy violations, all Atlantis does is create a comment in the PR. In our case, when that happens, the users themselves escalate to the SRE team for review or approval, but this might not work for you.

If you need other ways to be notified about policy violations (e.g. via Slack), you’ll need to implement it yourself.

One way could be writing a small GitHub App that receives hooks for all comments in a PR, detecting the policy violation comments, parsing/forwarding them to the owners via Slack, and maybe also providing actions directly in the notification for approval.⁴

Policy Approvals vs Branch Syncs

If you approve a policy for a branch that’s out of sync with the main branch, the policy approval will be “erased” after the rebase and you’ll have to approve it again, even when none of the code in the PR changes. This is particularly annoying for changes that require lots of back and forth due to apply errors.

If you want to resolve this, you’ll have to cook something yourself.

Again, some sort of GitHub App that automatically re-approves a PR that was once approved by some human as long as the set of violated policies and resources involved do not change.

Atlantis API

I once tried to enable the Atlantis API to implement basic drift detection and correction, but I could not make it work. Since the Atlantis primitives work on top of pull requests, even when using the API, you’ll need to provide a valid PR number when asking for a plan or apply, and even then things didn’t quite work as I expected.

Recommendations

• Let me repeat myself: lint and write tests for your policies!
• Add some unique identifier for each policy error (i.e. GCP001); this would allow you to track the frequency each policy is being triggered, and maybe even provide a nice wiki for your users that explain each policy in detail, how to resolve the issue, etc.
• Avoid putting too much stuff in the same file; keep each policy in a separate file.
• Periodically review your policies; check whether they are blocking stuff that needs to be blocked, or just introducing noise without real benefits; remove bad/useless policies.
• Be cautious when adopting third-party policy packs unless you have a way of granularly enabling or disabling specific policies that do not make sense for your own situation.

About the periodically reviewing your policies part: In one of the reviews, we identified that a single policy targeting a single resource accounted for 46% of all policy violation errors we had in the previous 90 days!

Conclusion

If you ask me if I think this is the dream workflow for Terraform code, I’d be lying if I said that it is. But hey, it’s open source, works mostly fine as long as you don’t want to do anything too crazy, and helped us a lot. I even made a contribution to the project a couple of years ago.

If my employer had deeper pockets, we might have gone in other directions, but I’m glad Atlantis exists. Now that the project was accepted into the CNCF, my hopes are up!

For those that use Atlantis and stumbled across some of these issues, I’d be happy to learn how you addressed them!

The Project

This tutorial follows a more practical approach (with hopefully just the right amount of theory!), so we provide a simple Docker Compose configuration for simplifying the project bootstrap.

You can download the code from here.

Pre-Requisites

• Docker + Docker Compose

Running the Code

Run the following command to start everything up:

1
2
3
4

$ docker-compose up -d

# Or, if you use podman:
$ podman-compose up -d

• Alertmanager: http://localhost:9093
• Grafana: http://localhost:3000 (user/password: admin)
• Prometheus: http://localhost:9090
• Sample Node.js Application: http://localhost:4000

Cleaning Up

Run the following command to stop all running containers from this project:

1

$ docker-compose rm -fs

Prometheus Overview

Prometheus is an open source monitoring and time-series database (TSDB) designed after Borgmon, the monitoring tool created internally at Google for collecting metrics from jobs running in their cluster orchestration platform, Borg.

The following image shows an overview of the Prometheus architecture.

In the center we have a Prometheus server, which is the component responsible for periodically collecting and storing metrics from various targets (e.g. the services you want to collect metrics from).

The list of targets can be statically defined in the Prometheus configuration file, or we can use other means to automatically discover those targets via Service discovery. For instance, if you want to monitor a service that’s deployed in EC2 instances in AWS, you can configure Prometheus to use the AWS EC2 API to discover which instances are running a particular service and then scrape metrics from those servers; this is preferred over statically listing the IP addresses for our application in the Prometheus configuration file, which will eventually get out of sync, especially in a dynamic environment such as a public cloud provider.

Prometheus also provides a basic Web UI for running queries on the stored data, as well as integrations with popular visualization tools, such as Grafana.

Push vs Pull

Previously, we mentioned that the Prometheus server scrapes (or pulls) metrics from our target applications.

This means Prometheus took a different approach than other “traditional” monitoring tools, such as StatsD, in which applications push metrics to the metrics server or aggregator, instead of having the metrics server pulling metrics from applications.

The consequence of this design is a better separation of concerns; when the application pushes metrics to a metrics server or aggregator, it has to make decisions like: where to push the metrics to; how often to push the metrics; should the application aggregate/consolidate any metrics before pushing them; among other things.

In pull-based monitoring systems like Prometheus, these decisions go away; for instance, we no longer have to re-deploy our applications if we want to change the metrics resolution (how many data points collected per minute) or the monitoring server endpoint (we can architect the monitoring system in a way completely transparent to application developers).

Want to know more? The Prometheus documentation provides a comparison with other tools in the monitoring space regarding scope, data model, and storage.

Now, if the application doesn’t push metrics to the metrics server, how does the applications metrics end up in Prometheus?

Metrics Endpoint

Applications expose metrics to Prometheus via a metrics endpoint. To see how this works, let’s start everything by running docker-compose up -d if you haven’t already.

Visit http://localhost:3000 to open Grafana and log in with the default admin user and password. Then, click on the top link “Home” and select the “Prometheus 2.0 Stats” dashboard.

Yes, Prometheus is scraping metrics from itself!

Let’s pause for a moment to understand what happened. First, Grafana is already configured with a Prometheus data source that points to the local Prometheus server. This is how Grafana is able to display data from Prometheus. Also, if you look at the Prometheus configuration file, you can see that we listed Prometheus itself as a target.

1
2
3
4
5
6
7
8

# config/prometheus/prometheus.yml

# Simple scrape configuration for each service
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - localhost:9090

By default, Prometheus gets metrics via the /metrics endpoint in each target, so if you hit http://localhost:9090/metrics, you should see something like this:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.95e-05
go_gc_duration_seconds{quantile="0.25"} 0.0001589
go_gc_duration_seconds{quantile="0.5"} 0.0002188
go_gc_duration_seconds{quantile="0.75"} 0.0004158
go_gc_duration_seconds{quantile="1"} 0.0090565
go_gc_duration_seconds_sum 0.0331214
go_gc_duration_seconds_count 47
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 39
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.10.3"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.7429992e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.37005104e+08
...

In this snippet alone we can notice a few interesting things:

1. Each metric has a user friendly description that explains its purpose
2. Each metric may define additional dimensions, also known as labels. For instance, the metric go_info has a version label
   • Every time series is uniquely identified by its metric name and the set of label-value pairs
3. Each metric is defined as a specific type, such as summary, gauge, counter, and histogram. More information on each data type can be found here

But how does this text-based response turns into data points in a time series database?

The best way to understand this is by running a few simple queries.

Open the Prometheus UI at http://localhost:9090/graph, type process_resident_memory_bytes in the text field and hit Execute.

You can use the graph controls to zoom into a specific region.

This first query is very simple as it only plots the value of the process_resident_memory_bytes gauge as time passes, and as you might have guessed, that query displays the resident memory usage for each target, in bytes.

Since our setup uses a 5-second scrape interval, Prometheus will hit the /metrics endpoint of our targets every 5 seconds to fetch the current metrics and store those data points sequentially, indexed by timestamp.

1
2
3

# In prometheus.yml
global:
  scrape_interval: 5s

You can see the all samples from that metric in the past minute by querying process_resident_memory_bytes{job="grafana"}[1m] (select Console in the Prometheus UI):

Queries that have an appended range duration in square brackets after the metric name (i.e. <metric>[<duration>]) returns what is called a range vector, in which <duration> specify how far back in time values should be fetched for each resulting range vector element.

In this example, the value for the process_resident_memory_bytes metric at the timestamp 1530461477.446 was 40861696, and so on.

Duplicate Metrics Names?

If you inspect the contents of the /metrics endpoint at all our targets, you’ll see that multiple targets export metrics under the same name.

But isn’t this a problem? If we are exporting metrics under the same name, how can we be sure we are not mixing metrics between different applications into the same time series data?

Consider the previous metric, process_resident_memory_bytes: Grafana, Prometheus, and our sample application all export a gauge metric under the same name. However, did you notice in the previous plot that somehow we were able to get a separate time series from each application?

Quoting the documentation:

In Prometheus terms, an endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job.

When Prometheus scrapes a target, it attaches some labels automatically to the scraped time series which serve to identify the scraped target:

• job - The configured job name that the target belongs to.
• instance - The <host>:<port> part of the target’s URL that was scraped.

Since our configuration has three different targets (with one instance each) exposing this metric, we can see three lines in that plot.

Monitoring Uptime

For each instance scrape, Prometheus stores a up metric with the value 1 when the instance is healthy, i.e. reachable, or 0 if the scrape failed.

Try plotting the query up in the Prometheus UI.

If you followed every instruction up until this point, you’ll notice that so far all targets were reachable at all times.

Let’s change that. Run docker-compose stop sample-app and after a few seconds you should see the up metric reporting our sample application is down.

Now run docker-compose restart sample-app and the up metric should report the application is back up again.

Want to know more? The Prometheus query UI provides a combo box with all available metric names registered in its database. Do some exploring, try querying different ones. For instance, can you plot the file descriptor handles usage (in %) for all targets? Tip: the metric names end with _fds.

A Basic Uptime Alert

We don’t want to keep staring at dashboards in a big TV screen all day to be able to quickly detect issues in our applications, after all, we have better things to do with our time, right?

Luckily, Prometheus provides a facility for defining alerting rules that, when triggered, will notify Alertmanager, which is the component that takes care of deduplicating, grouping, and routing them to the correct receiver integration (i.e. email, Slack, PagerDuty, OpsGenie). It also takes care of silencing and inhibition of alerts.

Configuring Alertmanager to send metrics to PagerDuty, or Slack, or whatever, is out of the scope of this tutorial, but we can still play around with alerts.

We already have the following alerting rule defined in config/prometheus/prometheus.rules.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

groups:
  - name: uptime
    rules:
      # Uptime alerting rule
      # Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
      - alert: ServerDown
        expr: up == 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: One or more targets are down
          description: Instance {{ $labels.instance }} of {{ $labels.job }} is down

Each alerting rule in Prometheus is also a time series, so in this case you can query ALERTS{alertname="ServerDown"} to see the state of that alert at any point in time; this metric will not return any data points now because so far no alerts have been triggered.

Let’s change that. Run docker-compose stop grafana to kill Grafana. After a few seconds you should see the ServerDown alert transition to a yellow state, or PENDING.

The alert will stay as PENDING for one minute, which is the threshold we configured in our alerting rule. After that, the alert will transition to a red state, or FIRING.

After that point, the alert will show up in Alertmanager. Visit http://localhost:9093 to open the Alertmanager UI.

Let’s restore Grafana. Run docker-compose restart grafana and the alert should go back to a green state after a few seconds.

Want to know more? There are several alerting rule examples in the awesome-prometheus-alerts repository for common scenarios and popular systems.

Instrumenting Your Applications

Let’s examine a sample Node.js application we created for this tutorial.

Open the ./sample-app/index.js file in your favorite text editor. The code is fully commented, so you should not have a hard time understanding it.

Measuring Request Durations

We can measure request durations with percentiles or averages. It’s not recommended relying on averages to track request durations because averages can be very misleading (see the References for a few posts on the pitfalls of averages and how percentiles can help). A better way for measuring durations is via percentiles as it tracks the user experience more closely:

In Prometheus, we can generate percentiles with summaries or histograms.

To show the differences between these two, our sample application exposes two custom metrics for measuring request durations with, one via a summary and the other via a histogram:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// Summary metric for measuring request durations
const requestDurationSummary = new prometheusClient.Summary({
  // Metric name
  name: 'sample_app_summary_request_duration_seconds',

  // Metric description
  help: 'Summary of request durations',

  // Extra dimensions, or labels
  // HTTP method (GET, POST, etc), and status code (200, 500, etc)
  labelNames: ['method', 'status'],

  // 50th (median), 75th, 90th, 95th, and 99th percentiles
  percentiles: [0.5, 0.75, 0.9, 0,95, 0.99]
});

// Histogram metric for measuring request durations
const requestDurationHistogram = new prometheusClient.Histogram({
  // Metric name
  name: 'sample_app_histogram_request_duration_seconds',

  // Metric description
  help: 'Histogram of request durations',

  // Extra dimensions, or labels
  // HTTP method (GET, POST, etc), and status code (200, 500, etc)
  labelNames: ['method', 'status'],

  // Duration buckets, in seconds
  // 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
});

As you can see, in a summary we specify the percentiles in which we want the Prometheus client to calculate and report latencies for, while in a histogram we specify the duration buckets in which the observed durations will be stored as a counter (i.e. a 300ms observation will be stored by incrementing the counter corresponding to the 250ms-500ms bucket).

Our sample application introduces a one-second delay in approximately 5% of requests, just so we can compare the average response time with 99th percentiles:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// Main route
app.get('/', async (req, res) => {
  // Simulate a 1s delay in ~5% of all requests
  if (Math.random() <= 0.05) {
    const sleep = (ms) => {
      return new Promise((resolve) => {
        setTimeout(resolve, ms);
      });
    };
    await sleep(1000);
  }
  res.set('Content-Type', 'text/plain');
  res.send('Hello, world!');
});

Let’s put some load on this server to generate some metrics for us to play with:

1
2
3
4
5
6
7
8
9

$ docker run --rm -it --net host williamyeh/wrk -c 4 -t 2 -d 900  http://localhost:4000
Running 15m test @ http://localhost:4000
  2 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   269.03ms  334.46ms   1.20s    78.31%
    Req/Sec    85.61    135.58     1.28k    89.33%
  72170 requests in 15.00m, 14.94MB read
Requests/sec:     80.18
Transfer/sec:     16.99KB

Now run the following queries in the Prometheus UI:

1
2
3
4
5
6
7
8

# Average response time
rate(sample_app_summary_request_duration_seconds_sum[15s]) / rate(sample_app_summary_request_duration_seconds_count[15s])

# 99th percentile (via summary)
sample_app_summary_request_duration_seconds{quantile="0.99"}

# 99th percentile (via histogram)
histogram_quantile(0.99, sum(rate(sample_app_histogram_request_duration_seconds_bucket[15s])) by (le, method, status))

The result of these queries may seem surprising.

The first thing to notice is how the average response time fails to communicate the actual behavior of the response duration distribution (avg: 50ms; p99: 1s); the second is how the 99th percentile reported by the the summary (1s) is quite different than the one estimated by the histogram_quantile() function (~2.2s). How can this be?

Quantile Estimation Errors

Quoting the documentation:

You can use both summaries and histograms to calculate so-called φ-quantiles, where 0 ≤ φ ≤ 1. The φ-quantile is the observation value that ranks at number φ*N among the N observations. Examples for φ-quantiles: The 0.5-quantile is known as the median. The 0.95-quantile is the 95th percentile.

The essential difference between summaries and histograms is that summaries calculate streaming φ-quantiles on the client side and expose them directly, while histograms expose bucketed observation counts and the calculation of quantiles from the buckets of a histogram happens on the server side using the histogram_quantile() function.

In other words, for the quantile estimation from the buckets of a histogram to be accurate, we need to be careful when choosing the bucket layout; if it doesn’t match the range and distribution of the actual observed durations, you will get inaccurate quantiles as a result.

Remembering our current histogram configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// Histogram metric for measuring request durations
const requestDurationHistogram = new prometheusClient.Histogram({
  // Metric name
  name: 'sample_app_histogram_request_duration_seconds',

  // Metric description
  help: 'Histogram of request durations',

  // Extra dimensions, or labels
  // HTTP method (GET, POST, etc), and status code (200, 500, etc)
  labelNames: ['method', 'status'],

  // Duration buckets, in seconds
  // 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
});

Here we are using a exponential bucket configuration in which the buckets double in size at every step. This is a widely used pattern; since we always expect our services to respond quickly (i.e. with response time between 0 and 300ms), we specify more buckets for that range, and fewer buckets for request durations we think are less likely to occur.

According to the previous plot, all slow requests from our application are falling into the 1s-2.5s bucket, resulting in this loss of precision when calculating the 99th percentile.

Since we know our application will take at most ~1s to respond, we can choose a more appropriate bucket layout:

1
2
3
4
5
6
7

// Histogram metric for measuring request durations
const requestDurationHistogram = new prometheusClient.Histogram({
  // ...

  // Experimenting a different bucket layout
  buckets: [0.005, 0.01, 0.02, 0.05, 0.1, 0.25, 0.5, 0.8, 1, 1.2, 1.5]
});

Let’s start a clean Prometheus server with the modified bucket configuration to see if the quantile estimation improves:

1
2

$ docker-compose rm -fs
$ docker-compose up -d

If you re-run the load test, now you should get something like this:

Not quite there, but it’s an improvement!

Want to know more? If all it takes for us to achieve high accuracy histogram data is to use more buckets, why not use a large number of small buckets?

The reason is efficiency. Remember:

more buckets == more time series == more space == slower queries

Let’s say you have an SLO (more details on SLOs later) to serve 99% of requests within 300ms. If all you want to know is whether you are honoring your SLO or not, it doesn’t really matter if the quantile estimation is not accurate for requests slower than 300ms.

You might also be wondering: if summaries are more precise, why not use summaries instead of histograms?

Quoting the documentation:

A summary would have had no problem calculating the correct percentile value in most cases. Unfortunately, we cannot use a summary if we need to aggregate the observations from a number of instances.

Histograms are more versatile in this regard. If you have an application with multiple replicas, you can safely use the histogram_quantile() function to calculate the 99th percentile across all requests to all replicas. You cannot do this with summaries. I mean, you can avg() the 99th percentiles of all replicas, or take the max(), but the value you get will be statistically incorrect and could not be used as a proxy to the 99th percentile.

Measuring Throughput

If you are using a histogram to measure request duration, you can use the <basename>_count timeseries to measure throughput without having to introduce another metric.

For instance, if your histogram metric name is sample_app_histogram_request_duration_seconds, then you can use the sample_app_histogram_request_duration_seconds_count metric to measure throughput:

1
2

# Number of requests per second (data from the past 30s)
rate(sample_app_histogram_request_duration_seconds_count[30s])

Measuring Memory/CPU Usage

Most Prometheus clients already provide a default set of metrics; prom-client, the Prometheus client for Node.js, does this as well.

Try these queries in the Prometheus UI:

1
2
3
4
5

# Gauge that provides the current memory usage, in bytes
process_resident_memory_bytes

# Gauge that provides the usage in CPU seconds per second
rate(process_cpu_seconds_total[30s])

If you use wrk to put some load into our sample application you might see something like this:

You can compare these metrics with the data given by docker stats to see if they agree with each other.

Want to know more? Our sample application exports different metrics to expose some internal Node.js information, such as GC runs, heap usage by type, event loop lag, and current active handles/requests. Plot those metrics in the Prometheus UI, and see how they behave when you put some load to the application.

A sample dashboard containing all those metrics is also available in our Grafana server at http://localhost:3000.

Measuring SLOs and Error Budgets

Managing service reliability is largely about managing risk, and managing risk can be costly.

100% is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice.

SLOs, or Service Level Objectives, is one of the main tools employed by Site Reliability Engineers (SREs) for making data-driven decisions about reliability.

SLOs are based on SLIs, or Service Level Indicators, which are the key metrics that define how well (or how poorly) a given service is operating. Common SLIs would be the number of failed requests, the number of requests slower than some threshold, etc. Although different types of SLOs can be useful for different types of systems, most HTTP-based services will have SLOs that can be classified into two categories: availability and latency.

For instance, let’s say these are the SLOs for our sample application:

The difference between 100% and the SLO is what we call the Error Budget. In this example, the error budget for both SLOs is 5%; if the application receives 1,000 requests during the SLO window (let’s say one minute for the purposes of this tutorial), it means that 50 requests can fail and we’ll still meet our SLO.

But do we need additional metrics for keeping track of these SLOs? Probably not. If you are tracking request durations with a histogram (as we are here), chances are you don’t need to do anything else. You already got all the metrics you need!

Let’s send a few requests to the server so we can play around with the metrics:

1

$ while true; do curl -s http://localhost:4000 > /dev/null ; done

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Number of requests served in the SLO window
sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job)

# Number of requests that violated the latency SLO (all requests that took more than 100ms to be served)
sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job)

# Number of requests in the error budget: (100% - [slo threshold]) * [number of requests served]
(1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job)

# Remaining requests in the error budget: [number of requests in the error budget] - [number of requests that violated the latency SLO]
(1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - (sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job))

# Remaining requests in the error budget as a ratio: ([number of requests in the error budget] - [number of requests that violated the SLO]) / [number of requests in the error budget]
((1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - (sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job))) / ((1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job))

Due to the simulated scenario in which ~5% of requests takes 1s to complete, if you try the last query you should see that the average budget available is around 0%, that is, we have no more budget to spend and will inevitably break the latency SLO if more requests start to take more time to be served. This is not a good place to be.

But what if we had a more strict SLO, say, 99% instead of 95%? What would be the impact of these slow requests on the error budget?

Just replace all 0.95 by 0.99 in that query to see what would happen:

In the previous scenario with the 95% SLO, the SLO burn rate was ~1x, which means the whole error budget was being consumed during the SLO window, that is, in 60 seconds. Now, with the 99% SLO, the burn rate was ~3x, which means that instead of taking one minute for the error budget to exhaust, it now takes only ~20 seconds!

Now change the curl to point to the /metrics endpoint, which do not have the simulated long latency for 5% of the requests, and you should see the error budget go back to 100% again:

1

$ while true; do curl -s http://localhost:4000/metrics > /dev/null ; done

Want to know more? These queries are for calculating the error budget for the latency SLO by measuring the number of requests slower than 100ms. Now try to modify those queries to calculate the error budget for the availability SLO (requests with status=~"5.."), and modify the sample application to return a HTTP 5xx error for some requests so you can validate the queries.

The Site Reliability Workbook is a great resource on this topic and includes more advanced concepts such as how to alert based on SLO burn rate as a way to improve alert precision/recall and detection/reset times.

Monitoring Applications Without a Metrics Endpoint

We learned that Prometheus needs all applications to expose a /metrics HTTP endpoint for it to scrape metrics. But what if you want to monitor a MySQL instance, which does not provide a Prometheus metrics endpoint? What can we do?

That’s where exporters come in. The documentation lists a comprehensive list of official and third-party exporters for a variety of systems, such as databases, messaging systems, cloud providers, and so forth.

For a very simplistic example, check out the aws-limits-exporter project, which is about 200 lines of Go code.

Final Gotchas

The Prometheus documentation page on instrumentation does a pretty good job in laying out some of the things to watch out for when instrumenting your applications.

Also, beware that there are conventions on what makes a good metric name; poorly (or wrongly) named metrics will cause you a hard time when creating queries later.

References

• Prometheus documentation
• Prometheus example queries
• Prometheus client for Node.js
• Keynote: Monitoring, the Prometheus Way (DockerCon 2017)
• Blog Post: Understanding Machine CPU usage
• Blog Post: #LatencyTipOfTheDay: You can’t average percentiles. Period.
• Blog Post: Why Averages Suck and Percentiles are Great
• Site Reliability Engineering books

As of March 2026, ingress-nginx will no longer receive new releases, bugfixes, or updates to resolve any security vulnerabilities that may be discovered.

So you have a Kubernetes cluster and are using (or considering using) the NGINX ingress controller to forward outside traffic to in-cluster services. That’s awesome!

The first time I looked at it, everything looked so easy; installing the NGINX ingress controller was one helm install away, so I did it. Then, after hooking up the DNS to the load balancer and creating a few Ingress resources, I was in business.

Fast-forward a few months, all external traffic for all environments (dev, staging, production) was going through the ingress servers. Everything was good. Until it wasn’t.

We all know how it happens. First, you get excited about that shiny new thing. You start using it. Then, eventually, some shit happens.

My First Ingress Outage

Let me start by saying that if you are not alerting on accept queue overflows, well, you should.

What happened was that one of the applications being proxied through NGINX started taking too long to respond, causing connections to completely fill the NGINX listen backlog, which caused NGINX to quickly start dropping connections, including the ones being made by Kubernetes' liveness/readiness probes.

What happens when some pod fails to respond to the liveness probes? Kubernetes thinks there’s something wrong with the pod and restarts it. The problem is that this is one of those situations where restarting a pod will actually make more harm than good; the accept queue will overflow, again and again, causing Kubernetes to keep restarting the NGINX pods until they all started to crash-loop.

What are the lessons learned from this incident?

• Know every bit of your NGINX configuration. Look for anything that should (or should not) be there, and don’t blindingly trust any default values.
• Most Linux distributions do not provide an optimal configuration for running high load web servers out-of-the-box; double-check the values for each kernel param via sysctl -a.
• Make sure to measure the latency across your services and set the various timeouts based on the expected upper bound + some slack to accommodate slight variations.
• Change your applications to drop requests or degrade gracefully when overloaded. For instance, in NodeJS applications, latency increases in the event loop might indicate the server is in trouble keeping up with the current traffic.
• Do not use just one NGINX ingress controller deployment for balancing across all types of workloads/environments.

The Importance of Observability

Before detailing each of the previous points, my 0th advice is to never run a production Kubernetes cluster (or anything else for that matter) without proper monitoring; by itself, monitoring won’t prevent bad things from happening, but collecting telemetry data during such incidents will give you means to root-cause and fix most issues you’ll find along the way.

If you choose to jump on the Prometheus bandwagon, you can leverage node_exporter in order to collect node-level metrics that could help you detect situations like the one I’ve just described.

Also, the NGINX ingress controller itself exposes Prometheus metrics; make sure to collect those as well.

Know Your Config

The beauty of ingress controllers is that you delegate the task of generating and reloading the proxy configuration to this fine piece of software and never worry about it; you don’t even have to be familiar with the underlying technology (NGINX in this case). Right? Wrong!

If you haven’t done that already, I urge you to take a look at the configuration your ingress controller generated for you. For the NGINX ingress controller, all you need to do is grab the contents of /etc/nginx/nginx.conf via kubectl.

1
2

kubectl -n <namespace> exec <nginx-ingress-controller-pod-name> -- /
   cat /etc/nginx/nginx.conf > ./nginx.conf

Now look for anything that’s not compatible with your setup. Want an example? Let’s start with worker_processes auto;

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# $ cat ./nginx.conf
daemon off;

worker_processes auto;
pid /run/nginx.pid;

worker_rlimit_nofile 1047552;
worker_shutdown_timeout 10s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    real_ip_header      X-Forwarded-For;
    # ...
}

# ...

The optimal value depends on many factors including (but not limited to) the number of CPU cores, the number of hard disk drives that store data, and load pattern. When one is in doubt, setting it to the number of available CPU cores would be a good start (the value “auto” will try to autodetect it).

Here’s the first gotcha: as of now (will it ever be?), NGINX is not Cgroups-aware, which means the auto value will use the number of physical CPU cores on the host machine, not the number of “virtual” CPUs as defined by the Kubernetes resource requests/limits.

Let’s run a little experiment. What happens when you try to load the following NGINX configuration file from a container limited to only one CPU in a dual-core server? Will it spawn one or two worker processes?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# $ cat ./minimal-nginx.conf

worker_processes auto;

events {
  worker_connections 1024;
}

http {
  server {
    listen 80;
    server_name localhost;

    location / {
      root  html;
      index index.html index.htm;
    }
  }
}

Thus, if you intend to restrict the NGINX ingress CPU share, it might not make sense to spawn a large number of workers per container. If that’s the case, make sure to explicitly set the desired number in the worker_processes directive.

1
2
3
4
5
6
7

$ docker run --rm --cpus="1" -v `pwd`/minimal-nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
fc7d98c412a9b90a217388a094de4c4810241be62c4f7501e59cc1c968434d4c

$ docker exec fc7 ps -ef | grep nginx
root         1     0  0 21:49 pts/0    00:00:00 nginx: master process nginx -g daemon off;
nginx        6     1  0 21:49 pts/0    00:00:00 nginx: worker process
nginx        7     1  0 21:49 pts/0    00:00:00 nginx: worker process

Now take the listen directive; it does not specify the backlog parameter (which is 511 by default on Linux). If your kernel’s net.core.somaxconn is set to, say, 1024, you should also specify the backlog=X parameter accordingly. In other words, make sure your config is in tune with your kernel.

And please, don’t stop there. Do this thought exercise to every line of the generated config. Hell, take at look at all the things the ingress controller will let you change, and don’t hesitate to change anything that does not fit your use case. Most NGINX directives can be customized via ConfigMap entries and/or annotations.

Kernel Params

Using ingress or not, make sure to always review and tune the kernel params of your nodes according to the expected workloads.

This is a rather complex subject on its own, so I have no intention of covering everything in this post; take a look at the References section for more pointers in this area.

Kube-Proxy: Conntrack Table

If you are using Kubernetes, I don’t need to explain to you what Services are and what they are used for. However, I think it’s important to understand in more detail how they work.

Every node in a Kubernetes cluster runs a kube-proxy, which is responsible for implementing a form of virtual IP for Services of type other than ExternalName. In Kubernetes v1.0 the proxy was purely in userspace. In Kubernetes v1.1 an iptables proxy was added, but was not the default operating mode. Since Kubernetes v1.2, the iptables proxy is the default.

In other words, all packets sent to a Service IP are forwarded/load-balanced to the corresponding Endpoints (address:port tuples for all pods that match the Service label selector) via iptables rules managed by kube-proxy; connections to Service IPs are tracked by the kernel via the nf_conntrack module, and, as you might have guessed, this connection tracking information is stored in RAM.

As the values of different conntrack params need to be set in conformance with each other (ie. nf_conntrack_max and nf_conntrack_buckets), kube-proxy configures sane defaults for those as part of its bootstrapping procedure.

$ kubectl -n kube-system logs <some-kube-proxy-pod>
I0829 22:23:43.455969       1 server.go:478] Using iptables Proxier.
I0829 22:23:43.473356       1 server.go:513] Tearing down userspace rules.
I0829 22:23:43.498529       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
I0829 22:23:43.498696       1 conntrack.go:52] Setting nf_conntrack_max to 524288
I0829 22:23:43.499167       1 conntrack.go:83] Setting conntrack hashsize to 131072
I0829 22:23:43.503607       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0829 22:23:43.503718       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0829 22:23:43.504052       1 config.go:102] Starting endpoints config controller
...

These are good defaults, but you might want to increase those if your monitoring data shows you’re running out of conntrack space. However, bear in mind that increasing these params will result in increased memory usage, so be gentle.

Sharing Is (Not) Caring

We used to have just a single NGINX ingress deployment responsible for proxying requests to all applications in all environments (dev, staging, production) until recently. I can say from experience this is bad practice; don’t put all your eggs in one basket.

I guess the same could be said about sharing one cluster for all environments, but we found that, by doing this, we get better resource utilization by allowing dev/staging pods to run on a best-effort QoS tier, taking up resources not used by production applications.

The trade-off is that this limits the things we can do to our cluster. For instance, if we decide to run a load test on a staging service, we need to be really careful or we risk affecting production services running in the same cluster.

Even though the level of isolation provided by containers is generally good, they still rely on shared kernel resources that are subject to abuse.

Split Ingress Deployments Per Environment

That being said, there’s no reason not to use dedicated ingresses per environment. This will give you an extra layer of protection in case your dev/staging services get misused.

Some other benefits of doing so:

• You get the chance to use different settings for each environment if needed
• Allow testing ingress upgrades in a more forgiving environment before rolling out to production
• Avoid bloating the NGINX configuration with lots of upstreams and servers associated with ephemeral and/or unstable environments
• As a consequence, your configuration reloads will be faster, and you’ll have fewer configuration reload events during the day (we’ll discuss later why you should strive to keep the number of reloads to a minimum)

Ingress Classes To The Rescue

One way to make different ingress controllers manage different Ingress resources in the same cluster is by using a different ingress class name per ingress deployment, and then annotate your Ingress resources to specify which one is responsible for controlling it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Ingress controller 1
apiVersion: extensions/v1beta1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - args:
            - /nginx-ingress-controller
            - --ingress-class=class-1
            - ...

# Ingress controller 2
apiVersion: extensions/v1beta1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - args:
            - /nginx-ingress-controller
            - --ingress-class=class-2
            - ...

# This Ingress resource will be managed by controller 1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: class-1
spec:
  rules: ...

# This Ingress resource will be managed by controller 2
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: class-2
spec:
  rules: ...

Ingress Reloads Gone Wrong

At this point, we were already running a dedicated ingress controller for the production environment. Everything was running pretty smoothly until we decided to migrate a WebSocket application to Kubernetes + ingress.

Shortly after the migration, I started noticing a strange trend in memory usage for the production ingress pods.

Why was the memory consumption skyrocketing like this? After I kubectl exec’d into one of the ingress containers, what I found was a bunch of worker processes stuck in shutting down state for several minutes.

root     17755 17739  0 19:47 ?        00:00:00 /usr/bin/dumb-init /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17765 17755  0 19:47 ?        00:00:08 /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17776 17765  0 19:47 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody   18866 17776  0 19:49 ?        00:00:05 nginx: worker process is shutting down
nobody   19466 17776  0 19:51 ?        00:00:01 nginx: worker process is shutting down
nobody   19698 17776  0 19:51 ?        00:00:05 nginx: worker process is shutting down
nobody   20331 17776  0 19:53 ?        00:00:05 nginx: worker process is shutting down
nobody   20947 17776  0 19:54 ?        00:00:03 nginx: worker process is shutting down
nobody   21390 17776  1 19:55 ?        00:00:05 nginx: worker process is shutting down
nobody   22139 17776  0 19:57 ?        00:00:00 nginx: worker process is shutting down
nobody   22251 17776  0 19:57 ?        00:00:01 nginx: worker process is shutting down
nobody   22510 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   22759 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   23038 17776  1 19:59 ?        00:00:03 nginx: worker process is shutting down
nobody   23476 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   23738 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   24026 17776  2 20:01 ?        00:00:02 nginx: worker process is shutting down
nobody   24408 17776  4 20:01 ?        00:00:01 nginx: worker process

In order to understand why this happened, we must take a step back and look at how configuration reloads is implemented in NGINX.

Once the master process receives the signal to reload configuration, it checks the syntax validity of the new configuration file and tries to apply the configuration provided in it. If this is a success, the master process starts new worker processes and sends messages to old worker processes, requesting them to shut down. Otherwise, the master process rolls back the changes and continues to work with the old configuration. Old worker processes, receiving a command to shut down, stop accepting new connections and continue to service current requests until all such requests are serviced. After that, the old worker processes exit.

Remember we are proxying WebSocket connections, which are long-running by nature; a WebSocket connection might take hours, or even days to close depending on the application. The NGINX server cannot know if it’s okay to break up a connection during a reload, so it’s up to you to make things easier for it. (One thing you can do is to have a strategy in place to actively close connections that are idle for far too long, both at the client and server-side; don’t leave this as an afterthought)

Now back to our problem. If we have that many workers in that state, this means the ingress configuration got reloaded many times, and workers were unable to terminate due to the long-running connections.

That’s indeed what happened. After some debugging, we found that the NGINX ingress controller was repeatedly generating a different configuration file due to changes in the ordering of upstreams and server IPs.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145

I0810 23:14:47.866939       5 nginx.go:300] NGINX configuration diff
I0810 23:14:47.866963       5 nginx.go:301] --- /tmp/a072836772	2017-08-10 23:14:47.000000000 +0000
+++ /tmp/b304986035	2017-08-10 23:14:47.000000000 +0000
@@ -163,32 +163,26 @@
 
     proxy_ssl_session_reuse on;
 
-    upstream production-app-1-80 {
+    upstream upstream-default-backend {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.71.14:3000 max_fails=0 fail_timeout=0;
-        server 10.2.32.22:3000 max_fails=0 fail_timeout=0;
+        server 10.2.157.13:8080 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-2-80 {
+    upstream production-app-3-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.110.13:3000 max_fails=0 fail_timeout=0;
-        server 10.2.109.195:3000 max_fails=0 fail_timeout=0;
+        server 10.2.82.66:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.124:3000 max_fails=0 fail_timeout=0;
+        server 10.2.59.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.45.219:3000 max_fails=0 fail_timeout=0;
     }
 
     upstream production-app-4-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.109.177:3000 max_fails=0 fail_timeout=0;
         server 10.2.12.161:3000 max_fails=0 fail_timeout=0;
-    }
-
-    upstream production-app-5-80 {
-        # Load balance algorithm; empty for round robin, which is the default
-        least_conn;
-        server 10.2.21.37:9292 max_fails=0 fail_timeout=0;
-        server 10.2.65.105:9292 max_fails=0 fail_timeout=0;
+        server 10.2.109.177:3000 max_fails=0 fail_timeout=0;
     }
 
     upstream production-app-6-80 {
@@ -201,61 +195,67 @@
     upstream production-lap-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.45.223:8000 max_fails=0 fail_timeout=0;
+        server 10.2.21.36:8000 max_fails=0 fail_timeout=0;
         server 10.2.78.36:8000 max_fails=0 fail_timeout=0;
+        server 10.2.45.223:8000 max_fails=0 fail_timeout=0;
         server 10.2.99.151:8000 max_fails=0 fail_timeout=0;
-        server 10.2.21.36:8000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-7-80{
+    upstream production-app-1-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.79.126:3000 max_fails=0 fail_timeout=0;
-        server 10.2.35.105:3000 max_fails=0 fail_timeout=0;
-        server 10.2.114.143:3000 max_fails=0 fail_timeout=0;
-        server 10.2.50.44:3000 max_fails=0 fail_timeout=0;
-        server 10.2.149.135:3000 max_fails=0 fail_timeout=0;
-        server 10.2.45.155:3000 max_fails=0 fail_timeout=0;
+        server 10.2.71.14:3000 max_fails=0 fail_timeout=0;
+        server 10.2.32.22:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-8-80 {
+    upstream production-app-2-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.53.23:5000 max_fails=0 fail_timeout=0;
-        server 10.2.110.22:5000 max_fails=0 fail_timeout=0;
-        server 10.2.35.91:5000 max_fails=0 fail_timeout=0;
-        server 10.2.45.221:5000 max_fails=0 fail_timeout=0;
+        server 10.2.110.13:3000 max_fails=0 fail_timeout=0;
+        server 10.2.109.195:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream upstream-default-backend {
+    upstream production-app-9-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.157.13:8080 max_fails=0 fail_timeout=0;
+        server 10.2.78.26:3000 max_fails=0 fail_timeout=0;
+        server 10.2.59.22:3000 max_fails=0 fail_timeout=0;
+        server 10.2.96.249:3000 max_fails=0 fail_timeout=0;
+        server 10.2.32.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.114.177:3000 max_fails=0 fail_timeout=0;
+        server 10.2.83.20:3000 max_fails=0 fail_timeout=0;
+        server 10.2.118.111:3000 max_fails=0 fail_timeout=0;
+        server 10.2.26.23:3000 max_fails=0 fail_timeout=0;
+        server 10.2.35.150:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.125:3000 max_fails=0 fail_timeout=0;
+        server 10.2.157.165:3000 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-3-80 {
+    upstream production-app-5-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.79.124:3000 max_fails=0 fail_timeout=0;
-        server 10.2.82.66:3000 max_fails=0 fail_timeout=0;
-        server 10.2.45.219:3000 max_fails=0 fail_timeout=0;
-        server 10.2.59.21:3000 max_fails=0 fail_timeout=0;
+        server 10.2.21.37:9292 max_fails=0 fail_timeout=0;
+        server 10.2.65.105:9292 max_fails=0 fail_timeout=0;
     }
 
-    upstream production-app-9-80 {
+    upstream production-app-7-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server 10.2.96.249:3000 max_fails=0 fail_timeout=0;
-        server 10.2.157.165:3000 max_fails=0 fail_timeout=0;
-        server 10.2.114.177:3000 max_fails=0 fail_timeout=0;
-        server 10.2.118.111:3000 max_fails=0 fail_timeout=0;
-        server 10.2.79.125:3000 max_fails=0 fail_timeout=0;
-        server 10.2.78.26:3000 max_fails=0 fail_timeout=0;
-        server 10.2.59.22:3000 max_fails=0 fail_timeout=0;
-        server 10.2.35.150:3000 max_fails=0 fail_timeout=0;
-        server 10.2.32.21:3000 max_fails=0 fail_timeout=0;
-        server 10.2.83.20:3000 max_fails=0 fail_timeout=0;
-        server 10.2.26.23:3000 max_fails=0 fail_timeout=0;
+        server 10.2.114.143:3000 max_fails=0 fail_timeout=0;
+        server 10.2.79.126:3000 max_fails=0 fail_timeout=0;
+        server 10.2.45.155:3000 max_fails=0 fail_timeout=0;
+        server 10.2.35.105:3000 max_fails=0 fail_timeout=0;
+        server 10.2.50.44:3000 max_fails=0 fail_timeout=0;
+        server 10.2.149.135:3000 max_fails=0 fail_timeout=0;
+    }
+
+    upstream production-app-8-80 {
+        # Load balance algorithm; empty for round robin, which is the default
+        least_conn;
+        server 10.2.53.23:5000 max_fails=0 fail_timeout=0;
+        server 10.2.45.221:5000 max_fails=0 fail_timeout=0;
+        server 10.2.35.91:5000 max_fails=0 fail_timeout=0;
+        server 10.2.110.22:5000 max_fails=0 fail_timeout=0;
     }
 
     server {

This caused the NGINX ingress controller to reload its configuration several times per minute, making these shutting down workers pile up until the pod got OOMKilled.

Things got a lot better once I upgraded the NGINX ingress controller to a fixed version and specified the --sort-backends=true command line flag.

Thanks to @aledbf for his assistance in finding and fixing this bug!

Further Minizing Config Reloads

The lesson here is to keep in mind that configuration reloads are expensive operations and it’s a good idea to avoid those especially when proxying WebSocket connections. This is why we decided to create a specific ingress controller deployment just for proxying these long-running connections.

In our case, changes to WebSocket applications happen much less frequently than other applications; by using a separate ingress controller, we avoid reloading the configuration for the WebSocket ingress whenever there are changes (or scaling events/restarts) to other applications.

Separating the deployment also gave us the ability to use a different ingress configuration that’s more suited to long-running connections.

Fine-Tune Pod Autoscalers

Since NGINX ingress uses pod IPs as upstream servers, every time the list of endpoints for a given Service changes, the ingress configuration must be regenerated and reloaded. Thus, if you are observing frequent autoscaling events for your applications during normal load, it might be a sign that your HorizontalPodAutoscalers need adjustment.

Another thing that most people don’t realize is that the horizontal pod autoscaler have a back-off timer that prevents the same target to be scaled several times in a short period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

Name:                                                   <app>
Namespace:                                              production
Labels:                                                 <none>
Annotations:                                            <none>
CreationTimestamp:                                      Fri, 23 Jun 2017 11:41:59 -0300
Reference:                                              Deployment/<app>
Metrics:                                                ( current / target )
  resource cpu on pods  (as a percentage of request):   46% (369m) / 60%
Min replicas:                                           8
Max replicas:                                           20
Conditions:
  Type                  Status  Reason                  Message
  ----                  ------  ------                  -------
  AbleToScale           False   BackoffBoth             the time since the previous scale is still within both the downscale and upscale forbidden windows
  ScalingActive         True    ValidMetricFound        the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited        True    TooFewReplicas          the desired replica count was less than the minimum replica count
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  14d           10m             39      horizontal-pod-autoscaler                       Normal          SuccessfulRescale       New size: 10; reason: cpu resource utilization (percentage of request) above target
  14d           3m              69      horizontal-pod-autoscaler                       Normal          SuccessfulRescale       New size: 8; reason: All metrics below target

According to the default value for the --horizontal-pod-autoscaler-upscale-delay flag in kube-controller-manager, if your application scaled up, it won’t be able to scale up again for 3 minutes.

Thus, in case your application really experiences an increased load, it might take ~4 minutes (3m from the autoscaler back-off + ~1m from the metrics sync) for the autoscaler to react to the increased load, which might be just enough time for your service to degrade.

References

• Tuning NGINX for Performance
• How TCP backlog works in Linux
• How TCP Sockets Work
• Netfilter Conntrack Memory Usage
• Optimizing web servers for high throuhgput and low latency
• Container isolation gone wrong

It was a good decision at the time. In general, Elastic Beanstalk works fine and has a very gentle learning curve; it didn’t take long for all teams to start using it for their projects.

Fast-forward a few months, everything was nice and good. Our old problems were solved, but - as you might have guessed - we had new ones to worry about.

Cost Issues

In Elastic Beanstalk, each EC2 instance runs exactly one application container.¹ This means that, if you follow reliability best practices, you’ll have two or more instances (spread across multiple availability zones) for each application. You might need even more instances if you have other environments besides the production one, i.e. staging.

Anyway, you’ll end up having multiple dedicated instances per service which, depending on your workload, will sit there doing nothing most of the time.

We needed to find a way to use our available compute resources more wisely.

The Winner

After looking around for alternatives to ECS, Kubernetes seemed to be the right one for us.

Kubernetes is a container orchestration tool that builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.

Although Kubernetes is a feature-rich project, a few key features caught our attention: namespaces, automated rollouts and rollbacks, service discovery via DNS, automated container scaling based on resource usage, and of course, the promise of a self-healing system.

Kubernetes is somewhat opinionated around how containers are supposed to be organized and networked, but this should not be a problem if your service follows the Twelve-Factor practices.

Our Path to Production

In order to ensure Kubernetes was a viable option for us, the first thing we did was perform some reliability tests to make sure it could handle failure modes such as dying nodes, killed Kubelet/Proxy/Docker daemons, and availability zone outages.

It’s impossible to anticipate all the ways things can go wrong, but in the end, we were very impressed by how Kubernetes managed to handle these failures.

At that time, we used kube-up to bootstrap our test clusters. This tool, although it served its purpose, not always worked as expected; it suffered from a number of issues, such as poorly chosen defaults, random timeouts that left the stack only half-created, and inconsistent behavior when destroying the cluster causing orphan resources to be left behind.²

Once we agreed that Kubernetes was the way to go, we needed a more reliable way to create and destroy our Kubernetes clusters.

Enter kube-aws

kube-aws is a tool created by some good guys from CoreOS. The cool thing about it is that it uses CloudFormation under the hoods, which gives us some neat advantages.

The first obvious advantage is that it’s very easy to create and destroy clusters without leaving anything silently hanging around.

Another feature is that, unlike kube-up, you can create a cluster in an existing VPC so all services running in Kubernetes have access to your existing AWS resources - such as relational databases - right off the bat.

In fact, you can run multiple clusters at the same time in the same VPC. This has a nice side-effect in which you can treat each cluster as an immutable piece of infrastructure; instead of modifying a running cluster - and risking break something - you simply create a new cluster and gradually shift traffic from the old one to the new in a way that any incidents has limited impact.

The final and probably the most useful feature is that you can easily customize nearly every aspect of the cluster provisioning configuration to make it fit your own needs. In our case, we added cluster level logging that ingests application logs to Sumologic, cluster monitoring with InfluxDB and Grafana, ABAC-based authorization, among other things.

The First Environment

After solving the problem of reliably creating and destroying clusters, we felt confident to start migrating our staging environment over to Kubernetes.

It was easy enough to manually create the yaml manifests for the first deployments, but we needed an automated way to deploy new application images as soon as they were built by our continuous integration system.

Just as a proof of concept, we quickly hacked together a small function in AWS Lambda (based on this article) that automatically updated the corresponding deployment object whenever it received a merge notification in which the tests passed.

This small Lambda function has now evolved into a major component in our delivery pipeline, orchestrating deployments to other environments as well, including production.

With this done, migrating staging services from Beanstalk to Kubernetes was pretty straightforward. First, we created one DNS record for each service (each initially pointing to the legacy deployment in Elastic Beanstalk) and made sure that all services referenced each other via this DNS. Then, it was just a matter of changing those DNS records to point the corresponding Kubernetes-managed load balancers.

To ensure every part of the pipeline was working as expected, we spent some time monitoring all staging deployments looking for bugs and polishing things up as we could.

More Tests, More Learning

Before deploying our first production service to Kubernetes, we did some load testing to find out the optimal configuration for the resource requirements needed by each service and out how many pods we needed to handle the current traffic.

Observing how your services behave under load and how much compute they need is essential.

Also take some time to understand how QoS classes work in Kubernetes so you have a more fine control over which pods gets killed in the case of memory pressure. This is particularly important if you, like us, share the same cluster for all environments.

Tip: Enable Cross-Zone Load Balancing (AWS)

This is already fixed in Kubernetes 1.4, but for now, if you expose your services via the LoadBalancer type, don’t forget to manually enable cross-zone load balancing for the corresponding ELB; if you don’t, you might notice uneven balancing across your application pods if they are spread in nodes from different availability zones.

Tip: Give the kube-system Namespace Some Love

If you ever tried Kubernetes, you probably noticed there’s a kube-system namespace there with a bunch of stuff in it; do yourself a favor and take some time to understand the role of each of those things.

For instance, take the DNS add-on; it’s rather common to see people having issues because they forgot to add more DNS pods to handle their ever-increasing workload.

Going Live

Instead of shifting all traffic at once, like we did in staging, we thought we needed to take a more careful approach and used weighted routing policy to gradually shift traffic to the Kubernetes cluster.

Once we noticed no more requests were reaching the legacy Beanstalk environments, we went ahead and killed them.

Update (Sep 21, 2016): All major services were migrated to our new platform! These are the final numbers:³

• ~53-63% decrease in monthly costs
• ~72-82% decrease in # of instances

Beyond Production

Kubernetes gave us the power to almost effortlessly mold our delivery pipeline in a way we never thought possible. One example of such improvement is what we call here development environments.

Whenever someone opens a Pull Request to one of our projects, the AWS Lambda function I mentioned earlier creates a temporary environment running the modifications introduced by the PR.

Also, whenever new code is pushed, this environment gets automatically updated as long as they pass the tests. Finally, when the PR is merged (or closed), the environment is deleted.

This feature made our code reviews more thorough because the developers can actually see the changes running. This is even more useful for UX changes in front-end services; artists and product owners get the chance to validate the changes and share their inputs before the PR is merged.

To send the GitHub Status notifications you see in these pictures, we implemented a small daemon in Go that monitors deployments to our development namespace and reconciles the deployment status for each revision.

Conclusion

Kubernetes is a very complex piece of software that aims to solve a very complex problem, so expect to spend some time learning how its many pieces fit together before using it in your projects.

Kubernetes is production-ready, but avoid the temptation of trying to run everything on it. In our experience, Kubernetes does not offer a clean solution for a number of problems you might face, such as stateful applications.

The documentation is not great as well, but initiatives like the Kubernetes Bootcamp and Kelsey Hightower’s Kubernetes The Hard Way gives me hope that this will no longer be a problem in the near future.

Without Kubernetes, I don’t know how - or if - we could have accomplished all the things we did in such a small period of time with such a small engineering team.⁴

We hope to continue building on Kubernetes to make our delivery platform even more dynamic and awesome!

According to the official website, Docker is…

…a platform for developers and sysadmins to develop, ship, and run applications. Docker lets you quickly assemble applications from components and eliminates the friction that can come when shipping code. Docker lets you get your code tested and deployed into production as fast as possible.

I’m not here to sell you anything; apparently there are too many people doing that already. Instead, I’m going to document my experiences trying to “Dockerize” a simple Rails application and show you some things I learned along the way.

The Application

I few months ago I built TeXBin, a simple Rails application where you can post a .tex file and get a URL for its PDF version. The code was sitting in my laptop without being used, so why not use it as guinea pig in my first attempt to use Docker? :-)

The proposed stack is composed by three components: the application itself, a MongoDB instance, and a Nginx server to both serve the static content and act as a reverse proxy to the application.

graph LR
    Users([Users])  <-->|Listen 0.0.0.0:80| Nginx

    subgraph HostMachine [Host machine]
        Nginx[Nginx]
        App[App]
        MongoDB[(MongoDB)]

        %% Relationships
        Nginx -- proxypass --> App
        App --> MongoDB
    end

WTF is a container?

Docker is built on top of Linux kernel facilities, like cgroups and namespaces, and provides a way to create lightweight workspaces – or containers – that run processes in isolation.

By using containers, resources can be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple containers can share the same kernel, but each container can be constrained to only use a defined amount of resources such as CPU, memory and I/O.

– Wikipedia

So, in short, you get nearly all the benefits of virtualization with barely none of the execution overhead that comes with it.

Why not put everything within the same container?

You get several benefits by exposing the different components of your application as different containers. Just to be clear, by component I mean some service that binds to a TCP port.

In particular, having different containers for different components gives us freedom to move the pieces around or add new pieces as we see fit, like:

• impose different usage limits (CPU shares and memory limits) for the database, the application, and the webserver
• change from a simple MongoDB instance to a replica set composed by several containers across multiple hosts
• spin up two or more application containers so you can perform blue-green deployments, improve concurrency and resource usage, etc

In other words: it’s a good idea to keep the moving parts, well, moving.

The Dockerfile

Containers are created from images, so first we need to create an image with the application code and all the required software packages.

Instead of doing things manually, Docker can build images automatically by reading the instructions from a Dockerfile, which is a text file that contains all the commands you would normally execute manually in order to build a Docker image.

This is the application’s Dockerfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Base image (https://registry.hub.docker.com/_/ubuntu/)
FROM ubuntu

# Install required packages
RUN apt-get update
RUN apt-get install -y ruby2.0 ruby2.0-dev bundler texlive-full

# Create directory from where the code will run
RUN mkdir -p /texbin/app
WORKDIR /texbin/app

# Make unicorn reachable to other containers
EXPOSE 3000

# Container should behave like a standalone executable
CMD ["start"]
ENTRYPOINT ["foreman"]

# Install the necessary gems
ADD Gemfile /texbin/app/Gemfile
ADD Gemfile.lock /texbin/app/Gemfile.lock
RUN bundle install

# Copy application code to container
ADD . /texbin/app/

# Try not to add steps after the last ADD so we can use the
# Docker build cache more efficiently

Information about the individual commands can be obtained here.

Tip: Kiss RVM Goodbye

Why do you need RVM if the application will live inside a controlled and isolated environment?

The only reason you might want to do that is because you need to install a particular version of Ruby that you can’t find via traditional OS package managers. If that’s the case, you’ll be better off installing the Ruby version you want from the source code.

Using RVM from within a Docker container is not a pleasant experience; every command must run inside a login shell session and you’ll have problems using CMD together with ENTRYPOINT.

Tip: Optimize for the Build Cache

Docker stores intermediate images after successfully executing each command in the Dockerfile. This is a great feature; if any step fails along the way, you can fix the problem and the next build will reuse the cache built up until that last successful command.

Instructions like ADD are not cache friendly though. That’s why it’s a good practice to only ADD stuff as late as possible in the Dockerfile since any changes in the files – or their metadata – will invalidate the build cache for all subsequent instructions.

Which leads us to…

Tip: Do Not Forget the .dockerignore

A really important step is to avoid ADDing irrelevant files to the container, like README, fig.yml, .git/, logs/, tmp/, and others.

If you are familiar with .gitignore, the idea is the same: just create a .dockerignore file and put there the patterns you want to ignore. This wil help keep the image small and the build fast by decreasing the chance of cache busting.

Testing the Images

To run the application, first we’ll need a container that exposes a single MongoDB server:

1

$ docker run --name texbin_mongodb_1 -d mongo

Then you have to build the application image and start a new container:

1
2

$ docker build -t texbin:dev .
$ docker run --name texbin_app_1 -d --link texbin_mongodb_1:mongodb -p 3000:3000 texbin:dev

Learning how container linking and volumes work is essential if you want to understand how to “plug” containers together.

Note: The project also includes a Dockerfile for the Nginx container which I won’t show here because it doesn’t bring anything new to the table.

Now docker ps should display two running containers. If everything’s working, you should be able to access the application at http://localhost:3000. To see the logs, run docker logs texbin_app_1.

Docker in Development

It turns out it’s quite easy to automate these last steps with Fig:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# fig.yml

mongodb:
  image: mongo

app:
  build: .
  ports:
    - 3000:3000
  links:
    - mongodb:mongodb
  volumes:
    - .:/texbin/app

Then, run fig up in the terminal in order to build the images, start the containers, and link them.

The only difference between this and the commands we ran manually before is that now we’re mounting the hosts’s current directory to container’s /texbin/app so that we can view our changes to the application in real time.

Try changing some .html.erb template and refreshing the browser.

Defining New Environments

The goal is to run the same application in production, but with a different configuration, right? A simple way to – sort of – solve this is by creating another image, based on the previous one, that changes the required configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Uses our previous image as base
FROM texbin

# Set the proper environment
ENV RAILS_ENV production

# Custom settings for that environment
ADD production_env /texbin/app/.env
ADD production_mongoid.yml /texbin/app/config/mongoid.yml

# Precompile the assets
RUN rake assets:precompile

# Exposes the public directory as a volume
VOLUME /texbin/app/public

If you know a better way to do this, please let me know in the comments.

Going Live

The first thing to do is to push your images to the server. There are plenty of ways to do that: the public registry, a private-hosted registry, git, etc. Once the images are built, just repeat the procedure we did earlier and you’re done.

But that’s not everything. As you probably know, deploying an application involves a lot more than just moving stuff to some remote servers. This means you’ll still have to worry with things like deployment automation, monitoring (at host and container levels), logging, data migrations and backup, etc.

Conclusion

I’m glad I took the time to look at Docker. Despite its young age, it’s a very impressive rapidly-evolving piece of technology with a lot of potential to radically change the DevOps landscape in the next couple of years.

However, Docker solves only one variable of a huge equation. You’ll still have to take care of boring things like monitoring, and I imagine it’s rather difficult – not to say impossible – to use Docker in production without some layer of automation on top of it.

Also, features like container linking, are somewhat limited and we’ll probably see substantial improvements in future releases. So stay tuned!

Continuations are the least understood of all control-flow constructs. This lack of understanding (or awareness) is unfortunate, given that continuations permit the programmer to implement powerful language features and algorithms.

– Matt Might, in Continuations By Example

The usual way to control the flow of execution of a computer program is via procedure calls and returns; a stack data structure is how high-level programming languages keep track of the point to which each active subroutine should return control when it finishes executing.

Unfortunately, you’ll need more than that if you intend to write useful programs to solve real-world problems. That’s why most high-level programming languages also provide other control-flow primitives, like the goto statement, loops, and exception handling.

I’m not saying that implementing a programming language is an easy task, but putting that aside for a moment, it’s like programming languages in general fight as hard as they can to make the call stack something as hidden and intangible as possible - something no one but itself are allowed to control.

What would happen if some programming languages, instead of keeping the call stack inside a 2" solid steel safe, actually gave the programmers the ability to “capture” them as functions that can be invoked, stored, and passed around as values?

In this post, I hope to show you what continuations are and how they can be used in practical situations. So grab Racket and let’s go!

Update (Jun 20, 2014): I’ve changed some things in this post in response to some great comments in this Reddit discussion and this blog post by jecxjo.

First Example

Note: The following problem is solvable without continuations, but I’d like to start with something simple enough.

Suppose you are writing code that interfaces with some API over HTTP. Also suppose this API requires a SessionId header to be sent over with the request in order to avoid CSRF attacks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#lang racket/base

;; Object to keep the session-id across requests
(struct session [id #:mutable])

(define (perform-request! session method params)

  ;; Performs the request
  (define headers  (session-id-headers session))
  (define response (http-request API-URL headers method params))

  ;; Retries the request with the given Session-Id
  ;; if necessary
  (when (request-denied? response)
    (update-session-id! session response)
    (perform-request! session method params))

  (parse-json response))

When the first request is sent - without the SessionId header - the server responds with an error, i.e. HTTP 409, in which case the procedure updates session with the session id given by the server and retries the request.

The code makes sense, but it’s broken.

The recursive call is not made in tail position. So, when it happens, another stack frame is pushed to the call stack and even though the retried request succeeds, what gets returned to the caller is the response to that first unauthorized request.

If only we had the chance to return that second response right to the caller instead of having the stack to unwind itself…

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

(define (perform-request! session method params)
  ;; ...

  (when (request-denied? response)
    (update-session-id! session response)

    ;; Something like this
    (return (perform-request! ...)))

  (parse-json response))

Enter call/cc

A continuation can be viewed as the evaluation context surrounding an expression or, in other words, a snapshot of the current control state of the program.

Here’s an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#lang racket/base

;; We'll keep the captured continuation here
(define cc #f)

;; This function returns the value 3 *and* stores the
;; continuation that represents the execution context
;; in which this function was called
(define (val!)
  (call/cc
   (lambda (k)
     (set! cc k)
     3)))

;; Stored continuation for this expression: (+ 1 (* 2 ?))
(+ 1 (* 2 (val!))) ;-> 7

;; Replays the continuation with different arguments
(cc 2) ;->  5, or (+ 1 (* 2 2))
(cc 6) ;-> 13, or (+ 1 (* 2 6))

It turns out that, if we rename k to return, this is exactly the thing we need in order to fix that broken API client example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

(define (perform-request! session method params)
  (let/cc return ; same as (call/cc (lambda (return) body...))
    ;; ...

    ;; Retries the request and gives the control
    ;; back to the caller if request-denied?
    (when (request-denied? response)
      (update-session-id! session response)
      (return (perform-request! session method params)))

    (parse-json response)))

Now the function captures the current continuation at the moment the procedure perform-request! is first called. Then, if the server denies a request, we re-send the request with the given SessionId and use that grabbed continuation to transfer the control back to the caller.

Nice, don’t you think? It’s like we’re freezing time at let/cc, doing some stuff, and then resuming from there.

This is a common use case of continuations. Check out this project if you want to read the code that inspired this example.

Generators

Generators can be viewed as special routines that behave like iterators. If you are familiar with Python, you’ve probably seen code like this one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

def iterate(list):
  "Generator function that iterates through list."
  for item in list:
    yield item

# Usage
it = iterate(range(2))

it.next() # -> 0
it.next() # -> 1
it.next() # -> raises StopIteration error

Do you see any resemblance between this example and the previous one? Although Python doesn’t provide a call/cc-like facility in the language, one can argue that its generators are like a poor man’s continuation.

Let’s pretend for a moment that Racket didn’t have a generator library that does exactly this. How could this be implemented Racket using continuations?

What we need is a function that returns another function which, when called, yields one item at a time, until the list is exhausted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

(define (iterate lst)
  (lambda ()
    (let/cc return
      (for-each
       (lambda (item)
         (return item))
       lst))))

;; Usage
(define next (iterate (range 3)))
(next) ;-> 0
(next) ;-> 0

This code follows the same pattern as the previous ones, but it doesn’t seem to work the way you might expect. The reason should be clear though: iterate returns a lambda that uses the captured continuation to yield the list’s first item.

To make this code work, we need to capture the current continuation from the inside of for-each and store it so it can be used to resume the computation when next is called again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

(define (iterate lst)

  ;; Defines `state` as being a function that starts the
  ;; iteration via `for-each`
  (define (state return)
    (for-each
     (lambda (item)

       ;; Here, we capture the continuation that represents the
       ;; current state of the iteration
       (let/cc item-cc

         ;; Before the item is yielded, we update `state` to
         ;; `item-cc` so the computation is resumed the next
         ;; time the generator is called
         (set! state item-cc)

         ;; Yields the current item to the caller
         (return item)))
     lst)

    ;; Yields 'done when the list is exhausted
    (return 'done))

  ;; Returns a function that calls the stored `state` with the
  ;; current continuation so we can yield one item at a time
  (define (generator)
    (call/cc state))
  generator)

;; Usage
(define next (iterate '(0 1)))

(next) ;-> 0
(next) ;-> 1
(next) ;-> 'done

If you are having trouble understanding how this code works, the following diagram might help.

stateDiagram-v2
  direction LR
  s1: State
  s2: (set! state item-cc)
  s3: (return 'item)
  state if_state <>

  [*] --> s1: (next)
  s1 --> if_state: (for-each)
  
  if_state --> [*]: (return 'done)
  if_state --> s2: Has items left

  s2 --> s3
  s3 --> if_state: Yield to caller

Other Examples

Moving along to more high level stuff, in this example Matthew Flatt explains how to build a continuation-based web server in Racket. Still in the realm of continuation-based-web-something, if you are into Smalltalk, don’t forget to check Seaside, a web application framework that uses continuations to model multiple independent flows between different components.

If you don’t code Scheme or Smalltalk for a living, don’t worry. The chances are your language does support some flavor of continuations, either natively or via some third-party library.

Conclusion

It seems that continuations can be used to implement a wide variety of advanced control constructs including non-local exits, exception handling, backtracking, and coroutines.

In this post, I hope to have clarified some of the key aspects about continuations. If you have any suggestion on how to improve this text, please let me know.

You are Hercules, about to fight the dreaded Hydra. The Hydra has 9 heads. When a head is chopped off, it spawns 8 more heads. When one of these 8 heads is cut off, each one spawns out 7 more heads. Chopping one of these spawns 6 more heads, and so on until the weakest head of the hydra will not spawn out any more heads.

Our job is to figure out how many chops Hercules needs to make in order to kill all heads of the Hydra. And no, it’s not n!.

We can start by defining a function that returns a n-headed Hydra.

1
2
3
4
5
6
7

(defn new-hydra
  "Returns a Hydra with n heads."
  [n]
  (repeat n n))

(new-hydra 3)
;; => (3 3 3)

To make it easy to compare both solutions, the data structure I’m using here is the same one used by Dinkar: a list. In this list, each number represents a living head and its level of strength.

Now, according to the problem description, when Hercules chops off a level 3 head, the Hydra grows two level 2 heads.

1
2

(chop-head (new-hydra 3))
;; => (2 2 3 3)

Here’s one possible implementation for such a function.

1
2
3
4
5
6

(defn chop-head
  "Returns a new Hydra after chop off its first head."
  [hydra]
  (let [head (first hydra)]
    (into (rest hydra)
          (new-hydra (dec head)))))

This code should make sense even if you are not familiar with Clojure.

What happens if Hercules tries to cut off the head of a headless Hydra?

Most functional programming languages I know are laid on top of a strong principle called the closure property.

In general, an operation for combining data objects satisfies the closure property if the results of combining things with that operation can themselves be combined using the same operation. Closure is the key to power in any means of combination because it permits us to create hierarchical structures – structures made up of parts, which themselves are made up of parts, and so on.

– Gerald Jay Sussman, Hal Abelson

To illustrate this concept with code, let’s consider Clojure’s cons function.

1
2
3
4
5

(cons 1 (cons 2 '()))
;; => (1 2)

(cons 1 (cons 2 (cons 3 nil)))
;; => (1 2 3)

That means cons follows the closure principle. But what about our chop-head function? Does the principle hold?

1
2

(chop-head (chop-head (chop-head '(2))))
;; => NullPointerException

Apparently not. To fix that, we need to make sure dec is not called with nil, since it’s not possible to decrement a null value.

1
2
3
4
5
6

(defn chop-head
  "Returns a new Hydra after chop off its first head."
  [hydra]
  (let [head (first hydra)]
    (into (rest hydra)
          (new-hydra (dec (or head 1))))))

What about now?

1
2

(chop-head (chop-head (chop-head '(2))))
;; => ()

Killing The Hydra

In order for Hecules to kill the Hydra, he needs to repeatedly chop off Hydra’s heads while it still has them.

1
2
3
4
5

(defn chop-until-dead
  "Repeatedly chops Hydra's heads until no head is left."
  [hydra]
  (take-while #(not (empty? %))
              (iterate #(chop-head %) hydra)))

The (iterate f x) function returns a lazy (infinite) sequence of x, (f x), (f (f x)), etc, given that f is a function free of side-effects.

1
2

(take 3 (iterate inc 0))
;; => (0 1 2)

Since chop-head respects the closure principle by always returning a list, we can use it in iterate until we get an empty list, which means the Hydra is DEAD.

Let’s test it on a 3-headed baby Hydra.

1
2
3

(chop-until-dead (new-hydra 3))
;; => ((3 3 3) (2 2 3 3) (1 2 3 3) (2 3 3) (1 3 3) (3 3)
;;     (2 2 3) (1 2 3) (2 3) (1 3) (3) (2 2) (1 2) (2) (1))

How many chops are needed in order to kill the original 9-headed Hydra?

1
2

(count (chop-until-dead (new-hydra 9)))
;; => 986409

Another interesting question: what is the maximum number of heads Hercules fought at once?

1
2
3
4

(apply max
       (map count
            (chop-until-dead (new-hydra 9))))
;; => 37

Ah. Beautiful.

As a contribution to this wonderful initiative, I gather here my impressions regarding some aspects of the course.

Prerequisites

Basic probability, matrices, and calculus. According to the course website, this is all you need to know. I would add programming to that list though, since it’s necessary to write some tricky programs in order to answer to some questions.

Broadcast

The Ustream platform were chosen to broadcast the course and, apart from some small glitches, it worked nicely the times I tried it. This is how I’d watch the lectures if I lived somewhere with a more favorable time zone. (2:30 P.M. to 3:30 P.M. in Brazil)

Material

Lucky for us, the lecture videos were made available in several formats – in low and high quality versions – just a couple of days after its broadcast. The recorded Ustream session was accessible shortly after each lecture. The lectures were also posted on YouTube and iTunes U course app for iOS devices. This was really impressive.

I liked the quality of the slides, which were made available in PDF format. All pictures, plots, and equations are sharp and easy to read. Another plus: the notation chosen to explain each concept and algorithm relate to the notation adopted in popular articles on the same subject, for example, in Wikipedia. This makes things easy for those who seek to enrich their knowledge even further.

I ordered the textbook for the course, but unfortunately it didn’t arrive in time for this post. However, according to the few reviews on Amazon and the feedback given in the course forum, it does its job pretty well.

As the course website states, this really isn’t a watered-down material and I’m glad for that.

The subjects I enjoyed the most: VC Theory (Growth Function, VC Dimension), Regularization, Validation, and Support Vector Machines.

Professors

I must say the course teacher, Professor Yaser Abu-Mostafa, is one of the most talented teachers I’ve ever had, online or otherwise. He truly masters the subject and knows what is relevant and what is noise when explaining something.

Another impressive thing was how he was available in the Q&A forum, together with the Associate Professors, answering student’s questions and collecting feedback in order to keep the course at its highest level. Associate Professor Malik Magdon-Ismail also helped a lot in some discussions by writing some real gems. Some of his posts could be turn into nice articles very easily.

Assignments

I enjoyed every single one of them. They were thorough, challenging, and fun to work on. Some of them made me “spend” the entire weekend in order to answer all the questions! Well, I wasn’t looking for an easy time anyway. :-)

In fact, I would stop using the word homework because what we did at the end of each week was actually an experiment. We played with every aspect involved in the design and implementation of simple machine learning systems, either by implementing everything from scratch or by using third-party packages.

The ambiguity present in several answers (they being intentional or not) made me more skeptical about what was coming out of my programs. This encouraged students to engage in enlightening discussions in the Q&A forum.

To be honest, I don’t like vBulletin, the platform chosen for the Q&A forum. I think even more users would have joined the discussions if a more friendly (and reward oriented) option were chosen instead. (some Stack Overflow clone would have been better)

Still, I’m impressed by the quality of the discussions started there. Thanks to the very gifted and dedicated colleagues who helped me clear my doubts – and even change my mind. I owe you! :-)

My Solutions

I chose Octave to be my programming language throughout the course, and this choice proved to be the right one. Some questions involved heavy matrix-based calculations and quadratic programming problems, and this is where Octave shines.

Unfortunately, I’m not allowed to post my solutions online because of the honor code, but ask me if you want to discuss any question in particular.

Update (Jan 20, 2014): I participated in the reedition of this course on edX so I could earn a certificate as a reward for my hard work. In this edition, the course staff encouraged students to discuss their solutions after each week’s deadline. My solutions are available here.