You can't have your cake and eat it too. Not even if you say please. <img src="/heheheheheh.gif" alt="heheheheheh" style="display:inline;height:1em;vertical-align:middle;" />

Every distributed system is a choice between what you want and what you can actually have. The CAP theorem is the most famous way to say this, and also the most misunderstood.

### The theorem

Eric Brewer proposed it in 2000, and Gilbert & Lynch proved it in 2002.<a href="#ref-1">[1]</a> It says: a distributed system can provide at most two of three guarantees:

- **Consistency**: every read sees the most recent write
- **Availability**: every request gets a response (no timeouts, no errors)
- **Partition tolerance**: the system keeps working when network messages get dropped

Pick two. You can't have all three.

### Why not all three?

Imagine two database nodes, A and B. A client writes to A. Before A can replicate to B, the network between them fails. Now a different client reads from B.

What do you do?

- **Return stale data** (sacrifice consistency): B responds with old data. The system stayed available.
- **Block or error** (sacrifice availability): B refuses to respond until it can check with A. The system stayed consistent.
- **Don't distribute** (sacrifice partition tolerance): run on a single node. No network between nodes means no partition. A single PostgreSQL instance is consistent and available, until the machine dies or becomes unreachable. Then you have nothing.

That's the trade-off. CA systems exist, but they're not distributed in the fault-tolerant sense. The moment you add a second node to survive failures, you've introduced a network, and networks partition. Then you're choosing between C and A.

### The misconception

People read CAP and think they're choosing a permanent category: "We're a CP system" or "We're an AP system."

That's not (quite) how it works.

The choice only matters during a partition (when something goes wrong). Most of the time, the network is fine, and you can have all three. CAP is about what you give up when things go wrong.

| State | What you get |
| ------------------- | --------------------- |
| Network healthy | C + A + P (all three) |
| Network partitioned | Pick C or A |

A system might be consistent 99.9% of the time and only go available-but-stale during the 0.1% when partitions happen. Or vice versa. It's a policy for failure modes, and has implications for the rest of your system.

### Where you see this

**CP systems** (consistent, partition-tolerant):

| System | What happens during partition |
| --------- | ----------------------------------------------------- |
| etcd | Minority partition can't reach quorum; rejects writes |
| ZooKeeper | Minority can't reach leader; rejects writes |
| MongoDB | With majority write concern, blocks until quorum acks |

These systems prioritize consistency. They'll reject writes they can't guarantee.

**AP systems** (available, partition-tolerant):

| System | What happens during partition |
| --------- | ----------------------------------------------- |
| DynamoDB | Eventually consistent reads by default |
| Cassandra | Nodes serve local data; conflicts resolve later |
| DNS | Caches serve stale records until TTL expires |

These systems keep responding, even if the data might be out of date.

### Consistency has levels

Between "every read sees the latest write" and "anything goes," there's a range. Weaker models can still provide useful guarantees:

- **Read-your-writes**: you see your own writes, others might not yet. You post a comment, refresh, your comment is there.
- **Monotonic reads**: your view of time doesn't go backwards. You see a video has 50k views, refresh, it won't show 48k. Someone else might still see 48k.
- **Causal consistency**: causes precede effects, globally. If Joe asks "Lunch?" and Emily replies "I'm in", no one sees Emily's reply without Joe's question.

These are weaker than strong consistency but stronger than "anything goes." And some applications can design with them in mind.

### The first rule

Systems can't be perfect, they will fail. CAP helps you understand the trade-offs. Every system you build on has made them. Understanding them tells you where it will fail. Every system you build will make them too.

That's the first rule. You can't have it all. 🎂



[1] Gilbert & Lynch. ["Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services"](https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf). ACM SIGACT News, 2002.

The first rule of systems engineering

Code used to be scarce. Humans wrote it, reviewed it, deployed it. You knew where it came from.

AI changes that. Agents generate code, pull from repos, execute user input. Code is abundant now, and most of it is untrusted. If you're building infrastructure for this world, you need isolation at every layer.

### MicroVMs aren't enough

Firecracker and similar microVM technologies provide strong isolation. Each execution gets its own kernel, filesystem, and network. A guest escape has to break through KVM virtualization, then past the jailer's seccomp filters, cgroups, and namespaces. That's a lot of layers.<a href="#ref-1">[1]</a>

But security research has found gaps. Firecracker relies on host kernel mitigations for speculative execution attacks. It doesn't add protection against cross-hyperthread attacks like L1 Terminal Fault or Microarchitectural Data Sampling.<a href="#ref-2">[2]</a>

These attacks exploit shared CPU resources. Sibling hyperthreads share the L1 data cache. A malicious VM on one hyperthread can sample data from whatever's running on the other, even across VM boundaries.<a href="#ref-3">[3]</a><a href="#ref-4">[4]</a>

The kernel documentation is direct: for full protection, disable hyperthreading.<a href="#ref-3">[3]</a> That cuts your vCPU count in half.

### Core scheduling

The kernel community built a middle ground. Core scheduling landed in Linux 5.14.<a href="#ref-5">[5]</a>

You tag processes with trust boundaries. The scheduler ensures only processes with the same tag can run on sibling hyperthreads simultaneously. Different tags? One idles.

It works through cookies assigned via prctl(). Cookies inherit across fork and exec, so you can tag an entire VM's process tree. The kernel prevents VMs with different cookies from sharing a physical core.<a href="#ref-6">[6]</a>

Benchmarks showed virtualization workloads running at 96% throughput with core scheduling enabled.<a href="#ref-5">[5]</a> You keep your full vCPU count for a small overhead.

Firecracker plus core scheduling gives you both layers: VM isolation for the software boundary, core scheduling for the hardware boundary. Defense in depth.

### The ARM path

There's another approach: use hardware that doesn't have the problem.

AWS Graviton, Ampere Altra, and most ARM server chips don't implement hyperthreading. One vCPU equals one physical core. No sibling threads, no shared cache to leak across.<a href="#ref-7">[7]</a>

ARM's simpler instruction set is more power efficient, so vendors pack more physical cores into the same power envelope. Ampere Altra ships 128 cores. You get the thread count without the security trade-off.

No hyperthreads also means predictable performance. Two VMs can't contend for the same execution units. When you're running thousands of isolated executions and billing for CPU time, predictability matters.

### Why not just use ARM?

Compatibility. x86 binaries don't run on ARM. Some workloads need AVX-512 or VNNI for ML acceleration. Legacy software may not have ARM builds. And x86 has stronger memory ordering guarantees, so concurrent code that works on x86 can have subtle race conditions on ARM.<a href="#ref-9">[9]</a>

For portable code, ARM works. For arbitrary user code that might depend on x86, you need x86.

### Why x86 still ships hyperthreading

x86 cores use more power, so hyperthreading is how Intel and AMD get more threads without blowing the power budget. AMD claims 30-50% throughput gains for less than 5% additional die area.<a href="#ref-8">[8]</a>

If you can layer the right mitigations, x86 with hyperthreading is still viable for untrusted workloads.

### Choosing your approach

| Approach | Security | vCPU count | Complexity |
| ------------------------- | -------- | ---------- | ---------- |
| Disable hyperthreading | Full | 50% fewer | Low |
| MicroVM + core scheduling | Strong | Same | Medium |
| ARM (no hyperthreading) | Full | N/A | Low |

For x86, the combination of Firecracker-style isolation and core scheduling makes untrusted multi-tenant compute practical. For ARM, the problem doesn't exist in the first place.

In a world where AI generates more code than humans review, running untrusted code efficiently isn't optional. The isolation primitives exist. The question is whether your stack uses them.



[1] AWS. ["Firecracker - Lightweight Virtualization for Serverless Computing"](https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/). 2018.

[2] Schlüter et al. ["Microarchitectural Security of AWS Firecracker VMM"](https://arxiv.org/abs/2311.15999). 2024.

[3] Linux Kernel. ["MDS - Microarchitectural Data Sampling"](https://docs.kernel.org/admin-guide/hw-vuln/mds.html).

[4] Linux Kernel. ["L1TF - L1 Terminal Fault"](https://docs.kernel.org/admin-guide/hw-vuln/l1tf.html).

[5] Corbet. ["Completing and merging core scheduling"](https://lwn.net/Articles/820321/). LWN, 2020.

[6] Linux Kernel. ["Core Scheduling"](https://docs.kernel.org/admin-guide/hw-vuln/core-scheduling.html).

[7] Ampere Computing. ["Looking Beyond SMT in the Cloud"](https://amperecomputing.com/en/blogs/looking-beyond-smt-in-the-cloud).

[8] AMD. ["Simultaneous Multithreading: Driving Performance and Efficiency on AMD EPYC CPUs"](https://www.amd.com/en/blogs/2025/simultaneous-multithreading-driving-performance-a.html). 2025.

[9] ARM. ["Learn about the C++ memory model for porting applications to Arm"](https://learn.arm.com/learning-paths/servers-and-cloud-computing/arm-cpp-memory-model/1/).

Running untrusted code in the age of agents


Where does the write go? To the cache, to the backing store, or both?

This question defines your caching strategy. Get it wrong and you lose data or tank performance. Get it right and you unlock the same trick that makes [LSM trees](/post/btrees-vs-lsm) and NVMe drives fast.

### Write-through

Every write goes to both the cache and the backing store. The write isn't acknowledged until both complete.

```
App → Cache → Backing Store
         ↓
      (wait for both)
```

Simple and safe. If the cache disappears, your data is already persisted. The downside: you're bottlenecked by the slower backing store on every write.

### Write-back

Writes go to the cache only. The backing store gets updated later, asynchronously.

```
App → Cache → (ack immediately)
         ↓
      (flush later)
         ↓
   Backing Store
```

Your writes are fast because you're only hitting the cache. The backing store gets batched updates when convenient.

The risk: if the cache dies before flushing, you lose data. For important data, you need replication or a write-ahead log to recover.

This is how LSM trees work. The memtable is the cache. Disk is the backing store. Writes hit memory, get acknowledged, and flush later as SSTables.

### Write-around

Writes bypass the cache entirely and go straight to the backing store. The cache only gets populated on reads.

```
App → Backing Store (writes)
App → Cache → Backing Store (reads, on miss)
```

Useful when you're writing data you don't expect to read soon. Avoids polluting the cache with write-once data.

### Read-through (pull-through)

On a cache miss, the cache fetches from the backing store, stores it, then returns it. The application doesn't talk to the backing store directly.

```
App → Cache (miss) → Backing Store
         ↓
   (cache stores it)
         ↓
   (return to app)
```

The cache is transparent. Your app just asks for data; the cache handles the rest.

### Cache-aside

The application manages everything. On a miss, the app fetches from the backing store and populates the cache itself.

```
App → Cache (miss)
App → Backing Store (app fetches directly)
App → Cache (app stores it)
```

More control, more code. You decide what gets cached and when.

### The trade-off matrix

| Strategy      | Write latency | Read latency | Durability        | Complexity |
| ------------- | ------------- | ------------ | ----------------- | ---------- |
| Write-through | Slow          | Fast (hit)   | High              | Low        |
| Write-back    | Fast          | Fast (hit)   | Needs replication | Medium     |
| Write-around  | Slow          | Fast (hit)   | High              | Low        |
| Read-through  | N/A           | Fast (hit)   | N/A               | Low        |
| Cache-aside   | N/A           | Fast (hit)   | N/A               | High       |

### Where you see this

Write-back shows up everywhere performance matters:

| System               | Cache layer    | Backing store   | Durability strategy       |
| -------------------- | -------------- | --------------- | ------------------------- |
| LSM trees            | Memtable (RAM) | SSTables (disk) | WAL (write-ahead log)     |
| NVMe SSDs            | DRAM cache     | NAND flash      | Flush on fsync            |
| CPU                  | L1/L2/L3 cache | RAM             | None (volatile anyway)    |
| Linux page cache     | RAM            | Disk            | fsync, O_DIRECT to bypass |
| Database buffer pool | RAM            | Disk            | WAL, checkpoints          |

The pattern is the same: absorb writes in fast storage, flush to slow storage in batches.

### The NVMe parallel

NVMe drives have a DRAM write-back cache in front of the NAND flash. When you write, data lands in DRAM first. The drive reports success immediately, then flushes to flash in the background.

This is why databases use fsync to force data to stable storage. Without it, a power loss can lose recent writes that the drive already acknowledged.

Same trade-off as LSM trees: write-back gives you speed, but you need a strategy for failures.

### The combo: write-back + read-through

To make a disk fast, you want both:

1. Write-back for writes: data lands in cache, gets acknowledged immediately, syncs to backing store lazily
2. Read-through for reads: reads hit the cache first, which has the freshest data (including unflushed writes)

The cache becomes the source of truth for recent data. Reads never see stale data because they check the cache first, which contains writes that haven't hit disk yet. The backing store is just a durability layer that catches up asynchronously.

```
Write: App → Cache (ack) .......... (lazy sync) → Backing Store
Read:  App → Cache (has fresh data, including unflushed writes)
              ↓ (miss)
          Backing Store
```

This is how LSM trees work: reads check the memtable first (the write-back cache), then fall through to SSTables on disk. The memtable always has the freshest view. Compaction syncs everything to disk in the background.

Same pattern in a database buffer pool, CPU cache, or NVMe drive. The fast layer absorbs writes and serves reads. The slow layer provides durability.

### When to use what

- **Write-through**: durability matters more than speed, or you can't trust your cache
- **Write-back**: you need write performance and have a replication or WAL strategy
- **Write-around**: you're writing data that won't be read soon (bulk loads, cold data)
- **Read-through/cache-aside**: orthogonal to write strategy; depends on whether you want the cache to be transparent

### So what

Write-back is the key to making writes fast. But it's a bet: your cache won't die before flushing. LSM trees, NVMe drives, and CPU caches all make that bet. The difference is how they handle losing it.


Cache strategies, explained

When do you organize your data? Every write, or... later?

Hash tables and B-trees said: every write. LSM trees said: later. Turns out, procrastinating is sometimes the right call.

### Hash tables: The baseline

Hash tables give you O(1) lookups. You hash a key, go directly to that slot, done. Hard to beat.

But they have limitations:

- **No ordering**: you can't ask "give me all keys between A and M"
- **No range queries**: scanning requires touching every slot
- **Resizing is expensive**: when the table fills up, you rehash everything

For pure key-value lookups (memcached, Redis), hash tables are great. But databases need more.

### B-trees: The workhorse

B-trees solved the ordering problem. They keep keys sorted in a tree structure, giving you:

- O(log n) lookups
- Efficient range scans (just walk the leaves)
- Good cache/disk utilization (nodes sized to pages)

This is the backbone of most relational databases: PostgreSQL, MySQL InnoDB, SQLite. On every write, you:

1. Traverse the tree to find the right page
2. Read that page from disk
3. Modify it in place
4. Write it back
5. Maybe rebalance, triggering more writes

Your data is always organized. Lookups are predictable. But you're paying the bookkeeping cost on every operation, and that cost is random I/O.<a href="#ref-1">[1]</a>

### The log-structured insight

In 1991, Rosenblum and Ousterhout asked a different question: what if we never modified data in place?<a href="#ref-2">[2]</a>

Their Log-structured File System (LFS) treated the entire disk as an append-only log. Writes always go to the end. Old data gets garbage collected later.

This sounds wasteful, but it converts random writes into sequential writes. Sequential I/O is 10-100x faster on spinning disks, still 5-20x faster on SSDs.<a href="#ref-3">[3]</a>

The same insight shows up in filesystems:

| Filesystem | Write strategy | Philosophy |
| ---------- | ------------------- | ----------------------------------------- |
| ext4, XFS | In-place + journal | Established: fix it now |
| btrfs, ZFS | Copy-on-write | Next-gen: never overwrite, clean up later |
| bcachefs | Copy-on-write + LSM | Next-gen: both techniques combined |

Copy-on-write filesystems are basically applying log-structured patterns to the filesystem layer.

### LSM trees: Log-structured meets sorted

LSM trees (Log-Structured Merge trees) took the append-only insight and applied it to sorted indexes.<a href="#ref-4">[4]</a> This is what powers RocksDB, LevelDB, Cassandra, and ScyllaDB.

The design:

1. Writes go to memory first, in a sorted buffer called a memtable
2. When it fills up, flush to disk as a sorted file called an SSTable
3. Periodically merge SSTables in the background (compaction)

Instead of organizing on every write, you batch it up and do it asynchronously. The bookkeeping still happens, just later, in bulk, when sequential I/O makes it cheap.

### Don't you end up doing the same work?

No. Way less.

By batching hundreds of writes into a single sequential flush, you're amortizing the I/O cost across all of them. One disk seek instead of hundreds.

### But reads must be slower?

In theory, yes. You might check multiple SSTables to find your data.

In practice:

- **Bloom filters** on each SSTable tell you which files definitely don't have your key
- Recent data is usually still in the memtable (in memory)
- Hot data gets cached at multiple levels

For many workloads, read performance is comparable to B-trees. For write-heavy workloads, LSM trees dominate.

### Temporal vs spatial locality

Here's where it gets interesting.

Spatial locality assumes data near each other in address space should be stored together. B-trees optimize for this: record #1000 is stored near record #1001.

Temporal locality assumes data written at the same time should be stored together. LSM trees optimize for this: everything flushed in the same batch ends up in the same SSTable.

Why does temporal locality matter? Because it matches how systems actually behave:

| Scenario | Why temporal locality helps |
| ------------------ | ----------------------------------------------------------- |
| `npm install` | Thousands of files written in a burst; they belong together |
| App startup | Reads config and state that was all persisted together |
| Checkpoint restore | Reading a consistent snapshot |
| Transactions | Records committed together are accessed together |

Data you need right now was usually written together at some point. Grouping by write-time often matches access patterns better than grouping by key.

### The trade-offs (there's always trade-offs)

LSM trees optimize for writes at some cost:

- **Write amplification**: data gets rewritten multiple times during compaction
- **Space amplification**: you temporarily store redundant data
- **Read amplification**: worst-case reads check multiple levels

For write-heavy workloads (logging, time-series, event streams), the trade-off is worth it. For read-heavy OLTP with lots of point queries, B-trees still make sense.

### So what?

Back to our opening question: when do you pay the bookkeeping cost? B-trees say now, on every write. LSM trees say later, in bulk, when sequential I/O makes it cheap.

That's the trick. Procrastination, done right, can be an advantage.

We're seeing this pattern show up in unexpected places. Cloud block storage systems are adopting log-structured designs to handle virtualized storage at scale.<a href="#ref-5">[5]</a> And with AI training flushing massive checkpoints every few minutes, the demand for write-optimized storage is only growing. This might be the architecture for the next generation of storage infrastructure.

And that's my love letter to the LSM tree. It's not real magic, but it's damn close.



[1] Graefe. ["Modern B-Tree Techniques"](https://doi.org/10.1561/1900000028). Foundations and Trends in Databases, 2011.

[2] Rosenblum & Ousterhout. ["The Design and Implementation of a Log-Structured File System"](https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf). SOSP, 1991.

[3] Jacobs. ["The Pathologies of Big Data"](https://queue.acm.org/detail.cfm?id=1563874). ACM Queue, 2009.

[4] O'Neil et al. ["The Log-Structured Merge-Tree"](https://www.cs.umb.edu/~poneil/lsmtree.pdf). Acta Informatica, 1996.

[5] Zhang et al. ["What's the Story in EBS Glory: Evolutions and Lessons in Building Cloud Block Store"](https://www.usenix.org/conference/fast24/presentation/zhang-weidong). USENIX FAST, 2024.

The art of procrastination, in storage

### Creating the Systemd Unit

Systemd is a process manager that can start, stop, and manage processes ("_Units_") on boot or shutdown of your machine!<a href="#ref-1">[1]</a>
In this walkthrough, we'll be starting a long-running Node.js web server that lives in `/var/www/`.

To create a new _Unit_, we need to create a file in `/etc/systemd/system/` (the default location on most Linux distributions).
This file should have the `.service` extension.

Let's create `/etc/systemd/system/webserver.service`:

```ini
[Unit]
# Since we're going to bind to a port on localhost, we need to wait for the
# network to be available:
After=network.target
Description=My fancy web server

[Service]
# Set the working directory to where your application lives
WorkingDirectory=/var/www
# Set any environment variables your application needs
Environment=NODE_ENV=production
Environment=PORT=3000
# Start your webserver (absolute path required - find yours with `which node`)
ExecStart=/usr/bin/node server.js
# Restart the service if it crashes
Restart=always
# Wait 3 seconds before restarting
RestartSec=3

[Install]
# Start this service when the system boots to multi-user mode
WantedBy=multi-user.target
```

After you've created the file, update it to have the correct permissions (`664`):

```bash
chmod 664 /etc/systemd/system/webserver.service
```

Don't be intimidated by the file above! A lot of it is boilerplate.
Read the comments placed in the above file to understand what each section is for!

> **Note**: Comments in systemd unit files must be on their own line—inline comments after values are not supported.<a href="#ref-2">[2]</a> The `ExecStart` directive requires an absolute path to the executable.<a href="#ref-3">[3]</a>

You can learn more about the [_Unit_](https://www.freedesktop.org/software/systemd/man/systemd.unit.html) or [_Service_](https://www.freedesktop.org/software/systemd/man/systemd.service.html) sections in the systemd documentation.

The _Install_ section is used to configure when the service will start. Most commonly, you'll want `multi-user.target`.<a href="#ref-4">[4]</a>
The `WantedBy` value directly corresponds to different [Linux runlevels](https://en.wikipedia.org/wiki/Runlevel):

| runlevel | WantedBy value | Description |
| -------- | ------------------- | ------------------------------------------------------ |
| 0 | `poweroff.target` | Run before the computer shuts down |
| 1 | `rescue.target` | Run when the system is in single-user mode |
| 2-4 | `multi-user.target` | Non-graphical multi-user system (networking available) |
| 5 | `graphical.target` | Run when the display manager has started |
| 6 | `reboot.target` | Run before the computer reboots |

### Enabling the Systemd Unit

Now that the unit has been created, we need to reload the Systemd process.
This will allow Systemd to recognize the new file.

Run the following command:

```bash
systemctl daemon-reload
```

> **Tip**: You need to execute the above command to reload Systemd **every time** you edit your `.service` file(s).

Next, you'll need to enable the service. Running this command will create a symlink into your `WantedBy` target.

```bash
systemctl enable webserver
```

Finally, since your system is already booted, run the following command to start the service:

```bash
systemctl start webserver
```

### Cheatsheet

**Enabling (creating) or Disabling (deleting) your service**

```bash
systemctl enable webserver # enable service to start on boot
systemctl disable webserver # disable service from starting on boot
```

**Starting, Restarting, or Stopping your service**

```bash
systemctl start webserver
systemctl restart webserver
systemctl stop webserver
```

**Checking the status of your service**

```bash
systemctl status webserver
```



[1] freedesktop.org. ["systemd.unit — Unit configuration"](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html). systemd documentation.

[2] freedesktop.org. ["systemd.syntax — General syntax of systemd configuration files"](https://www.freedesktop.org/software/systemd/man/systemd.syntax.html). "Lines beginning with '#' or ';' are ignored."

[3] freedesktop.org. ["systemd.service — Service unit configuration"](https://www.freedesktop.org/software/systemd/man/systemd.service.html). "The first argument must be either an absolute path or a simple file name without any slashes."

[4] Red Hat. ["Working with systemd unit files"](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/using_systemd_unit_files_to_customize_and_optimize_your_system/assembly_working-with-systemd-unit-files_working-with-systemd). RHEL 9 Documentation.

Walkthrough: How to start a long-running command on boot with systemd

`xargs` reads space and line delimited strings from stdin and executes a specified command with the input as arguments.<a href="#ref-1">[1]</a>

This makes `xargs` a very powerful command when you want to run a command against arbitrary input.

For example, let's run the following command:

```bash
❯ echo 'hello there\nbig world' | xargs echo
```

`xargs` automatically parses `hello`, `there`, `big`, and `world` as individual inputs.
Then, these arguments are passed to the specified command (in this case, also `echo`).

```bash
❯ echo 'hello there\nbig world' | xargs echo

# What gets executed:
# echo 'hello' 'there' 'big' 'world'

# Output:
hello there big world
```

> **Note**: All computed input arguments were passed to a single `echo` command.

For more practical usage, you may find yourself wanting to run a command once per argument.
For example, when the command you're using only accepts a single argument.

A common command I find this useful for is `aws s3`.
`aws s3` doesn't support passing multiple arguments, so each command may only accept a single S3 path.

For example, using brace expansion and `xargs` to pipe S3 paths to `aws s3 rm` doesn't work:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs aws s3 rm --recursive

# What gets executed:
# aws s3 rm --recursive 's3://my-bucket/partition-1/' 's3://my-bucket/partition-2/' 's3://my-bucket/partition-3/'

# Output:
Unknown options: s3://my-bucket/partition-2/,s3://my-bucket/partition-3/
```

To solve this, `xargs` accepts an `-n <number>` argument.<a href="#ref-2">[2]</a>
The `-n` argument configures how many arguments should be per command invocation.

So, since `aws s3 rm` only accepts one argument, we can pass `-n 1` to `xargs` to execute one command per argument:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 aws s3 rm --recursive

# What gets executed (sequentially):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success
```

> **Note**: The `aws s3 rm` command was invoked three times, once per argument.

Success! 🎉

Now that we're running one command per argument, it's important to note it's done so sequentially.
However, you may also want to parallelize execution of long-running commands.

`xargs` has an argument for parallelizing execution: `-P <number>`.<a href="#ref-3">[3]</a>

The above `aws s3 rm --recursive` example may take a long time per path, so we can maximize efficiency by running all the commands at once.
To do this, we'll pass the `-P 3` argument:

```bash
❯ echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 -P 3 aws s3 rm --recursive

# What gets executed (in parallel):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success
```

> **Tip**: To run one command per CPU, you can use this handy shortcut:
>
> ```bash
> xargs -n 1 -P $(getconf _NPROCESSORS_ONLN) command
> ```

Good luck! 🏎



[1] IEEE Std 1003.1. ["xargs — construct argument lists and invoke utility"](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html). The Open Group Base Specifications.

[2] GNU. ["xargs invocation"](https://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html). GNU Findutils Manual. <code style="font-size:inherit">-n max-args</code>: Use at most max-args arguments per command line.

[3] GNU. ["Controlling Parallelism"](https://www.gnu.org/software/findutils/manual/html_node/find_html/Controlling-Parallelism.html). GNU Findutils Manual. <code style="font-size:inherit">-P max-procs</code>: Run up to max-procs processes at a time.

timer.blog

Posts

The first rule of systems engineering

Running untrusted code in the age of agents

Cache strategies, explained

The art of procrastination, in storage

Walkthrough: How to start a long-running command on boot with systemd

xargs(1), Shell Brace Expansion, and Single-Line Execution