Switching from Synchronous to Asynchronous Mode of Coding

When I blogged about the challenges and opportunities of AI assisted coding, there were a fair number of questions about details of our experience. I know many are eager to learn more, and we can't wait to share details of what we are building in a few weeks at re:Invent. However, among the questions, there was one that caught my attention:

Looking at the git activity heatmap, you started coding on the weekends. Why the sudden change?

I wanted to dig into that question, because in many ways the answer captures how AI assisted coding has changed my approach.

For starters, I recommend folks find their own personal balance, and working on the weekends is generally not a great idea. Personally, I really love what I do and have always been energized by finding opportunities to write code, even as time became hard to find at the more senior levels. To that end, I'd often carve off an hour or two to write code on the weekends, if not for work then for personal projects. But that time was often insufficient - an hour isn't enough to get into flow, page in context, make meaningful progress, and wrap up cleanly. 

However, the biggest factor is that agentic coding has changed my approach to writing code in a fundamental way. Coding has always been a synchronous activity for me. I'd start working on a change or a feature, make some progress, gain new insights that would cause me to alter my approach, and keep iterating. When finished, tests wouldn't always pass on the first try, so I'd spend some time debugging and fixing problems.

Over the past 7 years, I've also been primarily developing in Rust. Rust is famous for prioritizing correctness, and that's what I like about it. The compiler will let you know if your code is potentially incorrect, which is great when trying to build robust software. However, that tends to add time into the implementation phase, as you work through the issues flagged by the compiler (the upside being that catching these issues at compile time beats catching them in production). Lastly, as I've grown more senior, finding continuous "focus" time has been a challenge too. Working on a coding task would often span multiple meetings, each time requiring me to page-in all the context again.

My coding flow would typically look something like this:

With agentic coding, my flow has changed in a fundamental way. Instead of synchronous flow of continuous attention, I usually spend five minutes breaking down the problem and getting to a place of "shared understanding" with the agent on how to tackle the task (that in itself is an interesting topic!). I then give the AI agent green light and disengage. I'll admit, for the first few months I stayed glued to my screen, watching the agent work through each implementation. That initial hand-holding helped me build intuitions for structuring requests in ways that help the agent succeed. These days though, I genuinely step away.

The agent then gets to work to implement my request, run the build (sparring with the Rust compiler instead of me!), run the unit and the integration tests, and iterate a few times until it believes it has succeeded. Within our project, that can take anywhere from 15 to 30 minutes. Once done, I review the code produced by the agent, and decide if it meets my bar. This review is a critical step, because I need to verify not only correctness and implementation choices, but also that the agent interpreted my request correctly within the broader context. Occasionally, I may decide that my request was too ambiguous or ambitious, so I would throw away the results and try again with a narrower request. Most of the time, I give the agent feedback, point out issues or parts that could be improved and let it go another round. In the end, my coding flow now looks more like this:

The net result is that my coding style switched from synchronous to asynchronous, with only needing to spend focused attention a few times throughout the implementation of the change. At work, these "idle" times is when I do all the other things a senior IC would typically do. On the weekends, the asynchronous approach allows me to do something I enjoy, without needing to block continuous time or step away from my other commitments.

The New Calculus of AI-based Coding

Over the past three months, a team of experienced, like-minded engineers and I have been building something really cool within Amazon Bedrock. While I'm pretty excited about what we are building, there is another unique thing about our team  - most of our code is written by AI agents such as Amazon Q or Kiro. Before you roll your eyes: no, we're not vibe coding. I don't believe that's the right way to build robust software.

Instead, we use an approach where a human and AI agent collaborate to produce the code changes. For our team, every commit has an engineer's name attached to it, and that engineer ultimately needs to review and stand behind the code. We use steering rules to setup constraints for how the AI agent should operate within our codebase, and writing in Rust has been a great benefit. Rust compiler is famous for focusing on correctness and safety, catching many problems at compile time and providing helpful error messages that help the agent iterate. As a juxtaposition to vibe coding, I prefer the term "agentic coding." Much less exciting but in our industry boring is usually good.

For me, roughly 80% of the code I commit these days is written by the AI agent. My personal workflow: break down the task until I have clarity in my own head (often using AI to explore approaches), prompt the AI agent, review its output, iterate with it until I like the results, and occasionally take over the change set and finish it myself. I pay attention to every line of code the agent produces, and don't accept them until I am fully satisfied with the quality of what is being produced - no different than if I wrote every line myself.

I've always been an efficient coder, finding time to write code even in limited amount of time I've been able to dedicate to coding over the past few years. And the last few months have been the highest coding throughput months of my career. My team is no different—we are producing code at 10x of typical high-velocity team. That's not hyperbole - we've actually collected and analyzed the metrics.

Driving at 200mph

Here's where it gets interesting. A typical software team, even an experienced one, doesn't get things right all the time. Even with good testing and engineering practices, bugs occasionally make it through. We've all heard the phrase "testing in production." That reality is the main reason I've always believed that focusing on testing alone is not enough, and investing in blast radius and time to recovery is just as important.

AI assisted code is no different, it may contain bugs even when thoroughly reviewed by a human, and I suspect the probabilities are not significantly different.  However, when teams ship commits at 10x the rate, the overall math changes. What used to be a production impacting bug once or twice a year, can become a weekly occurrence. Even if most bugs get caught in integration or testing environments, they will still impact the shared code base, requiring investigation and slowing the rest of the team down. Once again, this is not just hyperbole—our team sees signs that these are the challenges that pop up with a step function increase in throughput.

I am increasingly convinced that in order for agentic development to increase engineering velocity by an order of magnitude, we need to decrease the probability of problematic commits by an order of magnitude too. And likely by even more than that, since at high velocities individual commits can begin interacting with each other in unexpected ways too.

In other words, driving at 200mph, you need a lot of downforce to keep the car on the track!

The Cost-Benefit Rebalance

One of the best ways to reduce the chance of bugs is to improve testing. I'm an airplane geek, and have always admired the testing ideas used by the airplane manufacturers. From early simulations, to component testing, to wind tunnel testing, to testing to breaking point, and ultimately test flights of fully assembled aircraft. Even flight simulators play a role in improving the overall safety of the industry. Some of these ideas have been tried in the software industry, but they are far from ubiquitous. 

As an example, I've always liked "wind tunnel" style tests, that test fully assembled system in a controlled environment. To achieve that, one pattern I've used is implementing high fidelity "fake" versions of external dependencies that can be run locally. If you do that, you can then write build-time tests that run locally and verify end-to-end behavior of the whole system. You can even inject unexpected behaviors and failures into fake dependencies, to test how the system handles them. Such tests are easy to write and execute because they run locally, and they are great at catching those sneaky bugs in the seams between components.

Unfortunately, faking all the external dependencies isn't always easy for a service with moderate level of complexity. And even if you do, you now have to own keeping up with the real dependencies as they evolve. For those reasons, in my experience most teams don't write such tests.

I think we are seeing early signs that agentic coding can change the calculus here. AI agents are great at spitting out large volumes of code, especially when the desired behavior is well known and there's little ambiguity. Ideas that were sound in principle, but too expensive to implement and maintain just had their costs decrease by an order of magnitude. I really love riding such shifts in the industry, because they open the doors to new approaches that weren't practical in the past.

Our project (with the help of an AI agent) maintains fake implementations of external dependencies like authentication, storage, chain replication, and inference engine to be used in tests. We then wrote a test harness that uses those fakes to spin up our entire distributed system, including all the micro-services, on developers' machines. Build-time tests then spin up our canaries against that fully assembled stack verifying the system as a whole works. 

I'm really bullish on this approach catching a category of bugs that in the past could only be caught once the change was committed and made it to the test environment. A few years ago, ideas like these would receive resistance as nice, but too expensive. This time around, it took just a few days to implement for a relatively complex system.

Driving Fast Requires Tighter Feedback Loop

At the end of the day, all these changes need to be built, tested, and deployed in order to bring customer value. It's not uncommon for software teams to have a CICD pipeline that takes several hours to build, package, and then test software changes. That pipeline can then take a few days to roll a batch of changes across pre-prod stages and eventually production stages. Typically, a team with that level of automation would be considered a healthy team.

Agentic coding changes that dynamic. In the amount of time it takes to build, package, and test one set of commits, another dozen might be waiting to go out. By the time a change set is ready to deploy to production, it may contain 100 or more commits. And if one of those commits contains a problem, the deployment needs to be rolled back grinding the pipeline to a halt. In the meantime, even more changes accumulate, adding to the chaos and the risk.  

I'm a Formula 1 fan, and this reminds me of how an accident on the track can cause a Yellow Flag to be raised. Normally, the cars zoom around the track at immense speeds and accelerations. But if an accident occurs, the race marshals raise a yellow flag, which requires all the cars to slow down behind the pace car. An exciting race turns into a leisurely drive around the track until the debris is cleaned up and the track is safe again. To minimize such slow downs, race organizers go to great lengths to prepare for all types of accidents, and make sure they can clean up the track and restart the race in minutes.

Just like whole-system local tests help tighten the feedback loop for catching certain bugs, we may need to think similarly about how we implement our CICD pipelines. When teams are moving at the speed of dozen of commits per hour, problematic issues will need to be identified, isolated, and reverted in minutes instead of hours or days. That means that a typical build and test infrastructure will need to become an order of magnitude faster than it is today. Just like online video games become unplayable when there is high lag between player's inputs and the game's reaction, it's really hard to move 10x faster if every commit still requires a lengthy delay before you see the feedback. 

The communication bottleneck

I enjoy observing well-run operations. If you've ever peeked behind the curtain of a busy restaurant, then at first sight you may think it's chaos. But if you take a second to notice the details, you'll see that all members are constantly coordinating with each other. Chefs, cooks, wait staff, bussers, and managers pass information back and forth in a continuous stream. By staying in constant sync, a well run restaurant manages to serve its patrons even during peak times, without sacrificing on quality or latency.

I believe that achieving similar increase in velocity for a software team requires constraints on how teams communicate. When your throughput increases by an order of magnitude, you're not just writing more code - you're making more decisions. Should we use this caching strategy or that one? How should we handle this edge case? What's the right abstraction here? At normal velocity, a team might make one or two of these decisions per week. At 10x velocity, they are making multiple each day.

The challenge is that many of these decisions impact what others are working on. Engineer A decides to refactor the authentication flow, which affects the API that Engineer B is about to extend. These aren't just implementation details - they're architectural choices that ripple through the codebase.

I find that traditional coordination mechanisms introduce too much latency here. Waiting for a Slack response or scheduling a quick sync for later in the day means either creating a bottleneck - the decision blocks progress - or risking going down the wrong path before realizing the conflict. At high throughput, the cost of coordination can dominate!

One approach is to eliminate coordination - if everybody works on independent components, they are unlikely to need to coordinate. But I find that ideal impractical in most real-world systems. So another alternative is to significantly decrease the cost of coordination. Our team sits on the same floor, and I think that's been critical to our velocity. When someone needs to make a decision that might impact others, they can walk over and hash it out in minutes in front of a whiteboard. We align on the approach, discuss trade-offs in real time, and both engineers get back to work. The decision gets made quickly, correctly, and without creating a pile-up of blocked work.

I recognize this doesn't solve the problem for distributed teams—that remains an open challenge.

The Path Forward

I'm really excited about the potential of agentic development. I think it has the capability to not only improve the efficiency of software development, but also allow us to tackle problems that were previously too niche or expensive to solve. The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable.

But here's the critical part: these gains won't materialize if we simply bolt AI agents onto our existing development practices. Like adding a turbocharger to a car with narrow tires and old brakes, the result won't be faster lap times - it will be crashes. At 10x code velocity, our current approaches to testing, deployment, and team coordination become the limiting factors. The bottleneck just moves.

This means we need to fundamentally rethink how we approach building software. CICD pipelines designed for 10 commits per day will buckle under 100. Testing strategies that were "good enough" at normal velocity will let too many bugs through at high velocity. Communication patterns that worked fine before will create constant pile-ups of blocked work.

The good news is that we already have great ideas for comprehensive testing, rapid deployment, and efficient coordination - ideas that have shown promise but haven't seen wide adoption because they were too expensive to implement and maintain. What's changed is that agentic development itself can dramatically lower those costs. The same AI agents that are increasing our code throughput can also help us build the infrastructure needed to sustain that throughput.

This is the real opportunity: not just writing more code faster, but using AI to make previously impractical engineering practices practical. The teams that succeed with agentic development will be the ones who recognize that the entire software development lifecycle needs to evolve in concert.

The Nuanced Reality of Throttling: It's Not Just About Preventing Abuse

If you work with multi-tenant systems you are probably familiar with the concept of throttling or admission control. The idea is pretty simple and is rooted in the common human desire for fairness: when using a shared system or resource no customer should be able to consume "too much" of that resource and negatively impact other customers. What constitutes "too much" can vary a lot and will usually depend on technical, product, business, and even social factors.

Yet when most engineering teams think about throttling, the first thing that comes to mind is often protecting from bad actors who may intentionally or accidentally try to knock the system down. It's a clean, morally satisfying mental model. Bad actors perform unreasonable actions, so we put up guardrails to protect everyone else. Justice served, system protected, everyone goes home happy.

But here's the thing - protecting against bad actors is just a small fraction of the throttling story. The reality of throttling is far more nuanced, and frankly, more interesting than the "prevent abuse" story we often tell ourselves.

Two Sides of the Same Coin

I want to start with a distinction that influences how I think about throttling. When we implement an admission control like throttling, we're typically optimizing for one of two parties: the customer or the system operator. And these two scenarios are fundamentally different beasts.

Quotas and Limits for the Customers's Benefit

This is the "helpful" throttling. Think about a scenario where a developer accidentally writes a runaway script that starts making thousands of paid API calls per second to your service. Without throttling, they might wake up to a bill that they do not love. In this case, throttling is essentially a safety net - it prevents their own code from causing financial harm.

Similarly, consumption limits can be a mechanism to steer customers towards more efficient patterns. For example, by preventing the customer from making thousands of "describe resource" API calls we could steer them towards a more efficient "generate report" API. This could become a win-win situation: the customer gets the data they need more easily, and the system operator gets to improve the efficiency of their system.

Load Shedding for the System's Benefit

Now here's where things get nuanced. Sometimes you implement throttling not to help the customer, but to protect your system from legitimate traffic that just happens to be inconveniently timed. Maybe one of your customers is dealing with their own traffic surge - perhaps they just got featured on the front page of Reddit, or their marketing campaign went viral.

In this scenario, you're potentially hurting a customer who's doing absolutely nothing wrong. They're not trying to abuse your system; they're just experiencing success in their own business. But if you let their traffic through, it might overload the system and impact all your other customers. Now, technically we could argue that this type of throttling also helps the customer - nobody wins when the system is overloaded and suffers a congestion collapse. However, the point is that the customer isn't going to thank you for throttling them here!

I find it helpful to think of these as different concepts entirely. The first is quotas or limits - helping customers avoid surprises or use your system more efficiently. The second is load shedding - protecting your system from legitimate but inconvenient demand. 

The Uncomfortable Truth About Load Shedding

This distinction matters because it forces us to confront an uncomfortable reality: sometimes we're actively hurting our customers to protect our system. The "preventing abuse" mental model breaks down completely here, and we need a more honest framework.

A healthier way to think about load shedding is that we want to protect our system in a way that causes the least amount of harm to our customers. It's not about good guys and bad guys anymore - it's about making difficult trade-offs when resources are constrained.

This reframing changes how we approach the problem. Instead of thinking "how do we stop bad actors," we start thinking "how do we gracefully degrade when we hit capacity limits while minimizing customer impact?"

The Scaling Dance

Here's where throttling gets really interesting. Load shedding doesn't have to be a permanent punishment. If you're dealing with legitimate traffic spikes, throttling can be a temporary protective measure while you scale up your system to handle the demand.

Think of a restaurant during an unexpectedly busy dinner rush. If they are short staffed, a restaurant may choose to keep some tables empty and turn away customers to make sure the customers who do get in still have a pleasant experience. Then, once additional staff arrive, they may open additional tables and begin accepting walk ins again.

In practice, this means your load-shedding system should be closely integrated with your auto-scaling infrastructure. When you start load shedding, that should trigger scaling decisions. The goal is to make load shedding temporary - a protective measure that buys you time to add capacity.

However, you also want to be careful to avoid problems like run away scaling, where the system scales up to unreasonable sizes because load shedding does not stop. Or oscillations, where the system wastes resources by continuously scaling up and down due to hysteresis. In both of these scenarios, placing velocity controls on scaling decisions can be a reasonable mechanism. 

Beyond Static Limits

Many load shedding systems I've encountered use static limits. "Customer A gets 100 requests per minute, Customer B gets 100 requests per minute, everyone gets 100 requests per minute." It's simple, it's fair, and it's probably insufficient.

It's a simple system to implement and to explain, but static limits assume that every customer has the same needs from your system, regardless of their scale. But in reality, your customers exist in a wide spectrum of use cases. Some are weekend hobbyists making a few API calls. Others are large companies whose entire business depends on your service. 

Static limits also assume that the customers of your system act in uncorrelated fashion. If multiple customers of the system hit their limit at the same time, the system could still get overloaded. There are lots of real-world reasons such a correlated behavior could occur. Perhaps all these customers are different teams within the same company, and the whole company is seeing a big increase in workload. Or perhaps they are using the same client software that contains the same bug. Or, my personal favorite, perhaps they've all configured their system to perform some intensive action on a cron at midnight, because humans love round numbers! 

An interesting alternative is capacity-based throttling. Instead of hard limits, you admit new requests as long as your system has capacity. Think of it like a highway onramp - when the traffic on the highway is flowing freely the onramp lets new cars in without any constraints. But as soon as congestion builds up, the traffic lights on the onramp activate and begin metering new cars.

The Top Talker Dilemma

But what happens when you hit capacity limits? The naive approach is to shed load indiscriminately, but that's almost as bad as experiencing an overload. Almost, because you are avoiding congestion collapse - so many requests will still go through. However, such indiscriminate load shedding will make most of your customers see some failures - from their point of view the system is experiencing an outage. 

A different option might be to shed load from your top talkers first. They're using the most resources, so cutting them off gives you the biggest bang for your buck in terms of freeing up capacity. The problem is that your top talkers are often your biggest customers. Cutting them off first is like a retailer turning away their top spending customers. Not exactly a winning business strategy.

One approach I think can work well is to shed load from "new" top talkers - customers whose traffic has recently spiked above their normal patterns. This gives you the capacity relief you need while protecting established usage patterns. The assumption is that sudden spikes are more likely to be temporary or problematic, while established high usage represents legitimate business needs.

One way you could implement this behavior is by starting with low static throttling limits, but then automatically increasing those limits whenever a customer reaches it, as long as the system has capacity. In happy state, no customer experiences load shedding and the throttling limits are increased to meet new demand. However, if the system is at capacity, new increases are temporarily halted and customers who need an increase may get throttled, until the system is scaled up and new headroom is created.

A Different Mental Model

I think the key insight here is that throttling is not primarily about preventing abuse - it's about resource allocation under constraints. Sometimes those constraints are financial (protecting customers from runaway bills), sometimes they're technical (preventing system overload), and sometimes they're business-related (product tier differentiation).

When we frame throttling as resource allocation rather than abuse prevention, we start asking better questions:

• How do we allocate limited resources fairly?
• How do we balance individual customer needs against system stability?
• How do we minimize harm when we have to make difficult trade-offs?
• How do we use throttling as a signal to guide scaling decisions?

These are more nuanced questions than "how do we stop bad actors," and they lead to more sophisticated solutions.

The Path Forward

None of this is to say that traditional abuse prevention doesn't matter. There are definitely bad actors out there trying to overwhelm systems, and throttling is one tool in your arsenal to deal with them. But I think we do ourselves a disservice when we reduce all throttling to abuse prevention.

The reality is that throttling is a complex, multi-faceted tool that touches on resource allocation, system reliability, product design, and business strategy. The sooner we embrace that complexity, the better solutions we'll build.

In my experience, the most effective throttling systems are those that:

1. Clearly distinguish between customer protection and system protection use cases
2. Integrate closely with auto-scaling infrastructure
3. Use capacity-based limits rather than static ones where possible
4. Prioritize established usage patterns over new spikes
5. Treat throttling as a resource allocation problem, not just an abuse prevention one

The next time you're designing a throttling system, I'd encourage you to think beyond the "prevent abuse" narrative. Ask yourself: who is this throttling protecting, and what are the trade-offs involved? The answers might surprise you, and they'll almost certainly lead to a better system.

The Trouble with Leader Elections (in distributed systems)

When working on distributed systems, it's not uncommon to need a piece of logic that executes across the entire system. Some examples of that can be a task that deletes tomb-stoned records, or a task that periodically updates system-wide configuration, or perhaps a piece of code that continuously scans the entire fleet replacing unhealthy hosts. In my experience, most distributed systems have at least one such component.

The simplest way to implement such logic is to execute it on a single host - non-distributed algorithms tend to be simpler and more efficient to implement. However, because we still want redundancy a leader election algorithm is commonly used to elect one of a small number of hosts to act as the leader and execute the task. In a nutshell, most widely used leader election algorithms have a concept of leases (or timed locks) at their core. Each host continuously attempts to acquire a lease from a data store with appropriate atomicity and consistency guarantees. This could be a relational database, a datastore like DynamoDB, or a Paxos-based locking client. If successfully acquired, the host can begin executing the task, continuously renewing the lease. If the host dies, the lease will eventually expire and another host will win the race and take over as the leader. This Amazon Builders Library article goes into a lot of helpful details on implementing leader election: Leader election in distributed systems.

On the surface, leader election is a simple approach that fills a need for many distributed systems. So why the dislike? In my experience, leader election based systems have a couple of downsides:

Blast radius

Just like in the real world, leader election elevates a single participant into a position of immense power. If the leader is good, the system chugs along. If the leader makes bad decisions, it can very quickly impact the entire system. I find this concern most visible during deployments. In most systems, deployments tend to be the leading cause of failures. No matter how well you test the system, the real moment of truth hits when the new code and configuration meet the diversity and complexity of the real production environment. Most distributed systems deal with that by rolling new changes slowly across the fleet, perhaps starting with a single host and waiting for signs of success before incrementally rolling out the changes to the rest of the fleet. Deployments to a leader election based systems tend to go from 0% to 100% as soon as the changes hit the currently elected leader. Depending on the task performed by the leader, any bug that sneaks through testing can be catastrophic for the health of the overall system.

Liveness vs split-leader tension

One of the most important decisions in the leader-elected systems is the length of the lease. If the lease expires while the task is still executing, we run the risk of another host grabbing the lease and starting its own execution. All of a sudden, a task that was designed to run once at a time can begin running concurrently on multiple hosts. Unless the system has been designed and is continuously tested to support that, this could lead to subtle concurrency bugs that impact the entire system.

On the other hand, if the lease is long enough to cover even the most pessimistic task execution time, then an actual leader failure will need to wait that long until a new leader acquires the lease. This can lead to stalling and liveness problems.

We can attempt to work around this tension by using short-lived leases and having the leader continuously extend the lease while the task is running. That can quickly get tricky. For example, a blocking IO call or a stop-the-world garbage collection pause can push us past lease expiration time without an opportunity to extend. Or an attempt to extend can fail, and we will need to decide whether to abort the task and deal with partial progress or risk multiple tasks executing concurrently.

Liveness vs faux-leader tension

I find that leader election based system are also weak to ambiguous gray failure scenarios. The leader may be happily acquiring and renewing the lease, but it may have trouble executing the task. Perhaps a downstream dependency is timing out, or the task is terminating abnormally. When that happens, the leader gets to make a really tough decision. It could continue renewing the lease, in the hope that the task will eventually succeed. But that runs the risk of getting stuck in the faux-leader mode, where a problem with the host (e.g. a broken volume) prevents it from making progress, while it happily renews the lease.

Alternatively, the leader could decide to give up and release the lease, in the hope that a different host will have better luck with the task. But that runs a different risk. If the task was failing due to outside factors (perhaps a dependency was down), no leader will be willing to hold the lease. That can cause the system to enter into a leader churn, which can lead to various types of cascading failures.

So what should we do?

I think it's always worth starting by deciding if these problems matter to your system. There are systems where these problems are not deal breakers, and leader election can be the simplest solution. For example, a component that sends out an operational report once a day can likely ignore these tensions. In the worst case, operators may occasionally receive duplicate e-mails or a report may get delayed by a few hours. 

However, for many real world distributed systems these tensions matter and need to be considered. When dealing with those kinds of systems, I prefer to steer away from approaches that require trading off between multiple desirable properties. I like systems that have both liveness and predictability. I also like systems where no single fallible component has the power to impact the entire system.  So what are some improvements or alternatives?

Localized leaders

In United States, a common debate is how much power should be wielded by individual states and how much should be in the hands of the central federal government. It's a nuanced trade off with many strong opinions on both sides. Luckily, in distributed systems it's almost always better to have smaller blast radii. Instead of having a single leader that operates on the entire distributed system, we could have smaller sub-leaders that each operate on a portion of our distributed system. This can help reduce the blast radius of failures, as well as reduce the amount of work each leader needs to perform, making it easier to maintain liveness in the system.

Idempotent co-leaders

In the real world, deputizing multiple leaders can lead to chaos and indecision. In distributed systems, if we can make the task that is being executed idempotent and atomic, we can skip leader election altogether. Each participating host could execute the task at the same time. If some hosts fail, the system will still make progress as long as there's at least one healthy host. This approach introduces extra load into the system, but on the other hand that load tends to be consistent and predictable. Such systems don't suffer from surprising mode shifts during failures, if anything the load tends to decrease when things fail. I find that to be a desirable property for most systems!

Different architectures

Sometimes, it's possible to make small tweaks to our distributed system to avoid needing centralized logic altogether. Some alternatives that I've run into:

• Using a queue (like SQS) to enqueue housekeeping items as they arise and then processing those using a small fleet of subscribers.
• Using capabilities of the platform to perform housekeeping tasks (e.g. using AutoScaling Groups to replace unhealthy hosts or S3 lifecycles to delete expired objects).
• Using event driven approaches (e.g. using a Lambda to trigger an action when S3 object changes, instead of centrally recomputing all files in the bucket).
  

The mathematics of redundancy

I was at an airport during a recent business trip, watching planes take off and land when I saw the iconic Boeing 747 take off. This reminded me how growing up, 747 was a favorite. There is just something special about a double-decker jumbo jet capable of flying long range routes. But the one factor that endeared me to 747 above all else were its four engines. Perhaps as a foreshadowing of my future interests, it just felt inherently more reliable to fly on a plane with that level of redundancy. After all, even if two of its engines were to fail, the rest would make it possible to safely land the plane.

These days, it's fairly common to use redundancy to build systems that are more resilient than the individual parts they are made from. In the software world, it's a common practice to build a system that runs on multiple hosts, and that can continue operating even if one or more of those hosts fail. Many of us have probably seen the formula that looks like this: 

Where P is the failure probability between 0 and 1, and n is the number of redundant components, or more specifically the number of components a system could lose before a failure would occur. What's important is that the relationship is exponential, and we love exponents when they act in our favor. This means that small increases in redundancy will bring large reductions in failure probability. Or put another way, small increases in cost will bring disproportionally large increases in reliability. Looking at this mathematical model, it's easy to arrive at the conclusion that planes should have as many engines as possible. Especially if you are 14 years old. Unfortunately, the reality is far more nuanced.

The complication comes from the fact that there are different failure modes engineers working on complex systems need to account for. Two of the most important failure modes in redundant systems are correlated failures and cascading failures. Correlated failures are perhaps the easier of the two to understand. As most math textbooks tend to warn, probabilities compound only if the individual events are completely independent, or uncorrelated. In plain language, if there are failure modes that can impact multiple redundant components (for example, servers sharing a single power feed, or a single fuel pump feeding multiple engines), then we lose the benefit of exponentiation. And our mathematical model begins looking more like this:

For systems that have many correlated failure modes, increased redundancy (and cost) no longer increase reliability! 

The second type of failures that should concern engineers building redundant systems is cascading failures. In a cascading failure, a fault in a single redundant component causes other components to also fail. A concrete example of a cascading failure could be an explosion in one engine damaging other critical components of the plane (Qantas Flight 32 was a recent example of that, where luckily the crew still managed to safely land the badly damaged plane). Or a replicated database system, where issues in the replication protocol could cause a failure in one host to propagate to the others. In mathematics, cascading failures can be modeled by complement probability. In other words, our mathematical model looks more like this:

The important observation about cascading failures is that the exponent no longer works in our in our favor! In systems where most failures cascade, adding more redundancy is actually counter-productive. This is an easy intuition to test - if most engine failures risked bringing the whole plane down, then we would want as few engines on our planes as possible.

Conclusion

It goes without saying that this model is just a toy representation of the real world. All models are wrong, some are useful. But what does this mean for building resilient systems through redundancy? For starters, it means that simply adding redundancy into the system does not immediately make it more resilient. Unless the engineers put concerted effort into turning as many correlated and cascading failure modes into uncorrelated ones, the benefit of redundancy on resilience can be greatly diminished, or worse yet, turned negative. And the planes with four engines are not necessarily safer than the ones with two. 

Notable mention

It's hard to have a conversation about redundancy and resilience without talking about MTBF and MTTR (mean time between failures and mean time to repair). The simplistic mathematical model above only holds if the operators replace failing components much faster than subsequent components fail. The math behind that can itself be equally fascinating, especially when dealing with systems where state is involved.

Batching: Efficiency under load

In this post, I wanted to make a quick observation about batching as an underused technique in distributed systems. Systems folks have long used batching as an effective way to increase the throughput of the system. This works by amortizing the overhead of an expensive action (such as device IO or a syscall) over multiple operations. Yet, outside of non-realtime systems, batching is rarely used within distributed services.

The common reasoning is that while batching can increase throughput, it also increases latency. But that doesn't need to be true. A great example of having our cake and eating it too can be found within software defined networking systems. Throughput and latency are critically important for such services, so they are written in systems languages like C, C++, or Rust, rely on lock-free data structures, and tend to use a variety of kernel-bypass techniques to avoid the overhead of sys calls.

A common way such services are implemented is by polling the network device driver queue in an infinite loop (usually with the aid of a framework like DPDK) and then performing their business function on individual packets as they arrive. Most of the time the system is lightly loaded, and the queue depth never exceeds one. However, if the load on the system increases, the arrival rate of new packets can temporarily exceed the rate at which worker threads can handle them. When that happens, the bounded queue builds up a short backlog of packets waiting to be processed. This dynamic isn't that different from how web-servers that make up most of our distributed systems operate.

This is where networking services tend to take advantage of batching. Instead of dequeuing individual packets, the polling thread will try to dequeue multiple packets and submit them for processing as a single batch. Individual stages of the application are then written in a way that takes advantage of such batching, amortizing expensive lookups and operations over multiple packets. Most of the time, the system is lightly loaded, and each batch contains a single packet. However, if the system begins building up a backlog, the batch sizes will increase, improving both latency and efficiency.

This creates a dynamic where the system becomes more efficient as the load increases, and that's such a desirable property! I believe that the same type of opportunistic batching could benefit our distributed systems as well, by improving peak efficiency without sacrificing latency.

Performance and efficiency

The topic of software performance and efficiency has been making rounds this month, especially around engineers not being able to influence their leadership to invest in performance. For many engineers, performance work tends to be some of the most fun and satisfying engineering projects. If you are like me, you love seeing some metric or graph show a step function improvement - my last code commit this year was one such effort, and it felt great seeing the results:

However, judging by the replies on John Carmack's post, many feel they are unable to influence their leadership to give such projects appropriate priority. Upwards influence is a complex subject, and cultural norms at individual companies will have a big impact on how (or if) engineers get to influence their roadmaps. However, one tool that has been useful for me in prioritizing performance work is tying it to specific business outcomes. And when it comes to performance and efficiency work, there are multiple distinct ways they could impact the business, and I think it's worth being explicit about those.

Performance is a feature

We frequently see this in infrastructure services like virtualized compute, storage, and networking. Databases and various storage and network appliances are another example. Performance of these products is frequently one of the dimensions customers evaluate, and that tends to shape product roadmaps. This is also where we see benchmarks that fuel performance arms race, and never-ending debates of how representative these benchmarks are of the true customer experience. In my experience, teams working in these domains have the least trouble prioritizing performance related work.

Better performance drives down costs

We tend to see this in distributed systems a lot. By improving the performance of distributed system's main unit of work code path, we can increase the throughput of any single host. That in turn decreases the count (or size) of hosts needed to handle the aggregate work that the distributed system faces, driving down the costs. One obvious caveat is that many distributed systems have a minimum footprint that is needed to meet their redundancy and availability goals. A lightly loaded distributed system that operates within that minimum footprint is unlikely to see any cost savings from performance optimizations, at least not without additional and often non-trivial investments that would allow the service to "scale to zero."

Better performance improves customer experience

This category of performance projects improve customers experience, typically by improving the latency or quality of some process. Perhaps the best known example is website load times, but any other process where humans (or machines!) wait for something to happen is a good candidate. Yet another example from my youth is video games - the same hardware would run some games much smoother and at higher resolution than others. This was typically the case for any game where John Carmack had a hand!

The biggest challenge in advocating for this category of performance improvement projects is quantifying the impact. Over the past decades multiple studies demonstrated strong correlation between fast page load times and positive business outcomes. However, gaining that conviction required a number of time and effort consuming experiments, an investment that may be impractical for most teams. And even then, at some points the returns likely become diminishing. Will a video game sell more copies if it runs at 90 fps rather than 60? 

Besides time consuming A/B studies, one of the best ways to advocate for these kinds of performance improvement projects, in my experience, is to deeply understand how the customers use your product. Then you can bring concrete customer anecdotes to make the case.

Better performance reduces availability risks

These types of performance projects tend to come up anytime we have performance modes in our system, that is the behavior where some units of work are less performant than others. An example of that could be an O(N²) algorithm, which sometimes faces large Ns. Such inefficiencies in the system create a situation where a shift in the composition of various modes can tip the system over the edge, creating a DOS vector. Throttling and load shedding more expensive units of work is one common approach, however improving their performance to be on par with the rest can be even more effective. 

One way to identify and drive down these kinds of performance modality issues is by plotting tail and median latencies on the same graph, or perhaps even using fancier graph types like histogram heatmaps. Seeing multiple distinct modes in the graph is a telltale sign that your system has such an intrinsic risk.

Performance has security impact

Security sensitive domains are unique because more important that the raw performance is the lack of performance variability. This is because any variability can expose potential side channels to the attacker. For example, if an encryption API takes different lengths of time depending on the content being encrypted, then a determined adversary can use that timing information as a side channel to infer information about the plaintext that is being encrypted. Luckily, very few teams are writing code with that kind of blast radius. The amount of expertise and considerations that go into getting such code right is truly mind boggling!

Conclusion

Like all classifications, this one is imperfect. For starters, a project can fall into multiple categories. For example, improving P50 request latency on a web service is likely to both improve customer experience and also service costs. But like they say, "All models are wrong, some are useful", and hopefully this model will be useful in helping folks tie performance work to desirable business outcomes.