On the morning of August 1, 2012, Knight Capital - then one of the biggest market maker on Wall Street - deployed new code to its high-speed trading system—but one of eight production servers never got the update. That lone machine started running an old, dormant module called “Power Peg,” flooding the market with errant trades. In just 45 minutes, Knight amassed nearly $7 billion in accidental positions and lost $460 million.
It was one of the most expensive software failures in Wall Street history—driven by a rushed deployment, missing checks, legacy code left behind, and no clear plan to roll back. This is the story of how a routine release turned into a company-ending event—and what leaders today can learn from it.
Founded in 1995, the Knight Capital Group was the largest market maker in US equities. Knight’s electronic trading group covered more than 19000 securities and it’s high frequency trading algorithms processed a daily trading volume of $20 billions which was 15% of the volume of trades on NYSE and Nasdaq in 2012.
The Lead up
On July 3rd (less than a month before the incident occurred), the SEC approved the Retail Liquidity Program (RLP). The RLP was a New York Stock Exchange initiative launched to improve trade execution for retail investors. The goal? Give mom-and-pop traders better prices by matching their orders with market makers willing to offer price improvement — even fractions of a penny better than the national best bid or offer (NBBO).
Knight Capital, as a major market maker, wanted to participate. To do so, they had to update their systems — including SMARS — to handle the new RLP order types and routing logic.
SMARS
SMARS stood for Smart Market Access Routing System — Knight Capital’s proprietary high-speed order router. Its job was to take in “parent” orders (typically from clients), break them into smaller “child” orders, and then send those to the right stock exchanges for execution. Think of it as an air traffic controller for millions of trades per day, deciding how and where to route orders for best execution across multiple venues.
The SMARS production system ran on a cluster of eight servers, all of which were expected to run the same version of the code at all times to ensure consistency and reliability.
SMARS wasn’t a trading algorithm itself — it didn’t decide whether to buy or sell. But once an order came in, it handled the how and where, executing orders fast and at scale. It was a critical piece of infrastructure that had to be airtight.
Lurking inside the SMARS code base was a deprecated legacy piece of software called the “power peg”.
Power Peg
Power Peg was a legacy order execution logic embedded within Knight Capital’s SMARS order router. Its original purpose was to dynamically and aggressively peg child orders to the prevailing market price in an attempt to rapidly fill a parent order. The logic would monitor execution progress and continue sending child orders to various venues until the total filled quantity matched the parent order’s intended size.
In normal operation, a cumulative quantity counter, a circuit breaker of sorts, tracked how many shares had been executed against each parent order. Once the desired quantity was reached, the router would stop dispatching further child orders.
The last time the power peg code was used was in 2003, Knight hadn’t used Power Peg in 9 years. This deprecated code was behind a feature flag that still lived quietly on the production servers since 2003.
Repurposing a feature flag
Fast forward to 2012: engineers working on support for the NYSE’s Retail Liquidity Program (RLP) decided to repurpose the same flag used in the power-peg code to trigger the new RLP routing logic. The assumption was simple — Power Peg was dead, so the flag was available.
In hindsight, reassigning an old feature flag without fully deprecating the legacy power-peg code was the decision that lit the fuse.
The faulty deployment
The deployment process at Knight Capital lacked basic safeguards and automation — and that’s exactly how one of the eight SMARS servers was left running old code.
Specifically, the rollout of the new Retail Liquidity Program (RLP) code was done a manually and in stages (few days before 1st August), with individual technicians copying the updated code to each server. But during this process, one server was accidentally skipped — and crucially, Knight had no second person verify the deployment. There was no deployment checklist, no script to confirm version consistency across servers, and no automation to catch drift between environments.
As a result, seven servers got the new RLP logic, but the eighth continued running the outdated version — one that still included the legacy Power Peg code.
Event Day - The “Knightmare”
At market open 9:30AM EST 1-August-2012, seven out of the eight SMARS servers started correctly processing the RLP orders. The eight server which was not updated with the latest code, inadvertently triggered the deprecated power-peg code because the same feature flag was repurposed.
Within seconds, this rogue server started submitting millions of orders spanning 212 stock tickers. By the time Knight personnel started noticing something abnormal, 30 minutes had passed.
Fix made the problem worse
After engineers realized that one of the eight SMARS servers hadn’t been updated with the new Retail Liquidity Program (RLP) code, they tried to revert all servers back to the old version to regain consistency.
But here’s the catch:
The old version of the code still had the Power Peg logic wired to the reused feature flag — and now, all eight servers had that same bug.
So instead of isolating the issue to one server, the revert propagated the defective behavior to all servers in production.
The volume of erroneous trades increased dramatically after the revert. What had been bad — one server sending millions of unintended child orders — turned into a full-system meltdown, with all SMARS servers aggressively placing orders due to the now-active Power Peg logic.
Circuit breaker failure
Now, we discussed earlier that power peg had a circuit breaker feature which was a cumulative quantity counter and made sure that the child orders only filled the total quantity of the parent orders. Once, the threshold was reached, circuit breaker ensured no more orders were submitted. But as part of code refactoring in the past, the circuit breaker part of the power peg code was moved out.
So, the absence of the circuit breaker resulted in the power peg code submitting orders without any limits. We are talking millions of market buy and sell orders every minute.
The resolution
Around 10:40AM, more than an hour after the issue started, the problem was isolated and the servers were taken offline. This stopped the high frequency order submission.
The Aftermath
During the one hour the issue lasted, the business impact was the following -
More than 397 million shares traded
A net long position of roughly $3.5 billion across 80 stocks.
A net short position of roughly $3.15 billion across 74 stocks.
A net loss of $460 million
Public Market reaction
The stock price plunged 75% over the next two days in fears of Knight Capital going bankrupt.
Regulatory impact
Knight Capital filed a request with SEC for cancelation of the unintended trades but the then SEC chair Mary Schapiro declined the request citing poor risk controls at the firm.
SEC also launched an investigation and filed charges.
Regulatory impact
Knight eventually merged with Getco in 2013, forming KCG Holdings, effectively ending Knight as an independent firm.
Lessons from the Knight Capital Meltdown
If you're in tech, finance, or anywhere near software that moves money, this incident is a masterclass in what not to do. Here are some of the biggest lessons we should take away from it:
Best practices with feature flags
Don’t Reuse Old Flags
Never recycle a flag without confirming what legacy logic it’s tied to. Old flags might still be hooked into deprecated code, which can reactivate behavior you don’t expect — like Knight’s "Power Peg" strategy.
Set Expiration Dates
Flags should have an expiration date or be reviewed regularly. Knight’s original power peg flag was in production long after (9 years) it had served its purpose. Temporary flags can become permanent landmines if left in place too long. This practice ensures you are not leaving dead code in production too long.
Test Flag Combinations
Complex systems with many flags should be tested for key permutations. A rarely-used flag combo might trigger bugs in “zombie code” no one remembers exists. Also it is critical to test the flag in control or OFF configuration which can result in interesting issues at times.
Automated Deployments and Version Control
Deploy Atomically Across All Servers
Manual deployments were clearly one of the key misses that caused the Knight Capital disasters so clearly we should have fully automated deployments. And if we want to have run two versions of the code we should use different feature flag treatments for different customer cohorts.
Verification of post deployment consistency
Knight’s team didn’t realize one server missed the update. A simple consistency check would’ve caught it.
Run post-deploy health checks that confirm all servers are running the same code version, commit hash, and config.
Code Reviews and Approvals
The reused feature flag and associated logic likely passed through with little scrutiny.
Require approvals for code changes involving flags, legacy code paths, or deployment logic. These are high-risk areas.
Rollbacks are not always safe
While in most cases rolling back to the previous known working version is a good practice, we should not always assume it will fix the problem. Knight capital’s engineers rolled back to the previous version but they ended up making the problem worse. So, it is prudent to consider any potential failure modes in case of rollbacks.
Monitoring & Alerting best practices
Alerts should have clear owners and action
In the case of the Knight capital disaster, about 1 hour before the market opened, there were email alerts that were sent to a distribution list cautioning the potential issue but nobody looked into them. While configuring alerts, we have to make sure there are defined with the right owners and clear action expected when they get triggered.
“Worst case scenario” alerting / anomaly detection
During the incident, Knight’s high frequency trading system triggered more than 300 million orders in a few minutes. This was a highly anomalous situation. But it took more than 30 mins to be detected. We should build alerts for these type of worst case scenarios where we configure alerts for disasters. For example for an e-commerce application, we should have alerts if order volume goes to zero but also if the order volume unexpectedly doubles for a duration.
Circuit Breakers Must Be Real and Functional
Knight’s Power Peg logic did have a circuit breaker — but it was coded to only activate under a very narrow, inapplicable condition, which never triggered.
Your fail-safes must be tested, reliable, and trigger under real-world failure scenarios, not just theoretical ones.
Unused Legacy Code Can Be a Landmine
Removing unused legacy code is one of the highest-leverage ways to reduce risk, improve maintainability, and prevent disasters like Knight Capital. But it has to be done safely and systematically, especially when that code has been dormant for years. This is a type of tech debt that leaders would need to prioritize vs other feature work to reduce engineering risk.
QA Sign off & Go-No-Gos before releases
The Retail Liquidity Program (RLP) was launching for the first time and it was a critical release for Knight Capital. But there were clearly many of the above best practices missing that led to the disaster that eventually led to the company’s bankruptcy. Having explicit QA sign off and Go-No-Gos before major releases helps highlight such blindspots and reduces chances of catastrophic problems. I write about how to evaluate whether a project is ready for production release in another post here.
Culture and Organizational Factors: When Process Fails, Culture Must Catch It
What happened at Knight Capital wasn’t just a glitch in the code—it was a failure of organizational discipline and culture.
The technical trigger was simple: legacy code, mistakenly reactivated via an old feature flag, ran on just one of eight servers. But the real question is: how did a high-stakes deployment like this happen with no code review, no version control enforcement, and no system-wide validation?
The answer lies in culture. There was no strong process for escalation, no clear ownership of deployment hygiene, and seemingly no environment where engineers felt empowered—or expected—to raise concerns. A critical piece of obsolete functionality was still lurking in production, and it got triggered not by malice or even incompetence, but by a rush to ship, siloed knowledge, and no cultural safety net to catch a bad decision before it hit the market.
During the 45-minute crisis, the slow response exposed another weakness: unclear roles and no practiced incident protocol. In fast-moving systems, speed must be backed by preparedness. Knight had one, but not the other.
Executives need to take note: technical debt is a leadership issue, not just an engineering one. The best engineers can’t save a system from failure if they operate in a culture where asking “why are we doing this?” isn’t encouraged—or where shipping fast matters more than shipping safely.
Wrapping Up
The Knight Capital incident wasn’t just a $460 million loss—it was a wake-up call. A reminder that in complex, high-speed systems, failure is rarely caused by a single bug. It’s the result of overlooked processes, brittle systems, siloed knowledge, and cultures that don’t encourage scrutiny.
Executives often ask how to “move fast.” The better question is: how do we move fast with control? Knight’s engineers were smart. Their systems were fast. But they were operating without circuit breakers—both literal and organizational.
In today’s world of continuous deployment, feature flags, and real-time systems, the risks are higher than ever. But so are the opportunities to build things right.