On HFT (Part II): Bugs, Features, and Aggressive Incompetence

On High-Frequency Trading (Part II)

Since my last post on HFT, Michael Lewis has released a new book heavily criticizing the field, even going as far as saying that the markets are “rigged”. Explosive debate soon followed. For an accurate and colorful analysis of Mr. Lewis’ book, see Scott Locklin.

This post focuses on a couple specific programming and infrastructure aspects of HFT. After a brief anecdote explaining general challenges in coding, I address the following claims:

  1. That the SIP feed engages in “timestamp fraud
  2. That “quote stuffing” and “spoofing” are mainstays of HFT

Cliff’s Notes: it doesn’t; they aren’t (there is always a non-zero probability that I err. I reserve the right to correct and be corrected.)

A Saga: The Updates

The updates regarding Nanex’s various retractions and childish behavior have been moved to their own section, appropriately titled: The Hunsader Follies

Below contains the original post

A Sandwich

Software does not know the programmer’s intent, it simply does what it’s told to do. I first encountered the frustrations embedded in this truism in my high school AP Computer Science class. On the first day of class, the animated and brilliant Mr. Percival asked us to write out and hand him specific instructions for making a peanut butter and jelly sandwich. After all of us would-be clever boys had turned in our papers, he whipped out a knife, a loaf of bread, and jars of jelly and peanut butter. He read aloud from the top sheet: “1) grab knife, stick into peanut butter jar.” Not instructed to remove the lid, Percival promptly harpooned the jar.[1]

Many students have gone through something similar to the PB&J exercise which visualizes the importance of thoroughness and specification when coding. It also brings up an important point about assumptions. The programmer must be acutely aware of what can and cannot be assumed. Take the knife: would it have been possible for Mr. Percival to grab the knife by the blade? How do we know? The instruction “grab the knife” can fail even if we previously witnessed him grab the handle! This is how bugs happen.

Bug: when a piece of software behaves in a manner other than that which was intended.[2]

All code has bugs. All code has the potential to fail to do what we want. Industry averages of coding errors are in the range of 15 – 50 per thousand lines of code. Production-ready trading systems are on the order of hundreds of thousands to a few million lines of code. Concepts like peer review, bug tracking, best practices, and abstraction help to catch and reduce these errors before the code leaves development. But in any release cycle, there will be bugs. Often they present only in rare conditions — called edge cases — with dire consequences, like Knight Capital’s $440-million loss in under an hour.

Large software projects involve several developers all working on individual pieces of the digital pie. Frequently these pieces overlap which means people use and test each other’s code. Sometimes one developer isn’t quite sure of the expected behavior of another’s code, resulting in a brief intraoffice game show: Bug or Feature? This is where a lot of normal use cases and edge cases are corrected. I bring this up because much of the recent commentary on HFT mistakes features for bugs, or bugs for features. Listen…

A Blowhard

While Mr. Lewis sufficiently toured his book, other self-promoters rapidly took to social media to piggyback on the #HFT shenanigans — notably the aggressive and outspoken Eric Hunsader, founder of the data services company Nanex. Best known for making pretty pictures using his company’s charting software, he’s also been spinning yarns about HFT for a few years. The good news is that Eric Hunsader doesn’t know what he’s talking about. The better news is that I get to explain why. Here’s a typical specious claim:

On October 16, 2013, starting about an hour before the European open at 3:00 ET, an unusually large buy order appeared near the top of book in the December 2013 eMini (ES) futures contract: 5 lots of 1000 contracts, or orders to buy 5000 contracts total. That is a huge order, even during regular trading hours.

A Nanex "Panther"

About 1 minute before 3:00, the buy orders were moved up to the top of the book. Trades that appear at this level were executions against other existing orders in the book (and not against the large buy orders). Just before selling began in earnest at 4 seconds after 3am, all 5 orders, comprising of 5,000 contracts, were canceled. This behavior is similar to the Panther case. It shouldn’t take the regulator more than an hour to figure out who did it, and a day to find out the intent. We’ll wait.

Recap: over the course of an hour, a large buy order was placed away from the best bid and slowly moved up until it became the best bid. It was then cancelled three minutes later.

The most obvious question is: what does this have to do with HFT? There’s nothing “high-frequency” about it. Apparently in Hunsader’s mind, placing a large order and leaving it there for an hour now qualifies as high-frequency trading. Anytime you place an order, it has the chance to be filled. Automated algorithms have to be very careful about the sizes they place, because they have to assume they can be filled. As a result, algorithms have very sensitive risk metrics. It is entirely probable that there was a good ole fashioned human being working this order. Yet he refers to it as an “algo” out of either ignorance or malice.

The more important problem is with his claim of events. On the above image, Hunsader wrote “these small executions are against other buy orders.” He’s saying that the person with the 5,000 contract order did not receive any trades at that price and never intended to. The problem: The CME does not disseminate individual order data. Therefore Hunsader has no way of knowing which orders were filled; he is flat out lying.

Let me try to explain this concisely. When receiving a direct feed from the CME, there are two channels that have market data: the recovery (snapshot) channel which contains full order books on each update, and the incremental refresh channel. The incremental refresh channel is the fastest and highest granularity. Each update either contains instructions to add or remove some volume of contracts at some price level, or informs you that a trade has occurred at some price level and which side was the aggressor. It does not divulge information about individual traders’ orders or announce which orders have been filled.

These are fundamentals that anyone who has worked with a direct feed to the CME would know. Yet Hunsader does not know this. Instead he routinely makes similar claims about information he couldn’t possibly have. He calls something spoofing because he’s “never seen it execute.” I say base rate fallacy. He doesn’t have the ability to see anything execute.[3]

A Feature

Next in Nanex’s campaign of misinformation is the claim that the SIP feed (the consolidated information feed that all equities exchanges send their quotes to) engages in “timestamp fraud.” Last week, Hunsader posted an animated graphic with commentary to the Nanex blog purportedly showing that the SIP is doing something “illegal.” Here’s his argument:

… In the animation below, note that the information sent to the SIP has to travel significantly farther distances (40 miles vs 1000 feet), on a slower network (1 GBps vs 40 GBps) with a protocol that adds more latency (TCP vs UDP) than the same information sent to the direct feed. Sometimes this latency on the input side of the SIP shows up in SIP data as fantaseconds (a term we coined to describe trades printing before quotes). …

2014-04-13 15_34_35-Nanex ~ 04-Apr-2014 ~ Direct vs SIP Data Feed The animation also shows something that many aren’t aware of: the original timestamp gets stripped, and replaced with a fresh timestamp when the SIP transmits it to a subscriber! Watch the timestamp in the box get stripped when it enters, and replaced when it leaves, the circle labeled “SIP Tape A”. Keeping original timestamps is crucial for constructing audit trails, or for detecting system delays …

Recap: exchanges send information to SIP using a slower, more latent method and then the SIP replaces the original timestamp with a new one before sending it to the subscriber.

First we should go over Hunsader’s very disingenuous remarks about the methods exchanges use to disseminate information…

(Correction 4/17: An earlier version of this post incorrectly nit-picked about Hunsader’s incredulity with respect to bandwidth. While it had nothing to do with the main point in this section or the overall fraud committed by Nanex, I was unfairly critical of one of Hunsader’s points — and also wrong. The venerable Dave Lauer set me straight with respect to the differences in overhead at the switch level. Thanks Dave!)

But Hunsader also notes that the protocol for the SIP feed (TCP) is more latent than what exchanges use for their direct subscribers (UDP). There is a simple and non-threatening reason for this: TCP has flow control and error correction, UDP has neither. The SIP feed aggregates many exchanges together, it must ensure that all packets provide correct information. When a direct subscriber to a UDP feed drops a packet, it must initiate a time-and-bandwidth-consuming recovery protocol. This information is inconvenient to Nanex’s narrative so he ignores it, again out of either ignorance or malice.

The main point of the argument, that the “original timestamp gets stripped, and replaced with a fresh timestamp when the SIP transmits it to a subscriber” is somehow fraudulent behavior, is yet another outright lie. Each packet from each feed from each exchange is numbered with a unique number (to that feed) called a Sequence Number. This number allows the receiving server to verify that it has received every packet in the correct order. The timestamp of the packet is the time the sequence number was generated. In the context of the SIP feed, it aggregates information from many different exchanges. The timestamp of when the packet was generated at the exchange does not matter. It matters when the SIP feed received it and generated its own sequence number before sending to subscribers. If the SIP feed did not issue new timestamps and sequence numbers, subscribers would receive packets with out of order timestamps. That would be illogical.

Further, when Hunsader claims that “keeping original timestamps is crucial for constructing audit trails” he completely misunderstands how audit trails work. Every message that is decoded or encoded by an exchange, or by any server communicating with an exchange, is written to an audit trail. This means that by definition, the SIP feed does have an audit trail of the original decoded message with timestamp, and also the message it was turned into before broadcasting to subscribers. Again, Hunsader would easily know all of this if he actually worked with these feeds. My suspicion is that he has not.

A Bug

When a person or an algorithm rapidly places and cancels an order, it is often called “flickering.” Nanex coined the term “quote stuffing” when they found instances of certain financial products having a higher than normal rate of messages per minute. Without a second thought (or proof), they immediately labeled these as acts of intentional manipulation. The term caught on with journalists (obviously incapable of questioning the logic) and spread until it reached the SEC who investigated it in 2010 and turned up nothing. Despite this, Nanex frequently repeats, “the practice of spoofing, or sending in fake orders in order to gain information about what other investors, traders and other algorithms are doing, is the corner stone [sic] of most High Frequency Trading strategies.” But as we’ve seen, Hunsader’s imagination is far more active than his fact-based reasoning. There is a simple explanation for nearly all instances of flickering: bugs.[4]

Example Market
Example Market

Flickering is primarily caused by feedback loops. Example: imagine you have a strategy that wants to join (i.e. place an order on) whichever side of the Best Bid or Best Ask is showing fewer quantity available. In the above example market, that would be the Best Ask (price 16127 with a quantity of 1). Now let’s say your strategy is set to place orders with a size of 4. Your strategy places the order for 4. Then you receive a new order book showing the price 16127 has a quantity of 5. This is higher than the quantity of the best bid. So what does your strategy do? Does it cancel the order because it’s on the bigger side? Does it place another order on the bid? If you haven’t put in safeguards, you can easily get unwanted behavior.

That seems like a simple example (it is), and most strategies know to remove their own orders from calculations. But it gets complicated very quickly. Now imagine the order your strategy placed is “in flight” (i.e. it hasn’t yet been acknowledged by the exchange). Before you get acknowledgement, you receive another order book showing the Best Ask with a size of 10. What does your strategy do? Does it send an order for the Best Bid? When does it stop sending orders on the weaker side?

What happens if your order is immediately filled, before you receive the corresponding orderbook update?

Now imagine you trade more than one financial product. Your strategies might be using information from multiple sources. What happens if there’s a “bad” print at a abnormally high price? What if one of those products goes into a trading halt? Your trading system is likely multi-threaded. What happens if there’s a read-write collision?

One of the biggest sources of flickering is in how a system calculates risk. A trading system which manages a large number of products uses many factors in how it calculates risk, including: its current position in each product, the number of its own orders in each product, and various statistical characteristics of each product and how they relate to each other. As any model increases in complexity, the number of edge cases usually increases geometrically.

Hanlon’s Razor: “Never attribute to malice that which is adequately explained by stupidity.”[5]

Update 4/24

Former HFT programmer Chris Stucchio has posted detailed walk-through on how “quote stuffing” happens unintentionally. See: Quote Stuffing is a Software Bug.

The End

I’ve previously written about how easy it is to make mistakes, and how careful we must be about checking both our data and our assumptions. Eric Hunsader clearly does not take care with his assumptions. He continually reasons from obviously flawed information. I can’t speak for his data, but if he’s quick to invent new words like “fantaseconds” rather than make sure he’s done things correctly, I’d wager that Nanex makes countless mistakes there too. But the good news is: a man as aggressively incompetent as Hunsader is only good for media interviews. Regulators and traders alike give no credence to his lunatic ranting. And now you know why you shouldn’t either.



Footnotes    (↵ returns to text)

  1. That was probably the most important class I’ve ever taken, taught by the greatest teacher I’ve ever had. I went to an above-average school which constantly reminded the students that they were above-average. This bred a somewhat bratty atmosphere where we frequently challenged teachers or simply became bored when we felt under-stimulated. Percival also taught physics and astronomy, which when paired with computer science has the potential to be the trifecta of boredom. But no one knew how to work a classroom full of hyperactive sub geniuses like Percival. He had a satisfying answer for pretty much every oddball inquiry. Because of a genuine love for each subject, he’d already thought about all of the crazy questions we could come up with. The two years of programming I had with him are the reason I am able to work in this job that I’ve been doing for the past 5 years.
  2. While the term “bug” predates computers, the first computer bug was in fact a moth that became trapped in a relay. It was removed and taped to the error logbook. (link)
  3. Hunsader has a number of tells. He’s obsessed with the word “panther,” appearing in many of his ridiculously concocted, HFT blast pieces. (Panther Energy Trading was a firm that really got fined for order spoofing last July. Since then, everything he sees is a Panther.) In lieu of analysis he floods each post with 30+ images. His “research” is of course never peer reviewed, but you can purchase his data and software. And the email link on the bottom of every page is pr@nanex… I’ll leave the personality disorder speculation to you
  4. Yes there are some instances of “spoofing” that are legitimate manipulation. This is why the SEC frequently conducts audits. This is why the CME imposes messaging limits. There are organizations that are doing a good job of making sure this doesn’t happen, or that people are penalized and barred from trading when it does.
  5. And this is where I raise my hand and admit that I am often guilty of stupidity. Claiming knowledge of markets is an ego-driven exercise for academics and con artists. Actually participating in markets is humbling.