August 2, 2010
Horseflesh and Hypocrisy

T. Boone Pickens, from his autobiography The Luckiest Guy in the World:

I believe the greatest opportunity lies in a free marketplace. There are powerful forces afoot trying to restrict that freedom in the interests of the vested and already wealthy.

T. Boone Pickens, in congressional testimony on a bill to prevent the slaughter of horses for food:

The whole thing, it’s a boondoggle on the American people…People that are for the slaughter should be forced to go down on that kill floor…The brutal slaughter of horses for consumption by wealthy diners in Europe and Japan cuts against our moral and cultural fiber — it’s just plain un-American.

Remember, if they can come after the horse slaughterers, they can come after the hedge funds. So if you really believe in free markets, have some horse today!

8:00am  |   URL: http://tmblr.co/ZtlAMyrEH8w
(View comments) 
Filed under: markets 
July 23, 2010
Sarah Connor Comes to Wall Street

So if you haven’t read it, Joseph Fuller’s The Terminator Comes to Wall Street is a well-written policy-advocating gloss. However, its technical merit is exactly what you’d expect from the PBK pimps whose most significant recent article involved an English professor nicknamed “Cockmaster D” advocating for “erotic intensity” between professor and student. I’ll stick to my problem sets, thanks.

So, anyway, the article is bifurcated nicely between well-meaning policy suggestions (“Executives should take more time to understand what’s going on”) and technical ignorance (“At the cutting edge of modeling science, researchers are trying to move away from relatively crude rules-based models toward models that approximate the processes of human reason.”) The crux of Fuller’s argument is that computers are bad — and they’re bad for three reasons. Briefly:

  1. Modelers don’t understand markets - especially the psychology of markets.
  2. Managers don’t understand modelers - and even if they did, computers can easily get too complicated for anyone to understand.
  3. Models don’t understand each other, especially how they interact with one another.

I’ll move through these in reverse order. Number three is an easy one — you could make an identical argument for human traders. Computers don’t introduce any kind of systemic risk that isn’t already there — we still have a multi-agent system with a bunch of profit-seekers interacting with one another. It’s an unfair standard to insist that computers have to have some kind of global, system-wide knowledge while people can trade ignorantly and unfettered.

Number two is a sound point about better management being needed, but Fuller loses me here:

The problem here goes beyond comprehension. Even if the executives were Quants, they might well not understand as much as they would like about the programs running their businesses. The models themselves—and particularly the interaction among models—has grown so complex that it may have become impossible for any human to fully grasp the types and volumes of derivatives traded in this way or to predict how the models will interact with each other.

This is a strange point to make, because if the models were simple enough to fully grasp, then there really wouldn’t be a point for them in the first place. Put another way, the whole reason you’re using computers operating off of complex relationships is to make something you can’t fully understand.

And Fuller’s first point, about computers not understanding “psychology”, is irrelevant. If real phenomena like trend-following or whatever exist meaningfully, they’ll be found and exploited. Instead, this line of reasoning smacks of an NBA announcer declaring a player who happens to hit a couple jump shots to be “on fire”. It’s people that see patterns and symbols when they’re not actually there, not computers that have trouble detecting regularities.

Ultimately, I think much of Fuller’s argument is based around misconceptions from Classic AI: computers as deterministic automata, blindly and unceasingly following some fixed source code, with human intelligence as the ultimate standard. It sounds like a pre-Moravec’s Paradox view of the world, where anything people find hard must be a “true test” for computers. In reality, it’s entirely appropriate to expect that when there are well-defined objectives and computers have the complete access to the policy state (no falling over!), computers will outperform people. Comparatively, making great trading bots is easy. Opening doors is hard.

8:00am  |   URL: http://tmblr.co/ZtlAMyodnbd
(View comments) 
Filed under: AI markets 
July 9, 2010
The Real Money HSX is Bad Policy

Recently the HSX, a longtime favorite of prediction market advocates, was approved to start trading in real-money derivatives. The HSX using artificial currency has been used for many years to correctly estimate box-office receipts among a diehard but non-professional crowd; along with wikipedia, I think it’s one of the best examples of how “the crowd” can perform excellently without any financial inducement.

The chief mover behind the change was Cantor Fitzgerald, the new owners of the HSX. Cantor seems to be using its cachet as one of the largest 9/11 victims to push the limits of the “gambling versus trading” debate. I think Cantor’s motives here are transparent — they want to be playing bookie, and ideally exclusive bookie, to as many markets as possible. It doesn’t matter to Cantor whether those markets are on government debt, sports betting, or movie ticket sales.

The largest obstacle in moving to real money was the approval of the CFTC — a screening process by real adults who wear suits. There’s quite a bit of information on the HSX proposal scattered around the CFTC website, including all sorts of briefs and drafts and reports. Of all the stuff of it I’ve read, the MPAA brief against the real-money HSX made the most sense to me. Their two points that struck hardest for me were

  • The contracts will not be used for hedging. The most prominent story about why these contracts should be traded is that certain parties (mostly the studios but also the actors and movie houses, I suppose) have exposure risk to the success or failure of a film. Of course, on any contract you can always construct a theoretical hedger. Even for betting on a coin flip or a dice game; maybe someone has a substantial psychological revulsion to seeing George Washington’s face or the number four. However, the MPAA argues that the parties with material hedges on these events will not participate in their trade. I think given their status as an industry trade group this is a persuasive argument. The MPAA further alleges that because these contracts will not be used for hedging, they will only be used for gambling, and are therefore in violation of sundry statutes scattered about the states.

  • The data the contract settle on are manipulable. The data used to settle the contracts explicitly state that they are not intended for use in settling financial contracts. This is ultimately because they’re estimates based on assumptions, not a nitty-gritty verified and audited source.

Ironically, the one argument I don’t buy at all in the MPAA’s brief is that the HSX will impact the film-making industry at all. At worst, the markets will fizzle and have no impact, and at best they could provide free exposure for films. Then again, the MPAA doesn’t exactly have the best track record with knowing what’s good or bad for it.

Now, this all brackets the issue of whether the real-money HSX will actually work. I view it as very unlikely that the trading volume on these markets will come anywhere close to a significant fraction of movie revenue, and I view it as even more absurd that the HSX could somehow provide and IPO-style market for financing filmmaking outside of the traditional studio system.

And what we seem to be coming to, then, is a fundamental existential question. I ultimately believe that a real-money HSX is unnecessary. These markets are supposed to provide a way to hedge risk on people seeing films (of course, shouldering such calculated risks should be the whole purpose of a studio in the first place, but regardless…), but if I were a studio with exposure risk on box office returns I would shop that risk around to creative hedge funds with good models willing to take an equity gamble. The two big things a market provides, consolidation and anonymity, are just not necessary in this scenario. On every contract there’s just one movie and one seller of exposure risk.

Given that I’ve done some research in (and implementation of) prediction markets, my position might be a surprising one. Ultimately though, I think that speculative trade should always be secondary to real concerns; speculative trade for the sake of speculative trade is silly and non-productive at best and systemically dangerous at worst. The HSX decision is a bad policy and a worse precedent.

8:00am  |   URL: http://tmblr.co/ZtlAMyl2jQ_
(View comments) 
Filed under: markets 
July 2, 2010
Bets vs. Hedges, revisited

From the Daily Finance article on Cantor Fitzgerald becoming a bookie:

(Cantor Fitzgerald employee Andrew) Garrood seems less concerned. He makes the in-running operation sound easy for a company like Cantor, which he says handles $150 trillion in Treasury business each year. “This is an application of what Cantor has always understood,” he says. “It’s just dressing up the emperor in new clothes.”

He wouldn’t dare be implying that the finance industry exists to facilitate betting, would he?

4:36pm  |   URL: http://tmblr.co/ZtlAMyjTgsi
(View comments) 
Filed under: markets 
June 27, 2010
The Terminator Comes to Wall Street

10:38pm  |   URL: http://tmblr.co/ZtlAMyiMHZz
(View comments) 
Filed under: AI markets 
June 24, 2010
Flash Crash Analysis

The language is a bit hyperbolic and unscientific, but it’s certainly an interesting dataset.

I’ve been busy; real posts will return soon.

4:13pm  |   URL: http://tmblr.co/ZtlAMyhemRP
(View comments) 
Filed under: markets 
June 10, 2010
The Push to Obscurity

What struck me the most about Fab Fab’s emails wasn’t their maudlin tone or his low-grade ethical wrangling, it was this line:

When I think that I had some input into the creation of this product (which by the way is a product of pure intellectual masturbation, the type of thing which you invent telling yourself: “Well, what if we created a “thing”, which has no purpose, which is absolutely conceptual and highly theoretical and which nobody knows how to price?”) it sickens the heart to see it shot down in mid-flight…

What’s going on here? Why would anyone intentionally create something impossible to value? I think the answer is that it enables larger gains through speculation, and I’ll extrapolate and defend that position in this post.

So first, okay, we need an answer as to why trade occurs in the first place. Sustainable trade is positive-sum: both people benefit. An easy test to see how positive-sum the trade you’re doing is is by counterfactually jiggling the price around. I’d pay 20% more for bananas, and my local Giant Eagle would (and occasionally does) sell them to me for 20% less. Designer clothing is positive sum. Knockoff designer clothing is positive sum. Cigarettes and liquor and drugs in general are very positive sum.

Positive-sumness is not necessarily related to how repugnant a market is. Trade in horse flesh or marijuana would be very positive sum, but is outlawed. Numbers games and lottos are arguably zero-sum, but are run by the government.

Trade that is very positive sum — food, drugs, clothes, sex — essentially arises spontaneously. It doesn’t need a team at Goldman Sachs to create a new financial product for. And because of that, it’s reasonable to propose that all legal forms of trade up to some threshold of positive-sumness are currently taking place. That’s not the same as saying that capitalism is at the end of its rope or whatever, this isn’t a Marxist endorsement: there (probably) is a long tail of slightly positive-sum opportunities (tiny hedges all the way down) that add significant weight to overall well-being. One of these tiny hedges, somewhere along the line, is the ABS-CDO-squared Fab Fab was selling to Belgians.

Now if there’s not a lot of welfare gained in a transaction, there’s not a lot of profit that can be skimmed from it. But if trade is speculative — if it might not benefit both parties but result in one party losing and the other winning — the incentives are completely different. If your model of the world is better than your counterparty’s, you’re no longer limited by the narrow window of positive, welfare-creating value, but instead to a much larger world of profit. And the simplest way to make people disagree on the value of an item is to make it really hard to value. Obscurity emerges as a vehicle to get people to speculate against one another.

Speculative trade, of course, is very much unlike positive-sum, sustainable trade. And though Goldman will affirm endlessly that these trades were (positive-sum) “hedges” rather than (zero-sum) “bets”, it seems tautological that these trades were unsustainable: they are no longer being sustained.

4:52am  |   URL: http://tmblr.co/ZtlAMyejq2b
(View comments) 
Filed under: markets 
May 25, 2010
Are Prediction Markets Engines or Cameras?

Before I explain the metaphor, allow me to present the Health Care Reform Hustle. Let’s go way, way, back to the beginning of 2010, where the long-delayed bill on HCR is about to pass in the House, held together by a rickety Democratic coalition. The Hustle proceeds as follows:

  1. Spend a few thousand dollars consistently shorting Intrade’s contract on HCR passing successfully.
  2. Get the story picked up by conservative bloggers, which should be an easy sell because it combines two things they already love (namely HCR failing and markets).
  3. Get the story picked up by the wider media.
  4. Use the increased pressure to lobby the weakest members of the Democratic coalition to change their vote (“See? It’s all falling apart!”)
  5. Watch HCR flounder.
  6. Make tons of money on your short positions.

Now, of course, this actually happened, and it fell apart somewhere between steps two and three. Here was Gawker’s take on the issue:

I hate to break it to the manipulator, but making the chances of HCR passing seem lower on an Internet trading website does not actually have a real-world effect on the chance that HCR will pass. All you actually did is briefly make a bunch of conservative bloggers feel pangs of joy in their cold, cold hearts.

Back to that metaphor. Cameras take an objective (let’s not get all Errol Morris here) picture of reality. Engines create their own reality. The metaphor, as applied to markets, is shamelessly stolen from An Engine, Not a Camera, which is the best book I’ve read over the past few months.

Now, the shenanigans in the HCR market (and, for that matter, in the 2008 presidential market), seems to imply that the “Golden Age” of prediction markets, in which a small number of interested nerds could make bets in the comfort of their homes without impacting the events themselves, is fast coming to an end. It’s only a matter of time, a couple years I would guess, before the kind of manipulation I’ve described actually works.

Of course, this also hints at a much larger, more dangerous question. If the market manipulation I’ve described failed because the Intrade’s HCR market just wasn’t visible enough, what does that say about the markets we see every day?

5:00am  |   URL: http://tmblr.co/ZtlAMybcJ7q
(View comments) 
Filed under: markets 
May 18, 2010
Constantly Rebalanced Portfolios, Part Two

So last time I introduced the ideas behind CRPs. Now, why don’t they work in practice?

  1. They solve the wrong problem. One of the ideas central to CRP is that they are a long term strategy — that over time they will achieve sublinear regret with the best fixed strategy. But over long periods of time, the best fixed strategy becomes more and more meaningless relative to an adaptive adversary that can change its portfolio. Consider, for instance, investing in tech stocks in the ’90s, then cashing out, then investing in financial or oil companies in the ’00s, and then cashing out. Just because of the way the market is, the best fixed strategy in hindsight becomes less and less meaningful over the same long time spans over which CRPs are supposed to excel. Of course, CRPs do not achieve sublinear regret against the best adaptive strategy in hindsight — but this is, in a larger sense, the correct problem to solve.

  2. The bounds are meaningless. Let’s say we’ll use the Online Newton Step algorithm on the S&P 500, for a year. This algorithm achieves regret no larger than 5(1/a + GD)n log(T) where a, G, and D are pretty technical but take on values of about 1, T is the number of time steps (say 250 trading days in a year), and n is the dimension (here, 500). So, the bound is about 12,500. But this is an additive regret bound in log space. It seems reasonable to assume that even the best fixed CRP will not exceed a 55x return (which is e to the fourth power). Therefore, using this algorithm only proves that for each dollar you put into the algorithm, you will not end the period with fewer than exp(-12,496) dollars, which is close enough to zero to be zero. So just like in many other AI problems, the worst-case bound is meaningless here. Of course, this is not to say that such algorithms will work poorly in practice, but just to argue that there’s nothing magical about using one of these algorithms.

  3. The convex model is fundamentally flawed. Convex optimization requires a convex domain, in this case taken to be the simplex. But in the real world, nobody will let you put 100% of your money into a single name. There is, perhaps, a 10 or 20% maximum, and that’s if you have particularly willing investors. While the domain you are optimizing over is still convex, the optimal CRP in hindsight over the simplex could do much better than the optimal CRP over your restricted domain.

  4. They don’t take into account transaction costs. This is an issue that has been brought up before, notably by Blum and Kalai (warning, that link is a .ps.gz, ew). Essentially, introducing friction into your model slows down (and can even totally erase) your returns. Though this is the only issue that has been brought up in the literature, I think it’s actually the least important. As Cover alluded to in his article, by simply “caching” the moves you would have made and moving only when you have enough benefit to overcome transaction prices, you can accommodate these costs. Also, transaction costs just aren’t that significant anymore for hedge funds — my impression is that they would only slightly lower returns (say, in the 3-6% range).

  5. There’s no way to get the prices you see in hindsight. In the model, the CRP sees the daily return and then moves into the next day losslessly. But in practice, you don’t know the final prices until the end of the day, at which point you’re supposed to have already moved into your new position! This complication could be solved by moving throughout the day, or at the next day’s open, or what have you. Regardless, it’s a significant hit to the model.

  6. They don’t take into account market slippage. Even trading at the end of the day, in the auction, you’re likely to start moving prices if you put a lot of money into a single name. And of course, every time you move prices that hurts your return.

All of these factors combined conspired to kill Cover’s Mountain View Analytics. Still, I think the story is a wonderful illustration of the chasm between interesting, mathematically satisfying theory and nasty, brutish practice.

6:49pm  |   URL: http://tmblr.co/ZtlAMyaRjWQ
(View comments) 
Filed under: AI markets 
May 16, 2010
Constantly Rebalanced Portfolios, Part One

A decade ago, the Stanford Report was excited about a new hedge fund, Mountain View Analytics, run by the well-pedigreed Thomas Cover. To wit:

Tom Cover has the next-best thing to a time machine: He has an algorithm — a computational procedure — that uses the past to predict the future. It works as well or better than hindsight, outperforming a pretty good investment strategy: diversifying your stock portfolio and hoping that performance of superstars will more than make up for money wasted on losers. [several paragraphs redacted] So who wants to be a millionaire?

Sometime earlier this year, the domain registration on mountainviewanalytics.com expired, the page having not been updated for many years. These posts are intended to answer why constantly rebalanced portfolios (CRPs) — the broad class of investment strategies of which Cover’s is one — don’t actually work. In this post, I’ll give the technical and theoretical background, and in the next post, I’ll talk about the myriad of ways CRPs come up short in practice.

Why bother? Well, I think that CRPs are an interesting problem for two reasons:

  1. They provide a very illuminating example of the issues involved in bringing academic theory into practice, and the risks of hubris inherent in that process.

  2. They involve some very interesting math. They are featured at the end of the book Prediction, Learning, and Games (which I have considered buying if only for its cool cover art) which suggests they can be thought of as representing a culmination of learning theory, one of the more interesting topics I’ve had the opportunity to study in grad school.

The idea behind a CRP is that you always maintain a constant portfolio (distribution) of your wealth regardless of how the underlying assets in the portfolio change in value. So, if you’re mixing fifty-fifty between two stocks and one of them doubles in value while the other stays the same, you’d sell a quarter of your expensive stock holdings and use the proceeds to buy the cheaper stock — this maintains your holdings at an even split of wealth. Ideally, this scheme allows you to capture the value from positive swings in price while priming your portfolio to be ready for when undervalued assets become more in line. I probably should know a real answer to this, but my guess would be that if all the movements of the assets in your portfolio are completely arbitrary and not reflective of any kind of underlying value, then a CRP is your optimal policy.

Now, okay, let’s get into some math (sorry for the formatting, tumblr is nice for many things but not for this). Let x be a CRP (row vector of non-negative values, without loss of generality have it sum to 1 by introducing a “cash” asset if you’d like). Let aN represent a column vector of the return of the assets of the N-th day. (So if they were stocks, it would be a column vector that looks like [.97, 1.04, 1.01, …]).

Then we see:

return from best CRP in hindsight= max over x: (x * a1)(x * a2)(x * a3)*…

Now consider the RHS. By monotonicity of log, the same x will also solve:

max over x: log(x * a1) + log(x * a2) + log(x * a3) + … which is equivalent to

min over x: -log(x * a1) - log(x * a2) - log(x * a3) - …

So this is the insight transforms the task into an online convex optimization problem: minimizing a convex function (the sum of all those negative logs), over a convex domain (the simplex), which changes in an online fashion (you’re not statically optimizing, instead you get a new update every day).

If this sounds cool to you, you’ll probably want to check out Elad Hazan’s recent paper summarizing online convex optimization in this context. But essentially, there are now a large number of algorithms which stay competitive with the best CRP in hindsight. What’s neat about these algorithms is that, to achieve these bounds, you generally act to optimize over what you’ve currently seen, performing gradient-descent type algorithms over your historical data set. While gradient descent makes lots of sense in offline convex optimization, It’s very neat to me that such algorithms are successful in the online setting, where the shape of the space you’re optimizing over can change so dramatically from time step to time step, because assets could plummet or soar in value and you have no control over that.

So, I hope I’ve given a good illustration of how this should work in theory. Next time: why doesn’t it work in practice.

8:44pm  |   URL: http://tmblr.co/ZtlAMya551q
(View comments) 
Filed under: AI markets