StreetEYE Blog

The Dark Web Stack, Or How To Eff Up The Net

You have a choice of trusting the natural stability of gold or the honesty and intelligence of members of the government, and with all due respect to these gentleman, I advise you as long as the capitalist system lasts, vote for gold. – George Bernard Shaw

Do You Want To Eff Up The Net? Because That's How You Eff Up The NetAn interesting dive into “Deep Web Marketplaces” by the folks at avc.com and USV.

A deep web marketplace is like the Ebay of anonymous e-commerce. Other elements of the stack include:

  • Payments: Bitcoin
  • Transactional database: the blockchain
  • Non-transactional text database1: Pastebin (Images: Imgur; generic NoSQL database… I don’t know of one, so build on top of those, or left an exercise for an aspiring entrepreneur)
  • Networking: TOR
  • CDN: Bittorrent, Pirate Bay

Could you could go down the list of Amazon Web Services and come up with p2p distributed versions of each one? I don’t see why not.

Could you have a distributed version of Heroku/Amazon Web Services, an anonymous, distributed platform with all the services and APIs to create any app or marketplace from Ebay to Uber running in your browser or on your phone? I don’t see why not.

Could you have a distributed p2p version of UPS like TOR, with people handing each other anonymous packages and delivering in some dead drop or to the holder of dollar bill number B12345678?

Of course, that’s what they would do in an underworld network, or in a totalitarian state. Like samizdat publishing.

Should we?

There’s a constant ebb and flow between centralization (mainframes/AOL) and decentralization (PCs/Web) and back (Mobile/cloud).

Similarly there has been an ebb and flow from relatively anonymous Web protocols like Web, SMTP etc. to trusted IDs, Twitter/Facebook/SSH, and back with Bitcoin and dark web.

There’s a dichotomy: A world where much of the communication, transactions, commerce have to be over a dark web would be a pretty effed up place, like one where people had to pass along literature through samizdat and do commerce in back alleys.

The dark web stack is kind of an effed version of the legit web, and yet necessary.

It’s just that, being nexus-free, it’s more resistant to control.

Not totally resistant…the Internet is the granddaddy of distributed, anonymous networks, and it’s impressive what China can do to control it.

What can a dark web stack do, that a more centralized client/server or mobile/cloud stack can’t?

Absolutely nothing.

How much of your life needs to be conducted in a secure, ‘erasable’ Internet? Sure, some part of communications should be kept very private, and quickly forgotten.

Some, but by no means most of it.

And there’s a cost. It may avoid an ATM charge, but the cost of clearing a Bitcoin transaction is still staggering compared to institutional FX markets. The time to clear is staggering. And fairly fixing any glitch in centralized markets is maybe tough and possibly political, but it’s practically impossible on the blockchain.

And yet, it’s insane for someone like David Cameron to say the government needs the keys to everything and there can be no true dark web. If you had to put a back door in every communication or ecommerce system, impossible to believe anything would be safe against black hat hackers and foreign governments. And of course bad guys would always find a way around it. And there’s a free speech issue: what kind of world is it where you can’t have a private conversation in the safety of your own device? It’s unachievable, dangerous to try, and wrong.

The more you use control of legit platforms to enforce political goals, whether it’s against Falun Gong or Russia, the more you create demand for a dark, private web.

If people trust the government and the monetary and financial system, they don’t hoard gold. If they hoard gold, the response is to fix the monetary system, not to ban the ownership of gold. Which may be the only alternative when the monetary system is irretrievably broken in times of war or crisis. But it’s not the way a free economy and democracy are supposed to work. And the same goes for the dark web.

If you don’t want people to use the dark net, don’t mess up the legit networks with back doors and warrantless wiretaps, ‘express lanes,’ censorship, using them for political pressure. That’s not the way freedom of speech and democracy are supposed to work.

Or people will create worse versions and route around you.


1 Technically, these aren’t distributed in the same sense as the blockchain is. Many distributed apps and use cases could probably use them as a storage layer, though. For instance, to build a distributed p2p Uber, drivers could have an app that posts availability and reservation responses to a Pastebin type public space signed with their key, and riders could likewise post reservation requests. Perhaps there’s an opportunity for a NoSQL nontransactional distributed p2p database counterpart to the blockchain, which is transactional, strictly enforces no double-spend, but takes a long time to commit.

UberFail – a few thoughts on Uber

To lose one parent, Mr. Worthing, may be regarded as a misfortune; to lose both looks like carelessness. – Oscar Wilde, The Importance of Being Earnest

Some thoughts on Uber, some blindingly obvious, some maybe not. I’ll say at the outset that I think Uber is a 10x improvement on existing cabs, and I have no problem in principle with a supply/demand based surge pricing model. But here goes…

1) Uber doesn’t have it together operationally.

Exhibit 1: Outraged passengers post bills online.

B6Ro1GxCQAAUQ_J
Exhibit 2: Outraged drivers say they couldn’t find passengers and it didn’t pay to work on New Year’s Eve. Also on Reddit.

Exhibit 3: Combining 1 and 2 into a singularity of Uber-fail, some drivers said that even as the map showed a huge pricing surge presumably due to overwhelming demand, they couldn’t get passengers. Apparently the surge drove passengers away…or in any event something was awry.


cV2J9Wd

Capture

If you’re managing to piss off the riders because they’re feeling gouged, and drivers who feel they’re being taken advantage of, well, that seems less like misfortune and griping, and more like carelessness in running the system.

2) The frequent surges seem rather extreme. Hotels might have a 10x change in price from bottom of low season to the peak convention season. But…hotel room supply is perfectly inelastic in the short run. Uber, in most cities, should be able to tap casual drivers and get many more cars on the road at peak times. Car rides should be closer to other seasonal products and services like airfares, where maybe you see a 3x differential on a peak date.

Maybe, over time they’ll have more data and get better at forecasting demand on the one hand. And drivers and passengers will understand the system better and not get shocked and outraged. But…

3) There is very limited transparency, and data for riders and drivers to plan ahead. A fundamental problem with Uber’s surge pricing model is that it is a black box, and seems self-serving to Uber.

Even Uber’s exceptional pre-NYE blog was of limited value in providing actionable data about how high the surge was expected to go, beyond don’t travel between 12 and 2. (And they didn’t even provide lip service as to how they were working to get as many drivers as possible on the road to limit the surge, which is tone-deaf).

When they surge and then drivers sit around doing nothing, either the algorithm is crap, or they are raising prices on a hair-trigger whenever they think they can to maximize revenue, which clearly would be in their financial interest.

Consider one extreme – the algorithm always adjusts price as much as necessary to keep a 1-minute wait. Or the other extreme – no adjustment – and wait times go up. Well, some passengers might prefer a brief queue. But clearly, maximizing revenue is to Uber’s advantage.

Say your time is worth $30 an hour. Presumably you would be indifferent between a price with $15 dispersion and a wait time with 30 minutes dispersion. In fact the wait time might be easier to plan for, if it looks busy you call for the car 15 minutes before your dinner is over, maybe you have to rush out a little earlier, maybe you linger a little later. You’ll be happier with that than the current price dispersion with up to 9x surges on New Year’s Eve.

4) Uber wants to maximize profitability, but both drivers and riders value predictability. (see Steve Randy Waldman’s excellent discussion). Economists sometimes (not always) have a blind spot in only considering efficiency in terms of price and quantity. But of course, behaviorally, whenever something unexpected happens, it leads to unhappiness and inefficiency beyond the purely financial cost. Loss-aversion kicks in, and you feel gouged when you expected to pay $10 and had to pay $50 through no fault of your own. Maybe you don’t take a $20 round trip that would otherwise make sense, because you’re not sure if it will turn out to cost $50 to get back.

So I think that 1) Uber’s execution, even though outstanding in many respects, has room to improve, and 2) their market design is not ideal from a consumer standpoint.

5) Without further ado, here are some things Uber could do to improve:

  • Provide transparency on the algorithm used to set surge pricing. Provide data on number and location of drivers and riders in real time. Do a Netflix-style competition on finding the optimal market-clearing model.
  • Let you put in for a reservation hours or days in advance, and let drivers commit to pick up a reservation. You have a time and price guarantee, Uber has a valuable signal in advance on demand. And the driver gets predictability.
  • Uber’s slogan “Everyone’s private driver” becomes more literally true if people can request specific drivers in advance.
  • Make the surge pricing a true live auction. Let the rider say how much they are willing to pay, and Uber provides an estimate of how much of a wait that will give them, according to how many higher-paying riders are ahead of them. In other words, give the rider a choice – wait for an available car, with a predicted queuing time. Or bid a surge price to jump the queue. Then it is a true auction with transparency.1 [Edit: you could even bid below the normal price, if anyone's available off-peak to run a retiree to the drug store for 30% off, notify me.].
     
    Today, the model is Uber attempting to have their cake and eat it too – pretend they’re a two-way market while acting as a cartel to maximize revenue.
  • (Pipe dream) Charge their fee only on the non-surged rate. Otherwise it may be in their interest to surge, even when it hurts drivers and riders.
  • Push UberPOOL sharing – give people a prominent button to offer/accept ride shares to destinations along their route. For many people, sharing is a better alternative to bidding the price up or waiting.
  • (Pipe dream) Eliminate anticompetitive ToS termslet drivers join all services, allow open APIs for drivers to advertise their availability across multiple services, allow APIs to let riders to check all services from one app. [Edit: I haven't reviewed the driver agreement-I just assumed Uber would do everything possible to make it as sticky as possible for drivers to switch.]

If you have transparency, rider choice, and no anticompetitive and uber-aggressive tactics, that will go a long way toward improving rider and driver satisfaction.

Uber really doesn’t have it together on messaging: see the Uber-gate fiasco. (Even if Sarah Lacy and Pando have latched on to the anti-Uber angle and are prone to their own threatening meltdowns, Travis Kalanick can come across as a high-functioning psychopath. Here’s a balanced discussion, which I think pulls some punches over the narcissistic hacks at Pando.)

You might say this is beating a horse to death, it’s just a friggin’ car service.

Since we can now do complex market designs with networked smartphones, a lot of things are going to get allocated that way. It probably makes sense to put a little thought in how to come up with good market designs that make people happy. It would be pretty miraculous if Uber got the perfect design on the first try. If both drivers and customers are mad at Uber, it might be worth thinking about how to do it better, and not leave it to rapacious aggressive entrepreneurs and grasping politicos.

If Uber can come up with market designs that allocate scarce perishable resources in real time more efficiently and make people happier, that’s applicable in a lot of areas and worth far more than the taxi industry they’re disrupting. It’s a hundreds-of-billions of dollars type of problem.2

Hailing from an app is a 10x user experience improvement v. trying to hail from the street…you get your car faster, don’t have to wait on a street-corner with your hand out…it reduces empty cruising, which costs money and increases congestion…with some more thought, we can increase car pooling and let multiple people share rides.

Maybe the runt competitors like Lyft and prospect of competition will keep Uber from abusing its market position and extracting rents. My guess is, it’s more likely it becomes a Google or Amazon-type winner-take-all market. The biggest brand has a big, but maybe not insuperable advantage in attracting riders and drivers, and I would expect them to do everything possible to not become a commoditized airline ticket market, unless regulators force them too. Maybe that would be a good thing, as Waldman argues. But it would be a shame if ‘disruption’ just meant we went from a medallion cartel to a new tech cartel that extracted all the value from the technological improvement.

1In fairness, the practicality of a true live auction could turn out to be questionable. People may not want an Ebay or stock trader experience every time they take a cab. And the ultimate pricing outcome might not be that different from what it is now. When demand is high, bid high or don’t get a cab at all. Right at the marginal price one might save a few bucks by waiting. And there would still be volatility. And some customers fat-fingering their interactions with the system, or feeling gouged regardless. The key issue is transparency and a true sense that the price is supply and demand-driven. It might take a lot of product design / UI creativity to make a true live auction workable without defaulting a lot of user input to automatic defaults, but a black box clearing system seems hard to justify.

2Valuation of NYC taxi medallions: ~$13b. Uber valuation: ~$40b. I’m just sayin’… To those who think it’s a crazy bubble, there is a really big market opportunity here.

Sony is a pack of buffoons, and the farce is us.

A few points about the Sony debacle.

Point 1: Sony is a clown show.

Let’s be generous, and suppose the following is what happened. (We don’t know how the malware got in, because they seemingly have no clue, which makes me just weep with pity.)

  • Hackers send a carefully crafted email to a Sony dim bulb, like, “have you seen what Nikki Finke wrote about you today in Deadline: Hollywood?”
  • Aforementioned nitwit takes a break from mocking Angelina Jolie or whatever the f**k they do all day [ed.: Angelina's lips are a national treasure!], clicks on aforementioned link.
  • This installs some malware on their PC.
  • Malware
    • Surveys the entire network
    • Uploads tens of terabytes to a server in a foreign country
    • Infects every PC in the company
    • Erases all the PCs and flashes a Guardians of Peace banner
    • …and Sony’s IT buffoons never look up from eating donuts or whatever the f**k they do all day to notice a darn thing.

      At every stage, a proper infrastructure should have a good shot at stopping the attack.

      • Intercept the email with a disguised link to a non-whitelisted web site.
      • Disallow the download of an installer from an external website. Or let it run in a sandbox. Or download it but don’t let it install anything without adminstrator permissions.
      • Don’t allow it to remotely install itself throughout the firm.
      • But especially – don’t allow terabytes of data to be uploaded to an unknown IP address. I can’t even think down to the level of an IT team that would not detect that.

      In the words of Tina Fey: Shut it down!

      Now maybe there was an insider, a Snowden. You have to trust somebody, and it’s practically impossible to prevent them from walking out the door with a giant data dump. But even a Snowden shouldn’t be able to grab data, and install malware on every PC in the firm, and erase all trace so they don’t even know what happened.

      This just does not happen with a competent corporate IT team. And once you assume incompetence, it seems more likely that, rather than inordinately clever trickery, or an inside job, they just left vulnerable equipment wide open.

      If you get an STD and don’t have any idea how you got it, I’m going to say you were probably not using the safest practices.

      Here are a few other greatest hits from Sony IT:

      • 2011: PlayStation network down for 23 days, 77 million user records stolen after ‘external intrusion.’
      • 2007: Sony’s IT security chief says it’s a “valid business decision to accept the risk” of a security breach, like weak passwords, since requiring strong passwords might encourage people to put them on Post-Its.
      • 2005: Sony ships CDs with copy protection that secretly changes Windows to run the way they wish it did, opening users up to crashes and further malware exploits.

      Geniuses, clearly.

      Point 2: This hack doesn’t make the top 10 list of greatest hacks.

      Point 3: Maybe it was North Korea, maybe it wasn’t.

      The FBI says it “has enough information to conclude that the North Korean government is responsible for these actions.”

      These Sony clowns don’t even know what hit them. They and the FBI have provided no evidence it was North Korea.

      Why did the hackers demand monetary compensation, not mention “The Interview” until people started speculating about it? Could be Romanians, Chinese, anyone trying to make it look like North Korea.

      Lockerbie was constantly blamed on the enemy du jour until they settled on Libya. Do I believe the FBI now?

      The FT says there’s a long history of world class North Korean hacks on South Korea.

      So, either there’s a long history of attacks which could definitively be linked to North Korea, and this one bears the same signature in ways a copycat wouldn’t pick up, so the evidence, though circumstantial, is strong. Even then, the language the FBI uses is excessive, should be more like “we assess with high probability North Korea is linked to these actions.” Maybe they have some top-secret evidence, like a mole, or electronic surveillance.

      Or they’re just talking out of their asses, like Lockerbie, yellowcake, Atta’s meetings in Prague, etc. If something is in someone’s interest, they will believe it. If something is greatly feared, they will believe it. Who knows.

      Point 4: A rogue state maybe hacked a dipshit company. Who cares?

      The problem is this:

      So, do we put on our big boy pants, harden our security, keep calm and carry on?

      No…once again, faced with a serious, but not existential threat, we panic, run around like chickens with our heads cut off and beclown ourselves. And in the name of freedom, we’ll cancel movies, stop going to the theater, hire more hackers cyberwarriors and tap more phones and backbones.

      History repeats itself, first as tragedy, then as farce.

A Piketty counterargument: Is r > g a reasonable assumption?

A thought experiment:

If you could buy an asset that returned the GDP growth rate, would you, should you do it?

In other words, if you had a choice of Piketty’s r and g, from his ‘Iron Law’ equation r > g, which would you choose?

Marc Andreessen said, “The funny thing about Piketty is that he has a lot more faith in returns on invested capital than any professional investor I’ve ever met.” And I tend to agree with him. If someone offered me a guarantee of g, I would put a good chunk of my retirement portfolio into it.

Why?

  • US real GDP growth has averaged about 3.5% since 1930.
  • Bonds’ long run real returns are less than GDP growth, maybe 2% real (and remarkably less right now).
  • Stocks return more historically: on average about 8% real returns since 1930. (and also currently priced to underperform)
  • But stock returns are super bumpy. In 2008 they were down 37% on the year (and more at the trough).
  • Companies have financial and operating leverage, so in the medium term stocks are like an option on GDP struck at expected GDP. If GDP outperforms the expectations built into prices when you buy stocks, you’ll do really well, if GDP disappoints, you will lose quite a bit of money.
  • In the longer run, the outperformance of stocks over GDP seems to stem from 1) more leverage/risk and 2) historically, persistently better than expected outcomes, which is another way of saying a persistently excessive equity risk premium. People have historically feared stock volatility even more than they reasonably should have feared those 37% crashes, or stocks and the economy have historically on average outperformed people’s expectations.
  • The r Piketty refers to is the unlevered real return across all deployments of capital in GDP: Residential, non-residential construction, plant and equipment, inventories/working capital. 1

We can compute, historically, if g would have offered a good deal within the major investment classes. So let’s try it! Using historical real returns of US stocks, bonds, bills, gold, and this hypothetical GDP asset, the efficient frontier looks like this:

Efficient Frontier

And the transition map showing the efficient portfolio: As you go from left to right, you increase your risk and return, move out on the efficient frontier, and the composition of your portfolio changes.

Transition Map

The minimum risk portfolio starts with about 40% GDP. As you stretch for higher risk and returns it peaks at about 75%, before giving way to an increasing mix of higher returning stocks.

The bottom line is, if you could buy g, in most cases you would devote quite a big chunk of your portfolio to it. All the portfolios we would consider diversified have more ‘GDP’ than stocks.

Piketty makes the following claims:

    • When the rate of return on capital r is greater than the rate of GDP growth g, owners of capital get richer faster than wage-earners.
    • It is an ‘iron law of capitalism’ that in that situation inequality will increase.
    • This is a problem, and to avoid the ills associated with that, a global capital levy is an appropriate solution.

I would make the following counterclaims:

        • If people could buy an asset with r=g, they would.
        • If some people could persistently identify assets and strategies that earned r>g, they could make money supplying GDP-linked bonds or swaps which paid g, turn around and invest at r>g and make a profit.
        • We don’t see this sort of profitable banking or arbitrage in the marketplace. Even Warren Buffett wouldn’t offer g-linked bonds in size. Why not? Because he can put his hands on cash at better terms. And because if someone had offered such a deal at the beginning of 1930 (never mind the stock market peak) and invested in stocks, they wouldn’t have broken even until the 50s. And bonds don’t save you due to their low returns.

It seems to me the market is telling us it does not believe r > g.

r seems partly endogenous and partly exogenous: it depends on availability of capital and the return on the available investment opportunities, which can also be highly uncertain and idiosyncratic.

As a thought experiment, suppose cheap fusion was invented tomorrow. That would result in an investment boom. Enterprises making fusion reactors, fusion vehicles, all manner of cheap fusion energy-powered factories would suck in capital, crowd out other investments, r would go up, and an era of plenty for mankind would hopefully ensue as all those investments threw off great r initially, and eventually g. r would presumably rise above g initially, and then fall below it as the big rise in g threw off more surplus investment capital than people knew what to do with. That’s what boom/busts generally look like.

On the other hand, if oil suddenly runs out, or California tumbles into the Pacific, r will be negative, a lot of people with large capital investments will find them uneconomic, or lose them entirely to the whims of fate.

If you combine all the layers of the economy’s capital structure, including residential, non-residential structures and plant and equipment, working capital/inventories, I think the safer bet in the very long run is r ~= g. Piketty shows anything else is unsustainable. If r < g, there is too much capital and investors are foolishly investing in poor projects that lose money and the economy is getting less and less capital intensive over time. If there were no countervailing forces, this would presumably continue, until we’re back to performing surgery with stone knives and bear skins. If r > g, the economy is getting more capital-intensive over time, investors are under-investing in given growth opportunities due to scarcity of capital. If you believe Piketty, this continues until you somehow end up with a few capitalist lords supervising armies of robots and starving serfs.

Over the very long run, I don’t see any reason to think either situation qualifies as ‘iron law’. If anything the capitalist era has seen wealth distributed more widely than in the feudal era, and after a few hundred years we’re not once again ruled by a few kings and armies of serfs and robots (although recent tendencies in that direction are a bit concerning).

As an accounting identity you can say, when r>g, capital owners are doing better than everyone else.

You just can’t turn that into a statement about perpetually increasing inequality without heroic assumptions about how capital is distributed, and how the distribution of capital income, and wage income is changing. (Including how new wealth destroys old wealth, e.g. Apple destroys RIM and Nokia, how wealth gets dispersed over generations, how it eventually falls into the hands of fools and gets destroyed and dissipated.)

If that isn’t enough, let me throw out one more, entirely unrelated point. The distinction between return on capital and return on labor is ultimately more fuzzy and arbitrary than one might think.

Mitt Romney made a modest fortune from Bain Capital. Not only did he benefit from getting paid in ‘carried interest’ which was taxed as a capital gain at 15%, but he also put the most levered bets into his IRA, where they appreciated with no tax hit at all, until he decides to withdraw them.

Of course, if IRAs don’t get taxed, and capital gains are taxed at 15% instead of 39.6% on income, no one can blame him for structuring his compensation as capital gains.

And can anyone argue that if the tax rates were reversed, the payout would have come as bonuses and labor income for his talents and hard work, not as capital gains?

The distinction between return on capital and labor is in large part arbitrary when it comes to the top 1%. Whether you own a hot dog stand, a medical practice, or are a founder of Microsoft, it’s pretty much impossible to tell which part of the income is from your hard work and which is a return on investment from…your hard work in earlier years. And it’s a little pointless to try.

Goodhart’s Law: ‘Iron laws’ turn out to be made of straw once you depend on them for policy outcomes.

Democratic societies are founded on equality of opportunity, equality before the law, equality in the voting booth. When so much of our life takes place in the economic sphere, excessive inequality of wealth and income inevitably leads to inequality of opportunity and inequality before the law. r > g, the “iron law” and the global wealth tax, are an academic sideshow, and taken to an extreme, divorced from reality and common sense. We should strive for better education and opportunity for everyone, and a fairer tax system than the current abomination. Bill Gates has a thoughtful perspective on the subject, noteworthy because of where his own personal interests lie.

1 It’s a bit challenging to calculate the unlevered returns to all the capital deployed in GDP, but that hasn’t stopped some from trying. Logically, that rate of return should be in the realm of returns available from the universe of investable securities, after expenses, if for no other reason than that companies can issue stocks and bonds to fund capital projects, and when market rates are low and project returns are high, they will issue securities and fund projects until the gap no longer provides that incentive. In any event, that universe of securities represents the investments generally available to all those capital owners Piketty thinks have the tendency to get richer and richer.

2Let’s try anyway. In the case of the hot-dog vendor, you can ask what the cart-owner would have to pay to hire someone at salary to sell the dogs and say the balance is return on capital. But the wage-earner won’t work as hard and sell as many. Some of the difference is because the owner is in fact working harder. Same applies to the CEO. Bill Gates could have paid someone else to be CEO. But Microsoft wouldn’t have done as well. It’s quite hard to say how much of the money he made, or any of the Microsoft millionaires for that matter, was labor income, and how much was return on income foregone and re-invested in the company.

?View Code RSPLUS
 
install.packages('lpSolve') 
require(lpSolve)
 
install.packages('quadprog')
require(quadprog)
 
require(ggplot2)   
require("reshape")
 
SP500 = c(-0.2512,-0.4384,-0.0864,0.4998,-0.0119,0.4674,0.3194,-0.3534,0.2928,-0.011,-0.1067,-0.1277,0.1917,0.2506,0.1903,0.3582,-0.0843,0.052,0.057,0.183,0.3081,0.2368,0.1815,-0.0121,0.5256,0.326,0.0744,-0.1046,0.4372,0.1206,0.0034,0.2664,-0.0881,0.2261,0.1642,0.124,-0.0997,0.238,0.1081,-0.0824,0.0356,0.1422,0.1876,-0.1431,-0.259,0.37,0.2383,-0.0698,0.0651,0.1852,0.3174,-0.047,0.2042,0.2234,0.0615,0.3124,0.1849,0.0581,0.1654,0.3148,-0.0306,0.3023,0.0749,0.0997,0.0133,0.372,0.2382,0.3186,0.2834,0.2089,-0.0903,-0.1185,-0.2197,0.2836,0.1074,0.0483,0.1561,0.0548,-0.3658,0.2592,0.1486)
 
BONDS = c(0.0454,-0.0256,0.0879,0.0186,0.0796,0.0447,0.0502,0.0138,0.0421,0.0441,0.054,-0.0202,0.0229,0.0249,0.0258,0.038,0.0313,0.0092,0.0195,0.0466,0.0043,-0.003,0.0227,0.0414,0.0329,-0.0134,-0.0226,0.068,-0.021,-0.0265,0.1164,0.0206,0.0569,0.0168,0.0373,0.0072,0.0291,-0.0158,0.0327,-0.0501,0.1675,0.0979,0.0282,0.0366,0.0199,0.0361,0.1598,0.0129,-0.0078,0.0067,-0.0299,0.082,0.3281,0.032,0.1373,0.2571,0.2428,-0.0496,0.0822,0.1769,0.0624,0.15,0.0936,0.1421,-0.0804,0.2348,0.0143,0.0994,0.1492,-0.0825,0.1666,0.0557,0.1512,0.0038,0.0449,0.0287,0.0196,0.1021,0.201,-0.1112,0.0846)
 
BILLS = c(0.0455,0.0231,0.0107,0.0096,0.0032,0.0018,0.0017,0.003,0.0008,0.0004,0.0003,0.0008,0.0034,0.0038,0.0038,0.0038,0.0038,0.0057,0.0102,0.011,0.0117,0.0148,0.0167,0.0189,0.0096,0.0166,0.0256,0.0323,0.0178,0.0326,0.0305,0.0227,0.0278,0.0311,0.0351,0.039,0.0484,0.0433,0.0526,0.0656,0.0669,0.0454,0.0395,0.0673,0.0778,0.0599,0.0497,0.0513,0.0693,0.0994,0.1122,0.143,0.1101,0.0845,0.0961,0.0749,0.0604,0.0572,0.0645,0.0811,0.0755,0.0561,0.0341,0.0298,0.0399,0.0552,0.0502,0.0505,0.0473,0.0451,0.0576,0.0367,0.0166,0.0103,0.0123,0.0301,0.0468,0.0464,0.0159,0.0014,0.0013)
 
CPI = c(-0.0639535,-0.0931677,-0.1027397,0.0076336,0.0151515,0.0298507,0.0144928,0.0285714,-0.0277778,0,0.0071429,0.0992908,0.0903226,0.0295858,0.0229885,0.0224719,0.1813187,0.0883721,0.0299145,-0.0207469,0.059322,0.06,0.0075472,0.0074906,-0.0074349,0.0037453,0.0298507,0.0289855,0.0176056,0.017301,0.0136054,0.0067114,0.0133333,0.0164474,0.0097087,0.0192308,0.0345912,0.0303951,0.0471976,0.0619718,0.0557029,0.0326633,0.0340633,0.0870588,0.1233766,0.0693642,0.0486486,0.0670103,0.0901771,0.1329394,0.125163,0.0892236,0.0382979,0.0379098,0.0394867,0.0379867,0.010979,0.0443439,0.0441941,0.046473,0.0610626,0.0306428,0.0290065,0.0274841,0.026749,0.0253841,0.0332248,0.017024,0.016119,0.0268456,0.0338681,0.0155172,0.0237691,0.0187949,0.0325556,0.0341566,0.0254065,0.0408127,0.0009141,0.0272133,0.0149572)
 
GOLD = c(0,0,0,0.44700286,0.079661828,0,0,0,0,0,-0.014388737,0.028573372,0,0.027779564,-0.006872879,0.027212564,0.026491615,0.117056555,-0.023530497,-0.036367644,-0.00619197,-0.00623055,-0.033039854,-0.086306904,-0.007067167,-0.002840911,0.001421464,0.001419447,0,0,0.034846731,-0.027779564,-0.004234304,-0.002832863,0.002832863,0.004234304,-0.002820876,0.002820876,0.203228242,-0.059188871,-0.052577816,0.136739608,0.358646094,0.511577221,0.545727802,-0.132280611,-0.253090628,0.168898536,0.265477915,0.464157559,0.689884535,-0.285505793,-0.201637346,0.120144312,-0.163629424,-0.127202258,0.149181164,0.192236014,-0.020385757,-0.134512586,0.005221944,-0.058998341,-0.051002554,0.045462374,0.064538521,0.002600782,0.010336009,-0.155436854,-0.121167134,-0.052185753,0,-0.028987537,0.133990846,0.157360955,0.119003292,0.084161792,0.308209839,0.142551544,0.220855221,0.110501915,0.228258652)
 
GDP = c(-0.085084232,-0.064032275,-0.128868258,-0.012560264,0.107799049,0.089074461,0.129392971,0.051013673,-0.033106047,0.079706783,0.08808869,0.17707922,0.188888143,0.170391692,0.079905483,-0.009645441,-0.115835513,-0.010964353,0.041559245,-0.00549505,0.087162129,0.080586081,0.040720339,0.046944343,-0.005638952,0.071219054,0.02132165,0.021055266,-0.007352169,0.069022678,0.025635104,0.025541223,0.061164957,0.043539949,0.057670519,0.064997322,0.065934066,0.027436363,0.049090742,0.03140731,0.002015915,0.032952139,0.052628342,0.056443916,-0.005180583,-0.001964418,0.053849296,0.046093667,0.055617315,0.031752617,-0.002443475,0.025936376,-0.019100292,0.046323541,0.072585395,0.042388469,0.035120756,0.034616119,0.042040676,0.036804531,0.019188746,-0.000737018,0.03555943,0.027453435,0.04037391,0.027197286,0.03795652,0.044872645,0.044495193,0.046851005,0.040925252,0.009753418,0.017867562,0.028066125,0.037856696,0.033448288,0.026668165,0.01778456,-0.002911179,-0.027760546,0.025321284)
 
# put into a data frame (matrix)
 
fnominal=data.frame(stocks=SP500, bills=BILLS, bonds=BONDS, gold=GOLD, CPI=CPI, GDP=GDP)
freal=data.frame(stocks=SP500-CPI, bills=BILLS-CPI, bonds=BONDS-CPI, gold=GOLD-CPI, gdp=GDP)
 
# compute means
realreturns = apply(freal, 2, mean) 
realreturnspct = realreturns*100 
# print them 
realreturnspct 
 
# compute sds
realsds = apply(freal, 2, sd) 
realsdspct = realsds*100 
# print them 
 
realsdspct 
 
# find maximum return portfolio (rightmost point of efficient frontier) 
# will be 100% of highest return asset 
# maximize 
# w1 * stocks +w2 *bills +w3*bonds + w4 * gold 
# subject to 0 <= w <= 1 for each w 
# will pick highest return asset with w=1 
# skipping >0 constraint, no negative return assets, so not binding 
 
opt.objective <- realreturns 
opt.constraints <- matrix (c(1, 1, 1, 1, 1, # constrain sum of weights to 1 
1, 0, 0, 0, 0, # constrain w1 <= 1 
0, 1, 0, 0, 0, # constrain w2 <= 1 
0, 0, 1, 0, 0, # constrain w3 <= 1 
0, 0, 0, 1, 0, # constrain w4 <= 1 
0, 0, 0, 0, 1), # constrain w5 <= 1 
nrow=6, 
byrow=TRUE) 
 
opt.operator <- c("=", "<=", "<=", "<=", "<=", "<=") 
opt.rhs <- c(1, 1, 1, 1, 1, 1) 
opt.dir="max" 
 
solution.maxret = lp (direction = opt.dir, opt.objective, opt.constraints, opt.operator, opt.rhs) 
 
# portfolio weights for max return portfolio 
wts.maxret=solution.maxret$solution 
# return for max return portfolio 
ret.maxret=solution.maxret$objval 
# compute return covariance matrix to determine volatility of this portfolio 
covmatrix = cov(freal, use = 'complete.obs', method = 'pearson') 
# multiply weights x covariances x weights, gives variance 
var.maxret = wts.maxret %*% covmatrix %*% wts.maxret 
# square root gives standard deviation (volatility) 
vol.maxret = sqrt(var.maxret) 
 
wts.maxret
ret.maxret
vol.maxret
 
################################################################################
 
# find minimum volatility portfolio (leftmost point of efficient frontier) 
 
 
covmatrix = cov(freal, use = 'complete.obs', method = 'pearson') 
 
nObs <- length(freal$stocks) 
nAssets <- length(freal) 
 
# 1 x numassets array of 1s 
opt.constraints <- matrix (c(
1, 1, 1, 1, 1, # sum of weights =1 
1, 0, 0, 0, 0, # w1 >= 0 
0, 1, 0, 0, 0, # w2 >= 0 
0, 0, 1, 0, 0, # w3 >= 0 
0, 0, 0, 1, 0, # w4 >= 0 
0, 0, 0, 0, 1 # w5 >= 0 
)  
, nrow=6, byrow=TRUE) 
 
opt.rhs <- matrix(c(1, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001)) 
 
opt.meq <- 1 # first constraint is '=', rest are '>=' 
 
zeros <- array(0, dim = c(nAssets,1)) 
 
solution.minvol <- solve.QP(covmatrix, zeros, t(opt.constraints), opt.rhs, meq = opt.meq) 
 
wts.minvol = solution.minvol$solution 
var.minvol = solution.minvol$value *2 
ret.minvol = realreturns %*% wts.minvol 
vol.minvol = sqrt(var.minvol) 
 
 
 
################################################################################
 
 
# generate a sequence of 50 evenly spaced returns between min var return and max return 
lowreturn=ret.minvol 
highreturn=ret.maxret 
minreturns=seq(lowreturn, highreturn, length.out=50) 
 
# add a return constraint: sum of weight * return >= x 
retconst= rbind(opt.constraints, realreturns) 
retrhs=rbind(opt.rhs, ret.minvol) 
 
# create vectors for the returns, vols, and weights along the frontier, 
# starting with the minvol portfolio 
out.ret=c(ret.minvol) 
out.vol=c(vol.minvol) 
out.stocks=c(wts.minvol[1]) 
out.bills=c(wts.minvol[2]) 
out.bonds=c(wts.minvol[3]) 
out.gold=c(wts.minvol[4]) 
out.gdp=c(wts.minvol[5]) 
 
# loop and run a minimum volatility optimization for each return level from 2-49 
for(i in 2:(length(minreturns) - 1))
{
print(i)
# start with existing constraints, no return constraint
tmp.constraints = retconst
tmp.rhs=retrhs
# set return constraint
tmp.rhs[7] = minreturns[i]
tmpsol <- solve.QP(covmatrix, zeros, t(tmp.constraints), tmp.rhs, meq = opt.meq)
tmp.wts = tmpsol$solution
tmp.var = tmpsol$value *2
out.ret[i] = realreturns %*% tmp.wts
out.vol[i] = sqrt(tmp.var)
out.stocks[i]=tmp.wts[1]
out.bills[i]=tmp.wts[2]
out.bonds[i]=tmp.wts[3]
out.gold[i]=tmp.wts[4]
out.gdp[i]=tmp.wts[5]
 
} 
 
# put maxreturn portfolio in return series for max return, index =50 
out.ret[50]=c(ret.maxret) 
out.vol[50]=c(vol.maxret) 
out.stocks[50]=c(wts.maxret[1]) 
out.bills[50]=c(wts.maxret[2]) 
out.bonds[50]=c(wts.maxret[3]) 
out.gold[50]=c(wts.maxret[4]) 
out.gdp[50]=c(wts.maxret[5]) 
 
# organize in a data frame 
efrontier=data.frame(out.ret*100) 
efrontier$vol=out.vol*100 
efrontier$stocks=out.stocks*100 
efrontier$bills=out.bills*100 
efrontier$bonds=out.bonds*100 
efrontier$gold=out.gold*100 
efrontier$gdp=out.gdp*100 
names(efrontier) = c("Return", "Risk", "%Stocks", "%Bills", "%Bonds", "%Gold", "%GDP")
 
 
################################################################################
 
# define colors 
dvblue = "#000099" 
dvred = "#e41a1c" 
dvgreen = "#4daf4a" 
dvpurple = "#984ea3" 
dvorange = "#ff7f00" 
dvyellow = "#ffff33" 
dvgray="#666666"   
 
apoints=data.frame(realsdspct) 
apoints$realreturns = realreturnspct 
ggplot(data=efrontier, aes(x=Risk, y=Return)) + opts(title="Efficient Frontier") + theme_bw() + geom_line(size=1.4) + geom_point(aes(x=apoints$realsdspct, y=apoints$realreturns)) + scale_x_continuous(limits=c(1,24)) + annotate("text", apoints[1,1], apoints[1,2],label=" stocks", hjust=0) + annotate("text", apoints[2,1], apoints[2,2],label=" bills", hjust=0) + annotate("text", apoints[3,1], apoints[3,2],label=" bonds", hjust=0) + annotate("text", apoints[4,1], apoints[4,2],label=" gold", hjust=0) + annotate("text", apoints[5,1], apoints[5,2],label=" gdp", hjust=0) + annotate("text", 19,0.3,label="streeteye.com", hjust=0, alpha=0.5) 
 
################################################################################
 
 
keep=c("Risk", "%Stocks","%Bills","%Bonds","%Gold", "%GDP") 
efrontier.tmp = efrontier[keep] 
efrontier.m = melt(efrontier.tmp, id ='Risk') 
 
ggplot(data=efrontier.m, aes(x=Risk, y=value, colour=variable, fill=variable)) + theme_bw() + opts(title="Transition Map", legend.position="top", legend.direction="horizontal") + ylab('% Portfolio') + geom_area() + scale_colour_manual("", breaks=c("%Stocks", "%Bills", "%Bonds","%Gold","%GDP"), values = c(dvblue,dvgreen,dvred,dvyellow,dvgray), labels=c('%Stocks', '%Bills','%Bonds','%Gold','%GDP')) + scale_fill_manual("", breaks=c("%Stocks", "%Bills", "%Bonds","%Gold","%GDP"), values = c(dvblue,dvgreen,dvred,dvyellow,dvgray), labels=c('%Stocks', '%Bills','%Bonds','%Gold','%GDP')) + annotate("text", 16,-2.5,label="streeteye.com", hjust=0, alpha=0.5)

Easy Pieces on Risk #2: Visualizing Diversification With Risk Triangles

XKCD

Randall Munroe – XKCD.com – http://xkcd.com/552/

In this post we’re going to look at a simple way to visualize the power of diversification, and what correlation really tells us.

Suppose you have 2 investments, A and B.

You create a portfolio that consists of 1 share of A, and 1 share of B.

How risky is the combined portfolio? It depends on how risky the individual securities are, and on the correlation of the two stocks’ returns.

Suppose the volatilities of A and B are

A: σA = 6%
B: σB = 8%

and

correlation: ρ = 0

What does correlation ρ = 0 mean?

ρ means, when A’s return deviates from its mean by 1 standard deviation or σA, the expected deviation of B’s return from its mean is ρ standard deviations or ρσB .

In this example, since ρ is 0, we don’t expect B to deviate from its mean in any systematic way.

So, how risky is the combined portfolio? The formula for the volatility of the portfolio is1:

Eq 1

In this example, ρ is 0, so the correlation term goes away, and you are left with:

eq2

Which looks kind of like the Pythagorean theorem, right?

And the general formula including the correlation term looks like the law of cosines, which is the generalization of the Pythagorean theorem to non-right angles:

eq2

Where γ is the angle between the 2 sides.

So, to represent 2 securities with ρ correlation, we can draw a triangle where

  • The length of side 1 of the triangle represents the volatility of asset A: σA
  • The length of side 2 of the triangle represents the volatility of asset B: σB
  • The angle between sides 1 and 2 is the angle whose cosine equals , where ρ is the correlation of assets 1 and 2 (note the formulas are similar but the sign of the last term is reversed)

Figure 1: Risk Triangle, correlation = 0, cos 90° = 0

RiskTriangle1

 

Figure 2: Risk Triangle, correlation = -0.5, cos 60° = +0.5

RiskTriangle2

 

Figure 3: Risk Triangle, correlation = 0.5, cos 120° = -0.5

Risk Triangle 3

 

Figure 4: Table of portfolio vols for various correlations

Stock A
Vol
Stock A
weight
Stock A
wted vol
Stock B
vol
Stock B
weight
Stock B
wted vol
Correlation
ρAB
 
cos-1(-ρ)
 
cos
Portfolio
Vol
 6%  50%  3%  8%  50%  4%  100% 180° -1 7%
 6%  50%  3%  8%  50%  4%  50% 120° -0.5% 6.1%
 6%  50%  3%  8%  50%  4%  0 90° 0 5%
 6%  50%  3%  8%  50%  4%  -50% 60° 0.50 3.6%
 6%  50%  3%  8%  50%  4%  -100% 1 1%

Left as an exercise: draw the pictures for correlation = 1 and correlation = -1.

If you think visually, then these pictures can give you better intuition about what adding imperfectly correlated assets can do to your risk/reward.

“All nature is but art, unknown to thee;
All chance, direction, which thou canst not see;
All discord, harmony not understood;
All partial evil, universal good.” — Alexander Pope

Adendum – more intuitions and speculations about correlation and risk.

  • Diversification is, as Harry Markowitz said, one of the few free lunches in investing.
  • If you can reliably find low-correlation or negatively correlated bets, you can chop risk or supercharge returns.
  • Suppose you can find enough stocks that you can reliably expect to go down, to put 20% of your portfolio in short positions. Your market exposure just went down 20%, and your expected return went up! Alternatively, you could lever your stocks to 120% and go short 20%, keeping your market exposure at 100%, while levering long-term returns more than 20%. Of course, this is easier said than done, otherwise short-selling funds wouldn’t have such a low long-term survival rate, and hedge funds would consistently generate alpha and outperform the market.
  • Ray Dalio: “If you get 15 good — don’t have to be great — uncorrelated return streams, you’ll improve your return to your risk by a factor of 5. That means 5 times the return for the same amount of risk.” Again, easier said than done! To find 5 uncorrelated return streams, that would mean that the ‘dimensionality’ of the return space isn’t 2 dimensions, like the triangles above, but at least 5. Is it?
  • In the short run, and for small amounts of money, you can find a large number of uncorrelated things to bet on. You just need someone to take the other side of the bet. For the right price, Wall Street will make a market in almost anything legal. Of course only some of the bets have attractive, or even positive returns after fees. Nevertheless, anywhere there is someone who naturally is coming into a lot of risk, there is a decent chance they will be willing to pay a decent return to someone to unload or diversify that risk.
  • But you don’t need your money in the short run. You need it at the end of your investment horizon. What matters isn’t the distribution of returns over time. What matters is the distribution of the range of potential returns at the end of your investment horizon. As discussed in the last post, volatility is a proxy for what the people in the market today think risk is, not what it really is.
  • The longer time period you look at, the more things are cointegrated and the fewer dimensions the return space has. If the sun stops shining tomorrow or an asteroid strikes the earth, all your stocks and bonds are going to return the same thing, zero. Of course, we can’t evaluate the probability of such uncertain events, and if they happen, returns won’t make any difference to your outcome, so it’s reasonable to exclude them from the return distribution. The point is, when big historical regime shifts happen, things are more correlated than you think.
  • Just as every form of energy in the economy comes or came at some point from the Sun, all income streams get paid out of GDP or national income. All the investment returns come from GDP.
  • In the short run, in the current market regime, stocks and bonds are seen as good diversifiers, as changes in risk appetite drive people out of stocks and into bonds, and as good prospects for growth and inflation motivate investors to sell bonds and buy stocks. But in the long run, over which inflation expectations adjust, there isn’t supposed to be a long-run Phillips curve tradeoff between growth and inflation. And over a long-term horizon, animal spirits would be expected to even out and long-term returns would reflect their long-term averages.
  • Bond principal and interest must be paid out of national income. If there is a major shortfall in national income, bond defaults rise, governments have to raise taxes, or print money, and there is potential for stagflation in a supply shock scenario. Bonds don’t escape.
  • Of course, nominal bond returns are capped, you will never get more than principal and interest. So the return profile is less volatile. And unfortunately, less correlated in the best scenario for stocks. So in a strong growth scenario, “di-worse-ification.” At current rates, bonds are not priced to provide any long-term real return, merely a hedge against deflation and a really bad outcome for stocks.
  • You can’t make a large, long-term, positive expected real return bet that is negatively correlated with GDP or stocks. You can buy a long-term put from Warren Buffett, but it’s a negative return if the S&P goes nowhere, and if the S&P plunges to 666 so the option is in the money, you have to worry about even Warren Buffett’s ability to pay, and buy CDS to hedge your Berkshire Hathaway credit exposure. When it comes to GDP, we’re all in the same boat. (You could argue some people made such a bet against subprime mortgage CDOs in 2007. And they essentially broke the global financial system and put a lot of doubt on their ability to actually get paid on their bets. I would argue that e.g. gold is a zero long run real return asset that it makes sense to own a bit of, as an inflation/currency collapse hedge.)
  • Since the 80s, real returns for bonds have had a modestly negative correlation with stocks, but over the long haul, bond returns have had a modestly positive correlation with stocks. Over the crisis, they had a hugely negative correlation and offered tremendous diversification. But that has more to do with the specific nature of this crisis, and is not true over all long-term outcomes.
  • A cliché is that in a crisis, correlations go to one. Of course, this is just another way of saying, returns are not normally distributed. If they were, correlations would not go to one. We seem to have 100-year storms every 7 years. Because crises have abnormally large moves, the statement becomes a tautology. (Visualize a regime of tiny fluctuating random change, followed by a big move. It doesn’t matter how big the move is, it will make the correlation go toward one, because all the variance is in the last big move.)
  • Long-term future outcomes aren’t a binomial distribution. More like a binary distribution, things might be very good or very bad. Prices may be a random walk in the short run in a specific market regime, but in the long run, there are major regime shifts, and it’s a series of ‘punctuated equilibria’, long periods of life in Mediocristan followed by violent bouts of Extremistan.
  • Like the length of the coast of Britain, the value of correlation depends on the scale of time and size you’re looking at. In the long run, for institutional-size portfolios, everything is correlated, and the ‘fractal dimension’ of the return space is a very low number.
  • If you have a good intuition for how correlation works, you will have a good feel for how your portfolio will behave and what you can do to make it behave better. And realistic expectations are what gives you confidence to invest and take risk. If you really find good shorts and uncorrelated bets, they are invaluable. But in the long run, the free lunch is only so big.

 

1 To simplify the formulas a little, this is using 1 share of each and dollar volatility.

The more commonly cited formula relating portfolio volatility as a percentage of assets to percentage volatility and weights of underlying assets is:

Equation 3

which we simplified by setting σA = wAσA and σB = wBσB

Easy Pieces on Risk, #1 – Why is volatility a proxy for risk?

Why is volatility a proxy for investment risk?

This is a foundational question about risk, but I’m not sure it gets taught very often explicitly, it kind of gets lost in the math as just another simplifying assumption.

One post in particular that motivated this is Howard Marks’s recent letter. Howard Marks inhabits the same circle of the investing pantheon as Warren Buffett. He has a stellar long-term record and writes very lucidly about complex concepts. If you want to be a fully-fledged member of the fellowship of educated investors, but haven’t read his collection “The Most Important Thing” and made his letters a regular read, start now!

He says:

While volatility is quantifiable and machinable–and can also be an indicator or symptom of riskiness and even a specific form of risk–I think it falls far short as “the” definition of investment risk.

I think he could have been a little more specific if he had wanted, and said volatility is a good proxy for how risky the market thinks an investment is, not how risky it really is1.

Recall Ben Graham’s allegory of Mr. Market:

One of your partners, named Mr. Market, is very obliging indeed. Every day he tells you what he thinks your interest is worth and furthermore offers either to buy you out or to sell you an additional interest on that basis. Sometimes his idea of value appears plausible and justified by business developments and prospects as you know them. Often, on the other hand, Mr. Market lets his enthusiasm or his fears run away with him, and the value he proposes seems to you a little short of silly.

We don’t know with certainty the future cash flows we will harvest from a security. But people who own or might own a security have some idea of the range of potential outcomes. Otherwise they wouldn’t invest.

On a good day, Mr. Market values a security according to the best-case scenario: at the high end of the range of potential valuations. On a bad day he values it according to the worst-case scenario, at the low end of the range. On a typical day he moves a typical distance between the two extremes of fear and greed. ‘Volatility’ is how much a stock moves on a typical day based on Mr. Market’s mood. This amount is a proxy for how wide Mr. Market’s valuation range is, hence how risky the market thinks the security is. If the range is narrow, a typical day’s mood swing, from random news flow and shifts in buy v. sell volume and sentiment, won’t generate a big move in price. If Mr. Market’s valuation range is wide, the daily price swings will be wider2.

This is beautifully illustrated in Marks’s chart, which every investor would do well to think about and mentally internalize:

Figure 1: Howard Marks: Risk and Return
Howard Marks: Risk and Return

As you choose investments and portfolios that are more risky, the mean of the expected return distribution goes up, but the distribution of potential outcomes widens.

Suppose there is news out on the stock, and it drops 50%. A big drop makes its measured historical volatility, the standard deviation of daily returns, spike higher. This generally means a combination of two things. 1) As The Dude would say, “New shit has come to light.” The distribution of potential outcomes has changed. And 2), Mr. Market is reacting with his usual fickleness, and obliging you with an offer to buy or sell at some point in the distribution.

So, has the distribution of potential outcomes worsened and/or become more uncertain? Or is it little changed, and Mr. Market is just having a bad day and offering a price at the low end of the distribution?

If it’s the latter, then paradoxically, ‘quantifiable and machinable’ risk just soared, when actual downside risk of permanent loss plummeted.

And that’s the job of the stockpicker in a nutshell, to recognize when Mr. Market is panicked or depressed, and offering a price at the low end of true potential outcomes, and act dispassionately and decisively in the face of temperamental Mr. Market.


A subtle, but key point is that the volatility is a proxy for how risky the stock is as part of a diversified portfolio. For instance, a highly diversified but highly leveraged company may be very sensitive to economic conditions. A speculative company may be very sensitive to the risk of, for instance the outcome of a mining or pharmaceutical project. However, that risk is less correlated to the market and hence more diversifiable. So if they exhibit similar market volatility, the range of actually potential economic outcomes for the latter may be higher. But the latter would contribute a similar risk to a diversified portfolio, because even though it would have a higher range of outcomes, it would be more likely to be up when the rest of the market was down, and vice versa, dampening its effect on the risk of the whole portfolio.

Another way of visualizing Mr. Market is, on a good day, the most optimistic buyer is setting the price at the margin, on a bad day the most pessimistic seller or most forced liquidator is setting the price. If one stock is more volatile than another, maybe that means each individual market participant has a higher degree of uncertainty about its valuation. Or maybe that means there’s a wide dispersion of views among market participants. Either way, a typical daily swing in sentiment, and variations in buy/sell flow, will need to generate larger swings in prices to clear the market.

 

Retirement plans that maximize certainty-equivalent spending (conclusion)

In part 1 and part 2, we developed a framework for evaluating and identifying a good plan for retirement spending and asset allocation.

  • We discussed how a CRRA (constant relative risk aversion) utility function and the related concept of certainty-equivalent (CE) spending can discount a stream of future cash flows based on their risk and variability, and the retiree’s risk aversion.
  • We solved a toy problem: how the retiree maximizes certainty-equivalent spending if he or she can invest at a guaranteed fixed risk-free real return rate.
  • We generated CE-optimal spending schedules using that fixed risk-free real return rate for retirees with different levels of risk aversion (in this case, no investment risk, just longevity risk, trading off current income for future risk of outliving the portfolio)
  • We moved from a fixed rate assumption to using historical US real returns on stocks and bonds. We generated a spending schedule that maximized CE spending based on historical real returns for a 50% equity portfolio.
  • Using that spending schedule, we solved the other side of the problem, and generated an equity allocation that would have maximized CE spending for that spending schedule.
  • We looked at that solution, and found it seemed pretty good.

So, where we left off, we had independently solved the spending schedule and then the equity allocation schedule. Of course, that does not mean that when you put those two solutions together, they are the best we can do. It just means the equity allocation is the best available given that spending schedule. So today, we’ll try to solve them simultaneously.

The framework in which we try to solve the retirement spending problem is:

Maximize expected CE spending for a 25-year retirement…
Which is modeled as a function of a 2×25 matrix
• 25-years of retirement
• Spending % each year
• Equity % each year (the balance to be allocated to bonds)

And we find CE spending as:

Starting with the 2×25 vector of portfolio allocations and spending %
-> generate cash flows using historical returns for each retirement cohort
-> compute CE spending using CRRA function and gamma
-> compute expected CE spending for each cohort based on life table
-> compute expected CE spending across all cohorts and across all survival scenarios

That gives us a value: the CE spending a random retiree at any year 1926-1987, with the given life table, and given risk aversion, could have expected from that 2×25 spending/allocation schedule.

Now, the problem is to maximize that value: find the 2×25 spending/allocation schedule that maximizes the CE spending function.

So we fire up our optimizer, using this function, gamma=4, and the starting solution we previously found solving the two schedules independently. We try a few different optimization methods. Some of them fail, but the Powell method comes up with a pretty good solution after about six hours on our PC. We use that as our starting solution and run the optimization again using several different methods, and with a very slight improvement it holds up as the best we can find.

Age Equity % Spending %
65 63.8% 6.1%
66 63.7% 6.4%
67 72.1% 6.7%
68 73.7% 6.9%
69 74.4% 7.2%
70 75.2% 7.5%
71 85.3% 7.9%
72 86.1% 8.3%
73 75.3% 8.7%
74 76.8% 9.2%
75 80.9% 9.8%
76 86.3% 10.5%
77 91.6% 11.3%
78 100.0% 12.2%
79 98.9% 13.0%
80 100.0% 14.1%
81 100.0% 15.3%
82 100.0% 16.6%
83 100.0% 18.5%
84 100.0% 20.9%
85 100.0% 24.1%
86 100.0% 28.8%
87 100.0% 36.8%
88 100.0% 52.8%
89 50.0% 100.0%

We see that our initial spending is higher (6.1% vs. 5.9% when we optimized spending and equity independently). We see that in our median case, spending is flatter. We see that the worst-case outcome is a bit worse. Nevertheless it seems credible that the tradeoff is preferable for a moderately risk averse retiree.

Actual spending using computed schedule, % of initial portfolio, 25-year retirement cohorts 1926-1987

Actual Spending using computed schedule, 25-year retirement cohorts 1926-1987

It’s quite interesting that the equity % starts at 63.8% and rises throughout retirement. Conventional wisdom, as implemented in many target date funds would be to reduce the equity allocation as you get older, since you have less time to recover any shortfall from a major market decline. So that result bears investigation to see if there is an error, or if it’s inherent in the unconventional aspects of this approach.

Otherwise, this seems like an analytically sound approach that yields a good practical result.

Comments are invited.

Retirement plans that maximize certainty-equivalent spending, part 2

Last time we solved the problem of the perfect retirement spending plan, assuming a fixed known real return, and a CRRA utility function.

This time, we’ll try to look at the problem from the other angle:

  • Let’s assume a fixed spending schedule
  • Then, let’s solve the problem of the perfect portfolio allocation schedule between US stocks and bonds (of course, the approach could be generalized to other/more assets)

First, a brief digression to make the case that a CRRA utility function is a good thing to use.

CRRA utility

Here is a visualization of what a CRRA utility function looks like for different levels of gamma (use the slider to see how it changes as you adjust risk aversion parameter gamma.

Think of 1 as the ideal income or base case, where utility=0. With risk neutrality or gamma=0, gains and losses generate the same change in utility. As risk aversion increases and gamma goes up, small losses generate bigger and bigger drops in utility, while big gains generate smaller and smaller increases in utility. A ‘no-loss’ utility function with gamma=∞ would be utility=0 for consumption >=1 and a straight line down to -∞ for consumption<1.

One possible objection is, OMG, CRRA utility is such a strange complicated abstraction! No one actually thinks that way.

But we’re all used to thinking about mean-variance (or some of us, anyway). Clearly there is a tradeoff between the volatility of a portfolio, the distribution of potential outcomes, and the return we are willing to accept. So at some level a lot of our thinking about finance involves something very similar to applying a discount to future income streams based on how risky and volatile they are. That’s what the CRRA utility function does – apply a discount based on distribution of outcomes or volatility.

Another possible objection is, ‘utility’ is unobservable in the real world.

But if the utility function correctly ranks the outcomes consistently with the way a human would, at some level that’s all that matters1. The actual value is arbitrary. And as far as I know, any consistent ordinal ranking can be mapped to a cardinal utility function. And we can ask people which outcomes they prefer, either a priori asking them to rank risky outcomes to estimate their risk aversion, or simply generating CRRA-consistent retirement profiles with varying levels of risk aversion, and asking them to choose one.

Finally, why CRRA utility? The important property of CRRA utility is that it’s scale-invariant. A distribution of cashflows between 10 and 15 gets the same discount as a similar distribution between 100 and 150. So if you use a non-CRRA function, you’re going to get different answers depending on the size of the income streams that get generated.

So, if we think that, to reasonable approximations, humans are risk averse, they make consistent choices about risky outcomes, and their risk aversion is scale invariant over the range of outcomes we are studying, a CRRA utility function seems like a reasonable thing to use, as an approximation that leads to a problem we can solve.

Optimal allocation schedules

Last time we looked at the problem of finding the optimal spending schedule, given a known future return.

How do we find an optimal portfolio allocation schedule, given a known e.g. 25-year spending plan?

What we want: the expected CE spending for a given spending schedule and equity allocation schedule.

We write a function that for a given cohort, eg people who retired in 1987:

  • Takes as input the equity % in each year (a 1×25 vector – the bond% is implied as 1-equity%)
  • Uses the known returns for the 25 years 1987-2012, and the known spending schedule, to compute the retirement cash flows
  • Returns the certainty-equivalent cash flow for the 25-year period.

Now that we can calculate the CE spending over 25 years, we can write a second function that

  • Takes as input the cash flows for a period
  • Uses our life table with how many people survived in each year, and computes the expected value of the CE spending that 1987 retiree would have expected across all survival scenarios. (Since some people died in each year, it’s the CE cash flow through each year i of retirement, weighted by the percentage of retirees who lived to year i)

We can further write a third function that calls the second function on each cohort from 1926-1987, and computes the CE spending that each cohort retiring from 1926-1990 would have expected. That gives a distribution of outcomes which we also discount using the CRRA utility function, giving us what we want: the expected CE spending for someone who retired in a random year 1926-1990.

So the sequence is:

2×25 vector of portfolio allocations and spending %
-> cash flows
-> CE spending
-> expected CE spending for a cohort based on life table
-> expected CE spending across all cohorts and across all survival scenarios, e.g, for a random 25-year retiree at any year 1926-1987.

Finally, we can systematically try a universe of allocation schedules and spending schemes and find the one that maximized the expected CE spending for someone who retired in a random year in our history.

The optimal retirement plan is the one that would have maximized expected utility over all historical cohorts and survival timelines.

This does not seem completely computationally intractable in this day and age, so let’s try to compute it.

We add to our code from last time to

  1. 1) find an optimal spending schedule for a fixed return (what we did last time)
  2. 2) find an optimal spending schedule for a 50% equity portfolio that would have maximized CE spending using historical returns 1926-2012)
  3. 3) Use gamma=4, and the fixed spending schedule we found in 2), we find the equity allocation schedule that would have maximized CE spending using historical returns 1926-2012

In other words, to make the problem a little more tractable, first we find an optimum spending schedule for a 50% equity portfolio. Then, using that spending schedule, we find the optimal equity allocation schedule.

We end up with this schedule:

Age Equity % Spending %
65 61.5% 5.9%
66 61.9% 6.1%
67 70.7% 6.3%
68 72.4% 6.6%
69 73.2% 6.8%
70 73.6% 7.1%
71 83.7% 7.4%
72 84.9% 7.8%
73 75.1% 8.2%
74 76.3% 8.6%
75 80.2% 9.1%
76 85.6% 9.6%
77 89.7% 10.2%
78 99.8% 10.9%
79 96.4% 11.7%
80 100.0% 12.7%
81 100.0% 13.8%
82 100.0% 15.3%
83 100.0% 17.1%
84 100.0% 19.4%
85 100.0% 22.7%
86 100.0% 27.6%
87 100.0% 35.7%
88 100.0% 51.8%
89 50.0% 100.0%

This is somewhat counterintuitive insofar as conventional wisdom would be to reduce equity as your time horizon shrinks.

Let’s look at what that expected spending profile would have looked like.

Actual Spending using computed schedule, 25-year retirement cohorts 1926-1987

Actual Spending using computed schedule w/gamma=4, 25-year retirement cohorts 1926-1987

This schedule does in fact seem to perform pretty well. Pretty high initial spending, pretty smooth outcomes, pretty good median case, not very catastrophic worst case. Assuming I had a way to annuitize my longevity risk after 90 (which only about 20% of 65-year-olds will outlive), I would be pretty OK with this range of outcomes.

This seems like a sound approach, and the outcome looks like the kind of solution I would be hoping to find.

In a future post, we’ll see if we can determine both simultaneously – an optimal spending plan and portfolio allocation for a given level of risk aversion.

1If, on the other hand, humans care about things like the order in which income streams are experienced, ie you prefer an income of 10 followed by an income of 15 to the other way around, then CRRA utility is not going to capture that. Then maybe we need to move to a Kahneman-Tversky prospect theory utility function. And if people’s risk aversion changes over time, for instance at market peaks at troughs, then that’s also a problem.

Optimal certainty-equivalent spending retirements with DataNitro

Let’s see if we can come up with an ideal spending plan for a retirement, if you have a guaranteed annual return, for different levels of risk aversion.

It’s probably been done before, but seems like a fun illustration of the power of numerical optimization with Excel, Python and DataNitro.

(more…)

Bitcoin is the Linux of payments. And its killer apps will be for US dollars.

bernanke-ronpaulI was scanning the news the other day, and someone on Hacker News mentioned that half the items above the fold on StreetEYE were about Bitcoin. And I said to myself, I haven’t seen the neckbeards this excited since the early days of Linux.

And it hit me, Bitcoin is the new Linux.

Go back to 1998, the days of The Cathedral and the Bazaar and the ‘Halloween Document’, and open source zealots were gleefully foreseeing the day when freedom-loving hackers would take down the evil Microsoft empire.

Linux was how virtuous hackers were going to end the hegemony of robber barons who stifled freedom and innovation and extracted monopoly rents.
(more…)

The key to immortality is first living a life worth remembering. - Bruce Lee

23 queries in 0.590 seconds.