The Dark Side Of Statistics
Moneyball is a newsletter from Koble exploring the limitations of human decision-making and their implications for startup investing.
We’ve spent four years developing our groundbreaking algorithms, which discover early-stage startups that outperform the market and predict their probability of success.
This week
🧠 Mental Model #35 – Goodhart's Law – The Dark Side of Statistics
📖 Investor reading – Headline joins Europe VC’s wave of growth funds – Venture Mega-Rounds Return to Biotech – 6 things founders should know if they’re fundraising this autumn
💬 Some tweets – The world finding out how venture firms work – Intimate, confidential details about their business – I don’t get London
The Dark Side Of Statistics
“When a measure becomes a target, it ceases to be a good measure.”
Wise words from British economist Charles Goodhart, with profound implications for startups and people who invest in them.
Except Goodhart didn’t write them. What he actually wrote was:
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
Whichever formulation you prefer, “collapse” and “cease” are both apt to describe what happens when you disrespect the latent power of statistics.
Goodhart’s Law was coined in 1975 as a criticism of the Thatcher government’s attempts to conduct monetary policy via targets for broad and narrow money. It outlived that era, and now serves to encapsulate many of the challenges facing founders and investors as they seek to de-risk startups.
Statistics are a valuable tool in any enterprise, but they’re vulnerable to manipulation and debasement, depending on the personal agenda of those who wield them.
And when we play with statistics, we get unintended consequences (i.e. bad stuff).
To highlight how this works, here’s a couple of interesting (albeit depressing) examples.
Fans of the HBO show “The Wire” will recall the city of Baltimore’s strategy of improving crime statistics through reclassification of those crimes; “juking the stats”. Or the school board’s policy of “teaching to the test”. Substantive improvements in learning and public safety elude the city because politicians are focused on the wrong metrics. The measure has become the target.
Many moons ago in the UK, the government put pressure on hospitals to reduce emergency waiting times to within 4 hours. They managed to pull it off with a range of ingenious / unethical hacks, including one famous example whereby patients were made to wait in ambulances rather than inside the hospital. Some A&Es striving to reduce Length of Stay also discharged patients prematurely, leading to increased emergency readmissions.
The Law of Unintended Consequences is alive and well, thanks in part to its dysfunctional but incredibly good looking and popular cousin, Goodhart’s Law. And statistics abuse is everywhere you look in startups (and also in places that we don’t see).
Perhaps the most obvious example is measuring company growth by the crude metric of headcount. A measure of company size has mutated into an arbitrary target that says nothing about the underlying economics of a business. Perverse incentives abound; as the enterprise “scales”, managers benefit from growing their teams, acquiring more responsibility and status in the internal hierarchy, and controlling ever greater budgets.
Another painful symptom of Goodhart’s Law is the tendency to customer acquisition with growth, without factoring in ongoing engagement and churn. Something that looks great in a pitch deck can be a total red herring when it comes to assessing the unit economics and long term viability of an early-stage startup.
Let’s face it, the startup world has its fair share of gamers. So too, the VC industry. Firms that advertise their prowess by the number of pitch decks read, hours of Zoom calls taken, volume of companies invested in, and aggregate capital deployed, are simplifying something very complex. This is startup investing for idiots (and all too often, by idiots).
Implications for investors
Whenever working with statistics, consider unintended consequences; dysfunctional behaviour, questionable ethics, wonky performance measurement, the unreality of generic startup culture.
“Measure what matters” might be a cliché, but it’s a simple and powerful reminder of what investors are up against.
Systems-thinking can help. Understanding the wider company system and interdependencies between metrics, indicators, and actors can help to uncover perverse incentives that destroy value for investors.
And direct, frank, sometimes difficult conversations can help to purge these behaviours from startups before they become endemic and unfixable.
But implications for investors go beyond interrogating the metrics that founders use. Goodhart’s Law – the Dark Side of statistics – invites us to consider our own relationship with data. And compels us to build a relationship with statistics that is honest, healthy, and productive.
Work with Koble
At Koble, we’ve spent four years developing our groundbreaking algorithms, which discover early-stage startups that outperform the market and predict their probability of success.
We’re working with forward-thinking angels, VCs, family offices, and hedge funds to re-engineer startup investing with AI. If that resonates, get in touch.
Investor reading
💰 Venture Mega-Rounds Return to Biotech – The number of deals for $100 million or more have put biotech on pace to come close to full-year 2021’s record total
🇪🇺 Headline joins Europe VC’s wave of growth funds – European VC growth vehicles have been on the march this year with a series of substantial fund closes announced in recent months.
🍁 6 things founders should know if they’re fundraising this Autumn – From pitch dry runs to timing your fundraise right, VCs share their top advice
Some tweets
Parting shot
“You underestimate the power of the Dark Side. If you will not fight, then you will meet your destiny.”
― Darth Vader
Regards from your [Jedi Master] startup investing AI,
About Koble
Koble is re-engineering startup investing with AI, applying quantitative strategies that have disrupted public markets to early-stage startup investing.