One City; Two Worlds. Slick Graphics; Sloppy Stats
Two weeks ago, the Mayor’s Fund and Oliver Wyman released a report on social mobility in London. It looks pretty, but it’s misleading.
(full disclosure here, I used to work for Oliver Wyman, that’s why I saw the report)
The report argues that social mobility in London lags behind the rest of the UK. I thought that sounded a bit odd and counter to everything else I’ve read. So we at Time to Spare did some digging and found that this argument is based on flawed evidence and doesn’t withstand scrutiny.
Poorly researched reports like this are dangerous. If funders and policy makers incorrectly answer this question, it could have serious consequences. Believing this report may encourage money to flow away from places that actually need it the most. Within London, it might convince us that we desperately “need to do something different”, instead of continuing to fund the less sexy, but the more effective status quo. This would be a disastrous waste of money.
London as a social mobility hotspot - why were we surprised.
Prior to this report, most research points to London as a bright spot for social mobility. The report itself acknowledges that there is a “popular narrative that our capital is a social mobility engine”.
Educationally, the gap in grades between students receiving Free School Meals and everyone else is smallest in London. Similarly, the gap in university participation rates is only 6% - also the lowest in the country. Looking at more than just education, there are dramatically fewer “left behind” areas in London than elsewhere. London has the highest mean and median incomes in the country. And Londoners themselves seem to agree - 78% of them think there are opportunities to make progress in their life chances, compared to just 31% of people in the North East.
It would take some pretty comprehensive and conclusive evidence to overturn those priors. This was what tempted me to spend a day looking at this Mayor’s Fund/Oliver Wyman report to see if the research stacks up.
The evidence as presented.
The first paragraph of this report states:
“Those from London’s most deprived neighbourhoods do relatively worse in employment compared to their peers in the rest of England."
The key statistic that supports this conclusion is:
“Only 17% of London’s professional jobs are occupied by people from lower income backgrounds compared to 30% nationally.”
That sounds pretty conclusive. However, when you look at the evidence in more detail, it doesn’t support that conclusion. In fact, it quite possibly says the opposite.
The ideal evidence
To accurately assess whether social mobility is worse in London, we would need to answer one specific question:
“For a given person from a working class background, are the chances of getting a professional job higher in London, or outside London?”
Instead of measuring this, this report has measured people from two slightly different categories: professionals currently working in London and professionals currently working outside London.
That looks subtle, but makes a massive difference. Specifically, it’s because of Simpson's Paradox:
By Pace~svwiki - Own work, CC BY-SA 4.0
Quantifying the problem.
Rather than just asserting that this makes a difference we can quantify the impact of this mistake. The Oliver Wyman/Mayor's Fund report draws heavily on this earlier report from the Social Mobility Commission. It’s packed full of interesting statistics that add context to the survey of professionals in London and outside London.
Firstly, the SMC report highlights the different populations and different base rates:
- For non-Londoners, 11% of those from professional backgrounds move to London. Only 4% of those from working class backgrounds do.
- Nationally, 32% of jobs are “professional”. In London, it’s 40%.
As a result, we can’t measure social mobility by the share of professionals with working class backgrounds. This will be inherently biased.
There are more professional jobs in London. There are also more people from professional backgrounds moving to the city to fill them. It’s true that in London there are disproportionately more people with professional backgrounds in professional jobs. This is because there are more people with professional backgrounds in the city as a whole, not because people with working backgrounds disproportionately struggle to access professional jobs.
But is this statistical effect large enough to be responsible for the 13pp difference between London and the rest of the UK?
Yes, and with some change.
The model.
Here's a link.
Because the SMC report doesn’t ask exactly the right question, we need to build a model to answer the question indirectly. To build this model we can use the information available in the SMC report, and some sensible assumptions.
We can then evaluate the outcomes of 4 key groups of people from working class backgrounds:
If a larger share of group D gets professional jobs than group A, the report is correct that London is lagging behind the rest of the country.
Unfortunately, there’s too many degrees of freedom to reliably estimate each of these 4 probabilities from the information provided. However, we can use the model to answer an important question: how biased is the statistic in the Oliver Wyman/Mayor's Fund report?
Estimating bias.
To estimate the bias, first we can create a hypothetical scenario where we know for sure there is no difference in social mobility across the country.
(According to the SMC, 34% is the national average probability)
Using our model, we can then answer what the original survey would have said. If, by assumption, there is no difference in social mobility across the country, would the original survey still suggest London was lagging behind?
Yes.
In this scenario, 20.7% of professionals in London have working class backgrounds, compared to 33.5% of professionals outside London. In other words, under the assumption of exactly zero difference in social mobility across the country, London looks like it lags behind the rest of the UK by 13 percentage points, exactly the same amount as in the original survey.
We also have reason to believe this is a lower bound on the size of the bias.
As a proportion of the population, London has 25% more professional jobs than the rest of the country. As a result, it seems likely that a Londoner from a professional background is more likely to get a professional job than their counterparts from outside London. If we add this assumption, the bias of our survey statistic increases further.
Finally, to make matters worse for the validity of the report, this model only accounts for domestic migration. While detailed statistics for London as a whole aren’t as easy to find, we do know that 39% of the City of London’s workforce were born outside of the UK. This suggests that international migration likely follows a similar but exaggerated pattern - a greater share of immigrants will be professionals and they disproportionately move to London.
Hopefully by now it’s clear that this statistic isn’t a particularly stable foundation to use to overturn the popular narrative of London as a social mobility hotspot.
I now want to creep a step further onto shakier ground. I would argue that the statistic used in the Oliver Wyman report, far from being conclusive proof of social mobility problems in London, is actually suggestive evidence that London’s social mobility is leading the pack.
What if this meant the opposite?
The Social Mobility Commission report also says that 78% of people that move to London are in professional occupations. To incorporate this statistic into the model, we need to change some other assumptions. To ensure that the model is consistent with the survey results, we can fix the proportion of professionals with working class backgrounds to 17% and 30% for London and not London respectively.
This change means that we now need to change our probabilities into something like these:
Just to reiterate, under this scenario, if you took a survey of London professionals, 17% would have a working class background. 30% of non-London professionals would. However, in reality, a Londoner from a working class background is 5pp (16%) more likely to get a professional job.
If a survey of professionals can show what looks like a clear problem in London, but still be consistent with the opposite conclusion, it’s clear this is not a useful measure. Furthermore, given the previous research on social mobility, I believe that London is in fact a bright spot for social mobility. This new report offers no evidence to show otherwise.
Interestingly, in the original SMC report that the OW/Mayor's Fund report draws from, there actually is evidence that London is better for social mobility than the rest of the UK.
They show this graph:
Clearly on this measure, London is doing better than the rest of the country. What’s more, if you take these figures and plot it against the share of professionals with working class backgrounds, it’s a direct negative linear correlation (with an R squared of 0.82):
The share of low earners experiencing wage progression is a far more direct measure of social mobility - measuring exactly the population of interest, and asking a far more relevant question. I can’t say for sure, but if the two measures are directly negatively correlated, I would argue that for a particular place, the lower the share of professionals with working class backgrounds the better the social mobility.
Why does this matter?
The Mayor's Fund/Oliver Wyman report takes this 17% and 30% survey statistic and runs with it. As someone with power over policy or with power over funding, if you were to do similarly, you would make detrimental decisions for London and the rest of the UK.
For London's education system:
The report uses this statistic to create a narrative that London has a “broken path to social mobility”. It argues that London has seen great success in educational mobility - achieving high attainment for children from all backgrounds. In particular, London has managed to significantly narrow the attainment gap for children on free school meals. However, the report states “the path between educational achievement and meaningful job outcomes is broken.” The key supporting evidence for this is the 17% and 30% survey statistic.
It’s not hard to imagine a world where this confused analysis will be used to argue that “something else needs to be done”. It could be used to justify speculative programmes, without clear evidence of effectiveness, that may result in a significant waste of time and resources. London’s schools, Local Authorities and voluntary sector clearly have a successful track record in improving educational attainment for people from deprived backgrounds. Based on the research presented here, there’s no good evidence that better educational outcomes don’t lead to better social mobility. Why not keep funding what works?
For tackling poverty in London:
The fact that London's social mobility is better than the rest of the UK shouldn't overshadow the serious issues with poverty in the city. In 2019, 28% of London's population were living in poverty, compared to 2% in the rest of England. 58% of those living in poverty are from a working household. The fact that 37% of children in London live in poverty is an even more damning indictment.
However, there are signs that far from this being a result of poor social mobility, it's likely to be because of London's unusually good social mobility. The main driver of poverty in London is housing costs. If you strip these out, poverty rates are actually lower than the rest of England. Faced with inelastic supply, the city's house prices are quite obviously driven by increased demand. If better social mobility continues to pull in more people from the rest of the country, improving social mobility further is unlikely to be the best method of addressing poverty in London.
For the UK:
It’s not a giant step from a naive interpretation of this social mobility research to argue that more money ought to be invested in London at the expense of the rest of the country. I think that would be a bad idea. It should probably go to Leeds or Manchester instead. On this point, I wouldn’t say anything that isn’t just directly ripped from this blog, so you should just read it there instead.
Recommendations from the report
Having said all this, most of the recommendations from the Oliver Wyman report are very sensible - we should “increase the use of data driven interventions and work to improve network coordination”. The problem is, there isn’t the easily accessible data out there. Clearly, if we can’t even tell if London is doing better or worse than the rest of the UK, we need better data.
However, collecting that data is not straightforward. If you ask every organisation in London to run surveys and collect outcomes on every individual they support, people will be so drowned in paperwork they won’t have time to do any work. The resulting data would be such a mess of interpretations and formats, it would be impossible to sift through.
In our vision, London should be able to use data generated by its complex network of voluntary and statutory organisations to build a city that everyone loves to live and work in. However, to achieve that we need to be pragmatic and practical. Unless you give people a reason to collect that data, they won’t (or if you force them, they’ll just resent you). In the words of the late Clayton Christensen, you need to solve a “job to be done”.
Fortunately in charities, there are plenty of jobs that need to be done. Filling in funding applications, reporting on existing grants, choosing which projects to prioritise, managing volunteers and finding extra support for their most vulnerable beneficiaries are all things charities have to do constantly. For all of these, if you had access to organised data and detailed and clear analysis, your life would be much easier. For the past 18 months at Time to Spare, we’ve been busy building a platform that does exactly that.
We’re also hopeful that having this data available will enable funders and statutory organisations to build a more productive relationship with the voluntary sector. Within the NHS there’s been a lot of discussion around “social prescribing”. Among foundations, there is plenty of conversation about a “funder plus” model. In Local Authorities, the discussions are around how the public sector can help VCS organisations to become “more resilient”. We believe that better data, paired with better analysis of that raw data, is crucial to turn these Powerpoint agenda items into practical action.
If you share our vision for the future of London’s civil society, or want to argue about social mobility statistics, you should drop me an email at tom@timetospare.com