What might “Progress 8” for universities look like?

“The Japanese have a saying,” Sean Connery drawls in the 1993 crime thriller Rising Sun, “‘Fix the problem, not the blame.’”

The Japanese do not, in fact, have this saying: it rests on a double meaning of the word “fix” that the Japanese language does not make, so it is almost certainly an English coinage, given artificial weight by its ascription to a different culture.

Perhaps it is scepticism about its provenance that has prevented high-minded professors and evidence-informed policymakers from applying it within English higher education, because apply there it does not. Pretty much all that system is currently concerned with is finding out whose fault something is and appending the blame to a person, institution, or policy, and considering that sufficient to resolve the issue.

Take the matter of funding, high on everyone in the sector’s worry list right now. For the past three years, the HE regulator, the Office for Students (OfS), has publicly enumerated how many institutions on its register are in dire financial trouble. The Department for Education has been very clear that it will not “bail out” any such institution if (more likely, when) they reach the end of the financial road. Financial instability is the fault of the individual provider, it seems clear, and it is for them alone to resolve.

Against this, pressured institutions point the finger back at government: on the one hand, a decade-long unwillingness to raise the value of the government-imposed fee cap, on the other, a recent willingness to actively dissuade international students (who are not covered by that cap) from attending English universities – and let us not even begin to talk about national insurance rises, deliberately underfunded research commissions, or reductions in an already shrunken central grant.

Perhaps it is natural to focus on these things. Fee caps, regulatory conditions, financial strategies – each has an author, and an author can be argued with, lobbied, replaced. Find who made the decision, change the decision (and probably get rid of the person at the same time), and all will be well.

That assumes, of course, a linear causal connection from the decisions made to the outputs generated. But there is plenty of evidence that any system can produce “emergent behaviour” – outputs which can only be the result of the interaction of the system’s parts, not ascribed to any one part alone. That output might be planned for – neither water nor heat alone will make a steam engine move, but the creation of steam is not only not a surprise, it is why the engine is built as it is – but it might also be unexpected.

It is likely such unexpected outputs will also be, initially at least, unexplained, else they would have already been accounted for in the design of the system.

In theory, English post-18 education ought to be very good at providing explanations for its outputs. A significant portion of the system is made up of universities, institutions specifically commissioned to create, curate, and communicate knowledge of how and why the world works as it does. Moreover, the past thirty years have seen an intense focus on constructing accountability structures, including an annual survey of all graduating undergraduates, agreements to work on equality issues in return for increased fees, and reports on which students reach the start of their second year, the end of their final year, and professional employment a year or so after leaving.

But despite this over-abundance of data, too little of it is being turned into genuinely useful information, capable of being deployed to generate credible explanations of what is happening in the system.

To its credit, the government has noticed the gap. Its Post-16 Skills White Paper commits ministers to working with the OfS on a way to “measure and compare progress in higher education”, on the model of Progress 8 in schools.

We asked Professor Antony Moss to take that commitment at its word, and to show what such a measure could look like if it were built for higher education as it is, and not borrowed wholesale from a school system that works quite differently. His paper, Happily Heva After: A new value add framework for higher education, is published this morning. The paper addresses this challenge directly, taking a well-known data set (the statistics for continuation, completion, and attainment) and interrogating it more thoroughly than has previously been done, to create a valuable source of information about the success, or otherwise, of English institutions registered with the OfS in serving the needs of students from disadvantaged backgrounds.

I would draw out two particularly strong contributions which ought to be in front of every senior leader in the sector.

The first is a function of what data is publicly available – in a characteristically modest way, Tony makes the point almost in passing, but the numbers he examines here make it clear that the institutions for which we currently have no data must, in aggregate, have wider gaps in outcomes for less privileged students than the sector as a whole. Only the regulator can correct that, because it holds the unredacted data – but at present, for three-quarters of registered providers we know nothing of their impact on students from free-school-meal backgrounds, because the data cannot be published.

The second point is to say that Tony’s intention-to-treat approach is one of those ideas that is so powerful once outlined that it seems bizarre it has not been presented before. Because the regulatory metrics are non-cumulative, a provider that loses disadvantaged students in the first year can present a perfectly respectable attainment gap, calculated only across the survivors. The students who left are nobody’s statistic. Tony’s cumulative calculations close that escape hatch: every student a provider chose to recruit stays in the reckoning, and the whole journey becomes attributable to the institution that admitted them.

Thus, the information we can present is powerful – and that which we cannot shows how much more powerful the calculation of these numbers for all providers could be.

However, a word of warning: this paper does not fix the blame. Tony has made no claims here to explain the data he has uncovered. Instead, he has, rightly and usefully, offered a way of better measuring where students from disadvantaged backgrounds are achieving, and where they are not. That is the start of fixing the problem – it may well be that some providers are behaving in reprehensible ways to generate this data, but given that the outcomes show few patterns of institutions by type or region, it seems more plausible that unexamined interactions between different parts of the system are producing emergent behaviour no one wants.

This seems especially likely in the English HE system because of its incoherence – most of its parts are badly or barely governed, and the interplay between them is no one’s responsibility at all. Other work by The Post-18 Project will address this phenomenon and propose solutions. But for now, please read Tony’s excellent and thought-provoking work in the spirit of fixing the problem through greater understanding.

The full provider data set behind this paper is available to download as an Excel workbook or as a CSV file, and you can look up any English provider’s index – or see exactly which figure is missing – at Find your provider.