Impact Report

Investigating Impact

Opportuned

Opportuned is an independent educational consultancy that works to put data and research at the heart of the design and evaluation of university access, outreach and widening participation initiatives.

Their team unites access and widening participation specialists with data analysts, social scientists, teachers and education leaders to provide a unique combination of sector knowledge and innovative analysis techniques refined in business and academia.

The importance of quantitative evaluation

Whilst the Access Office has previously used external education evaluation specialists for qualitative evaluation to improve its programmes, it is imperative that it also commits to quantitative investigations of programme impact

To properly analyse St John’s programmes, we need to understand who engages with them and to what effect The Access Office has a large amount of data from entries on the Higher Education Access Tracker (HEAT) This can build a picture of the historic impact of the College’s programmes on participants' educational outcomes.

Last year, the College took the first steps in analysing this - commissioning external evaluators to compare which universities participants attended after leaving school compared with the national average. Data relating to socio-economic background was included to contextualise the results.

Whilst this analysis produced encouraging results, the dataset reflected fewer than 500 participants. This figure is now 2,114 and will continue to grow each year. Opportuned have drawn on their expertise working with big datasets to independently investigate the impact of St John’s Access and Outreach work

St John’s Impact

Yet again, we have seen that young learners on St John’s programmes significantly outperform their peers

76% of POLAR Q1 tracked students on a St John's programme attend a top third institution, with 11% attending one within the top three. By comparison, the national average shows that just 35% of Q1 students who go to university will study at a top third institution.

Whilst St John’s Access initiatives are not the only reason for participants’ success, it is reasonable to conclude that these interventions improve students’ outcomes and encourage them to consider elite universities as an achievable goal.

It's been really rewarding to analyse St John's College access data to show the impact of the programmes, and inspiring to see the educational pathways that students go on.

Alastair

Understanding Proxy Metrics

Measuring disadvantage

There are many ways of understanding and quantifying disadvantage. The simple splits between state and independent, selective and non-selective schools have never been sufficient to encompass the spectrum of the multiple disadvantages that pupils face Indeed, even on a more granular level, there is not a single metric that encapsulates this spectrum - hence the emergence of composite contextual bands that triangulate between multiple metrics.

The Access Office records participant data on the individual, school and postcode level

This gives a range of proxy data to understand the disadvantages faced by pupils Due to the project's scope, it has focussed in particular on POLAR4.

POLAR4

POLAR4 is a nationally calculated geographical metric that assesses the likelihood of progression to any HE institution for those in a given postcode region POLAR classifies local areas into five quintiles based on the proportion of people who enter higher education aged 18 or 19 The 20% of areas with the lowest progression are designated Quintile 1

POLAR4 does not necessarily converge with other ways of measuring disadvantage, useful proxy measure of disadvantage fers widely available comparisons

POLAR4: The National Picture

Q5 students outnumber Q1 students at university by a 3:1 margin.

The gap between Q5 students and all other quintiles becomes much more pronounced in top half institutions.

73% of Q5 students who go to university attend a top half institution, as opposed to 53% of Q1 students.

This graph tracks 'placed applicants’ (UCAS data) across the five quintiles in universities.

Students from less advantaged backgrounds make up a lower proportion of the student body at higher ranked universities. Only Q5 students make up greater proportions of the student body at higher ranked institutions than lower ranked institutions.

Programme Impact: POLAR4 Quintile 1

76% of tracked POLAR4 Q1 students on a St John's programme attend a top third institution, with 11% attending one within the top three.

On average, only 35% of Q1 students who go to university will study at a top third institution.

Understanding this analysis

How does the graph work?

On the X axis are university rankings. The higher-ranked universities (Guardian League Tables 2017-23) are further right

On the Y axis is the cumulative % of students who attend a university of this ranking or less, from POLAR4 Quintile 1 based on the national data for the grey data points, and from the St. John’s participant cohorts for the red data points.

Each dot represents a university.

The red flag signifies the top third of universities; every dot to the right of this flag is a top third university. This is a group which has significant overlap with the Russell Group, but includes competitive, high-tariff institutions such as St Andrew's that are not otherwise included.

What do the two shapes show?

The upper row of dots represents national data available from UCAS This works as a loose control.

The lower, concave line of dots represents Inspire data. This represents the same universities as the upper row of grey dots, but using the inspire data rather than national data.

Across the national average, only 35% of Q1 students who go to university will study at a top third institution. This is represented by the upper row of dots.

However, 76% of Q1 students who engage with St John’s access programmes attend a top third institution, with 11% attending one within the top three. This is represented by the lower row of dots.

We can see this through the point at which each line intersects the red flag A steeper curve to the right of the red flag means that a greater percentage of the sample attended a top third university

Programme Impact: POLAR4 Q2-5

Understanding this analysis

These graphs apply the same analysis to the remaining POLAR4 Quintiles

There is a strong correlation between engaging with St John’s access programmes and attending significantly higher-ranked universities than peers from the same POLAR4 quintile

The same curve is echoed across all five groups, which shows that the impact of the activities is not unique to students of any one background

What else can we find out?

These graphs also demonstrate that a high proportion of participants go on to study at the country’s most elite institutions. 16.5% of participants from Q2-Q5 backgrounds go on to study at a university ranked within the top three (Oxford, Cambridge or St Andrews in these years).

This analysis also shows that there is a high floor for student performance, with 82% of St John’s participants attending a top third institution across Q2-Q5.

What does this mean?

Proving causality is challenging, as it would rely on both complete base data and the ability to conduct randomised control trials. This standard of evidence poses questions of feasibility and ethics.

However, it is clear that ambitious and talented students (regardless of social background) were keen to get involved in the College’s programmes, and that those who did outperformed their national peers.

Whilst St John’s Access initiatives are not the only reason that programme participants significantly outperform their peers, it is reasonable to conclude that their work improves students’ outcomes and encourages them to consider elite universities as an achievable goal.