Ofqual defended that figure by pointing out that agreement between an examiner and a senior examiner is:

Also around 60% — Which exposed the underlying uncertainty in assessing students at a single point in time.

The clearest symptom of that objective was that:

A students in historically low-performing classes were downgraded to Bs and Cs — Schools with more resources tend to outperform schools with fewer, so the grader punished students from low-resource schools.

For small schools, Ofqual used teacher-assessed grades only. The unintended effect was:

Better grades for private school students, who tend to have smaller classes — A carve-out for a data problem that produced a fairness problem, which fine-grained evaluation would have caught.

Ofqual withheld the model and its objective until results day in order to:

Stop teachers altering assessments to game the model — Good intentions — but it meant the system did not get sufficient independent, external scrutiny.

The transparency principle the book draws from the case is that:

Such systems should be reviewable by independent experts trusted by the public — The Royal Statistical Society questioned the composition of Ofqual's technical advisory group on exactly this basis.

In the Strava case, the culprit was:

How the interface and the collection of data were designed — The algorithm was not an explicit culprit at all.

The Strava heatmap aggregated roughly how many activities?

One billion — Recorded between 2015 and September 2017, covering 27 billion kilometres.

The US Department of Labor definition of PII notably includes information allowing identity to be inferred by:

Either direct or indirect means — Which is exactly how an aggregated exercise heatmap becomes personally identifying.

ML-SYSTEMS

intermediate

Responsible AI in Practice

ML Engineer path · lesson 80 of 80

A grading model that matched human examiners at 60% agreement and failed anyway, and a heatmap of anonymized data that exposed military bases. Step the three Ofqual failures, flip a privacy default, and drag a compression slider until the headline metric and the long-tail metric come apart.

The idea

A building that passes every load test and still has to be demolished, because nobody asked whether it should have been built on that plot. The engineering was never the question.

How to use

On the trade-offs tab, drag the compression slider and watch the two lines separate — the headline number barely moves while the long-tail number collapses. That gap is the whole finding.

Summer 2020

A levels were cancelled because of COVID-19, and Ofqual, the UK regulatory body for education and examinations, sanctioned an automated system to assign final grades without students taking the test. Teacher assessment alone had been rejected “on the grounds of unfairness between schools, incomparability across generations and devaluing of results because of grade inflation” — so the fairer option, Ofqual surmised, was to combine previous attainment data and teacher assessment using a statistical model.

The accuracy number, and why it is not the story

The symmetry exposes the underlying uncertainty in assessing students at a single point in time, which only fuelled public frustration. The lesson the book draws: “coarse-grained accuracy alone is nowhere close to being sufficient to evaluate a model's performance, especially for a model whose performance can influence the future of so many students.”

The expected objective was grading accuracy for students. The objective Ofqual actually optimized was "maintaining standards" across schools — fitting predicted grades to each school's historical grade distribution. "If school A had historically outperformed school B in the past, Ofqual wanted an algorithm that, on average, also gives students from school A higher grades than students from school B." In one line: Ofqual prioritized fairness between schools over fairness between students.

The model disproportionately downgraded high-performing cohorts from historically low-performing schools. A students from classes where students had historically received straight Ds were downgraded to Bs and Cs — and because schools with more resources tend to outperform schools with fewer, the auto-grader punished students from low-resource schools, which tend to have more students from underprivileged backgrounds.

Key insight

The case also raises a question the book leaves open: it “exemplifies the currently mucky boundary between what should be automated by algorithms and what should not” — and “it's also possible to argue that due to the potential for catastrophic consequences of the A-level grading, it should never have been automated in the first place.”

What kind of failure?

Sort

Sort each incident into the kind of responsible-AI failure it is.

Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.

Fairness

Drop items here

Privacy

Drop items here

Transparency & accountability

Drop items here

6 pieces left to place

Test your knowledge

20 questions

Question 1 of 20

Score: 0

Ofqual's A-level model had an average accuracy across subjects of about:

The short version

A building that passes every load test and still has to be demolished, because nobody asked whether it should have been built on that plot. The engineering was never the question.

In more depth

Responsible AI in Practice

Definition: *"Responsible AI is the practice of designing, developing, and deploying AI systems with good intention and sufficient awareness to empower users, to engender trust, and to ensure fair and positive impact to society. It consists of areas like fairness, privacy, transparency, and accountability."*

There are applications where the use of AI "might altogether be inappropriate or unethical (e.g., criminal sentencing decisions, predictive policing), regardless of which framework you follow."

Case study I — the UK A-level auto-grader, summer 2020

Ofqual's model, tested on 2019 data, had about 60% average accuracy — so they expected 40% of grades to differ from students' actual grades. Their defence: an examiner against a senior examiner also agrees around 60%. "Coarse-grained accuracy alone is nowhere close to being sufficient."

Failure 1 — the wrong objective. Ofqual optimized "maintaining standards" across schools, fitting predicted grades to each school's historical distribution. Ofqual prioritized fairness between schools over fairness between students. Result: A students from classes that had historically received straight Ds were downgraded to Bs and Cs.

Failure 2 — insufficient fine-grained evaluation. It failed to address teachers' inconsistency in evaluation across demographic groups, and ignored multiple disadvantages under the 2010 Equalities Act. The small-school carve-out used teacher-assessed grades only, which produced better grades for private school students, who tend to have smaller classes.

Failure 3 — lack of transparency. "Transparency is the first step in building trust in systems." Ofqual didn't disclose the objective until the day grades were published. "Ofqual's decision to keep their model development in the dark meant that their system didn't get sufficient independent, external scrutiny." Any system that operates on the trust of the public should be reviewable by independent experts trusted by the public.

Case study II — "anonymized" data, Strava 2018

*"Here, the algorithm is not an explicit culprit. Rather it's how the interface and collection of data is designed that allows the leakage of sensitive data."*

One billion activities, 2015 to September 2017, 27 billion kilometres. The public data exposed US forward operating bases in Afghanistan, Turkish military patrols in Syria, and a possible guard patrol in the Russian operating area of Syria.

PII covers information permitting identity to be inferred "by either direct or indirect means." Strava's default was opt-out, and some settings could only be changed through the website, not the mobile app.

"Data opt-in (data collecting isn't by default), not opt-out, should be the default." And: "developers must proactively work to make the right settings the default, even at the cost of gathering less data."

Five sources of bias

Training data · Labeling ("the more annotators have to rely on their subjective experience, the more room for human biases") · Feature engineering — disparate impact "occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral", as when hiring leverages zip code and high school diplomas · The model's objective · Evaluation — "fair, adequate evaluation depends on the existence of fair, adequate evaluation data."

Trade-offs the literature assumes away

Privacy vs accuracy — the higher the level of privacy differential privacy provides, the lower the accuracy — and per Bagdasaryan and Shmatikov (2019), "the accuracy of differential privacy models drops much more for the underrepresented classes and subgroups."

Compactness vs fairness — compression cuts parameter count by 90% with minimal accuracy cost, but "the minimal accuracy cost is indeed minimal if it's spread uniformly across all classes, but what if the cost is concentrated in only a few classes?" Hooker et al. (2019): "models with radically different numbers of weights have comparable top-line performance metrics but diverge considerably in behavior on a narrow subset of the dataset." And pruning incurs a far higher disparate impact than the quantization techniques they evaluated.

If you are working with a system that is compressed or differentially private, allocating more resources to auditing model behavior is recommended.

Act early, and write it down

"A study by NASA shows that for software development, the cost of errors goes up by an order of magnitude at every stage of your project lifecycle."

Model cards (Mitchell et al., 2018), nine sections: model details · intended use · factors · metrics · evaluation data · training data · quantitative analyses · ethical considerations · caveats and recommendations. Because their contents overlap with what a model store should track, "I wouldn't be surprised if in the near future, model stores evolve to automatically generate model cards."

"It is important to not treat responsible AI as merely a checkbox ticking activity... it won't be a replacement for critical thinking on whether a product or service should be built in the first place."