# Datasheet — Police Sexual-Offence Outcomes (structured + modelled)

Following the *Datasheets for Datasets* tradition (Gebru et al.). This release is a
**structured, modelled derivative** of Home Office police-recorded-crime open data. It
contains **no personal data**: every figure is an aggregate count of offences or outcomes
by police force, offence type and period.

## Motivation
To turn the Home Office police-outcomes open data — published as large, flat spreadsheets —
into tidy tables and calibrated statistical models that answer a specific question: what is
the probability that a recorded sexual offence is charged or shed, and how much of that
probability is driven by the offence versus the police-force area? Built for the Inference
Institute paper *Three in a Hundred* (v6.0).

## Composition
- **Unit:** a (police force × offence × period × outcome) cell, plus modelled summaries.
- **Source rows processed:** ≈ 2.05 million — Home Office modern outcomes 2023/24
  (≈ 712k rows), the 2005/06–2013/14 sanctions series (≈ 287k), and Community-Safety-
  Partnership recorded-crime volumes 2015/16–2024/25 (≈ 1.05M, 359 local areas).
- **Published derivatives** (this release):
  - `police-outcomes-by-force-offence-2023-24.csv` — every force × offence-subgroup with
    counts per outcome group and derived rates (all crime types; `is_sexual` flag).
  - `police-outcomes-charge-model-by-force.csv` — model-predicted P(charge) for a recorded
    rape by force, with 95% confidence intervals.
  - `police-outcomes-funnel-2023-24.csv` — force charge/withdrawal rates with Spiegelhalter
    funnel control limits and outlier flags.
  - `police-recorded-sexual-by-area-csp.csv` — recorded sexual-offence volume by Community
    Safety Partnership (local-authority granularity), latest complete year.
  - `police-outcomes-model-results.json` — headline rates, deviance decomposition, odds
    ratios, funnel and Theil summaries, model AUC/Brier.
- **No missing-data imputation**; partial financial years are excluded from trends.

## Collection process
Downloaded from `assets.publishing.service.gov.uk` (GOV.UK), the Home Office *Police
recorded crime and outcomes open data tables*. Raw files are git-ignored; the parser
(`rape_pipeline.outcomes`) maps the twelve-group outcome taxonomy to analytic categories and
recovers the recorded-offence denominator by summing outcome groups within each cell.

## Preprocessing / cleaning / labelling
Tidy long → wide pivots; outcome-group → category mapping; rate derivation over recorded and
over resolved offences; complete-year filtering for trends. Models (`rape_pipeline.models`):
binomial GLM (odds ratios, predicted probabilities, nested-deviance decomposition), exact
binomial funnel limits, Theil index, and a histogram gradient-boosting classifier with
permutation importance and a calibration check.

## Validation
- **Denominator integrity:** for each force × offence × quarter cell the outcome groups sum to
  the recorded-offence total; rates are computed against that recovered denominator and checked
  to lie in [0, 1].
- **External cross-check:** the derived national sexual-offence charge rate (~5.7%) and the
  rape charge rate (~3.5%) reproduce the published Home Office / HMICFRS figures for 2023/24,
  confirming the parser and category mapping.
- **Statistical validation of claims:** force outliers are flagged by exact binomial funnel
  limits (95% and 99.8%), not eyeballing; the predictive model is judged by **calibration**
  (predicted vs observed charge rate by decile), ROC area (≈ 0.73) and Brier score (≈ 0.05) on
  the weighted cells, and the GLM odds ratios carry confidence intervals.
- **Human review:** outputs were sanity-checked against the known direction of the literature
  (rape charged far less than other sexual offences; victim withdrawal dominant; a wide force
  spread). No individual-level validation is possible or attempted — the data are aggregate.

## Uses
- **Appropriate:** system-level analysis of charging and case-closure patterns; force
  benchmarking with proper control limits; reform monitoring; teaching.
- **Out of scope / prohibited:** predicting or scoring any individual case, suspect or
  complainant; inferring guilt; reading the area-level "force effect" as an individual-level
  cause. *"Victim does not support action"* is the police-recorded outcome category, **not** a
  judgement about a complainant, and must not be used as one.

## Distribution & licence
Source data © Crown copyright, Home Office, under the Open Government Licence v3.0. Derived
tables released CC-BY 4.0. Cite via the paper's inference.institute URL (no DOI).

## Maintenance
Regenerated offline from committed aggregates via `make reproduce`; refreshed from source via
`make outcomes` when the Home Office publishes a new period.

## Ethical considerations
Aggregate official statistics only; no personal or victim-identifying data. The analysis is
observational and ecological — associations at the force/offence level, stated as hypotheses,
never as individual-level causation or credibility judgement.