A failed validation, and the amendment it produced

The Clinical Outcomes Research Group validates predictive models of clinic-level ART outcomes against public CDC National ART Surveillance System data, under protocols registered before outcome data are accessed. Criteria are frozen at registration, models are locked before the test partition is evaluated, and results are published whether they pass or fail. This note reports the first executed test under that discipline and the amendment it produced.

Amendment N1 to the group's ART outcome modeling protocol executed the clinic-level arm on the frozen panel of Final reporting years 2020 to 2022, with 2021 as the training year, 2022 as the test year, and 375 clinics in the balanced panel. The primary model, an elastic net on one-year-lagged predictors, returned a calibration slope of 0.907 against a registered band of 0.80 to 1.25, a mean absolute error of 0.0573 against a registered maximum of 0.05, and a Spearman rank correlation of 0.644 against a registered minimum of 0.70. It failed on two of its three criteria. The standardization model placed 60.5 percent of test-year clinics inside its 95 percent funnel limits, against a registered requirement of 90 to 98 percent. It also failed, and stratum calibration left the 35 to 37 and 38 to 40 age bands not validated. The standardization model's test computation was performed three times, once for each of two construction corrections justified by internal-consistency checks independent of the verdict; no model, predictor, or criterion changed in response to test performance, and the model failed in all three computations, with coverage of 57.1, 58.9, and finally 60.5 percent.

The record supports a narrow reading. The standardization was centered correctly, with a median observed-to-expected ratio of 0.99, but the spread around it was far wider than sampling variation allows: the signature of genuine between-clinic heterogeneity that age and diagnosis mix do not capture. A gradient-boosted sensitivity analysis failed identically, so the failure is not a model-class failure. The published lagged clinic variables carry real but insufficient signal about next-year clinic rates.

Amendment N2 registers the revised execution designed from that record. Predictors are rebuilt as two-year histories rather than single lagged values. Band-resolution models address the two unvalidated age bands, with recalibration fitted inside the training years only. The funnel limits are rebuilt on an overdispersion factor estimated from training data, and the amendment states openly that the estimand changes with that construction, so a pass under it is a weaker claim than a pass under the original would have been. The frozen criteria of Amendment N1 are carried over numerically unchanged, and two criteria are added, not substituted: coverage of 99 to 100 percent inside the 99.8 percent funnel limits, and a ranking floor of Spearman correlation at least 0.50 that a band's model must meet before the band may be declared validated. The amendment also registers a delimitation statement as a named deliverable: a stratum-by-stratum account of what the evidence supports reporting.

Since Amendment N2 was registered, the position has moved, and this note records the movement rather than presenting the amendment as the next step. Amendment N2 has not been executed and cannot be executed on the data now posted: its registered test-year rule requires a Final reporting year whose outcome files were never opened under a prior amendment, every posted Final year, 2020 to 2022, was opened under Amendment N1, and the Centers for Disease Control and Prevention has not posted 2023 in any form. A run of the Amendment N2 design on a development split shifted one year back, reported as development only and carrying no verdict, would not have met either funnel coverage bar (89.3 percent against 90 to 98 percent at the 95 percent limits, and 97.3 percent against 99 to 100 percent at the 99.8 percent limits) and would still have left the 35 to 37 and 38 to 40 age bands not validated. Amendment N3, drafted and gated on 2026-10-25 and awaiting approval and registration, supersedes Amendment N2 as the operative construction for the confirmatory evaluation on the first never-opened Final year: it replaces the multiplicative dispersion factor with an additive clinic-effect variance, shrinks the lagged band rates, fits the age bands jointly, and removes per-band recalibration from the primary construction. The frozen criteria are numerically unchanged, no verdict under either amendment can overturn the Amendment N1 verdict, and Amendment N2 remains registered and unexecuted of record.

These models predict published clinic-level aggregate rates, and nothing else. They license no statement about an individual patient's prognosis, they are not a clinical tool, and no clinic is identified in any deposited table. That boundary travels with every result the program publishes.

Amendment N2: 10.5281/zenodo.23630227

Parent protocol: https://doi.org/10.5281/zenodo.23541714