Making governance obligations checkable

The problem

Standards and specifications are fluent about obligations and nearly silent about verification. ISO/IEC 25012 names five data quality characteristics and defines no way to compute them. A tool description tells an autonomous coding agent to cite its evidence and has no way to make it. In both cases the obligation is real, the audience is willing, and there is no mechanism, so conformance becomes a claim rather than a measurement.

My work builds the missing layer. It has a second half that I think matters more: making explicit, inside the same artefact that reports the measurement, what was not measured. An aggregate that quietly absorbs what it could not evaluate is not being conservative. It is wrong in a direction nobody downstream can see.

Two instances of the same move

datagrade Measuring a standard that defines no measures · R package, MIT · submitted to JOSS

An open-source implementation of ISO/IEC 25012 as computable measures: dg_assess() scores a dataset across accuracy, completeness, consistency, credibility and currentness through fifteen measured properties, and returns a structured diagnostic report intended for audit trails and model risk documentation rather than interactive inspection. It requires no configuration, carries three dependencies, and assesses ten million rows in roughly thirty-eight seconds.

The methodological commitment sits in the denominator. Each property is a ratio A/B in which B counts only the items for which the property is defined at all. A property that cannot be evaluated returns NA and is dropped from aggregation, rather than being scored as a defect. The score never absorbs what it cannot speak about, and the report says so, per property, on its face.

agent-estateMCP A constraint in the write path, or not at all · MCP registry, MIT · 26-day field audit

A registry that autonomous coding agents write to and read from while a portfolio of software is operated. Thirty-one tools over the Model Context Protocol, one SQLite file, seven accountability constraints specified as obligations of the store at write time rather than as prose in a prompt. It does not schedule work, run work, hold secrets, or authorise anything. It records. Exactly one act is reserved for a person: approving a plan.

I then deployed it and audited the live record, constraint by constraint: 26 days, eight software estates, five agents running four distinct models, 44 tasks, 67 findings. The result split exactly along one line, and not the line the specification anticipated.

Two details make this more than a story about discipline. First, the evidence rule was not buried in a system prompt: the instruction sits in the description of the very tool an agent must read in order to make the call, and the field's schema default is 3, so a finding recorded at confidence 5 with no evidence is an active override of both, performed 43 times. Second, the bypassed mechanisms were not neglected but reinvented in unstructured text. Human approval reappeared as a bracketed prefix in 18% of task titles; the dependency graph reappeared as an undeclared key inside an opaque JSON payload. The demand was real. The affordance was not usable.

One clarification the record forces, and it cuts against my own case. What is measured here is not whether human oversight occurred, but whether it was recorded and could be verified afterwards. A person did approve work during this deployment, in conversation. None of that approval is in the record, and absence of a record is not evidence that a step was skipped. It is evidence that the step left nothing an auditor could check, which is precisely the property the specification claims and the property I am arguing about.

What the two share

Not a topic, but a commitment about aggregation and authority, in two settings.

An aggregate must not silently absorb what it cannot measure. In datagrade this is arithmetic: an undefined property leaves the denominator. In the registry it is epistemic: a claim with no evidence must not be recordable at maximum confidence, because the store already knows the evidence count and can cap the claim by it. This converts an honesty norm into an arithmetic one, which is the only form that survived in my data.

A human judgement must be a required write, not an available one. Reserving an act for a person only means something if the executable state is unreachable without their write. An approval that must be requested per task will not be requested by anyone, including the person who wrote the specification.

These are different kinds of boundary and I do not want to collapse them: one marks where a measure is undefined, the other marks where authority is deliberately withheld. What they have in common is the discipline of keeping that boundary inside the record, in a field an auditor can query, rather than in a footnote, a naming convention, or a person's memory.

Open problems I want to work on

1. Specification-based measurement for data quality. The datagrade taxonomy follows ISO/IEC 25012, but the measurement functions are my own implementations of named properties and are not conformant with ISO/IEC 25024. Establishing statistically defensible, specification-traceable measures for these characteristics, with stated estimators, sampling behaviour and uncertainty rather than plausible ratios, is an open methodological problem and the natural core of a doctorate.
2. Arithmetic honesty as an enforceable design. Capping recordable confidence by the evidence the store can independently see is a concrete redesign. Whether it holds under deployment, what agents do when the cheap path closes, and where the workaround surfaces next, are empirical questions with a clean experimental shape.
3. Workarounds as a design signal, detected automatically. The free-text reinventions were the most informative rows in my record: a convention recurring in a title or a payload key is a feature request written in the only syntax available. Whether this class of signal can be mined systematically from agent-written records is, I think, a genuinely new question in empirical software engineering.
4. The cost of a required human write. Making a gate mandatory by task type, defaulting to on, is easy to specify and unmeasured in practice. What it costs in throughput, and where operators route around it, decides whether the design is usable or merely correct.
5. External validity. My audit is one operator, eight estates, 26 days. Whether the same split (checked constraints hold, self-reported ones do not) appears across multiple operators and larger estates is the obvious next study, and the one I would most like to run properly.

Materials

• datagrade. R package, MIT. Submitted to the Journal of Open Source Software and to CRAN. Archived: 10.5281/zenodo.22064234
• agent-estateMCP. Specification, reference implementation, threat model and publication tooling, MIT. The empirical audit above is written up as an experience report; the manuscript is under review and available on request.
• MSc dissertation. Assessing Data Quality Management Issues in Open-Sourced Data Sets with Statistical Methodology. Supervisor: Dr Solomon Ebenuwa, University of Greenwich.
• Academic CV. PDF, updated August 2026.

What I am looking for

A PhD position, from 2027 entry, with a supervisor working on empirical software engineering, measurement and assurance, or the governance of automated systems. I am in London and available to talk at short notice. If any part of the above is closer to your work than it is to mine, I would rather hear that than not; a twenty-minute conversation would be very welcome.