Automating Judgement: AI Risk and Four Steps to Governance
AI governance is trailing AI progress. Guidance is published, revised, and withdrawn. Laws are proposed, delayed, and rewritten. Meanwhile, agencies are already using AI, through agency-provided tools and through those that employees adopt on their own.
That leaves government executives in a difficult position. You are accountable for the risks your agency takes on today, under whatever rules exist today. So, the practical question is this: how do you manage the risk of a technology when the rules trail the technology?
And beyond risk, a second question: when judgment is outsourced to software, who protects the values the law was written to enforce?
Principles outlast rules
I'm not a lawyer, and nothing here is legal advice.* What follows are impressions I took from Cornell's training, Generative AI Law and Ethics. These ideas help executives operationalize AI with prudent guardrails. One point from that training, made by Cornell law professor Frank Pasquale, has stayed with me: one of the primary roles of law is to enforce the shared values of a society. Rules may change, but the values underneath them are stable. Three of those values appear in American decision-making law again and again:
- Accurate and relevant data. People should be judged on data that is correct and that belongs to them. The Fair Credit Reporting Act requires that inaccurate information be corrected, and it limits how long negative information can follow a person.
- An explanation for adverse decisions. Under the Equal Credit Opportunity Act, a creditor who denies credit must give specific reasons. “Your score was too low” is not sufficient.
- Notice that a machine is involved. California's privacy laws give consumers the right to meaningful information about the logic involved in automated decision-making.
Why should consumer laws matter to an agency? Sometimes they apply directly; here is just one example: a utility that checks the credit of a potential customer to set a deposit takes on Fair Credit Reporting Act duties. But even where these laws don't apply, they are the plainest written record of what the public expects from any decision that affects a person.
Here's what these particular laws have in common: none of them regulates a technology. Each one regulates a decision: who was denied, on what data, with what explanation.
This suggests a way to manage AI risk without waiting for the rules to settle. Trying to govern AI as a technology is impractical; the category is too broad, and often the rules for it keep moving. Relying on guardrails that only prohibit risky behavior is also impractical; prohibitions cannot anticipate every behavior. What remains is the use case. A use case is where a system meets a decision, and decisions are what the durable principles govern.
The stakes rise with the decision. Seton Hall legal scholar Brian Sheppard calls deploying technology before it is ready “premature disruption.” That may be tolerable in low-stakes settings, like recommending a movie. It is another matter in regulated settings, where the bureaucratic barriers exist for a reason. Michigan learned this in 2013, when its automated unemployment fraud system falsely accused more than 40,000 residents. When auditors reviewed 22,000 flagged cases, 93 percent involved no fraud. The problem was not the technology itself. It was an ungoverned use case: accusing people of fraud with no human review.
Four steps to AI risk governance
If the principles are stable and the use case is the unit of risk, then governance becomes manageable work. Here are four steps you can use to help manage risk:
- Inventory. List every AI use case in your agency. Include the ones staff adopted on their own.
- Rank. Order use cases along two criteria: (1) by the impact of each decision, and (2) the difficulty in correcting or reversing an error. This means human review may be required for anything (a) touching eligibility, enforcement, money, or liberty, and/or (b) that is difficult to correct.
- Assign. Name one accountable owner per use case. A name, not a committee.
- Review. Re-check each use case on a schedule. Computer scientist Deb Raji and her coauthors call the assumption that AI simply works “the fallacy of AI functionality.” That assumption is itself a risk to manage.
I don't expect the rules to settle any time soon. But the decisions your agency makes—who receives benefits, who gets flagged, who gets a permit—are already governed by durable principles. Start there.
*For decisions that carry legal consequence, involve your counsel.
___
For more information, check out our Tech Modernization services.
Carlos Venegas helps leaders in government connect people, process, and technology into systems that actually work. For over 30 years he has helped government agencies simplify complex systems, implement technology with confidence, and lead change through clarity and empathy. He is the author of three books on Lean process improvement, including Flow in the Office, which is about improving office processes with office automation. Learn more at carlosvenegas.com.
Carlos works in collaboration with The Athena Group, a human-centered technology modernization consultancy serving state and local government leaders. Learn more at athenaplace.com.
