Government AI Risk: Questions Leaders Should Ask Before Launching AI
A practical framework for government leaders making AI deployment decisions
Imagine a government agency is hiring a program director. A vendor offers an AI-powered screening tool that promises to cut time-to-hire in half by ranking applicants before a human ever reads a resume. The tool works. The vendor is credible. And the deployment decision is about to go sideways — not because anyone did anything wrong, but because nobody asked the one right question first.
Should AI be used for this decision?
Government leaders, and it seems everyone else, are under pressure to adopt AI. That pressure can move AI adoption too fast, leading to failures, or worse, litigation. Many of these problems stem from due-process violations and administrative procedure breakdowns where automated tools affected people's rights without adequate notice or quality controls. [1]
Yet over 1,200 AI-related bills were introduced across U.S. states in 2025, and government organizations have adopted AI policies to help protect their organizations and those they serve. [2] While this is progress, the question missing from many governance conversations isn't "how do we oversee this AI" — it's "should we be using AI for this decision at all?"
Four conditions that answer that question
Prof.Karan Girotra of Cornell University has identified four factors that determine whether a learning algorithm — the kind powering most AI tools today — is appropriate for a given use.[3]
The hiring scenario fails all four.
Here are the four factors and how the scenario fails.
1. Stakes: Are errors reversible?
Learning algorithms cannot provide performance guarantees, and mistakes are difficult to reverse. [4] A candidate screened out before a human sees their application has no path to appeal. When courts have examined AI hiring tools, they’ve ruled that the accountability doesn’t disappear because an algorithm made the decision; it transfers to the agency that deployed it. [5]
2. Stability: Does your historical data reflect outcomes you would choose today?
Learning algorithms find patterns in historical data and apply them forward. This works well when the past is a reliable guide, when conditions are stable, and the outcomes your data reflects are ones you'd deliberately produce again.
But when the data is comprised of the past encoded outcomes you're trying to move away from, the algorithm will faithfully replicate them. A University of Washington study found that AI resume screening tools preferred white-associated names in 85% of cases and Black-associated names in only 9%. [6] The AI wasn't told to discriminate. It learned discrimination from the data.
3. Interpretability: Can you explain the output to the person it affected?
Learning algorithms produce outputs that cannot be explained in plain terms — not because the output is hidden, but because the internal reasoning that produced it is not interpretable by humans. In government, that matters: the obligation to explain a decision to the person it affects is grounded in due process and administrative law. "The AI said so" is not a valid reason for a decision; it does not reflect who bears true responsibility for the decision.
4. Rules: Is this decision governed by clear rules you can specify?
When the logic or rules are already known and can be written down, a well-designed process that applies it consistently is almost always the better fit. Bringing AI into that situation doesn't add intelligence; it adds unpredictability where consistency is required.
In the scenario above, government hiring decisions are rule-based, governed by merit principles, equal opportunity law, veterans’ preference, and civil service requirements. A rule-based system includes checklists, standard workflows, calculation-based determinations, to name a few, that can be manual or computerized.
No governance framework applied after the fact fixes an AI tool that shouldn’t have been deployed in the first place.
Where AI does belong in hiring
None of this means AI has no place in government processes, including hiring. It means AI belongs where the four factors are met, such as drafting job postings, developing interview questions, producing administrative documents. In each case, a human reviews the output before anything consequential happens. AI does what it’s suited for. The human owns the outcome.
The AI fit card
Whenever AI is on the table as a potential solution, whether in a planning meeting, a budget conversation, or a procurement discussion, this framework can help cut through the noise.
Two questions drive every row in this table: Does the history in this system reflect outcomes we'd choose to replicate — or ones we're trying to move away from? And if something goes wrong, can we explain this decision to the person it affected? If you can't answer both with confidence, that's a signal to change the design; keep AI in the drafting and support work, and keep humans in the decisions that determine what happens to people.
The leaders who get the most out of AI won’t be the ones who adopt it fastest.They’ll be the ones who deploy it deliberately.
___
For more information, check out our Tech Modernization services.
Carlos Venegas helps leaders in government connect people, process, and technology into systems that actually work. For over 30 years he has helped government agencies simplify complex systems, implement technology with confidence, and lead change through clarity and empathy. He is the author of three books on Lean process improvement, including Flow in the Office, which is about improving office processes with office automation. Learn more at carlosvenegas.com.
Carlos works in collaboration with The Athena Group, a human-centered technology modernization consultancy serving state and local government leaders. Learn more at athenaplace.com.
___
References
[1] National GovernorsAssociation, “Mitigating AI Risks in State Government”
[2] Baker Botts, “U.S.Artificial Intelligence Law Update,” January 2026
[3] Non-publiccourse material by Prof. Karan Girotra, Cornell University. Cornell Techfaculty page: https://tech.cornell.edu/people/karan-girotra/. eCornell course: link
[4] NIST AI Risk ManagementFramework (AI RMF 1.0), January 2023
