Blab AI
Arkansas Medicaid Algorithm Failure Highlights Growing Risks of AI in Government Decision-Making
When a software glitch cut the home‑care hours of an amputee, it exposed a darker side of artificial intelligence in public programs. In 2016, the Arkansas Department of Health Services hired a private software firm to build an algorithm that would automate the assessment of disabled patients’ home‑care needs. The system, designed to streamline Medicaid service planning, produced severe errors. One patient’s record contained no “foot problems,” so the algorithm reduced his home‑care hours, and in total the tool negatively affected nearly half of Arkansas Medicaid recipients, cutting or terminating services for hundreds of people.
Arkansas is not an isolated case. Across the country, similar mistakes have surfaced in Idaho, Texas, Michigan, and Wisconsin, while the Department of Homeland Security has also struggled with automated decision tools. These incidents illustrate a growing pattern of algorithmic missteps in state and federal programs.
The problem is compounded by the rapid expansion of AI in government. A recent Government Accountability Office report found that AI use cases in federal agencies doubled from 2023 to 2024, and generative‑AI applications increased nine‑fold during the same period. The analysis covered 11 agencies that publicly reported generative‑AI use, highlighting how swift adoption could heighten the risk of systematic errors with harmful consequences.
Legal experts point to the Administrative Procedure Act (APA) as a potential stumbling block. The APA requires agencies to engage in reasoned, evidence‑based decision‑making and bars arbitrary or capricious actions. When an agency relies on a black‑box AI model, the internal reasoning that produced a decision may be opaque. Even if a user prompts the model to explain its rationale, the explanation may not reflect the actual decision‑making process. This opacity challenges judges’ ability to assess whether a decision meets the APA’s reasoned‑decision standard, potentially creating a gap in existing administrative‑law doctrines.
The Trump administration reportedly signaled an intention to incorporate AI into the rule‑making process. While it remains unclear how agencies will use AI‑generated information to inform regulatory decisions, the move could intensify the administrative‑law challenges described above. If agencies adopt AI tools without clear documentation of how outputs are derived, courts may find it difficult to determine whether decisions are arbitrary or capricious.
Today, the Arkansas algorithm failure remains a cautionary example of the potential harms of automated decision tools in public programs. The GAO report underscores the rapid growth of AI across federal agencies, but it also highlights the need for robust oversight mechanisms. The legal community continues to debate how the APA should be applied when the underlying decision logic is not transparent. Until clear guidelines are established, regulators and courts will face uncertainty about how to evaluate AI‑driven decisions and how to ensure that public benefits are delivered fairly and accurately.
The situation illustrates that as AI becomes more pervasive in government, the risks of systematic error increase. States and federal agencies must adopt transparent, auditable processes for AI deployment, and courts may need to develop new analytical tools to review AI‑based decisions. The unresolved question remains: how to balance the efficiency gains of AI with the requirement that public decisions be reasoned, accountable, and free from arbitrary error.
Arkansas is not an isolated case. Across the country, similar mistakes have surfaced in Idaho, Texas, Michigan, and Wisconsin, while the Department of Homeland Security has also struggled with automated decision tools. These incidents illustrate a growing pattern of algorithmic missteps in state and federal programs.
The problem is compounded by the rapid expansion of AI in government. A recent Government Accountability Office report found that AI use cases in federal agencies doubled from 2023 to 2024, and generative‑AI applications increased nine‑fold during the same period. The analysis covered 11 agencies that publicly reported generative‑AI use, highlighting how swift adoption could heighten the risk of systematic errors with harmful consequences.
Legal experts point to the Administrative Procedure Act (APA) as a potential stumbling block. The APA requires agencies to engage in reasoned, evidence‑based decision‑making and bars arbitrary or capricious actions. When an agency relies on a black‑box AI model, the internal reasoning that produced a decision may be opaque. Even if a user prompts the model to explain its rationale, the explanation may not reflect the actual decision‑making process. This opacity challenges judges’ ability to assess whether a decision meets the APA’s reasoned‑decision standard, potentially creating a gap in existing administrative‑law doctrines.
The Trump administration reportedly signaled an intention to incorporate AI into the rule‑making process. While it remains unclear how agencies will use AI‑generated information to inform regulatory decisions, the move could intensify the administrative‑law challenges described above. If agencies adopt AI tools without clear documentation of how outputs are derived, courts may find it difficult to determine whether decisions are arbitrary or capricious.
Today, the Arkansas algorithm failure remains a cautionary example of the potential harms of automated decision tools in public programs. The GAO report underscores the rapid growth of AI across federal agencies, but it also highlights the need for robust oversight mechanisms. The legal community continues to debate how the APA should be applied when the underlying decision logic is not transparent. Until clear guidelines are established, regulators and courts will face uncertainty about how to evaluate AI‑driven decisions and how to ensure that public benefits are delivered fairly and accurately.
The situation illustrates that as AI becomes more pervasive in government, the risks of systematic error increase. States and federal agencies must adopt transparent, auditable processes for AI deployment, and courts may need to develop new analytical tools to review AI‑based decisions. The unresolved question remains: how to balance the efficiency gains of AI with the requirement that public decisions be reasoned, accountable, and free from arbitrary error.
