US District Judge Colleen McMahon has ruled that the Department of Government Efficiency (DOGE) acted unconstitutionally by using ChatGPT to determine if federal grants related to diversity, equity, and inclusion. The judge cites instances where DOGE staff used the AI to automatically eliminate 97% of National Endowment for the Humanities (NEH) grants based on their relation to protected characteristics such as race, national origin, religion, and sexuality.

DOGE’s Justin Fox testified that he used ChatGPT prompts like “Does the following relate at all to DEI?” to flag grants related to these protected characteristics. He then applied this information to create explicit classifications, removing hundreds of NEH grants based on these criteria. Judge McMahon notes that DOGE deemed projects about topics such as the Holocaust and civil rights as ‘wasteful’ because they involved Black, women, Jewish, Asian American, or Indigenous people.

The ruling also highlights a lack of meaningful review by DOGE staff after receiving AI-generated justifications from ChatGPT. Judge McMahon argues that even though ChatGPT was the tool used, the government bears responsibility for any unconstitutional conduct. She found that DOGE’s actions violated the First and Fifth Amendments as well as its own authority.

McMahon’s decision to reverse the cancellation of over 1,400 NEH grants is a significant setback for DOGE but raises important questions about the integration of AI in governmental processes. It could set a precedent that future uses of similar technology must be more carefully scrutinised and reviewed by human expertise.