Don't Trust the Process: When Verifiability Undermines AI Accountability

How do we know if Artificial Intelligence (AI ) systems are as performant and responsibly designed as the AI companies claimthem to be? In a race-driven innovation climate where responsive development requires time and resources, AI developers andproviders may be tempted to misrepresent system performance or overstate their commitment to responsible AI principles.Such circumvention is further enabled by limited access to system components and information by external stakeholders, arestriction commonly justified on the grounds of trade secret protection, privacy and security considerations, among others.In response, a growing community of scholars has been developing cryptographic and statistical solutions that aim to enablerobust verification of specific claims under constrained access. However, the construction of these solutions rely on a set ofshared, yet unexamined, assumptions required to abstract complex real-world governance challenges into computationalrepresentations. In this article, we examine the validity of these assumptions. After detailing the conceptual foundation andanalytical lens we used to interrogate these abstraction processes, we show that existing technical approaches to developingverifiable AI commit systematic fallacies that compromise the validity of these approaches. While the existing technicalverification processes aim to solve critical AI governance problems, we argue that these fallacies create loopholes that can beexploited by dishonest developers and providers, and therefore lead to misplaced trust in these processes. Finally, we discusshow the field of verifiability could be reoriented towards a more nuanced and interdisciplinary approach to develop rigorousverification processes, both technical and non-technical, that support effective AI governance.