A prominent law firm, Sullivan & Cromwell, has publicly acknowledged that a recent legal filing submitted to a U.S. bankruptcy court contained significant errors, including fabricated legal citations, generated by artificial intelligence. The firm stated that its internal safeguards designed to prevent such issues were bypassed, leading to the inclusion of “AI hallucinations” – fictitious references and distorted existing legal authorities.
- Key Takeaways
- Sullivan & Cromwell admitted to submitting AI-generated errors in a high-profile legal case.
- The filing included fabricated legal citations and misrepresented existing ones.
- The firm stated that its AI usage policies were not followed, resulting in improper verification of AI output.
- The case involves liquidators pursuing claims against Prince Group, linked to alleged global fraud operations.
- This incident highlights the ongoing challenges and risks associated with integrating AI in legal practice.
The admission was made in a letter to the U.S. Bankruptcy Court for the Southern District of New York. The firm is representing court-appointed liquidators from the British Virgin Islands in their efforts to pursue claims connected to Prince Group and its owner, Chen Zhi. U.S. prosecutors allege Chen Zhi directed fraudulent operations targeting victims globally, seeking to recover billions in cryptocurrency. Chen Zhi was detained earlier this year in Cambodia and subsequently repatriated to China.
Through Chapter 15 proceedings in the U.S., the liquidators aim to secure recognition of their authority to act on behalf of creditors and alleged victims. Prince Group, a British Virgin Islands-incorporated entity, has been identified by U.S. authorities as being associated with extensive fraud schemes in Southeast Asia and has faced sanctions from both the UK and U.S. governments.
A corrected submission revealed that the original April 9 motion contained multiple misstatements of case law and citations that did not support the assertions made. Some citations appeared to have no basis whatsoever. Sullivan & Cromwell has since withdrawn the initial motion and filed a revised version.
The inaccuracies were initially identified by legal representatives for Prince Group and Chen Zhi. They pointed out that language purportedly from the U.S. Bankruptcy Code could not be located, and several legal authorities were either mischaracterized or incorrectly identified. In one specific instance, a cited case was found to refer to a different decision in another judicial circuit.
In a separate filing, the defendants indicated that at least 28 citations were erroneous, including non-existent quotations attributed to the court. They argued that the timing of the correction was disadvantageous, as the revised filing was submitted after their objections had already been made. Consequently, they requested the court to adjourn a scheduled hearing and convene a status conference.
Sullivan & Cromwell emphasized that its established policies mandate that lawyers undergo mandatory training before utilizing AI tools and that all AI-generated output must be independently verified. The firm’s statement detailed:
“Before any Firm lawyer is granted access to generative AI tools, the lawyer must complete two required training modules, completion of which is tracked and verified. The training repeatedly emphasizes the risk of AI ‘hallucinations,’ including the fabrication of case citations, misinterpretation of authorities, and inaccurate quotations. It instructs lawyers to ‘trust nothing and verify everything’ and makes clear that failure to independently verify AI-generated output constitutes a violation of Firm policy.”
The firm also indicated that a comprehensive review uncovered additional minor drafting irregularities in other filings, which were attributed to human error rather than AI. The specific lawyers responsible for preparing the original motion were not identified.
Long-Term Technological Impact: AI Governance and Verification in Digital Law
This incident serves as a stark reminder of the critical need for robust governance and verification protocols as artificial intelligence becomes increasingly integrated into professional workflows, including the legal sector. The concept of “AI hallucinations” is not unique to legal applications; it represents a fundamental challenge in current generative AI models where the systems can produce plausible-sounding but factually incorrect or entirely fabricated information.
In the context of blockchain and Web3 development, where data integrity and verifiable transactions are paramount, the lessons from this legal case are highly relevant. As decentralized applications (dApps), smart contracts, and Layer 2 scaling solutions continue to evolve, they often rely on complex code and intricate legal frameworks. The integration of AI into these areas, whether for code auditing, legal compliance analysis, or even dispute resolution mechanisms, must be approached with extreme caution.
The Sullivan & Cromwell case underscores that AI, while a powerful tool for enhancing efficiency and potentially democratizing access to information, cannot replace human oversight and critical judgment. For the blockchain industry, this means that any AI-driven analysis of smart contracts, regulatory compliance, or the interpretation of decentralized governance protocols must undergo rigorous human review. Furthermore, the development of AI models specifically trained on verifiable blockchain data and established legal precedents, rather than broad internet datasets, could mitigate the risk of hallucinations.
The future of AI in legal and blockchain contexts will likely involve a hybrid approach, where AI assists human experts by performing initial analysis, identifying patterns, and drafting preliminary documents. However, the final validation, especially in high-stakes applications, must remain with human professionals. This incident may accelerate the development of specialized AI verification tools and ethical guidelines within both the legal and the burgeoning Web3 technology sectors, ensuring that innovation does not come at the expense of accuracy and trust.
Original article : decrypt.co