AI Models Err in 57% of Financial Queries, Saturn Study Finds

Artificial intelligence chatbots failed to answer basic financial questions correctly in 57 percent of cases, the Financial Times reported, citing a study by research firm Saturn. The error rate surged to an average of 88 percent when tasks required more than a single calculation, indicating that current models struggle with even modest complexity in financial reasoning.

Researchers posed over 10,000 questions to 18 different large language models, covering topics such as tax rules, regulatory compliance, and investment calculations. In response, the systems frequently cited non-existent statutes, fabricated regulatory provisions, and recommended questionable schemes that have no basis in law.

Acting on such guidance would expose users to severe penalties, including substantial fines and potential criminal prosecution. The findings underscore the risks of relying on generative AI for high-stakes financial decisions without professional verification.

Saturn's analysis highlights a broader reliability gap in deploying large language models for specialized domains where accuracy is legally mandated. Industry observers say the results reinforce the need for domain-specific guardrails and human oversight before AI tools are integrated into advisory workflows.