The Risks of AI: Hallucinations, Data Privacy, and What Should Never Go Into a Chatbot

Jamaican businesses are rapidly integrating artificial intelligence tools into their operations, but this adoption is outpacing the development of essential safe usage habits. The column has consistently advocated for AI's use in finance for tasks like cash flow forecasting and receipt processing, always accompanied by warnings to verify all outputs and refrain from inputting confidential data.

This week, the focus sharpens on two primary risk categories: the potential for AI to fabricate information and the unintended dissemination of sensitive data provided to AI systems. While these risks do not necessitate abandoning AI, they underscore the critical need for clearly defined operational rules.

A significant concern is AI's tendency to "hallucinate" – generating plausible-sounding but inaccurate information. This can manifest as incorrect financial totals, fabricated contract clauses, or outdated and imagined tax thresholds, all delivered with unwavering confidence. Newer AI models hallucinate less frequently, but the risk remains, particularly with precise figures, obscure facts, and citations where verification is most challenging.

To mitigate these risks, two key habits are essential. Firstly, AI should be used to read and process existing figures, not generate them. Any number or statutory information included in an AI output must originate from the source documents provided. Secondly, similar to due diligence practices, all AI-generated answers should be cross-referenced and verified, especially when dealing with specific financial data or legal requirements.