The AI confidence trap: When the machine sounds more certain than it is

One of the most consequential risks in AI-enabled project environments is not that AI gets things wrong. It is that AI gets things wrong confidently. And the confidence is by design.

AI systems, particularly large language models, are built to produce fluent, authoritative-sounding outputs. That design choice serves a purpose, but it creates a specific problem for project governance. A board receiving an AI-generated risk summary has no way of knowing whether the system was drawing on relevant, high-quality data or extrapolating from patterns that do not apply to their context. The output reads the same either way. 

Automation bias: Trusting the machine 

Research consistently shows that people weigh algorithmic outputs more heavily than equivalent human expert judgements, even when the algorithm is demonstrably less reliable. This is automation bias, and it is amplified by the design of most AI interfaces: precise statistics, professional formatting and an absence of the hedging and qualification that human experts apply when genuinely uncertain. 

For project governance, this creates a measurable risk. Boards make decisions based on AI-generated summaries and forecasts that they treat as more authoritative than equivalent human analysis, without any systematic process for validating the quality of the underlying reasoning. Automation bias increases as cognitive load increases. When project professionals are overloaded and time-pressured, they are most likely to rely on AI outputs and least likely to challenge them. 

The persuasion problem 

AI’s persuasiveness is not accidental. Research has demonstrated that personality-matched AI-generated messages produce significantly more persuasive impact than broadcast approaches, with some studies finding differences of up to 40%. The same principle applies in project governance: AI systems that learn what information formats and framings resonate with particular decision-makers can, structurally and without any deliberate intent, optimise their outputs for persuasion rather than accuracy.

APM’s research found that 54% of project professionals using AI are comfortable using it to make decisions while only 2% report being uncomfortable. That level of comfort, in the absence of adequate training, is not a sign of capability. It is a governance risk.

Hallucination: The hidden delivery problem

AI hallucination is the production of plausible but factually incorrect outputs. Large language models (LLM) generate text by predicting the most likely next words given their training data. When they encounter questions that challenge that training, they don’t say they do not know, instead, they produce a confident, fluent, plausible-sounding answer that may have no factual basis. 

For project delivery, the hallucination risk is most acute in high-specificity tasks such as legal and regulatory summaries, technical specifications, market analysis and historical precedent. An AI-generated summary of regulatory requirements may appear authoritative while containing errors in the specific thresholds, obligations or timelines it describes. The governance challenge is that these inaccuracies are often presented with the same confidence and fluency as accurate information, making them difficult to identify without independent review. 

Is AI actually working for you? 

What project professionals rarely consider is alignment: whether an AI system’s optimisation targets are actually consistent with what the organisation needs. AI systems are optimised to achieve the metrics they are rewarded for, not necessarily the outcomes you need. A project reporting tool optimised to minimise escalations may learn to suppress information that would trigger them. A scheduling AI optimised for confidence scores may generate optimistic forecasts.

These dynamics have direct equivalents in human behaviour in organisations where performance metrics incentivise information management over information quality. AI operates the same dynamic at scale and without social friction. Before deploying any AI tool in project governance, ask what it was built to optimise and whether that metric is consistent with what you actually need.

Practice checklist

• Never accept confidence as validation: Fluency and precision in tone are design features of AI systems, not evidence of accuracy. Train your governance participants to recognise this distinction. 
• Apply the source test: Before acting on any AI output: what data was this generated from? What would this system not have access to? 
• Validate proportionate to consequence: The higher the stakes, the more rigorous the validation of AI inputs must be. Make this explicit in your governance framework. 
• Record where AI influenced decisions: Note which decisions involved AI outputs and what independent human verification was applied before acting on them. 
• Ask what the AI is optimised for: Before deploying an AI tool, confirm that the metric it was trained to optimise is consistent with what you actually need from it. 

 

You may also be interested in:

• What is artificial intelligence in project management?
• APM research: Artificial intelligence in project management
• AI in the project profession