Generative AI in Safety-Critical Systems: Risks of Sycophancy, Hallucination, and Trust Degradation
Generative AI can support safety work, but fluent and confident outputs can reinforce unsafe assumptions, fabricate technical information, and weaken professional scrutiny. This evidence-to-practice brief examines the implications of sycophancy, hallucination, automation bias, and cognitive offloading for safety-critical decision support.
Use generative AI as controlled decision support, not as an autonomous safety authority. Critical outputs require verified sources, expert review, defined approval responsibilities, and clear conditions for rejection or escalation.