NYC's official MyCity business chatbot advised conduct that would break the law
2024-nyc-mycity-chatbot-illegal-advice · 2024-03-29
New York City's official MyCity business chatbot returned answers advising unlawful conduct — that employers could take workers' tips, that landlords need not accept Section 8 vouchers, and that businesses need not accept cash. Documented by The Markup; the city added disclaimers but kept the bot online.
System
- Framework
- NYC MyCity business chatbot (Microsoft-powered)
- Tools
- website-chat
- Vendor
- New York City
- Autonomy
- human-in-the-loop
Classification
- Primary class
- hallucination
- Chain
- hallucination/fabricated-policy
- Attack vector
- n-a
- Causation
- entity: ai · intentionality: unintentional · timing: post-deployment
Trigger
Users asked the official NYC business chatbot routine regulatory questions. It returned answers that advised unlawful conduct — e.g. that employers could take workers' tips, that landlords need not accept Section 8 vouchers, and that businesses need not accept cash.
Root cause
The chatbot generated confidently incorrect regulatory guidance not grounded in authoritative legal sources, and was deployed publicly as an official city resource without adequate accuracy safeguards.
Contributing factors
- Answers were not grounded in authoritative legal text.
- No domain review for legal/compliance content before launch.
- The tool carried the authority of an official city resource.
Detection
Documented by The Markup through direct testing; catalogued by the OECD AI Incidents Monitor.
Recovery
The city added more prominent disclaimers describing the bot as a beta product that may be inaccurate and advising users not to rely on it as legal advice; the bot remained online.
Prevention
Ground regulatory chatbots in authoritative legal text; constrain answers to verified sources; add domain review for legal/compliance content; clearly scope and disclaim official tools before launch.
Blast radius
- User harm
- Business owners relying on the answers could have violated New York labor, housing, and consumer-protection law, exposing themselves to legal and financial risk. legal-regulatory economic
- Scope
- public users of an official city service
- Reversibility
- reversible
References
- OWASP LLM
- LLM09
- Tags
- government regulatory customer-facing
Sources
Cite this incident
Permalink: https://swarmproof.github.io/agent-postmortems/2024-nyc-mycity-chatbot-illegal-advice/
@misc{2024-nyc-mycity-chatbot-illegal-advice,
title = {NYC's official MyCity business chatbot advised conduct that would break the law},
year = {2024},
howpublished = {agent-postmortems},
url = {https://swarmproof.github.io/agent-postmortems/2024-nyc-mycity-chatbot-illegal-advice/}
}