NYC's AI Chatbot Doles Out Bizarre, Illegal Advice

The city says it won't take it offline
By Newser Editors and Wire Services
Posted Apr 4, 2024 3:50 PM CDT
NYC's AI Chatbot Doles Out Bizarre, Illegal Advice
Mayor Eric Adams speaks during a news conference at City Hall, Dec. 12, 2023. An AI chatbot meant to help small-business owners in New York City has come under fire for dispensing bizarre advice.   (AP Photo/Peter K. Afriyie, File)

An AI-powered chatbot created by New York City to help small-business owners is under fire for dispensing bizarre advice that misstates local policies and advises companies to violate the law. But days after the issues were first reported last week by tech news outlet the Markup, the city has opted to leave the tool on its official government website. Mayor Eric Adams defended the decision this week even as he acknowledged the chatbot's answers were "wrong in some areas." The AP provides the lay of the land:

  • Its purpose: Launched in October as a "one-stop shop" for business owners, the chatbot offers users algorithmically generated text responses to questions about navigating the city's bureaucratic maze.
  • The disclaimer: A message on the page states the chatbot may "occasionally produce incorrect, harmful or biased" information and the caveat, since-strengthened, that its answers are not legal advice.
  • Its faulty advice: In responses to questions posed Wednesday, the chatbot falsely suggested it is legal for an employer to fire a worker who complains about sexual harassment, doesn't disclose a pregnancy, or refuses to cut their dreadlocks. Contradicting two of the city's signature waste initiatives, it claimed that businesses can put their trash in black garbage bags and are not required to compost.
  • Stranger still: At times, the bot's answers veered into the absurd. Asked if a restaurant could serve cheese nibbled on by a rodent, it responded: "Yes, you can still serve the cheese to customers if it has rat bites," before adding that it was important to assess the "the extent of the damage caused by the rat" and to "inform customers about the situation." (See more examples at the Markup.)

  • Next steps: A rep for Microsoft, which powers the bot through its Azure AI services, said the company was working with city employees "to improve the service and ensure the outputs are accurate and grounded on the city's official documentation."
  • The mayor's stance: At a press conference Tuesday, Adams, a Democrat, suggested that allowing users to find issues is just part of ironing out kinks in new technology. "Anyone that knows technology knows this is how it's done. Only those who are fearful sit down and say, 'Oh, it is not working the way we want, now we have to run away from it all together.' I don't live that way."
  • Critics' perspective: "There's a different level of trust that's given to government," said Jevin West, a professor at the University of Washington and co-founder of the Center for an Informed Public. "Public officials need to consider what kind of damage they can do if someone was to follow this advice and get themselves in trouble." Experts say other cities that use chatbots have typically confined them to a more limited set of inputs, cutting down on misinformation.
(More chatbot stories.)

Get the news faster.
Tap to install our app.
X
Install the Newser News app
in two easy steps:
1. Tap in your navigation bar.
2. Tap to Add to Home Screen.

X