New York City agencies need AI usage policies, audit finds

An audit by New York State Comptroller Thomas DiNapoli found that the New York City Office of Technology and Innovation needs to create better artificial intelligence usage policies for city agencies.

In examining the four agencies included in the Feb. 16 report — Administration for Children’s Services, Department of Education, New York City Police Department and Department of Buildings — auditors found “significant shortfalls in oversight” of AI use. According to a news release, that gap leaves the tools and systems “vulnerable to misguided, inaccurate or biased outcomes in several programs that can directly impact New Yorkers’ lives.”

When Mayor Eric Adams created OTI in 2022, the office inherited responsibility for the oversight and governance of AI usage across city agencies from the previous administration’s Algorithm Management and Policy Officer role. Former Mayor Bill de Blasio set goals to create guidelines for fair and responsible use of AI, but this task remains unfinished, the news release read.

The audit recommended OTI create a new structure governing the use of AI tools and systems by city agencies and assess the risk each tool poses. Auditors also recommended a review of de Blasio administration policies to identify areas that need to be strengthened by OTI.

Despite the alleged lack of guidance, some city agencies that use AI have taken steps internally to address the risks of biased outcomes. The children’s services agency removed certain types of racial and ethnicity data from its “severe harm predictive model,” which is designed to identify children most at risk of abuse and prioritize quality assurance reviews of cases, the news release said.

The children’s services agency has also established internal guidelines specific to the use of AI, and agency officials told auditors they were developing a more formal policy to ensure the guidelines are followed, according to the release.

Auditors found that the NYPD has also created an internal impact and use policy for certain tools, like facial recognition software, that are susceptible to bias, particularly for groups other than white males. Even so, the department has not set a standard of acceptable accuracy for that software, and the release said those efforts are part of NYPD’s broader surveillance policies and not specific to some of the unique risks posed by AI.

In addition to citing a need for guidance, the audit also found that none of the agencies keep a formal inventory of AI systems and tools they use and have in development. A local law enacted in January 2022 requires all city agencies to disclose the algorithmic tools they have used one or more times during the prior calendar year to the mayor’s office. Of the four agencies included in the audit, only the children’s services agency keeps an inventory of all of the datasets that its AI tools use, the news release said.

“Government’s use of artificial intelligence to improve public services is not new,” DiNapoli said in the release. “But there needs to be formal guidelines governing its use, a clear inventory of what’s being used and why, and accuracy standards, so that the algorithms that can help train educators, identify potential criminal suspects, prioritize child abuse cases and inspect buildings don’t do more harm than good.

“I hope the city’s Office of Technology and Innovation acts on my office’s recommendations to help ensure the AI being used by the city is not at risk of bias or inaccuracies.”

An OTI spokesperson told StateScoop that the audit focused on much of the de Blasio administration’s structure, but added that the office is currently looking to hire a Director of AI and Machine Learning, who will lead the work of developing a central AI framework and strategy under the new city structure.

“The administration fully understands the increasingly important role that AI plays in New York City’s technology landscape, and we welcome this opportunity to discuss and reflect on the city’s work to date,” the spokesperson wrote in an email. “While much of this audit focused on the work of the prior administration and a different government structure, this administration’s recent consolidation of technology agencies and entities under the Office of Technology and Innovation (OTI) umbrella puts the City in a strong position to approach AI in a more centralized, coordinated way.”