Nevada’s genAI-driven unemployment benefits system
As has been shown many times before, current generative AI systems encode bias and racism in their training data. This is not going to go well:
“There’s no AI [written decisions] that are going out without having human interaction and that human review,” DETR’s director told the website. “We can get decisions out quicker so that it actually helps the claimant.” […] “The time savings they’re looking for only happens if the review is very cursory,” explained Morgan Shah, the director of community engagement for Nevada Legal Services. “If someone is reviewing something thoroughly and properly, they’re really not saving that much time.” Ultimately, Shah said, workers using the system to breeze through claims may end up “being encouraged to take a shortcut.” […] As with most attempts at using this still-nascent technology in the public sector, we probably won’t know how well the Nevada unemployment AI works unless it’s shown to be doing a bad job — which feels like an experiment being conducted on some of the most vulnerable members of society without their consent.
Of course, the definition of a “bad job” depends who’s defining it. If the system is processing a high volume of applications, it may not matter to its operators if it’s processing them _correctly_ or not.(tags: generative-ai ai racism bias nevada detr benefits automation)