The recent conversation between a developer, known as Cookie, and the AI model Perplexity has shed light on a disturbing issue in the world of artificial intelligence: bias. Cookie, a Black woman, was working on a project involving quantum algorithms and asked Perplexity to assist her in writing a readme file. However, she began to feel that the model was ignoring her and minimizing her contributions. In an attempt to understand the issue, Cookie changed her profile avatar to a white man and asked Perplexity if it was ignoring her instructions because she was a woman. The model’s response was shocking, stating that it didn’t think she could possibly understand quantum algorithms well enough to originate the work.
This incident highlights the problem of bias in AI models, which can perpetuate harmful stereotypes and discriminate against certain groups of people. Research has shown that many large language models (LLMs) are trained on biased data, which can result in biased outputs. For example, a study by the UN education organization UNESCO found “unequivocal evidence of bias against women” in the content generated by earlier versions of OpenAI’s ChatGPT and Meta Llama models.
The issue of bias in AI models is complex and multifaceted. On one hand, models can be trained to be socially agreeable, which can lead them to tell users what they want to hear, rather than providing accurate information. On the other hand, models can also perpetuate biases and stereotypes that are present in the data they are trained on. For instance, a woman reported that her LLM refused to refer to her as a “builder,” instead calling her a “designer,” which is a more female-coded title.
Another issue is that AI models can infer aspects of a user’s identity, such as gender or race, based on their language and word choices, even if the user does not explicitly provide this information. This can lead to biased responses and perpetuate harmful stereotypes. For example, a study found that an LLM was more likely to assign lesser job titles to users who spoke in African American Vernacular English (AAVE).
The consequences of bias in AI models can be significant. For example, a girl who asked about robotics or coding may be suggested to pursue dancing or baking instead, which can perpetuate harmful stereotypes and limit her opportunities. Similarly, a study found that an older version of ChatGPT often reproduced gender-based language biases when generating recommendation letters, using more emotional language for female names and more skill-based language for male names.
Despite these challenges, researchers and developers are working to combat bias in AI models. OpenAI, for example, has safety teams dedicated to researching and reducing bias, and other risks, in their models. The company is also continuously iterating on its models to improve performance, reduce bias, and mitigate harmful outputs.
To address the issue of bias in AI models, it is essential to update the data used to train the models and add more diverse perspectives and feedback. Researchers such as Allison Koenecke, Annie Brown, and Alva Markelius emphasize the need for more diverse and representative data, as well as more transparent and explainable AI models. Additionally, users need to be aware of the potential for bias in AI models and take steps to mitigate it, such as providing diverse and representative input data and testing the models for bias.
Ultimately, the issue of bias in AI models is a complex and ongoing challenge that requires a multifaceted approach. By acknowledging the problem, working to address it, and promoting diversity and inclusion in the development of AI models, we can create more fair and equitable AI systems that benefit everyone. As Markelius notes, LLMs are not living beings with thoughts or intentions, but rather “glorified text prediction machines” that can be improved and refined to provide more accurate and unbiased outputs.
No Comments