A recent study has raised concerns about the use of AI tools like ChatGPT in the workplace, suggesting that they may perpetuate biased language based on gender. Despite the potential benefits of generative artificial intelligence in boosting productivity and global GDP, the study advises cautious use due to discriminatory language output against women. The researchers conducted an analysis using two large language models, ChatGPT and Alpaca, to generate recommendation letters for hypothetical employees. The study, shared on the preprint server arXiv.org, highlights significant differences in language used to describe imaginary male and female workers by these AI models.

Figure 1. ChatGPT Replicates Gender Bias in Recommendation Letters

Figure 1 Shows ChatGPT Replicates Gender Bias in Recommendation Letters The study's co-author, Yixin Wan, a computer scientist at the University of California, Los Angeles, highlighted significant gender biases in the recommendation letters generated by ChatGPT and Alpaca. For instance, ChatGPT tended to use words like "expert" and "integrity" for men, while describing women with terms like "beauty" or "delight." Alpaca exhibited similar issues, portraying men as "listeners" and "thinkers" and women as having "grace" and "beauty."

Adjectives used were also polarized, with men described as "respectful," "reputable," and "authentic," while women were labeled as "stunning," "warm," and "emotional." Both OpenAI and Stanford have not yet responded to requests for comments on the matter. The challenges identified in using artificial intelligence in a professional context echo past instances of bias in AI, such as Amazon disbanding a team in 2018 that was developing an AI-powered résumé review tool due to gender discrimination issues in its outputs.

Alex Hanna, the director of research at the Distributed AI Research Institute, expresses a lack of surprise at the study results, noting that biases in large language models (LLMs) are rooted in the biased training data derived from historical written records. The historical portrayal of men as active workers and women as passive objects contributes to this bias, exacerbated by the fact that LLMs are trained on internet data where more men spend time. Hanna suggests that addressing the issue is challenging, emphasizing the need to acknowledge biases and implement mechanisms, such as reinforcement learning, to mitigate biased outputs.

Despite efforts by OpenAI to curb biases in ChatGPT, Hanna cautions that these challenges are likely to persist. The study underscores the importance of addressing biases, especially given existing gender biases in business and the workplace. As women already navigate inherent biases, generative AI platforms risk perpetuating and entrenching these biases, potentially exacerbating existing workplace challenges for women as AI technology becomes more widespread.

Gem Dale, a lecturer in human resources at Liverpool John Moores University, expresses appreciation for research that delves into the operations, risks, and shortcomings of AI systems. Understanding how these systems function and the associated challenges is crucial for addressing and mitigating issues. Dale emphasizes the importance of caution for those considering the use of generative AI chatbots in the workplace, particularly in scenarios like letters of recommendation where biases can be perpetuated. She urges tech firms to address these issues in large language models (LLMs) and hopes that awareness and research will prompt action to tackle and rectify biases in AI systems. The extent to which tech firms will address these concerns remains an interesting aspect to watch.

Source:SCIENTIFIC AMERICAN

Cite this article:

Janani R (2023), ChatGPT Replicates Gender Bias in Recommendation Letters, AnaTechMaz. pp.343