Researchers from Ruhr University Bochum, in collaboration with experts from the CISPA Helmholtz Center for Information Security in Saarbrücken and the GESIS-Leibniz Institute for the Social Sciences in Cologne, have made a startling discovery: a small but impactful percentage of profile pictures on X (formerly Twitter) are AI-generated, often serving as avatars for fake accounts that spread disinformation.

Figure 1. X. (Credit: RUB, Marquard)

The team analyzed nearly 15 million accounts, revealing that about 0.052 percent used AI-generated images. While that may seem minimal, these fake avatars are far from inconspicuous on the platform. "That may not sound like much, but such images feature prominently on Twitter," states lead author Jonas Ricker of Ruhr University Bochum. Their findings were presented at the 27th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID) in Padua on October 1, 2024 [2]. Figure 1 shows X.

Detecting Fake AI-Generated Accounts

To uncover these accounts, researchers trained an AI tool capable of distinguishing between real and AI-generated images. The analysis was based on data collected in March 2023, focusing on accounts with portrait-style avatars, which made up roughly 43 percent of the sample. The AI flagged 7,723 accounts with AI-generated profile pictures, which were further verified through randomized manual checks [1].

The study found notable behavioral differences between accounts using AI avatars and those with genuine profile pictures. Fake accounts tended to have fewer followers and limited follow-backs. Interestingly, many of these accounts were created in 2023, with hundreds established within hours—strong evidence of inauthentic activity. Nine months later, more than half of these fake accounts had been blocked by the platform, reinforcing suspicions of malicious intent.

Disinformation Hotspots: AI-Generated Accounts and Propaganda

The team also examined the kind of content spread by these fake accounts. Topics frequently included politically charged themes, especially regarding former President Trump, Covid-19, vaccination, the Ukraine war, lotteries, and cryptocurrencies. "We can only speculate what’s going on there," says Ricker. "But it’s fair to assume that some accounts were created to spread targeted disinformation and political propaganda."

Future Research and Improved Detection

Looking ahead, the researchers plan to enhance automated detection methods for AI-generated images, particularly as new AI models emerge. For the current study, they limited their analysis to images produced by "StyleGAN 2," the model used by the website thispersondoesnotexist.com. Ricker notes, “We assume that this site is often used to produce AI-generated profile pictures. This is because such AI-generated images are more difficult to trace than when someone uses a real image of a stranger as their avatar.”

This research highlights the significant role AI-generated avatars play in spreading disinformation and underscores the need for ongoing development in automated detection to counteract this growing issue.

Source: Ruhr-University Bochum

References:

1. https://techxplore.com/news/2024-11-common-fake-profile-pictures-twitter.html
2. https://www.eurekalert.org/news-releases/1063917

Cite this article:

Hana M (2024), AI-Generated Profile Pictures on X (Formerly Twitter) Are Fueling Fake Accounts and Propaganda, Study Finds, AnaTechmaz, pp. 1039