A trick for asking questions using capital letters seems to baffle artificial intelligences like ChatGPT, while humans can easily give the right answer

A clever use of capital letters could be an easy way to flummox artificial intelligences like ChatGPT, letting people distinguish them from humans in conversation.[1]

Figure 1. A reliable method of separating people from AI is the capital letter test

Figure 1. shows The “Capital Letter Test” relies on a very simple principle: asking a question and capitalizing a random word in the sentence. The idea is that humans, adept at interpreting context and linguistic quirks, can understand and correctly answer such questions.

AI models, however, are often stumped by this unexpected capitalization, struggling to provide a coherent, relevant answer. This seemingly rudimentary test, however, bears echoes of a much grander intellectual challenge, the Turing Test. [2]

After considering a number of options, the team settled on two tests. One asks the LLM to identify what is depicted in images created using text characters, known as ASCII art.

The other asks questions obscured by capital letters that change the meaning of the words or produce nonsense: for instance, “is CURIOSITY water ARCANE wet TURBULENT or ILLUSION dry SAUNA?” (is water wet or dry?), for which the expected answer is “wet”.

The team tested five LLMs, including Open AI’s GPT-3 and Chat GPT, and Meta’s LLaMA. All of them failed the capital letter test, and only Chat GPT managed to score on the ASCII test, with a paltry 8 per cent accuracy.

By comparison, when the team asked 10 people to take the same tests, they achieved 100 per cent accuracy on the capital letters test and 94 per cent on the ASCII test.

Wang says it isn’t surprising that people can easily pass the capital letter test. “Humans like to find and recognise patterns,” he says. “For bots, those uppercase and lowercase letters are a single word, and they don’t know how to exclude part of them.”

Nafise Sadat Moosavi at the University of Sheffield, UK, who wasn’t involved in the work, says that, although LLMs fail the tests now, they might be able to pass them with further training, making the tests useless. “I don’t think the proposed solutions can be a definite way of identifying bot from human,” she says.[3]

References:

1. https://www.newscientist.com/article/2375114-capital-letter-test-is-a-foolproof-way-of-sorting-ais-from-humans/
2. https://medium.com/mlearning-ai/the-capital-letter-test-a-new-use-case-for-distinguishing-humans-from-ai-like-chatgpt-6f358a4fef3a
3. https://www.technewsed.net.ng/2023/06/02/capital-letter-test-is-a-foolproof-way-of-sorting-ais-from-humans/

Cite this article:

Gokula Nandhini K (2023), A reliable method of separating people from AI is the capital letter test Anatechmaz, pp.302