A new study reveals non-expert readers not only prefer AI-generated poetry to poetry created by humans, they are also more likely to judge AI poetry as human-authored than poems written by famous English-language poets, reports the Guardian.
Conducted by researchers at the University of Pittsburgh, the study showed participants 50 poems written by 10 famous English-language poets along with poems generated in the style of those poets by ChatGPT 3.5. None of the five poems selected for a given author were among their most well known and neither human nor AI poems were more than 30 lines. Results found that the odds of a human-authored poem being judged as human-authored were roughly 75 percent that of an AI-generated poem being judged as human-authored. Moreover, participants ranked AI-generated poems higher in terms of overall quality than those written by humans.
The researchers note this represents a stark shift from previous studies using GPT-2, an earlier iteration of the OpenAI platform, which found participants could reliably distinguish AI-generated poetry from poems written by humans.
[See more: AI will increase energy use and could worsen the climate crisis, report says]
The researchers posit that the misidentification of AI-generated poems and the preference for them is linked. While ChatGPT trained to mimic the style of each poet, ranging from Geoffrey Chaucer to Walt Whitman to Dorothea Lasky, the poems generated by AI lack the complexity and opacity of their human-authored counterparts. “Non-expert poetry readers expect to like human-authored poems more than they like AI-generated poems”, the study notes, but the accessibility of the more straight-forward AI-generated poems appealed more, and participants misinterpreted that preference as evidence of human authorship.
While it is understandable that non-experts wouldn’t derive that same enjoyment as literary buffs from the in-depth study and analysis of great human poetry, it is concerning to see the mismatch between their expectations and reality. “Our participants do not expect AI to be capable of producing poems that they like at least as much as they like human-written poetry; our results suggest that expectation is mistaken,” the study points out.
“As generative AI models become both more capable and more common, it is unclear whether ordinary people’s expectations for generative AI will catch up to the reality of generative AI,” the study authors conclude. With evidence indicating that simple disclosures of AI use are often overlooked, the authors emphasise the urgency of “identifying effective disclosure methods.”
