Why AI Couldn't Crack an 11th-Century Italian Mystery

An art historian in Florence decided to see if AI could crack the mystery he'd just spent years solving—and it couldn't. Elon Danziger's finding was this: that Florence's Baptistery of San Giovanni was not a homegrown project but a papal-led initiative under Pope Gregory VII in the late 11th century. He put three chatbots—ChatGPT, Claude, and Gemini—to the test, seeing if they could sift through scholarship, notice the same historical oddities he did, and arrive at his conclusion about who built the landmark and why. As Danziger writes in the New York Times, they managed the data-mining part but stalled at the leap of interpretation, never assembling a fresh explanation from scattered clues.

Danziger says the problem lies in how large language models work. They excel at finding and summarizing patterns in mainstream thinking, but that very strength makes them prone to ignoring "eccentric or contrarian ideas" that can trigger breakthroughs. A 2006 book by scholar Guido Tigler, which proposed a later date for the baptistery, was central to Danziger's own shift in perspective; AI never surfaced it when asked what texts it planned to rely on, perhaps because Tigler's idea is "not widely accepted." When prodded to examine a long-assumed 1059 consecration of the baptistery by Pope Nicholas II—an event for which no direct record exists—two bots noted the gap but didn't treat it as noteworthy, while Gemini invented supporting evidence.

For Danziger, the exercise doubles as a caution about what AI can and cannot do for scholarship. As for his own scholarship, Danziger zeroed in on the following: an 11th-century Florence arguably too poor to fund such a sophisticated building; the baptistery's heavy architectural debt to Rome's Pantheon, by then a papal church; and a local crisis of confidence in a corrupt bishop that pushed Florentines away from baptisms in their own city. Remove Nicholas II and focus on popes fixated on Rome's ancient heritage, and Gregory VII—who became pope in 1073—becomes, in his view, the only plausible driving force behind the project. (Read his full essay here.)