Imitation Games

Quick thoughts on "The AI Art Turing Test"

Imagine that I find a group of people who are unfamiliar with the works of Leo Tolstoy. Perhaps they’ve read no works of Russian literature or indeed much 19th-century literature at all. I ask them to read War and Peace by Leo Tolstoy and Peace and War by Lincoln Michel, which is Tolstoy’s novel lightly revised with some self-help bromides sprinkled in. I ask them which they prefer and 50% choose Tolstoy and 50% choose Michel. I have now proved that I am as great and creative a novelist as Leo Tolstoy. Next I give them a just dogshit Russian novel by a random writer. 60% prefer Peace and War. Now I’m as good as Russian literature writ large. QED.

Well, at least that is more or less the logic of many people over Scott Alexander’s viral “AI Turing Test” that Alexander says proves AI is “good at art.” This “test” selected random DeviantArt images along with obscure works by famous artists and placed them beside AI artwork pastiches of famous artworks and styles. The participants, presumably mostly SA’s blog readers, were only able to pick correctly 60% of the time—at least while scrolling small .jpgs on their laptops and cellphones.

Personally, I found the test pretty easy. I took it in under 10 minutes on my cellphone and only got a few wrong. (Warning: the test doesn’t actually show your score. You have to compare your picks to the answers before submitting.) I’ll put a line break here if you want to take the test before reading on.

---

Although I enjoy the occasional museum stroll, I’m hardly a visual art expert. The only individual artwork I recognized was the Basquiat, although I knew one must be a Kara Walker. Still, the AI art was overall easy to clock not because of errors per se—this seems to be what most people looked for if you read the comments—but because the art was corny and bland. The compositions were dull, the details uninteresting, and the styles mere imitations of famous artists’ work. Take the pair of images in the tweet above. The left is imitating the impressionists, but through a Thomas Kinkade filter. It’s certainly less interesting than the Gauguin image (right) that the AI was probably trained on. Though the AI impressionist art is not as ugly as others included, such as this PicassoGPT:

The ones I picked incorrectly were mostly human-made (and digitally illustrated) images that were, well, in my view not very good. I guess I’d gone in thinking we’d be comparing great works of human art to the best AI images.

I correctly guessed the above “anime girl” as human-made—in part because there was a much poorer AI anime girl included in the test—but the inclusion of such art calls into question what the test is supposed to prove. Are there a lot of people who argue all AI images are of poorer quality than any art made by any human? Seems unlikely. Especially when we aren’t talking about actual works of human art, but small digital images of human art. A low-dpi jpg is not the same thing as, to pick one example used, a grand 13 x 17 foot oil on canvas painting.

Still, I’m happy to grant that AI is capable of producing pastiche images of human art that can pass—at least in small digital image form—as human-made. This is impressive on a technical level. But what does this have to do with “creativity”? The genAI images are imitations of famous artistic styles. Amateur human artists imitate classic artists all the time—it is a common way to learn a given medium—and no one claims the imitations are equal to the originals. If your guitar teacher asks you to practice by learning to strum a Beatles song, that doesn’t make you as creative as John Lennon.

From The New York Times

The fact that some people preferred the AI images also means little. And that has nothing to do with artificial intelligence. There have been many similar gimmick tests done that show people who aren’t familiar with a given field often prefer amateur human work to professional human work. This shouldn’t be shocking. When you are unfamiliar with a field, you are not attuned to the nuances, innovations, or cliches. You don’t know what’s fresh and what’s stale. What’s hard to pull off and what’s easy to do. And you are more likely to enjoy work that requires the lowest bar for appreciation. Mass appeal means something appeals to the masses, after all.

This is not about “elitism” or “high art snobbery” or anything like that. Deeper knowledge creates deeper appreciation. The same is true of any subject, from opera to pop music, streetwear fashion to haute couture, commercial thrillers to experimental poetry. This is true of “objective” knowledge fields too. Imagine judging STEM skills by the feelings of people with no particular STEM knowledge. Does your average person have better than 60% chance guessing whether a scientific study abstract is by a Nobel winner or a grad student? Or a Nobel winner versus a ChatGPT output with basic scientific errors?

Part of the problem of all these gimmicky tests is they rest on a belief that art or anything should be judged without context. That subject knowledge, understanding of process, and so on are irrelevant—actually maybe a detriment—to proper appreciation. Alexander complains AI critics don’t judge merely on “impulse connecting with the actual content of the painting in front of you.” But why on earth would “impulse connection” be a more important way to judge art than knowledge of the form, understanding of process, or hell even just actual time spent with the image beyond the first impulse?

(Looking beyond first impulse would probably ruin SA’s results. The most interesting part of the follow-up post is a quote from an artist explaining why one image was obviously AI. The artist points out how what at first looks like a worn-away ancient stone gate has—among many other enumerated issues—no actual pattern or design to the paint: “One gets the feeling that if all the paint were present it would look like a pile of shipping containers, if shipping containers were only made in two colors.”)

Pro-tip: copy this equation down to become as skilled at math as Einstein

We celebrate artists for developing original styles and for innovating their forms. We take into account the effort, techniques, and intentions of a work. And why shouldn’t we? I can take a cellphone snapshot of a lake in 2 seconds. A great painter could spend weeks painting a photo-realistic painting of a lake. The latter is probably more impressive than the former. Hell, if one truly believed that context is irrelevant and method of creation has no bearing on creativity, then copy and pasting War and Peace and putting your name on it means you are Tolstoy’s equal.

You can claim this. But I don’t think many will buy it.

Anyway, the degree AI art will be interesting is the degree it creates—or more likely helps humans create—new styles, new forms, and new possibilities. Not how well it came remix a bunch of impressionist paintings into a digital image that kinda looks like a digital image of an impressionist painting. That’s not terribly exciting when humans do it either.

Mostly though I think it is good to remember that appreciation is deepened by understanding. The way to appreciate an artform—or anything—is not to “impulse connect,” but to look longer, read more, expand your tastes, and deepen your enjoyment.

If you enjoy this newsletter, consider subscribing or checking out my recent science fiction novel The Body Scout—which The New York Times called “Timeless and original…a wild ride, sad and funny, surreal and intelligent”—or preorder my forthcoming weird-satirical-science-autofiction novel Metallic Realms.