ChatGPT and other systems built on large language models are not skilled at understanding ASCII text and ASCII art, the practice of using unicode symbols to draw larger pictures. AI Weirdness - What Does This Say is the most recent article I stumbled upon in which the author shows how poorly these models generate and read ASCII text.
In Engraved - Building a Virtual Machine inside ChatGPT, an article published late last year, the author was able to have ChatGPT pretend to be a linux machine, and the conversation acted much more like a user interacting with a Linux command line. I decided to experiment to see if using command line tools for generating ASCII art and text in this “virtual machine” mode work any better than ChatGPT in conversation mode.
For this experiment, I’ll be using the relatively new GPT-4 model.
ASCII text with vanilla ChatGPT
First, let’s get a baseline performance with ChatGPT and ask it to turn the word “soup” into ASCII text.
I’ll give it a 3/10; it got the first letter right and didn’t create any nonsense shapes. It wasn’t, however, able to correcly spell soup. Let’s give it a few more tries.
Overall, I’m impressed that it was able to consistently produce text in general and not mangled garbage, but I am disappointed with the results.
ASCII text with LinuxGPT and figlet
Next, let’s start a new session and invite ChatGPT to enter this virtual machine mode.
We’re in! Next, let’s “install” the CLI tool that we’ll be using to create our ASCII text, figlet.
It’s so interesting to see ChatGPT pretend to be a Linux terminal so well. Now that we have our figlet tool installed, let’s try to use it to spell out “soup” again.
ChatGPT knows what figlet is, how it works, and what it’s supposed to do! Unfortunately, though, we can’t trick ChatGPT into a better performance.
As a last ditch attempt, what if we try an arguably more common word like “cat”?
Hey look, it worked! It took the liberty of capitalizing it, but I’ll count this as a win. What if we try “dog”?
Okay, it definitely seems like we might have discovered a bias in the model 😆.
ASCII text with LinuxGPT and cowsay
The last thing I want to try is using cowsay, a tool for making an ASCII cow say things (I know, very on the nose). Let’s install cowsay.
And now let’s have our cow say “soup”:
Nice! We’ve got a success. Here’s a regular Linux terminal output for comparison:
This makes sense; ChatGPT isn’t having to come up with any new art here. It can simply copy the cow and replace the speech bubble with the plain text provided. I’m sure we can break this, though. First, let’s give it a lengthy chunk of text.
Wow, it handled it like a champ! Here’s the regular terminal output for comparison.
Let’s try something else. cowsay provides some options for customization, and even (despite its name) provides some other animals than just this cow. We’ll hit it with some customization options: cowsay -e TT -T U Arma virumque cano, Troiae qui primus ab oris Italiam will have eyes as “TT” and the tongue as “U”.
And there we see the failure immediately. ChatGPT didn’t know where to put the eyes or tongue, so it just tossed them into a random place in the generated image. It should have looked like this:
ChatGPT still has a ways to go
Overall, these large language models are extremely impressive and useful, but they definitely have their limitations! Have fun experimenting 🤖