Are large language models dyslexic?

Like millions of Americans, I am dyslexic. You would probably never know, unless you happen to be sitting in the passenger seat of my car and find yourself yelling “I said left!” as I oddly turn right. Then, if you ask me why I turned the wrong way, you will be unable to comprehend that I just can’t remember which side is my right and which is my left. It is simply impossible for me. 

I know this makes no sense. After all, I have no problem telling the difference between other things. I know up from down. I know black from white. I know forks from spoons. And yet, I do not know left from right. My brain is not wired that way. This is true for many dyslexics, and I suspect multimodal large language models (MLLMs) may be dyslexic too. 

Before I describe a recent study that made me question whether MLLMs are dyslexic, let me tell you what it’s like to have the form of dyslexia that I do, and describe what I believe is happening inside my brain. I will also explain why being dyslexic, which makes life difficult for millions of students around the world, can also be a cognitive gift that fosters creativity and innovation.   

Living with dyslexia

As a kid with dyslexia, school was very hard for me. This is because many of the fundamentals that students need to learn were created by people who process spatial information differently than I do. For example, we humans created two lowercase letters in the English alphabet — “b” and “d” — that are only different because one points left and one points right. For decades, I could not tell the difference. This is a very common problem among dyslexics.  

Similarly, many of our math rules use algebraic steps that depend on left-right directionality. The same is true for telling time on traditional clock faces — it only makes sense if you know the difference between clockwise and counterclockwise. Calendars are hard, too, because the spatial layout depends on the left-right directionality. As a result, following math rules and reading clock faces or calendars are common challenges for many dyslexic kids. 

These challenges don’t end in elementary school. I still remember getting a problem wrong in a physics class during my freshman year at Stanford. There is a simple convention in physics called the “right-hand rule” for determining how vectors point. Unfortunately, when I took the test, I used my left hand. That is dyslexia. It has nothing to do with focus or intelligence — your brain just works differently from the people who created the cultural conventions we use in symbolic languages, mathematics, and many branches of science.

So, how is a dyslexic brain different? I can only speak for myself, but having spent years thinking about the odd mix of strengths and weaknesses that come from how I process spatial information, I am pretty sure I know what’s going on. It all relates to the “mind’s eye.” By this, I mean the way I visualize things inside my mind and store spatial elements in memory. 

For most people, their mind’s eye is oriented behind the bridge of their nose, looking out into the world, unless they make a concerted effort to diverge from that perspective. This makes sense because it’s how our brain receives visual content (i.e., from the first-person perspective). But when I recall things in my mind (objects, environments, images, or text), I don’t visualize them from a fixed first-person perspective. I think about them from all directions at once, more as a vague cloud of perspectives than a single, grounded orientation. 

The problem is, if your brain stores a “b” from all perspectives at once, it becomes an identical symbol to a “d.” It’s not that I confuse these two symbols. It’s that they are the same symbol, the only difference being whether you are visualizing each from the front or behind. The same is true of clock faces. How can you remember the difference between clockwise and counterclockwise if you simultaneously imagine the object from many directions?

This brings me back to multimodal large language models that process and interpret images and videos. These models are remarkable. They can match or exceed human performance on countless tasks, for example, diagnosing cancers from visual slides better than any human. And yet, a recent study found a surprising result: All major MLLMs currently struggle to tell time on analog clocks. According to the study, GPT-4o was only able to correctly read clock faces 8% of the time. Claude-3-5-sonnet was worse at 6%. Gemini 2.0 was the best, but still at only 20%.

These numbers are surprisingly low, especially when you consider that these AI models can perform so well in other contexts. In addition, the same study found that MLLMs also struggle when asked to interpret calendars. This is surprisingly similar to dyslexia in humans, not just in the simple artifacts that cause problems (clocks and calendars), but in the confounding mix of strengths and weaknesses that enables a person like me to earn a PhD and work successfully as a computer scientist and engineer, and yet still fail the “turn left here” test.

Before I move on, I had to test this for myself rather than rely on the academic paper cited above. So, I fired up two popular LLMs and asked them to tell me the number of seconds represented by the red hand on the following clock:

These are the two responses that I got back: 

The correct answer is just under 9 seconds, but both LLMs incorrectly reported the number (11 seconds for Gemini and 12 seconds for ChatGPT). This is a surprising error, especially since both LLMs approached the problem correctly by looking at the distance from the “2” on the dial.  

Now, I’m pretty sure the LLM can “see” which side of the “2” the second hand is pointing at. So why did both LLMs make this mistake, which happens to be the same type of mistake that I would have made as a kid? Well, if you mix up clockwise and counterclockwise, you might say that it’s “a little past the two” if you imagine the hand moving in the wrong direction.

What makes this error confounding is how well LLMs perform in other visual tasks. In 2023, I was involved in a spatial estimation study where we asked 240 people to estimate the number of gumballs in a jar from a photograph. The average person made a 55% error. We also asked ChatGPT 4, and it was significantly more accurate, estimating with a 42% error. Clearly, LLMs can outperform humans on complex visuospatial tasks, and yet, the average first-grader is likely better at reading clocks.

What does this teach us about current AI systems? 

For me, it suggests LLMs store and process spatial information so differently than humans, they sometimes struggle with cultural conventions that assume the viewer maintains a particular perspective. When you ask an AI to interpret a tissue sample and assess if it’s cancerous, the accuracy is not impacted by orientation. But when you ask it to read a clock face, it has to be conceptualized from a specific direction, or the system will make errors. 

In humans, such errors are considered a “learning disability,” and for millions of dyslexics, it creates daily challenges, especially for kids. That said, the ability to conceptualize the world from unconventional perspectives is also a cognitive gift. It may be one of the reasons why dyslexic people are often highly creative and innovative. In fact, research studies have shown that kids with dyslexia score significantly higher on creativity tests than the general public. In addition, many adult dyslexics credit their “disability” for their success in various fields. 

Personally, I am certain my career was transformed by dyslexia. In college, it gave me a deep fascination for how people process spatial information and inspired me to earn a PhD focused on enhancing human perception by adding virtual content to the real world. This led me to the Human Sensory Feedback Group at Air Force Research Laboratory, where I developed the first mixed reality system, and I’ve been working in the fields of VR, AR, and AI ever since. I’ve heard many similar stories from dyslexics who leverage their unique perspectives to innovate in wildly different fields, from artists and filmmakers to scientists, writers, and even many athletes.

As AI systems evolve, I suspect we will learn even more about the benefits and hindrances of perceiving the world in radically different ways. After all, we don’t know how smart AI systems will ultimately become, but we do know they do not learn, think, visualize, or reason in the same ways that our brains do — not even close.