How Much of the World do we Really See?

by Laura Spinney

It sounds impossible, but when Daniel Simons, a psychologist at Harvard University, and his colleague Daniel Levin of Kent State University in Ohio actually did this experiment, they found that fully HALF of those who took part failed to notice the substitution. The subjects had succumbed to what is called change blindness. Taken with a glut of recent experimental results, this phenomenon suggests we see far less than we think we do.

Rather than logging every detail of the visual scene, says Simons, we are actually highly selective about what we take in. Our impression of seeing everything is just that -- an impression. In fact we extract a few details and rely on memory, or perhaps even our imagination, for the rest. Others have a more radical interpretation: they say that we see nothing at all, and our belief that we have only to open our eyes to take in the entire visible world is mistaken -- an illusion.

We don't store elaborate pictures in short-term memory, Dennett said, because it isn't necessary and would take up valuable computing power. Rather, we log what has changed and assume the rest has stayed the same. Of course, this is bound to mean that we miss a few details. Experimenters had already shown that we may ignore items in the visual field if they appear not to be significant -- a repeated word or line on a page of text, for instance. But nobody, not even Dennett, realized quite how little we really do "see. "

Just a year later, at a conference on perception in Vancouver, British Columbia, John Grimes of the University of Illinois caused a stir when he described how people shown computer-generated pictures of natural scenes were blind to changes that were made during an eye movement. Dennett was delighted. "I wish in retrospect that I'd been more daring, since the effects are stronger than I claimed," he says. Since then, more and more examples have been found that show just how illusory our visual world is. It turns out that your eyes don't need to be moving to be fooled. In a typical lab demonstration, you might be shown a picture on a computer screen of, say, a couple dining on a terrace. The picture would disappear, to be replaced for a fraction of a second by a blank screen, before reappearing significantly altered -- by the raising of a railing behind the couple, perhaps. The picture flickers back and forth, and many people search the screen for up to a minute before they see the change. A few never spot it.

It's an unnerving experience. But to some extent "change blindness" is artificial because the change is masked in some way. In real life, there tends to be a visible movement that signals the change. But not always. As Simons points out, "We have all had the experience of not noticing a traffic signal change because we had briefly looked away." And there's a related phenomenon called inattentional blindness, that doesn't need any visual trick at all: if you are not paying attention to some feature of a scene, you won't see it.

Now imagine that the task absorbing their attention had been driving a car, and the gorilla-man had been a pedestrian crossing their path. According to some estimates, nearly half of all fatal auto accidents in the U.S. can be attributed to driver error, including lapses in attention. It is more than just academic interest that has made both forms of cognitive error hot research topics. Such errors raise important questions about vision. For instance, how can we reconcile these gross lapses with our subjective experience of having continuous access to a rich visual scene? Last year, Stephen Kosslyn of Harvard University showed that imagining a scene activates parts of the visual cortex in the same way as seeing it. He says that this supports the idea that we take in just what information we consider important at the time, and fill in the gaps where the details are less important. "The illusion that we see 'everything' is partly a result of filling in the gaps using memory," he says. "Such memories can be created based on beliefs and expectations."

Ronald Rensink of the University of British Columbia in Vancouver believes that our impression of a rich visual world comes from our building internal representations, though he accepts that they are far less detailed than was once thought. According to his "coherence theory," the brain first constructs a temporary layout of the visual scene -- not much more than the basic geometry and light distribution. Then attention comes along and pulls out a few of these "proto-objects" to a higher resolution. More importantly, he explains, "what attention does is to stabilise these representations so that they form an individual object, something with continuity in space and in time." The moment attention is released, they dissolve back into the volatile, unresolved landscape. In Rensink's view, focused attention is needed to perceive change.

But while Rensink or Kosslyn would argue that there is some role for internal images or memory, other researchers argue that we can get the impression of visual richness without holding any of that richness in our heads. Back in 1992, Kevin O'Regan, an experimental psychologist at the French National Centre for Scientific Research (CNRS) in Paris put forward what later became known as his "grand illusion" theory. He argued that we hold no picture of the visual world in our brains. Instead, we refer back to the external visual world as different aspects become important. The illusion arises from the fact that as soon as you ask yourself "am I seeing this or that?" you turn your attention to it and see it.

According to O'Regan, it's not just our impression of richness that is illusory, but also the sense of having control over what we see. "We have the illusion that when something flickers outside the window, we notice it flickering and decide to move our eyes and look," says Susan Blackmore of the University of the West of England, who supports O'Regan's views. "That's balderdash." In fact, she says, we are at the mercy of our change detection mechanisms, which automatically drag our attention here, there and everywhere.

According to this idea, the sensation of "redness" arises from knowing that moving your eyes onto a red patch will produce a certain change in the pattern of stimulation in line with laws of redness. In other words, the role of the brain is to initiate the exploratory action and to hold the knowledge of those laws: together this give rise to the sensation of redness.

Once you dismiss the need for visual memory, O'Regan says, many of the problems that vision researchers have grappled with for decades vanish. Namely, how does the ropy physics of the eye give rise to the largely flawless experience of visual perception? Leaving aside the blind spot in each retina and the fact that we view the world through a jerky sequence of eye movements or "saccades," we have two upside-down retinal images. If you assume that our brains build detailed reconstructions from such inadequate, distorted input, you have to postulate some kind of compensation mechanism in the visual system. In O'Regan's model no such mechanism is required because there is no reconstruction. His theory also explains change blindness. Take the example of the dining couple. The reason you don't notice the raising of the railing is because you didn't notice the railing in the first place-it wasn't relevant so it remained invisible.

O'Regan's ideas have not been generally accepted. "He's pushed the idea that we lack visual representations farther than most people in the field have been willing to," says Simons. But despite their differences, Simons, Rensink and O'Regan all say that of all the myriad visual details of any scene that you could record, you take only what is relevant to you at the time.

In the Simons-Levin experiment, for example, even the object to which the person is attending -- the stranger asking for directions -- can be swapped without them noticing. Despite the fact that they were looking at him for around a minute, half the subjects encoded none of the details of his physical appearance that were later to change. It was not relevant that the stranger had a certain haircut or that his trousers were a certain color. What was relevant was that he was a person in a certain location addressing them with a certain query. "Paying attention to an object does not give you all of that object's properties for free," says Simons. He points out that those who did notice the switch were students of about the same age as the "strangers." Being in the same social group, he and Levin speculated, they would be more inclined to take in individual details, whereas older subjects might categorise the stranger as "student" and leave it at that.

The relationships between attention, awareness and vision have yet to be clarified. But there is one thing on which most researchers agree: because we have a less than complete picture of the world at any one time, there is the potential for distortion and error. And that has all sorts of implications, not least for eyewitnesses. If it is possible to stand less than a metre from a person and talk to them for a minute without taking in more than a few basic facts, how reliable is the testimony of a person who witnesses a scene from a distance, when they were oblivious to its significance and only later came to recall it?

"In my view, imagery plays a key role in many sorts of false memories," says Kosslyn. "One is 'filling in' the gaps and later remembering not only what was attended to, but also what was filled in." In retrospect, he says, we don't make any distinction between the two types of information. For all our experience of a rich visual world, it seems that we take in no more than a handful of facts about the world, throw in a few stored images and beliefs, and produce a convincing whole in which it is impossible to tell what was real and what imagined. As Blackmore puts it: "There is a world and a brain in it, which together are building a construction, a story, a great confabulation."