you should. It exposes how much effort the brain goes through to make vision seem effortless.
Synchronizing the senses
So we’ve seen that our perception requires the brain to compare different streams of sensory data against one another. But there’s something which makes this sort of comparison a real challenge, and that is the issue of timing. All of the streams of sensory data – vision, hearing, touch, and so on – are processed by the brain at different speeds.
Consider sprinters at a racetrack. It appears that they get off the blocks at the instant the gun fires. But it’s not actually instantaneous: if you watch them in slow motion, you’ll see the sizeable gap between the bang and the start of their movement – almost two tenths of a second. (In fact, if they move off the blocks before that duration, they’re disqualified – they’ve “jumped the gun”.) Athletes train to make this gap as small as possible, but their biology imposes fundamental limits: the brain has to register the sound, send signals to the motor cortex, and then down the spinal cord to the muscles of the body. In a sport where thousandths of a second can be the difference between winning and losing, that response seems surprisingly slow.
Could the delay be shortened if we used, say, a flash instead of a pistol to start the racers? After all, light travels faster than sound – so wouldn’t that allow them to break off the blocks faster?
I gathered up some fellow sprinters to put this to the test. In the top photograph, we are triggered by a flash of light; in the bottom photo we’re triggered by the gun.
Sprinters can break off the blocks more quickly to a bang (bottom panel) than to a flash (top panel).
We responded more slowly to the light. At first this may seem counterintuitive, given the speed of light in the outside world. But to understand what’s happening we need to look at the speed of information processing on the inside. Visual data goes through more complex processing than auditory data. It takes longer for signals carrying flash information to work their way through the visual system than for bang signals to work through the auditory system. We were able to respond to the light at 190 milliseconds, but to a bang at only 160 milliseconds. That’s why a pistol is used to start sprinters.
But here’s where it gets strange. We’ve just seen that the brain processes sounds more quickly than sights. And yet take a careful look at what happens when you clap your hands in front of you. Try it. Everything seems synchronized. How can that be, given that sound is processed more quickly? What it means is that your perception of reality is the end result of fancy editing tricks: the brain hides the difference in arrival times. How? What it serves up as reality is actually a delayed version. Your brain collects up all the information from the senses before it decides upon a story of what happens.
These timing difficulties aren’t restricted to hearing and seeing: each type of sensory information takes a different amount of time to process. To complicate things even more, even within a sense there are time differences. For example, it takes longer for signals to reach your brain from your big toe than it does from your nose. But none of this is obvious to your perception: you collect up all the signals first, so that everything seems synchronized. The strange consequence of all this is that you live in the past. By the time you think the moment occurs, it’s already long gone. To synchronize the incoming information from the senses, the cost is that our conscious awareness lags behind the physical world. That’s the unbridgeable gap between an event occurring and your conscious experience of it.
When the senses are cut off, does the show stop?
Our experience of reality is the brain’s ultimate construction. Although it’s based on all the streams of data from our senses, it’s not dependent on them. How do we know? Because when you take it all away, your reality doesn’t stop. It just gets stranger.
On a sunny San Francisco day, I took a boat across the chilly waters to Alcatraz, the famous island prison. I was going to see a particular cell called the Hole. If you broke the rules in the outside world, you were sent to Alcatraz. If you broke the rules in Alcatraz, you were sent to the Hole.
I entered the Hole and closed the door behind me. It’s about ten by ten feet. It was pitch black: not a photon of light leaks in from anywhere. Sounds are cut off completely. In here, you are left utterly alone with yourself.
THE BRAIN IS LIKE A CITY
Just like a city, the brain’s overall operation emerges from the networked interaction of its innumerable parts. There is often a temptation to assign a function to each region of the brain, in the form of “this part does that”. But despite a long history of attempts, brain function cannot be understood as the sum of activity in a collection of well-defined modules.
Instead, think of the brain as a city. If you were to look out over a city and ask “where is the economy located?” you’d see there’s no good answer to the question. Instead, the economy emerges from the interaction of all the elements – from the stores and the banks to the merchants and the customers.
And so it is with the brain’s operation: it doesn’t happen in one spot. Just as in a city, no neighborhood of the brain operates in isolation. In brains and in cities, everything emerges from the interaction between residents, at all scales, locally and distantly. Just as trains bring materials and textiles into a city, which become processed into the economy, so the raw electrochemical signals from sensory organs are transported along super-highways of neurons. There the signals undergo processing and transformation into our conscious reality.
What would it be like to be locked in here for hours, or for days? To find out, I spoke to a surviving inmate who had been here. Armed robber Robert Luke – known as Cold Blue Luke – was sent to the Hole for twenty-nine days for smashing up his cell. Luke described his experience: “The dark Hole was a bad place. Some guys couldn’t take that. I mean, they were in there and in a couple of days they were banging their head on the wall. You didn’t know how you would act when you got in there. You didn’t want to find out.”
Completely isolated from the outside world, with no sound and no light, Luke’s eyes and ears were completely starved of input. But his mind didn’t abandon the notion of an outside world. It just continued to make one up. Luke describes the experience: “I remember going on these trips. One I used to remember was flying a kite. It got pretty real. But they were all in my head.” Luke’s brain continued to see.
Such experiences are common among prisoners in solitary confinement. Another resident of the Hole described seeing a spot of light in his mind’s eye; he would expand that spot into a television screen and watch TV. Deprived of new sensory information, prisoners said they went beyond daydreaming: instead, they spoke of experiences that seemed completely real. They didn’t just imagine pictures, they saw.
This testimony illuminates the relationship between the outside world and what we take to be reality. How can we understand what was going on with Luke? In the traditional model of vision, perception results from a procession of data that begins from the eyes and ends with some mysterious end point in the brain. But despite the simplicity of that assembly-line model of vision, it’s incorrect.
In fact, the brain generates its own reality, even before it receives information coming in from the eyes and the other senses. This is known as the internal model.
The basis of the internal model can be seen in the brain’s
