Monday, February 25th, 2013

How 'Minority Report' Trapped Us In A World Of Bad Interfaces


I wish I could get away with charging my clients a fee for every time they say "Minority Report" to me. I’m a commercial artist in L.A., and 90% of commercial art is shutting up and giving the client what they want. That means I spend a lot of time trying to repackage Steven Spielberg’s vision of the future: floating graphical windows with video hovering in them, typography flickering and animating in response to actors’ actions, interfaces appearing and disappearing when fingers reach out to poke them. In short, building a virtual iPad interface, hovering in front of the actor using it. In Spielberg's future, you only have to twirl your fingers at a computer screen to make it do what you want. It looks cool enough, but it's time for us to let it go: we’ve built our graphics and our electronics around interface eye candy, rather than trying to come up with new and more effective ways to control our real and imaginary gadgets. The best thing you can say about touchscreens are they look good on camera and they’re better than T9 texting, which is kind of like being better than fax machines.

The next in a short series about our strongest movie opinions,
past and present.

It’s important, of course, to put this in context. Minority Report came out in 2002, and we had touchscreens for a long time before then. If you want to feel really let down by the future, here’s the Prius computerscreen-o-matic as interpreted by a 1987 Buick Riviera. Even multitouch had been played with before the movie came out—just in labs, and very expensively. Minority Report’s cleverness was not in inventing new technology from whole cloth, but in extrapolating existing tech into practical, consumer-friendly products.

In the run-up to production on the movie, Spielberg invited a panel of tech experts and self-identified futurists to an "idea summit." Their goal was to create a plausible description of the world of 2054 based on what was current, cutting-edge technology, rather than just constructing one from nothing. Self-driving cars, retina-scanning billboards and criminal-identifying spiderbots all made the resulting 80-page "2054 bible," but the most influential invention of the futurists is the gesture-controlled display.

In the movie, when Tom Cruise straps on his infogloves and starts rummaging through the dreams of the psychic precogs, classical music begins to play. He stands in front of a semicircular computer screen, the size of a wall, and uses his hands to fast forward and rewind, to zoom in and out and rotate the screen. Many of them are laughable—he places one hand in front of another to zoom in, like a vertical hand jive. He goes to shake someone’s hand and all his files are thrown down into the corner. It’s, frankly, absurd—especially if you haven’t seen it since 2002. THIS is the thing tech reviewers are always comparing a new interface to? Even so, there are recognizable gestures that anyone with an iPhone has used. The pinch-zoom, the rotation, and the swipe-to-dismiss are all used daily by smartphone users. And while Cruise’s begloved gesticulation is silly on its face, everyone else in the movie has to use a regular old multitouch computer monitor.

If there’s anyone to blame for the gesture-based interface Cruise uses, it’s virtual reality engineer Jaron Lanier, and the most outspoken member of the think tank, entrepreneur and MIT Media Lab alum John Underkoffler. Lanier, who brought working prototypes of glove-tracking hardware to the idea summit, eventually went on to develop the Kinect gesture-recognition system for the Xbox 360. But it was Underkoffler who took Spielberg’s request for an interface that’s like "conducting an orchestra" and turned it into the gestures we see on screen. In a 2012 interview, Underkoffler said he "devised this whole kind of sign language for interacting with this computer, for controlling the flow of all this information."

The problem is, that sign language has gotten stuck in our cultural mind, like a particularly virulent earworm. In 2006, a year before the iPhone’s debut, Jeff Han gave a TED Talk about multitouch gestures, demonstrating the use of them to manipulate photos and globes. Throughout, he described gestures as an "interfaceless" technology, a way to intuitively zoom in and out and rotate around images without a "magnifying glass tool." This is, of course, nonsense. While touching something to get more info may be intuitive, every other gesture demonstrated is noteworthy for how NON-instinctive it is. Does pressing with one hand and dragging with another really intuitively represent rotation? Especially of a 3D object, like a globe?

There are better ways to handle spatial ideas, ways which are more in line with the way our bodies are built. Human hands and fingers are good at feeling texture and detail, and good at gripping things—neither of which touch interfaces take advantage of. The real future of interfaces will take advantage of our natural abilities to tell the difference between textures, to use our hands to do things without looking at them—they’ll involve haptic feedback and interfaces that don’t even exist, so your phone shows you information you might want without you even needing to unlock and interact with it. But these ideas are elegant, understated, and impossible to understand when shown on camera.

The reality is, there’s a huge gap between what looks good on film and what is natural to use. Movie computers are designed to look cinemagenic. Mostly this translates into transparent screens and huge fonts—things nobody would try and put on a phone. But touch-screen interfaces, which look great because of how easy it is to tell what a user is doing on camera, have managed to take over our lives.

This isn't to argue that touchscreens are useless. They’re a great way to cheaply interact with a small electronic device—like, say, a phone. But the problem is the outsized role the touchscreen has taken in our pop cultural understanding of computer interfaces. The "hovering multitouch" interfaces of Iron Man 2, Total Recall, and Tron have become pop culture’s vision of what's state-of-the-art, even outside of Spielberg’s movie. None of these are fundamentally different from Minority Report’s technoscreen—they just have varying distances between fingertip and graphic. But all of them are, essentially, what design critic Bret Victor has called "pictures-under-glass." They are interfaces that look good, rather than interfaces that work well.

Put another way: If Jeff Han had designed a keyless entry system for a car, it would’ve involved dragging a secret gesture on the car’s window instead of the car automatically unlocking when you open the handle if you have the key in your pocket.

And at the end of the day, it’s visual accessibility driving this trend. Hopefully one day we’ll reach the point where filmmakers don’t want computers to look like conducting an orchestra, and we’ll be able to back out of this interface cul-de-sac and find our way forward into a genuinely natural way of using our devices. Like porn, techno interfaces are more focused on what looks good than what feels good. And like porn, it's pretty hard to get people to stop buying. Here, I’ll make a deal: If we’re gonna be focusing all our cultural attention on something so impractically sexy, can it be jetpacks instead?

Previously in series: Why We Need Best Supporting-Supporting Actor & Actress Categories

Christian Brown is an animator in Los Angeles.

26 Comments / Post A Comment

scrooge (#2,697)

Interesting, thanks.

Really we should just do away with external interfaces and wire direct from the brain.

deepomega (#1,720)

@scrooge True. But that would look terrible on film.

scrooge (#2,697)

@deepomega Well, it was pretty cool in The Matrix!

weizhi (#241,997)

@scrooge Something like that (but wireless) is in anime Accel World where in year 2046 kids on birth get personalized device that is put around neck (looks like hairband) and with that they write on virtual keypad, send emails, playing virtual games, with one gesture close all opened applications…

Grue (#5,527)

The interface used in Minority Report always reminded me of non-linear editing software on steroids. I liked it but i reckon my arms would be tired by the end of the day.

deepomega (#1,720)

@Grue Yeah, and the dude who came up with it is now trying to shop around a real-world NLE based on it. It looks really terrible to use.

You had me until "jet packs." Good piece.

stuffisthings (#1,352)

@My Number Is My Address So basically you liked the whole article except for the word "instead"?

kikimonster@twitter (#241,935)

Interesting perspective. Not sure I agree with the author however I openly disagree with one point made.

"If Jeff Han had designed a keyless entry system for a car, it would’ve involved dragging a secret gesture on the car’s window instead of the car automatically unlocking when you open the handle if you have the key in your pocket."

Let's not ignore the fact that the "secret gesture" is far more secure than a simple RFID keyfob which could easily be stolen from a jacket or pocketbook thereby making the entire car susceptible to theft.

deepomega (#1,720)

@kikimonster@twitter I'm not sure I'd argue that code-based and key-based systems are more or less secure than each other. How many people would lock their car with a big zorro Z, for instance?

(Really the most secure would be some kind of, like, BRAIN-detecting unlock device)

169553143@twitter (#248,562)

@kikimonster@twitter Oh, you mean like having car keys? Something you HAVE on your person to access something is far more secure than something someone can watch, imitate and share.

Um i think star trek has had a bigger impact on design than minority report. The LCARS system of the TNG era is essentially a giant touchscreen. The PADDs are tablets.

deepomega (#1,720)

@Brandon Conver@facebook I'd argue that LCARS is basically just a functional single-point touchscreen. They'd already existed before TNG happened, and were already in some (expensive) consumer devices. That Buick Riviera came out the year before TNG.

Minority Report, on the other hand, came out at a critical time before consumer devices were using multitouch, and created a very visually compelling argument for them.

Astigmatism (#1,950)

Related: I blame Good Will Hunting for the terrible habit of some of my classmates in college of doing math problems on windows and mirrors in dry-erase marker. Matt Damon's character didn't have unlimited access to lecture halls and offices filled with whiteboards, guys. You're not badasses just because you do your surface integrals on a borderline-illegible medium.

"Human hands and fingers are good at feeling texture and detail, and good at gripping things—neither of which touch interfaces take advantage of. "

If you say so. Of course, grasping objects and pushing buttons and so on are what got us to the point of RSI and carpel tunnel syndrome. I had lots of RSI problems in the mouse world when you had to try to make fine-grained motions while simultaneously pressing down on buttons and scroll wheels.

Today almost all of my interaction is with a Magic Trackpad or touch on an iPad. Light, fluid gestures and taps.

Barry Grant (#239,287)

Also, moviemakers use impossibly beautiful people as actors.

Really, I feel for the pain this causes in the author's workplace but it's not like movies haven't been giving us impossible expectations since forever.

tuz (#241,951)

Popular computing is regressing.

It moved from the command line to the Graphical User Interface to touch.

The command line is the most efficient method of accessing a computer's potential. Touch is by far the worse.

Touch is the equivalent of a reverse industrial revolution. By taking a machine that is capable of processing a hundreds thousands lines of code or more, instantaneously and turning it into a basket weaver.

Omar (#242,236)

@tuz that's an interesting point. Do you think most common people including kids and elders would've been able to use the command line prompt? People can't even remember their passwords. It may seem touch is backwards from a computing perception, but don't you think more is being produced by people than ever before because of its simplicity. No function without form.

Bobby Parker (#241,985)

It's rather amazing how true this is.

I stand before you, having been accused of creating a "Minority report' interface. By the *fashion industry*.

So I wrote most of the front end of that. And let me tell you: Minority report was NOT what we after.

And to be honest, I've felt frankly rather snubbed that the design/engineering effort that went into that thing is so roundly dismissed as irrelevant in the face of it having been even compared to something that looked so hard to use.

hapax (#6,251)

PREACH IT. People look at me like I'm crazycakes when I express regret at the fact that I'm no longer able to skip a song on an MP3 player while the player's still in my pocket. And don't even get me started about the steady decline in my texting speed over the past ten years; I think even T9 was faster than Swype. Touchscreens are the devil.

Interesting to come across an article like this.

I've been developing an App that defines a unified UI to surf through all the information on the web. Still under heavy development, but if it sounds interesting to you, I could use the feedback =) There is a stable version up at:

Whenever I add iPad or Leap Motion support; there. Minority Report meets reality, with an usable UI =P.

Thanks for the article!

scrooge (#2,697)

@Pablo Neirotti@facebook It's not really developed enough to test yet… or maybe doesn't work with Firefox?

Joe Kvam@twitter (#242,006)

I understand the frustration but removing everything discussed here from the context of the movie as it's own animal is very problematic – especially if you are going to ultimately blame the movie. There is a textual purpose for evoking a conductor in that sequence and it's purposeful that the gestures reinforce that. It's more about the story being told than the world and gadgets used to dress it. If people keep asking for it it's because they intuitively like how it works, not how its depicted specifically in MI.

isaacson@twitter (#242,297)

be curious what you think of this concept video. Less for the messaging of how technology will ruin our lives than for the eye-controlled UI

Mathias Tierens (#257,359)

This is an interesting point of view. I however wouldn’t be so negative about this film. Since the start of science fiction, human computer interaction (HCI) is used in films. In the first SF movies since the computer existed, simple adaptations were made to the current technologies. Movie directors were inspired by technology and tried to embed it in their films. Later on, when technology and the movie business evolved, films also started inspiring researchers for making new technologies. Movie directors made their own vision of the future, without any regard to the possibility of those invented technologies being realistic. This changed with Minority Report, where HCI scientists were involved to make an authentic vision of the future. They looked at the current technologies at hand, thought about what the logical developments would be in this field and made a realistic vision of the future. So even without this movie, the current progress of touch devices and gestural interfaces would still be going on, not only because it looks cool, but also because consumers keep buying it. If touch interfaces wouldn’t feel natural to their users (that’s why they call it NUI’s), I would think that they wouldn’t buy it anymore and lose their interest in this technology. As for gestural interfaces like the big screen Tom Cruise interacts with, it is important to provide a good conceptual model and offer affordances. And like you said, they actually do this, the pinch-zoom, rotation and the swipe-to-dismiss feel natural to us, we intuitively know how to do these actions, as we all use it on our smartphones as well. We aren’t used to these actions for such a big screen yet, but as with all new technologies, it takes some time to experiment with it and get the hang of it. The bigger problem with this interface might be that our arms get tired by using it for too long, but I don’t think this computer is designed for extensive use. Its power lies in visualizing things in a quick and simple way, which it does perfectly. Of course, I don’t discourage the research of new kind of NUI’s like you suggested with more haptic feedback, but I wouldn’t repel the NUI’s we have at our disposal either. They are useful if the design is good and the way of interacting with it is clearly visualized.

lllxlll (#1,982)

So guess we are rehashing this.

Minority Report is just this weird standard and expectation in people's perception of tech and the future, and much smarter people than me have pointed out that Expectations are what drives people's commercial behavior and political beliefs.

I still think the "Computer" in the old Superman movies are going to be possible, organic units that interact with each other or whatever…

I find "talking to an interface" to be horribly useless and unnatural.

Anyways, my problems with Minority Report are this
1) They don't pay their fucking pre-cogs. Pay your fucking pre-cogs
2) Your envisioning a future where we have enough data to circumvent free will, but their choice in the movie was to punish the people that committed crimes, and not find ways to prevent the crime from happening.

It's Xeno's Paradox, data, and the equal enforcement of laws in a society where law is not applied evenly.

Ramble off..

Post a Comment