Augmented reality on mobile devices: a new way of seeing, and being?

Augmented reality is a coalescing of technologies that promises to create a new interactive relationship between mobile users and their surroundings. It's easy to make AR sound like the latest technology in search of an application, but an analogy with jet fighter pilots might help.

Pilots look through the cockpit window, or a helmet faceplate. The inside of these surfaces is treated to display transparent images of cockpit controls and navigation data, a technique called "head-up display." Data, images and text overlay a view of the real world.

Smartphones now can apply that basic concept using very different technologies and take it to another level: adding data about the phone's location and orientation to relate the data overlays to the actual objects you can see onscreen through the phone's camera: buildings, streets and monuments.

You're able to "augment" the reality you see with data drawn from a variety of different online sources, such as Wikipedia, YouTube, Flickr, or commercial content providers. Typically a middleware server application acts as an intermediary, and pulls down the geotagged remote content. The client application runs the calculations to position the content with the "point of interest" on the screen.

The mapping of data to image is done by using the phone's GPS data to pinpoint the location, coupled with its onboard accelerometer and compass to figure out the phone's orientation and the direction in which it's pointing. In the future, the mapping can be made more precise and personal by factoring in the user's identity.

Network World blogger Mark Gibbs recently analyzed the different techniques used to analyze a real world scene and figure out where to place the augmentations. He was impressed by a Web-based AR application from the U.S. Postal Service: the Virtual Box Simulator, which lets you check whether something you want to ship will fit in one of the USPS flat rate shipping boxes.

The surging interest in AR is being fueled by its potential for mobile users. Japan's two biggest cellular carriers, NTT DoCoMo and KDDI demonstrated at a trade show in July an AR system that populated browsers on Android-based smartphones with icons and information about restaurants or other sites near the subscriber's location.

Uncovering, discovering reality

Another example of AR in action is an early 2009 AR application from SPRXmobile, which lets you find ATMs from ING in The Netherlands. You point your Android phone's camera in a given direction, and information about nearby ATMs will pop up. SPRXmobile released the technology last June as Layar, billed as the world's first augmented reality browser.

Yelp, the social ranking site, introduced an AR application for the iPhone 3GS at the end of August. In one YouTube demonstration, the reviewer described it as a "ranked reality," enabling the phone's user to stand in front of a restaurant and see ratings of it created and posted by other Yelp users.

In the last few months an array of AR browsers, applications and SDKs, have been surfacing in the market, offering a growing number of developers the chance to explore what can be done when an object, place or person is augmented with additional information.

Metaio has just opened the API to its junaio AR platform, which includes a 3D engine. It's a consumer-oriented adaptation of its Unifeye business-to-business AR platform, which combines the company's code with the Adobe Shockwave player to let developers create and insert 2D and 3D images and animations directly into live video streams, and integrate images from the user's surroundings. With junaio, users can create 3D scenes, map them to points of interest, share them with friends and make 360-degree searches via Google Local Search for junaio scenes created by others.

Version 3.0 of the Layar Reality Browser, from Layar, was released in early December after its first introduction in mid-year. The name refers to the fact that the browser's image can be "layered" with data from various sources, creating in effect different perspectives on the same object or location. One addition in 3.0 is cookie support to authenticate users. Masternaut 3X used this to create a closed layer, open only to a client's employees, to track automobile or truck fleets: the employee points the phone's camera at a street or parking lot and sees any fleet assets tagged with a symbol, and accompanying details.

Mobilizy this month released a version of its Wikitude World Browser for Symbian-based phones, the first AR browser for that platform.GeoVector in September launched World Surfer, described as a directional search application for iPhones and Android-based smartphones. Users point their phones in any direction, and along that "vector" see information associated with objects of interest.

Those objects are filtered out by user-selected criteria -- distance, direction, category -- and World Surfer "links to publicly available Internet sights for more in-depth information," says Pamela Kerwin, head of strategic business investment for GeoVector. Users then can visit the store's Web site, see a restaurant review, search Google, or see the precise location on a map. The software supports plug-ins from third-party content providers.

Novelty gives way to new vision

These and similar tools are cropping up in a variety of travel/tourist and navigation applications, as well as in games. Many "applications" are nothing more than novelties, though the U.S. Army as far back as 2003 was using AR technology in prototype combat training systems. There's a flurry of marketing applications that border on outright gimmicky: both Popular Mechanics and Esquire created issues that users can hold up in front of a PC-based Webcam and, via a downloaded AR client, see additional images, video, and text, in 2D or 3D, that seem to pop out from the page.

Other examples include interactive AR kiosks set up in retail stores. Customers can hold up a brochure or a product box or package before the kiosk's camera, and see a variety of overlaid and associated information and media.

A range of industrial AR applications, often with special helmets or goggles, are intended to provide specialists, such as mechanics, with context-specific images and data for a task, such as voice, text and animations showing how to install a new part on an engine. BMW has a YouTube "concept" video showing exactly this.

More suggestive is the work by Columbia University AR researchers for the U.S. Marines in creating a prototype AR system -- a headset and Android-based G1 smartphone -- for mechanics doing maintenance work on a light-armored vehicle. The system displayed text instructions, floating labels, 3-D arrows pointing to various components, and animated 3-D models of tools and steps in the repair process. The wrist-mounted G1 phone provided touchscreen controls for queuing up the next sequence of instructions.

The AR mechanics on average were about twice as fast in finding and starting a repair task compared to a control group using a headset that only showed text instructions and another group that used a stationary computer screen, according the study.

There's growing interest in applying AR to at all grade levels of education. An example of the potentiality can be found in the work of Massachusetts Institute of Technology Professor Eric Klopfer, author of "Augmented Learning: Research and Design of Mobile Educational Games," from the MIT Press, and director of the MIT Teacher Education Program, where AR's role in teaching and learning is being explored. One recent mobile AR game is TimeLab, which starts with a video putting the players in a future Cambridge, Mass., (MIT's home) devastated by global warming. They're sent back to the present day to travel around today's MIT campus and collect real and virtual information that could be used to reduce climate change and its impact on the city.

In an interview earlier this year, Klopfer said that augmented reality adds much that purely virtual "reality" can't for these kinds of serious games.

"Many of our AR games are built around socio-scientific problems, that is, issues that require both an understanding of the underlying science as well as an understanding of the social and real world context for the problem," he said in the interview. "When using AR to study problems that are seemingly "entirely scientific," players tend to think more holistically considering many of the subtle real world constraints -- how will this impact me or the people I know? What will the community think? How will this impact what I see around me? It is much harder to generate these kinds of considerations in a purely virtual experience we have found."

The trend in education gives a sense of the potential for mobile AR. "As we move away from the desktop AR toys and start paying attention to where you are and what is around you, things get much more interesting," writes Robert Rice, author, entrepreneur and chairman of the AR Consortium. In this July 2009 blog post, Rice argues that the "mobile device becomes a lens that gives us the sensation of looking through and seeing the world around us layered with information, data and visualizations. As an industry, we are only beginning to explore the possibilities hereIt is one thing to associate or link media to a general location, but it is much better to link to specific objects and things."

Mobile AR challenges

And much more challenging. The key limitations are the accuracy of the consumer GPS fix and of the compass headings, and together, their relatively wide variance can baffle even the most promising AR software, as note earlier this year in a blog post by Jack Benoff, director of marketing strategy for Zugara, an interactive marketing and advertizing agency that's been tracking the potential, and the hype, of AR. The Zugara team experimented with the Layar AR browser on an HTC G1 phone on the T-Mobile network.

"Typically we experienced a GPS accuracy level of somewhere between 100 to 250 feet," Benoff wrote. "Now, remember, that could go 250 feet to the left, right, forward or backwards So really, the device was telling us that the piece of data that was overlaid on the phone's screen was somewhere within a 31,400 square foot (if accuracy was 100 feet) to 196,250 square foot (if accuracy was 250 feet) area. That means you won't be able to swing your phone around an Internet caf and match a tweet with a face. In fact, you can't even safely assume that the person whose tweets you're reading is even in the caf with you"

The GPS signal can also be flakey, as one former MIT student, Karen Schrier, discovered when she tested an AR education game she developed for her masters degree. Using the MIT AR toolkit, she modified one of the MIT games for GPS-enabled Microsoft PocketPC handheld, dubbing it "Reliving the Revolution," a mobile AR game that simulates the Battle of Lexington, which she tested out with some kids on the actual scene of battle in Lexington, Mass. Some GPS radios didn't work, or the signal was intermittent.

The current lack of location precision is one reason that metaio focused its junaio mobile AR platform initially on entertainment uses. The company uses that it calls picture tracking: "It recognizes the 3D image of a building and overlays the correct, relevant information," says junaio's Lisa Murphy.

"In the local search domain, we see AR as more of an embedded enrichment features," says GeoVector's Pamela Kerwin. "Based on the current state of sensors, data, battery life and processing capabilities within mobile devices, we believe augmented reality should be used to enhance the user experience once he has selected a place or object of interest."

Battery life may be a severe limiter, so mobile AR may be elective for some time to come: continuously running GPS, compass, Internet, and 3G or Wi-Fi connectivity can drain power in a very few hours.

Greater precision will let AR software open up new interactions based on data from social networking sites, such as location-based Twitter feeds, and geotagged photos and video clips.

In July, Mobilizy launched a Website called Wikitude.me to let users create and share with other Wikitude World Browser users their own points of interest and location-based hyperlinked media. Users can sign in via their accounts on Twitter, Facebook, Google and Yahoo, to post, update and view information about Wikitude locations marked on Google Maps.

Wikitude.me might be better named Wikitude.us: it opens the possibility of sharing individual knowledge and experience and perspective to create a comingled and communal view of the world through which we are moving. How that will change what we do in that world remains to be seen.

More from IDG:

Original story