Microsoft's Mixed Reality Plans Go Far Beyond HoloLens
When Microsoft made the surprise announcement of a new augmented reality (AR) product at a Windows 10 event in early 2015, the Microsoft HoloLens headset itself wasn't the first thing the audience saw. When unveiling the new technology to the world, Microsoft's Alex Kipman—creator of both the Microsoft Kinect motion controller as well as the HoloLens—instead showed a logo for Windows Holographic, the Windows 10-aligned software platform underpinning Microsoft's AR ambitions.
More than two years later, the software powering Microsoft's growing AR device ecosystem is still the best indicator of where the company plans to take its "mixed reality" (MR) future. It's got a different name now; Microsoft re-branded Windows Holographic to Windows Mixed Reality, but the goal hasn't changed. Microsoft wants to build an interoperable network of AR, VR, and mixed reality headsets from different manufacturers, all talking to one another and running Windows 10 Universal Apps.
There are three prongs to this approach: Microsoft's first-party hardware (HoloLens), the evolving software stack housed within the Windows Mixed Reality platform, and Microsoft's third-party hardware partnerships with Acer, Asus, Dell, HP, and Lenovo to build $299 mixed reality headsets running Windows 10. The first of these, the Acer Windows Mixed Reality Development Edition headset, is already shipping to developers.
What Is Mixed Reality?
Microsoft isn't the only AR company that has started using the term mixed reality to differentiate between different points on the AR/VR spectrum. On one side, you have the monocular glasses you see from companies such as Vuzix, and smartphone-based AR popularized by apps such as Pokemon Go.
On the other, there are the more immersive binocular head-mounted experiences (displays in both eyes with 360-degree field of vision) of devices such as HoloLens, the ODG R-9 glasses, or the still-unseen Magic Leap. In over-simplified terms, you can think of AR as adding overlaid information or virtual objects in more of a two-dimensional way on top of what's in front of you.
But the continuum in which these kinds of technologies exist goes deeper than that. If you imagine a spectrum with the physical world on one side and true virtual reality on the other, it would look like this:
Physical World—>Augmented Reality—>Mixed Reality<—Augmented Virtuality<—Virtual Reality
(Don't worry too much about augmented virtuality right now. We're not quite there yet).
The industry is mired in confusing terminology at the moment, but separating out all the jargon is important not only in understanding the ecosystem, but identifying all the points at which Microsoft plans to plug into it. If the tech giant's vision comes to pass, Microsoft's mixed reality software may be running in headsets ranging from $299 to $3,000, embedded in consumer and enterprise experiences across gaming, e-commerce, industrial design and manufacturing, interactive enterprise data visualizations, medicine, and beyond.
How You Build a Mixed Reality OS
During a recent briefing with Microsoft, PCMag got a chance to briefly check out the new Acer partner headset. We also got a breakdown of the Windows Mixed Reality software stack, including some insight into how Microsoft is working with OEMs to build the $299 headsets, and what the recent Windows 10 Creators Update for mixed reality Windows apps on a range of different computing devices.
The Windows Mixed Reality platform breaks down into a number of different layers. At the top of the stack, you have the head-mounted display (HMD) with six Degrees of Freedom (6DoF), representing the level of three-dimensional motion and position tracking supported. The HoloLens uses what Microsoft calls inside-out 6DoF, meaning position tracking that combines data from multiple input sensors and cameras with algorithms to keep the experience smooth.
For context, untethered VR headsets that use a smartphone display including Samsung Gear VR and Google Daydream View currently support 3DoF, sometimes known as pitch-yaw-roll. Tethered VR headsets such as Oculus Rift and HTC Vive have 6DoF position tracking.
The Windows 10 Creators Update includes support for 3DoF motion control in the Acer and other upcoming OEM headsets. This makes sense from a hardware perspective, since the Acer Windows Mixed Reality Development Edition includes two front-facing position tracking cameras and dual displays, compared with the four cameras you'll find on the more expensive HoloLens. An update coming later this year will add support for integrated graphics chips in mixed reality headsets, though the hardware will need a relatively recent Intel 7th generation chip to get the update.
The next layers in the Windows Mixed Reality stack handle input, middleware, and the application programming interfaces (APIs) for mixed reality and spatial perception. The input layer processes all the visual, gyroscopic, voice, and spatial input data that an AR/VR device collects at any given time. This might encompass data from sources including coordination systems, gaze and directionality, gestures, voice input, spatial sound, and spatial mapping.
The middleware and API layers are key to understanding how the platform is beginning to connect to the rest of the AR/VR software ecosystem. The Windows Mixed Reality developer documentation already includes instructions for building holographic apps using popular platforms including Unity (cross-platform development engine), Vuforia (open-source AR development platform), and a middleware protocol for building MR apps using the Microsoft DirectX runtime.
The platform isn't open, per se—you've got to play by Microsoft's Universal Windows App rules—but Microsoft is providing a number of options for plugging into its MR software stack. One example of this is Intel's Project Alloy VR headsets, expected later this year, which run a version of Windows Mixed Reality software.
The final layers of the stack add a custom user interface (UI) Microsoft calls a "shell," along with cloud services such as Xbox Live authentication and syncing, and collaborative messaging through services like Skype. In the Acer Windows Mixed Reality Development Edition we got a brief look at, the "shell" was essentially an operating system laid out like rooms in a house.
The user can then teleport into different rooms and launch various apps displayed as floating icons or occupying a tile on a wall. These can be simple apps such as Photos, Movies and TV, a Microsoft Edge, or even a full-fledged Windows 10 Start menu you could navigate within the experience, which worked with an Xbox controller. Of course there are also some fun 3D interactive modeling capabilities as with HoloLens. Demo examples ranged from elephants and tigers roaming around to a waving astronaut you can place in the room, and an app that launches you into a 3D solar system. These are just a couple examples, considering the OEM headsets can run any Universal Windows App. From a software perspective, Microsoft sees all these devices as Windows 10 computers for your head.
The company may release a standard mixed reality hardware specification at some point down the road, but at the moment Microsoft is focusing on the software stack and co-engineering the coming wave of partner headsets. If you take a closer look at the Windows 10 Anniversary Release (and the HoloLens Commercial Suite within it), you can also see the software features Microsoft has been adding under the hood to make it's enterprise-facing HoloLens more attractive for mainstream business deployment.
The HoloLens Commercial Suite bolsters the base feature set of the HoloLens Development Edition, allowing enterprises to treat the mixed reality headset as just another device in their corporate IT network. Microsoft released the HoloLens Commercial Suite last August for a price tag of $5,000, and for that extra $2K you get features including deeper mobile device management (MDM) with Microsoft Intune along with Kiosk Mode, which will launch the HoloLens directly into a specific app experience for the purposes of an in-store demo in a retail environment.
The Commercial Suite also comes with Microsoft BitLocker Drive Encryption and access to virtual private network (VPN) protection, which is now more important than ever. The shiny hardware will always get more attention, particularly as Asus, Dell, HP, and Lenovo roll out their own partner headsets in the coming months. But for the clearest picture of where Microsoft's mixed reality strategy is headed, keep a close eye on the software.
This article originally appeared on PCMag.com.