No, I’m not talking about the US elections, I’m talking about a profound change in technology that lies in our not too distant future. It will impact almost anything in our lives, the same way smartphones do. Just more profoundly.
Since the iPhone was introduced in 2007, things haven’t been the same. Now almost everybody carries one of those devices and they changed everything. Everybody now carries a camera all the time. Communications and social interactions have dramatically changed. Even seemingly disconnected things like film photography see their fair share of influence from smartphones. Physical rolodexes and address books are a thing of the past. I can (and do) produce entire podcast episodes using nothing but the device I carry in my pocket every day. Entire feature-length movies have been made using smart phones only.
What I’m talking about will surpass that. The next big thing is what some call Augmented or Mixed Reality (AR/MR) – every major player is invested in one way or another and there are multiple competing technologies being rolled out as we speak. Some of the early public players are Microsoft with their Hololens product (promotional video from Jan 2015) and Meta (an early promo video from Apr 2013 and a much more recent video from April 2016).
None of those devices are in a mass-marketable production state so far. They are bulky. Dev kits are available and applications are being built by developers as we speak. The devices are big and bulky and they all use a similar technology. A display above the eyes projecting downwards and a semitransparent diagonal mirror in front of the eyes that mixes that display with the real world around you. To allow things like attaching virtual objects to real-world things (think sticking a virtual display on a real wall), they also need 3D tracking of the room around you – and motion sensing. To allow interaction with your hands, they need to precisely track those too.
Kinect was one of the first products that would do some of that. Microsoft Kinect is the 3D room tracking bar for your Xbox. It was introduced in 2009 and here’s a video from 2013 showing how far the technology has come in only four years. And in this interview with the founder of Primesense, the company that made the Kinect, you can hear that by 2013, they had their sensors down to the size of a stick of gum with dramatically better precision. It’s 3 years later as I write this and development and further optimization and miniaturization has certainly continued. Google Tango is another 3D sensing technology that has been included in a first prototype smartphone as well as in a first production phone from Lenovo.
It’s only a small step to imagine those sensors being built into even smaller devices, to allow them to get a three-dimensional sense of the world surrounding them.
This company has been under tight wraps of secrecy for years. Only now are Magic Leap beginning to come out and publicly hint at their tech and their capabilities and I’m very excited about what I see. One of the reasons I’m excited is their state of funding. They don’t have a product in the market, they don’t talk much about their tech and they received venture capital from virtually all the big players – not just from the VC line of business, but also from the entertainment industry. As of early 2016 they were valued at USD 4.5 billion (yes, with a “b”) and the word is that with pretty much all of the funders it went like the following. Magic Leap: “here’s what we can do” – big company x: “No way” – Macig Leap: “here, let us give you a demo” – big company x: “shut up and take our money.” Magic Leap has just revealed that they’ve been working with Lucasfilm for a year to bring us this demo. This was shot through an actual prototype and it boggles my mind. Think C3PO as a personal assistant, interacting with you at life-size, right where you are. And thanks to 3D sensing, he won’t walk through tables, chairs or walls.
Their technology seems different from what Microsoft and Meta do. Remember Lytro? The lightfield camera that would let you re-focus photos after they were taken? Magic Leap speaks about lightfield displays, wave-guides, photonics.. from all I’ve heard and seen and read, they seem to allow much better immersion and a significantly smaller form factor than the competition. They call it “Mixed Reality Lightfield” – and think eye-glasses, not head-mounted display. If you’re into the tech, I recommend this video and its follow-up for an analysis based on what we know.
No matter if Microsoft, Meta, Google, Apple or Magic Leap, the applications of this are virtually endless and it will take some time until we have developed a new visual interaction language to deal with this. But let me take a stab at some of the most obvious changes that this will bring us.
Screens will turn virtual. Instead of going to a specific place to sit in front of a screen, you just conjure one up in mid-air. Or stick it to a wall, any size you like. Or create five screens. Or turn an entire wall into a screen. Here’s your home movie theater. Your window into another world.
Social interaction. You will be able to talk face-to-face to people who are not in the same physical space with you. And not through a screen, but by being in the same virtual 3D space. Have a meeting with your remote business partner while sitting with them at a table.
Learning. Setting up an IKEA shelf will no longer be an exercise to interpret 2-dimensional drawings and make sure the holes are on the right side but more like someone pointing to things with little arrows. Put these screws here, turn this around, attach this part to that part.
Gaming. One enticing example I’ve read about is Pokemon Go but with the Pokemon hiding behind your couch and turning off your lights to hide from you as it runs away. Hello Internet of Things.
I wouldn’t be me if I didn’t try to map this onto the future of photography. Off the top of my mind I can think of a few: Composition overlays over the real world. Framing without a camera. Learning camera operation. Remote photography teaching. Virtual photo classes and tours.
That’s all pretty low-hanging fruit. The reality will probably be a lot weirder and I’d love to hear your thoughts on it.
If I had to call on it (and based on what I’ve heard so far) I’d venture the guess that we will see the first consumer-grade devices hit the market in 2017. And I don’t count devices like Lenovo’s Google Tango phone into that, as they are not immersive but will show you the world through the small window of a smartphone only. I’m talking real immersion.
The iPhone took about 5 years to completely change the phone industry and it still had to grow up during that time. If we assume the same for Augmented/Mixed Reality, the next five years will be very exciting. As these developments tend to become exponentially faster, I’m even inclined to think that the timeframe will be shorter.