10 May 2018
VR and AR are on the wrong track. Says Tilman Aretz, editor in chief of n-tv, a German television news channel. Newsrooms do not need to bother, Arentz says. He will present his ideas at this years European Newspaper Congress. I am always in for provoking thoughts that challenge your own assumptions. But while n-tv might be the most frequently used news app in Germany these days, the future will certainly keep ready way more than apps to be opened on smartphones. I will not dig deeper into the likely end of smartphones here because I have done so before. I will rather provide an overview of actual examples in the field of #ARjournalism and add contextual infos and thoughts.
Let’s examine current events with a mindset rather coming from the future than from the past. Because the future is a better key to the present than the past.
When the future is finally there, you do not recognize it as such because it blends in so perfectly and feels all natural. And you might wonder how it could ever be any different. Like living without a smartphone, the internet, cars, penicillin or a letter press. It is not unlikely that future generations will solely rely on the so called AR cloud for nearly everything they are doing in their everyday life. I have written about the AR cloud but to sum it up: The AR Cloud is a persistent 3D digital copy of the real world that organizes the world’s information visually, and “in situ,” meaning where items actually are. Knowledge organized at its origin while today, most of the world’s information is organized in digital documents, videos, and information snippets stored on a server, and ubiquitously accessible on the net requiring some form of search or discovery. Amongst others Matt Miesnieks is working on the AR cloud with 6D.AI which aims to “synchronize 3D computer vision data across devices, time and space” in order to enable “persistent content, occlusion, and real-time shared experiences.” I recommend listening to Matt talk to Kent in the Voices of VR podcast.We are still lightyears away from that. But you can see babysteps being taken. And while a lot of valleys of delusions are waiting one thing is almost certainly true: The disruptors will be disrupted and technological development will continue to evolve in circles. Just think how you stored, consumed and owned audio in the last 10,20 or 30 years. So it is time to embrace our cyborg selves or at least to check out what the MIT Media Lab’s Fluid Interfaces research group has been doing throughout the last couple of years.
Right now we have a disconnect between the physical and the digital. Think about bridging both seamlessly together MIT Media Lab The internet evolves from 2D computer screens into a 3D internet that overlays the real world, an entirely new paradigm for navigating, identifying, and organizing data will be requiredMichael Park
Google presented a bunch of interesting new products and technical solutions at this years I/O developer conference. And while Google's robot assistant making an eerily lifelike phone call provided a proper shit look at that effect, I meanwhile concentrated on a virtual fox telling me where to go. Google showcased a new augmented reality feature that could soon put an end to the stress associated with finding a new location. Users will open their camera and will get a real world view of their surroundings, with Google’s directions overlayed in real-time, along with other points of interest. The feature combines the users’ camera view with Google’s extensive Street View imagery of the world. Sounds useful and quite natural.
Another headline that caught my attention a couple of days ago: Mercedes Expected to Ditch Owners' Manuals For Augmented Reality App. Sounds useful and quite natural as well. Why should you have a printed version of a car's owner's manual? It comes in way more handy to give detailed info while scanning the surrounding. The other day I saw a stack of telephone books in a hallway. It seems as if besides Google and Mercedes a lot of companies are (yet again) playing around and experimenting with phone based AR. I started to gather some use cases but eventually had to stop because things were getting out of hand. Amongst others Gucci, Eminem, U2, American Express, Zara, Saatchi, Shazam, Coca-Cola and Facebook implemented various experiences. But what about journalism? If all sectors of society start catching up where does that leave journalism?
But you might one soon day ask yourself how you managed to live without that feature.
Augmented Reality and journalism are made for each other. Journalism gathers information about the world around us. Thanks to augmented reality, this information can be displayed where it got picked up – which is especially interesting for event reporting.
This is how an article in the Guardian starts. It is from 2010. Helen Papagiannis who recently published the book Augmented Human sums up most AR experiences from this period or earlier as first wave of AR. The current wave, she says, moves beyond printed targets toward a new spatial understanding and deeper intelligence of your environment. More data, more convenience. Because technology starts understanding us. Papagiannis is not using the expression AR cloud but she is certainly arguing in the exact same direction.
Augmented Reality is a new approach to digital storytelling that we are exploring. It is a new pathway that can lead away from the abstract depiction of objects and toward a more visceral sense of real-life scale and physicality.
This is how the New York Times is embracing AR in 2018. They started with a retro vending machine that you could place in your living room or somewhere else – augmented on your smartphone screen. Followed by Four of the Best Olympians, as You’ve Never Seen Them, David Bowie in Three Dimensions and they let you Explore InSight, NASA’s Latest Mission to Mars. Here is a list with a bunch of other recent #ARjournalism examples.
You can do stuff like that. But you can certainly do more...
Are these recent examples any good? Do they justify costs and time needed to develop them? Time that could better be spent doing „proper journalism“? Besides all infatuation with technology these are crucial questions. And it definitely depends what kind of lenses (as in perspective) you wear when evaluating these experiments. We can – first if all – assess that smartphone screens are not perfectly suited for proper AR experiences most of the time. They are ubiquitous but they are too small, you have to hold them, they feel like a hurdle unless you re-design the user experience. Speaking in metaphors: For a window to the world the smartphone is very small, it might better be suited as a magnifying glass in the hand of an investigator. Maybe and only maybe – because i do not know if this observation stands the test of time – Ian Bogost is right with his article in the Atlantic: Why 'Stories' Took Over Your Smartphone. The format, made popular by Snapchat and Instagram, is the native genre of glass rectangles. Where does that leave AR? Until we have proper smartglasses we need to cope with smartphone AR. Here we can try out, fail and learn crucial design skills needed in the years to come. It was something totally different to design a responsive website, an app or a conversational user interface instead of a simple html-based website for a desktop pc 2002. And things will get way more complicated when it comes to spatial design. You can include multi sensory here if you want to multiply problems.
From an innovators perspective it is crucial to tap into the cold water. You only learn when you do stuff and mostly when you do mistakes. So, well done NYT. From a user perspective things are a bit more complicated. Because users only ask: Why should I care? The novelty factor quickly vanishes.
There is a lot AR can offer. Says Hisham Bedri, MIT Research Engineer. He held a short talk at this years AR in Action called Spatial news and AR journalism production where he listed spatial, perspective, interactive and social as fields of possibilities. While Erica Anderson from the Google News Lab named engagement, context and location as cornerstones of good AR in a SXSW2018 session on AR for newsrooms. A lot of well known buzzwords. But what do they really mean? And do they help you with developing a worthwhile experience? Going back to the one and only question asked by your audience: Why the heck should I care?
If you care about the users needs you will design better stuff, including AR experiences in the context of journalism. Engagement and social sound great but need to be fueled by actual people engaging and being social. And you can not force users to do that.
As long as our most fundamental needs are somewhat satisfied - that means we have enough air, food, water, sex and sleep - and we are save and have a somewhat social life then we can care about our social status and prestige. This is a strong driver for motivation. And might be a strong reason why to open up an AR experience. To help you get better and increase status. Especially in comparison with fellow human beings. Sounds corny? Well, just having fun together is a strong driver as well. So let us check a couple of possible or already accomplished projects in order to ask if they enhance and/or enrich or what it would take to do so. I will break it down in a bunch of typical news sections.
If you have watched sports on TV during the last couple of years you are pretty much aware of augmentation of information. Whether it is soccer, baseball, rowing, skiing or whatever - you get context sensitive info-overlays these days. That is because sports is all about numbers, distances, speed, angles, lines, trajectories. It is only fun to nerd around if you get into the details. This is why I totally agree that Augmented Reality Is The Next Big Move For The Sports Industry and hell yeah sports journalism is a part of it. You can learn how to perfectly play golf using a mixed reality headset or if you prefer soccer, here is your AR soccer trainer (Learn to Play Like The Pros with This AR Soccer Trainer) – that might give you an idea about the potential. The current #1app for live baseball in the app store – that by offers premium features for a yearly fee of $19.99 – includes „Live pitch tracking with hi-res ballpark renderings". I am pretty convinced that the combination of sports, data and AR are huge.
In the next phase of technology’s evolution, AI will increasingly be “the brain” with which we process the world around us, while AR will be “the eyes” through which we visually and physically engage the augmented reality of our urban environment Michael Park
This is why I am a bit disappointed with the AR sports projects by Washington Post and the New York Times. „See the athletes race across your table or floor in 3D augmented reality“ that is what the WP promises. And I can not help to ask myself: Why? Just because I can? The experience is nice but not much more.
It is a typical case for ASTTW (Add something to the world). One of two cases that Benedict Evans distinguishes when it comes to Mixed Reality applications. Check the talk here! ASTTW applications are fairly easy to make. The Quartz app for example started with a bunch of stories that let you place an object in AR: AR in the Quartz app brings stories to life right in front of you, whether it’s an object from outer space, an artifact from history, or the landscape of an important part of the world. We’ll add 3D objects to our news feed when it helps tell the story, and sometimes just for the fun of it. Yeah why not. Nevertheless augmenting the perfect line in the field of view while playing golf sounds way more convincing to me. Because the added value is way higher. Do not get me wrong. I embrace every attempt to make use of AR in the context of journalism. But future experiences should aim for more than just some digital 3d objects standing around somewhere.
Besides ASTTW (Add something to the world) Evans proposes LATWATMAI (Look at the world and tell me about it) a fascinating concept that needs way more machine learning and context but could one day be a major force behind most journalistic services: Who is this person? What kind of building is that? What do I need to know now? Depending on your situation you do not need to open an app but you get customized contextual information that you will pretty sure get used to. And that can of course be highly personal and relevant only to you.
Of all places it was Hamburgs Elbphilharmonie that was used for an interesting AR use case by the Washington Post last may: Using your phone’s camera, we’ll project the Elbphilharmonie’s unique acoustic panels on your own ceiling, explaining how the acoustic panels create a sound so clear and precise.How great this must feel on future smart glasses with a crystal clear surround sound. Just start to imagine what is possible if we integrate 6 degrees of freedom and you could wander around in your living room transforming to a dreamlike landscape that changes according to the music being played. Yeah sure, sounds a bit far off. But I am in it for the long run. I personally liked augmenting my kitchen with a David Bowie costume and I am pretty sure that we get used to augmenting digitized 3D models of everything. There are already 1,5 Million users of Sketchfab "the world's largest platform to publish, share, and discover 3D content on web, mobile, AR, and VR.“ Strong partners like Facebook (Facebook and Sketchfab partner to bring free 3D models to AR Studio) will help spread the word. In January 2017 over 500 cultural institutions had joined Sketchfab and you should check out the cultural heritage section. But only models do not make a promising AR app. We need story, context and interaction to provide a sustained experience. The whole topic of time travel is very much suited to accomplish that. In 2014 the Guardian augmented historical photos from the second world war on Google Street View imagery. That is a promising way that could and should be re-created with 3d animations or life-like 360 video footage. Our grandchildren will hopefully have access to good enough material from early 21st century. And I am pretty sure that you will have way more and way more immersive experiences about the Berlin Wall than today. But wait, there actually is one called MauAR.
That is a very nice idea. In theory. In practice the app again and again and again told me that „We´re having trouble recognizing your ceiling“. My fault. But I am just an ordinary DAU and a smartphone might be lightweight but if you keep holding it to the ceiling for a couple of minutes it feels like a brick.
In order to provide a proper journalistic AR experience it seems legit to get inspired here and there. Just two quick ideas for the culture section. Saatchi Art Adds the Power of AR to Its Mobile App, Letting You See High-Priced Art in Your Home Before You Buy It. It would be easy to copy and enhance this concept for people learning more about art in their living room. Could be a nice social experience as well. Come on over we have Karl Marx week in our living room. And Apparatus Studio AR just refurbished our bedroom. Not yet sure how virtual vandalism evolves. Similar creative ideas could be a thing and they could be reinterpreted as positive joint social culture events moderated by journalists.
The New York Times Mars experience is nice. The red planet just appeared in front of me. And while I started moving around the smartphone provided some neat tactical feedback to show me different infos. But: Where is the story? Where is the challenge? Where is the fun? I think the combination of science + journalism + interactive / game elements + AR is promising instead. I can not dive deep into details here. But I am happy to discuss game elements in the context of journalism. For a start I recommend to play a science game for instance Rugged Rover by Preloaded for the British Science Museum in London. Another promising area of science journalism in the context of AR is urban city planning. Just imagine to combine AR Minecraft and Education Minecraft. You could let users rebuilt certain parts of their city using Minecraft while storing their models in situ for others to see and discuss. And just start thinking about food. There is a lot happening in that area from coke cans telling you about - well - coke to augmented burger menus. Where does (service) journalism come in here? To wrap things up just think about sensors and data for a second. Your smartphone can already measure noise levels with a decibel meter app. This data could be gathered, stored and presented as an interactive chart comparing aircraft noise in the city and - of course - seamlessly integrated in an unobtrusive AR interface. And this is just one example. After having seen this piece of hardware i am pretty convinced that the add-on hardware market will totally go nuts.
It reminded me of an early iPad app I once tried. Nice but lacking the necessary elements to be engaging, fun or eye-opening for longer than a couple of seconds.
One of the most promising data visualizations in 3D when it comes to the stock market is from the Wall Street Journal: Viewers can project a holographic 3D view of the live U.S. stock market onto the real world, offering a groundbreaking way to see how the daily drama of the U.S. stock market is unfolding. Users can place the hologram on a table or floor, and explore the rising and falling sectors of the U.S. stock market with their camera.
Experience augmented reality in the WSJ app for iOS
An open source camera that employs object detection for data gathering
Here is another idea for the long run. Why not have your AR glasses integrated camera see and learn while you stroll the streets of Berlin Kreuzberg. After a couple of hours you could get a machine learning fueled analysis of let´s say sneakers being worn in this area. Maybe this knowledge gap could help you buy promising stocks and bonds. Yet again. This might be far off for now. But a future LATWATMAI use case. And way more interesting to me than this Reportages en réalité augmentée.
In the early 2000s I saw a man with a portable dvd-player sitting in the tube watching a film. That was extraordinary. In 2013 Googles Sergey Brin was riding the subway in google glasses. That was spooky. Whatever happens in 2026. It is still a very long way paved with a gazillion problems. To get an idea about some actual problems on the way to the super smart glasses I recommend listening to Karl Guttag on the AR Show Podcast.
It is hard to keep track with developments in the software sector when it comes to AR. There are a lot. And things are happening fast. For a start why don’t you check the AR Round-up: April 2018 by Tom Emrich.
You can not learn how to design emerging media and new formats if you do not try them out yourself, evaluate and learn. I collect some general use cases here for inspiration. Our studio Vragments has vast VR experience, prototyped an AR app called Explainer and we frequently run hackathons, popups and workshops to dig deeper. We are happy to talk, share knowledge or accompany your development process. And we certainly get you to suspend the disbelief about change. I will discuss VR / AR journalism at this years Mojo Fest 2018 in Ireland, end of may and introduce my concept of XRjournalism.
As long as there's sunAs long as there's sunAs long as there's sunAs long as there's sun