A Healthier AR Future (Part 1: The Home)
A visualization of how Augmented Reality can improve our lives.. and the paths to avoid.
The Opportunities and Dangers Ahead
With Great Power…
Whether you think the wide adoption of augmented reality is something that is about to happen or still many years away, given the pace of technology one thing stands is clear: it is inevitable. As such, it is important to talk about the paths where we can maximize the benefits- and avoid the worst scenarios it can bring us.
Computing has brought a lot of amazing benefits to humanity- and with it, it has also brought issues such as those around privacy, social polarization, and distraction. As we have migrated computing form our desks, to our pockets, to our wrists and ears… we now run the risk of exacerbating these problems through a medium that is capable of being ever-present in our field of view. Look no further for the dark side than Keiichi Matsuda’s disturbing visualization of an AR future:
Why AR then?
The Medium Matters
“If something changes by an order of magnitude along any meaningful dimension, it is no longer the same thing.” — Bill Buxton, HCI pioneer & Microsoft Partner Researcher
The size and spaces that interfaces occupy make a difference. Prior to the 19th century, it was common for students worldwide to learn how to write on individual chalk slates with a teacher having to come around and teach each student individually. It was around this time that a revolutionary concept came by: why not have a gigantic slate at the front of the classroom from which a teacher could teach the entire class. This of course, became the ubiquitous blackboard.
The same applies to digital devices. When the iPad came out, it was commonly derided as “an oversized iPod Touch.” And the critics weren’t wrong- but at the same time- that’s all it ever needed to be. Just as with the blackboard, a larger screen was a change in a meaningful dimension that opened up possibilities that were never possible on a smaller devices.
And so the same applies to AR and VR, where interfaces can exist in a medium that allows them to be as large as they need to be and also access the potential benefits of a 3rd dimension.
Embedded in Our Lives
Unlike VR, AR has the ability to fit seamlessly into our everyday lives as we move around our homes, our workplaces, and all the spaces in-between. As such, there is an opportunity to use intelligent design and principles to provide us with helpful experiences that flow with our day-to-day lives as opposed to obstructing them.
Scoping out the Project
Constraints
Thought it was clear I wanted to visualize as best as I could a handful of examples of how AR could fit realistically in our lives. There were several constraints put in place:
- Realism was key. While I would normally go for lower-fidelity experiments, I wanted to showcase them in as tangible and realistic a manner possible in order to convey the possibilities of AR to an audience which might be less familiar to it. Of course, to take it further into the realm of an actual product, much more testing, iterations, and engineering conversations would be required.
- Factor in Hardware Constraints. I tried to consider what was possible with currently available technologies as well as what will be coming in the next few years. I assumed the use of additive displays on the lens, with positional awareness using on-board Lidar scanning (and potentially UWB iOT devices embedded in the environment), as well as computing off-loaded to a phone. Some liberties were taken such as an expanded field-of-view which will become available in the coming years.
- Controls. This portion I intend to further elaborate on in the future. However a subtle wrist or ring-based means of detecting input (such as those being experimented with by Apple and Facebook) in conjunction with voice commands is what I have pictured as the most promising and primary means of interaction.
- VR Mockups. After several experiments, I decided to visualize the AR experience in VR and Unity (yeah AR/VR-ception for sure) due to the flexibility of the medium for prototyping, and trying out interfaces in different “spaces.”
- Limited to the Home. Some of the most amazing potential for AR exists outside of the home. However, due to time and effort required, I’ve decided to scope this initially to a household environment, with other visualizations of it outside reserved for a future post.
Principles
When thinking of how AR can help us lead better and healthier lives, it is important to have key principles to rely on. These are a few that I wanted to emphasize:
1. Healthy Prompts/Triggers.
Anyone familiar with the Hook Model by Nir Eyal, or the work of BJ Fogg of which much of the former is based off of, understands the importance of a combination of motivation (M), ability (A), and prompts (P) to create habitual behaviors.
The relative ease those three values can result in less desirable addictive behaviors such as endless scrolling of our feeds, or staying up just a bit too late being hooked on our latest mobile game. One of my goals is to use the following equation to create behaviors that are good for us.
2. Intentional Spaces
One of the best pieces of advice I’ve gotten was to keep certain spaces separate for certain activities. This means having separate spaces for work, and simple things like not bringing any of the distracting behaviors we do on our phone into our bedroom, particularly not at night. Check out:
Given the potential disaster of AR behaving in the complete opposite way, I wanted to investigate how this could be avoided. This leads into..
3. Focused Experiences
My favorite device purchases of the last 3 years weren’t the fanciest new phone or computer. (Though those were nice too). They were: a simple desktop visual timer, and my Google Home Mini (now Nest).
What I love about both devices is that while I can get the exact same functionality on my phone, their benefit to me is that they aren’t attached to everything else a phone is.
When I want to get work done, I set my visual timer up in front of me for 50 minute blocks. And that’s all it does.. you can’t play Candy Crush on there. Same with my Google Home Mini. Since I removed my phone from my bedroom, it still allows me to set up reminders or ask about tomorrow’s schedule without getting sucked into an endless hole on the internet. Based of these experiences I wanted to explore software design patterns that would emulate that of single purpose devices in Augmented Reality.
Okay shut up, and let’s get to the fancy visuals.
(Yeah I know- I’m getting impatient too.)
A Spatially-Aware Device
I decided the easiest way to visualize how AR could work in the home was to recreate a journey of a typical day in the household from waking up to arriving back home in the evening.
Smart Room Detection
The OS of such a device would have to intelligently handle the detection of what room you are in, based off of onboard-Lidar in the glasses and potentially other UWB linked devices in the household which are starting to seep into the IoT ecosystem.
Waking Up to a New Reality
I wanted the utility of the glasses to be clear even when they are not on. As such I imagined a gentle wake up to an alarm and softly glowing light resting on a charging pad on your bed stand as your cue to pick them up.
Upon putting on the glasses, you are greeted by only the most pertinent information during wakeup. (No other functionality is allowed in the bedroom, at least not during morning hours.)
There were several considerations when designing for this:
Spatial vs HUD-like Widgets
Option A: HUD-display
Early wireframes consisted of a lot of traditional “HUD” (heads-up-display) - like information pushed to the edges of your field of view that would persist in your field-of-view like you would see in video games. However, these do not bode well for head-mounted displays as these get pushed into your peripheral vision.. an area that is very hard to process information in as your eyes naturally want to dart your central vision (see below) towards those areas.
Option B: Centralized HUD in foveal range
A persistent UI centered right in the foveal area was also considered such as the UI by the Focals by North (now Google), but discarded as I wanted to explore experiences that could span beyond that region of vision.
Option 3: Spatially-Embedded/Diegetic Interface (THE CHOSEN ONE)
With the other two options discounted, allowing for an interface “embedded” in the environment allowed for maximum flexibility and a medium in which a variety of different interface
Readability was key. A subtle grey box is in the background for key information (darker colors are not preferred due to the current glasses display technologies using additive displays.. as pure blacks would render as transparencies.)
Deference was also a key visual principle as early experiments with a ton of background and container elements seemed to get in the way of the sense of the widgets being embedded in the environment.
Throughout the iterations, there we the general movement away from “heavy”-looking backgrounds and a more lightweight feel.
More Time for Sun
AR being a spatial medium allows for different formats of interfaces we’ve grown accustomed to. As someone who is greatly sensitive to the sun and tries to maximize the daylight I get in a day, having a clock like this helps me understand where my sleep and bedtimes are in relation to the sun.
There is a subtle billboarding (ie. it always faces the user) and resizing of the text for sunrise and sunset times, to enhance the readability to the user at all times.
No More Bed-Time Scrolling (Beta)
As stated- one of my key objectives was to demonstrate how we could move away from one of the biggest potential issues with AR.. which is the possibility of being distracted at all times. One means of doing this in the home is leveraging room and time-based awareness in order to limit things such as an app drawer. This is still being re-worked but you can see some visualizations of this in action with a proposed app drawer below:
A New Kind of Workout App
I tried to rely on subtle but noticeable prompts for a morning workout routine. First off, as the user I’ve placed this spatial app right outside my bedroom and cued to a morning routine whenever I wake up.
Other features of this app that prompt me are the glowing circle (taken from video games), the visually large amount of area it takes up in my field of view, as well as the “Step inside to start” call-to-action — which is referenced further down below.
A technique to increase the visibility of the prompt, also seen in the morning widgets is the use of billboarding for certain UI elements, so that it always orients the user as it rotates along the Y-axis.
Stepping inside as the strategy to start an app as opposed to a typical “tappable” CTA is meant to minimize the ability for a user to start up the behavior, and then lends itself to an easier way to establish a habit.
This lends itself to probably my favorite aspect of this app, which is the increased ability to follow along. Not only do you have the presence of a 3d video of your instructor (done via photogrammetry in a studio), you also, have the shadowy figure customized to your body shape/size following along, so you can match it’s movements with yours. An important note: this mockup assumes on-glasses Lidar in order to differentiate your limbs the background and allowing for the glasses to properly occlude the shadowy figure.
Special attention was given to making the shadowy figure that you are trying to follow along with intelligible as traditional transparency shaders can result in confusing overlapping geometries. (See below.) Choosing a shader with a transparent depth pre-pass allows for equal shading of overlapping geometries, making it easier to understand what you are looking at. Also, in this case, making the geometries closest to you (in this case the head) invisible eliminates awkward clipping and helps to help more clearly focus on the limbs.
A lot of consideration was also put into the placement of the UI relative to the figure of the instructor. It was interesting to find a lot of the preferred versions done in 2D with headings pointing down towards the instructor, did not work in 3D due to having to crane the head and also it potentially clipping into the ceiling. Once again as with the Morning UI, a general shift towards more deferential “lightweight” design was a theme.
NOTE ON FIELD-OF-VIEW: The effectiveness of this follow-along style of instruction only works with AR glasses with a very wide field of view. Modern AR headsets usually have a limited field of view which would limit the utility of this feature. Also, clipping distance, or the distance at which an object is close to you starts disappearing would have to be optimized.
Reminders/ Voice Interactions
COME BACK LATER FOR MORE DETAILS
A Little Help With Dinner
The idea here was to have contextually relevant suggestions. In this case, an recipe app/widget that shows up during mealtimes, providing relevant meals depending on the time of day. I did not want to make this app persistent as that would mean, a constant trigger to want to eat or cook.
In my ideal scenario, I imagine food that you buy through your card at the grocery store would be tracked which would also feed into the recipe recommendation algorithm.
Seeing as how entire OS ecosystems such as Apple’s WatchOS and iOS have great physical activity features, I thought it’d be great to see how similar features focused around diet would work. In this particular instance, the user is focused on tracking their macronutrients (carbs, fat, protein) through the day.
This app/widget also heavily makes use of one of the biggest advantages of AR.. the ability to convincingly convey real-world objects.. in this case photogrammetry of real recipes are used to create 3d models of the delicious outcome of your cooking.
Other Considerations
Battery life + Light/Sound Cues Given current battery technologies, these glasses are likely not to be devices that are constantly maxing out their processing potential. As such considerations need to be made about their utility in a passive mode as well as through lights and sounds (as seen in the morning visualization) when not on a user’s head.
Streaming Compute. Likely the most reasonable option for processing will not be on the glasses themselves in the near future but on the phone. Considerations for wireless bandwidth as well as how phone power consumption would have to be factored in.
Interactions with other devices. These glasses would not exist in a vacuum. That’s already a given based off the likeliness of streaming computing above. However, how does this interact with devices such as smartwatches and smart ear buds? How about traditional computers and displays?
Spatial Audio. Audio cues from 3-dimensional space are one of the most powerful and effective ways of providing messaging to a user in AR/VR. These can be somewhat alluded to in examples such as the Exercise widget where the audio of a guide track for their exercise would be coming from the direction of your instructor. However, the full extent of this was not scoped in this project.
Different-Sized Spaces. It was important to consider the fact that these kind of glasses would be used in housing with less room and less space. Some of the early explorations as a result dealt with fitting these UIs into a smaller single bedroom/studio apartment sized area. In these situations, the contextual and time-based appearance of certain widgets becomes extra important.
Reflections and Next Steps
There were many MANY more ideas I wanted to convey, however due to the scoping of the first part of the project, these will have to wait.
Related to that, due to the RFP (request for proposal) nature of this project, the ability to try out more possibilities were hampered by the high fidelity nature of this prototype. More experiments would likely be possible with prototypes done at a lesser graphical intensity. However, as a tool to convey how AR can effect our ordinary lives, the level of fidelity shown on this project was appropriate.
For my next explorations, I want to focus more on means of control (something of which I’ve thought a lot about), interactions with other devices, as well as the possibilities of AR outside the home.
