Skinned Reality: Creating common language for describing augmented and virtual experiences
Amid an industry that can’t agree on labels or descriptors for the range of augmented and virtual reality experiences possible, we propose a new methodology with standard dimensions and aspects so that we can practically categorize and visualize different experiences. Along with examples, we discuss how MR technologies and experiences are likely to advance in the future.
Mixed Reality, Virtual Reality, Augmented Reality, Augmented Virtuality, Extended Reality - So many terms to describe the range of mostly visual experiences that include some level of augmentation or virtuality. Talk to 10 different industry professionals and you'll get 10 different definitions. Each company and individual has their own take on what the terms mean and where they overlap or diverge. Despite a multitude of technical papers and blogs, we still seem to have no broad industry agreement or standard way to analyze and describe these experiences.
I encounter this confusion often as I talk to colleagues and customers who are trying to make sense of the landscape. (The addition of the term "Metaverse" to this conversation has made things even worse.) As I ponder the types of augmented and virtual experiences I believe will be available to the enterprise and entertainment markets over time and attempt to describe them, I've realized that this lack of accepted terminology is a real hinderance to growth. Product teams, marketers, analysts, and most importantly, customers, all need to have a clear understanding of what is meant when we assign these labels. But that is not enough. We also need to have a common language to describe the attributes that make up specific types of augmented and virtual experiences.
With this in mind, about a year ago I began thinking hard about standard language I could use to describe the types of experiences possible now and those I believe will dominate in the future. In addition to keeping up with industry newsletters, marketing, and pr, I've scoured the Web for research papers and blogs in hope of finding a common understanding of labels and a set of standard descriptive terminology to adopt, but I haven't found it yet. So, I decided to take the task up myself. There's a lot to consider, so I've broken the undertaking into three parts.
In Part 1, I'll propose a new version of the well-known Virtuality Continuum first conceived by Milgram and Kishino in 1994 and updated as the Reality-Virtuality (RV) Continuum by Milgram et al. in 1995. My new version tries to resolve some common confusion about the RV Continuum by reframing it as a range of Experiences rather than Environments. This allows us to include Virtual Reality (VR) in the Continuum for the first time. I call this reframed version the “Virtuality Experience (VE) Continuum” and hope it will provide a clear foundation for assigning basic categories to experiences.
In Part 2, I will describe a new taxonomy for analyzing Mixed Reality (MR) Experiences on a chart I call the "MR Experience Graph." This 3D chart will consist of three primary dimensions - “Mix of Virtuality,” “Level of Interaction,” and “Extent of World Knowledge.“ The third is adopted from Milgard and Kishino's 1994 paper entitled "A Taxonomy of Mixed Reality Visual Displays." To the three primary dimensions I add a fourth aspect I call “Agency.” My goal is to establish some practical, standard qualitative and quantitative measurements that can be used to clearly convey what an experience consists of and to be able to place any experience on the 3D graph. This will provide a convenient way to visually differentiate experiences.
Finally, in Part 3, I will talk about where continued technology advancement in this segment is likely to take us and the types of capabilities and resulting experiences I think will dominate in the near and distant future. This is an exciting and terrifying topic. Eventually, the advanced experiences we will be able to deliver with increased computing power, sensor capability, display quality, battery life, etc. will change our lives dramatically, for better, and likely, for worse.
Through writing this 3-part article I hope to clarify my own thoughts and to provide to developers, marketers, and analysts a straightforward method to describe the various components and capabilities that make up any MR Experience. I hope it will also be of interest to enthusiasts and researchers who want to gain a better understanding of the full range of MR Experiences possible now and in the future.
Most of what I'll present is informed by my 25 year career in the interactive digital media space and my more recent experience creating and using MR applications for entertainment and industrial use cases. I'll also utilize the unique knowledge I've gained through guiding thousands of people of all ages through VR experiences at Portal®, the VR arcade I established in Seattle, WA in 2016. Where I have pulled insights from previous research I will identify the sources, which are easily found online. However, this content, though technical, is not intended to be a formal research paper, so I will not iterate the previous literature comprehensively.