There has long been a line drawn between STEM and the arts. Between left-brain and right-brain, logic and creativity. The engineer thinks in numbers and mathematical relationships while the artist thinks in colors, shapes, and feelings. I’ve always been taught that when it came to my career, I’d have to choose between the two. However, I find myself crossing this line on a daily basis. The official name of my major is “Integrated Art, Technology, and Business” at the Iovine and Young Academy, and is exactly that. I start off my day learning about design history and principles, code in p5.js and SQL after lunch, and round off the day reading about Amazon’s revenue streams. My discipline is in creativity, to put it simply. I show up every day and ask myself, what will I create today?
The Academy focuses on teaching emerging technologies, which is where it slightly differs from your standard computer science or engineering college. My specific concentration lies in Extended Reality. XR is the overarching term encompassing several different tools we can use to digitally maneuver what we experience- in effect, we are using technology to extend our own perception of reality. There are different modes of doing so, including AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality). With AR, we are augmenting our existing reality by adding digital elements to it. A great example of this is the Pokemon Go app, which revolutionized the public’s understanding of AR technology. This game worked by placing 3D characters into the real world in users’ screen space. To anyone playing the game, it would look as though Pikachu was sitting right in front of them through the lens of their phone screen. Mixed Reality is similar, but has an additional interactive element to it. MR devices like Microsoft’s Hololens allow users to “touch” augmented objects. The headset tracks hand and finger movements to calculate how virtual materials might react. For example, when wearing an MR headset, I could shoot lightning out of my hands, or mold a virtual sculpture placed on my desk. In actuality, this sculpture does not exist. It only exists in the world of my Hololens. When using Virtual Reality, we become immersed in a new, entirely virtual space. Many video games and 360 movies take place in VR, which are experienced through a headset like an Oculus Quest or its Playstation counterpart. Inside each headset is a little computer and screen that displays a digital reality we can only interact with through game controllers. The reality we know completely fades away, and is no longer just being augmented by a digital component. This form of XR yields the highest level of sensory immersion, as there is no longer a layer between the user and the virtual world.
The field of Extended Reality truly encompasses the intersection of arts, technology, and business to a tee. Building virtual experiences requires both artistic and technological facets. There is the creative element of controlling color, lighting, and space to curate a feeling of embodiment. Even more important is the implementation of storytelling, regardless of how short or long the XR experience may be. All of these artistic pieces come together through code and 3D design, which requires an understanding of logic as well as very basic physics. The most underrated part of creating an XR app or experience is where the business angle comes in. Because AR, VR, and MR are emerging technologies that are just now starting to become known to the public majority, it is essential to consider how we market it in a way that people understand its value.
This is where my focus lies. I am a storyteller through art and code, but what most fascinates me about XR is the research there is still yet to do, and finding a way to tell this story in a way that catches people’s attention. There is a philosophical element to what we are doing. There’s an ethical element. These are all things that impact the way our society moves forward, especially with concepts like the Metaverse and biometric monitoring on the rise. These things tend to get brushed over in the heat of advancing tech to reach monetary goals. Moreover, many people don’t really understand what Extended Reality is, and therefore have a hard time grasping why it is such an up and coming field. I am interested in changing this, and have sought to do so inside and outside of the classroom. In class, I work to create accessible experiences that are simple enough for a user with any expertise to enjoy and understand. I spend a good amount of class time ideating and pitching XR experiences, a process which starts with identifying a pain point and deciding how extended reality might be an ideal solution. Once I have a concept, I have to pitch it to my classmates and professor for critique. This is where communication is really important: I have to sell my concept to the audience without having a finished product. Once I’ve received feedback and iterated on my initial idea, it’s time to make the XR experience itself. In the case of VR, it is most common to use Unity for creating and programming the experience. In Unity, you are essentially creating a miniature version of the VR world you want to place your viewer in. It’s as if your viewer is a doll, and you are building a to-scale dollhouse around them. Unity requires the use of C# to code in animations and scene transitions. In the case of AR, there are multiple solutions. The easiest to use is Reality Composer, an Apple product, which allows you to create a 3D scene and then impose it onto a real background through your camera lens. Outside of the classroom, I work towards building a broader public understanding of XR. I head an academic journal at USC called SCribe, and started an XR blog section to spread knowledge on applications and implications of the medium. I’ve covered everything from the history of immersive experiences to the use of XR in theme parks. Once readers understand just how many applications this technology has, including in parts of their own life, they find it less alienating.
It is apparent through observing the evolution of social media and the internet as a whole that our presences are becoming increasingly digital. Our way of interacting with information and communicating with one another is fundamentally changing, especially in a post-pandemic time. As my Immersive Media professor puts it, entering a virtual space is becoming synonymous with entering a physical space. Our digital selves and the virtual places we visit become a part of us. Suppose I play games online with my friends under a specific avatar- this avatar becomes a part of my identity, of who I am. It is an extension of myself that I put into the digital world. This is why Extended Reality is on the rise, in parallel with the Metaverse. We want to simulate our physical realities as much as possible in our digital realities- we want to be able to feel the digital grass brush against our legs as we walk through the fields of a VR video game, or be able to walk up to another player and see the details of their avatar up close, as we might a real human. It’s human nature to want more and more control over what we perceive. XR is able to make this a reality. It’s still an emerging field, but all trends in tech point in one direction: towards a world where our digital selves are just as, if not more important, than our physical selves. The ability to change reality is right at our fingertips now, and can allow us to do everything from elevating entertainment to curing mental ailments. It’s up to us to decide what realities we design from here.
Dear Shanna,
Appreciate your posts about XR!
I would love to contact you as I represent a startup building an AR solution to change the eCommerce experience for Home Decore items.