Mixed Reality: Students as multimodal creators
When it comes to harnessing the potential of augmented reality (AR), virtual reality (VR), and mixed reality (MR) in education, the focus has typically centered on how students consume educational content. However, our research focused on the significance of empowering students to become creators of narratives, rather than merely receivers (Mills & Brown, 2023). Specifically, we explored the concept of transmedia storytelling, which typically refers to a dynamic approach in which a story is told multiple mediums and technologies. For example, written stories transmediated across to a virtual world though MR technologies.
Photo by Thought Catalogue on Unsplash
During this phase of our study, we observed how students created stories by employing the capabilities of Microsoft Smart Glasses HoloLens 2™. The students’ written stories underwent a transformation, evolving into immersive digital narratives with appealing visual and audio features, including 3D holograms. Kathy Mills and Alinta Brown (2023, p.3) explored two key research questions: “(i) How can smart glasses support students’ multimodal composition in the classroom? (ii) What multimodal resources are available to students wearing smart glasses to compose narratives?”. From this, we observed the following themes.
The real and virtual world in story creation
In the process of crafting their narratives, our students had the opportunity to bridge the gap between the virtual and tangible spaces. What stood out in our study was how the holograms reacted to the students’ hand gestures, eliminating the need for any intermediary tools like game controllers. This direct interaction made the students feel connected to the holographic elements. Within the classroom environment, the virtual elements, displayed as sound effects and both still and animated images, introduced many ways for students to engage with their surroundings.
As seen below, students created unique landscapes, and adapted their storytelling skills to this unique context (see Figure 1). The blending of both the tangible and virtual worlds was a distinctive with this form of textual expression. As a result, the narratives developed a deeper connection with the physical spaces in the real world.
Figure 1. An example of a student created virtual landscape.
Additionally, the virtual elements responded dynamically to the students’ movements in their physical surroundings, all while being observed by their classmates who shared the same space (Mills & Brown, 2023). This interplay between the real and virtual marked a unique feature of our research.
Story composition and in-air haptics
As also noted in our earlier blog post, during the creation of their narratives, the students harnessed the power of haptic gestures to engage with 3D virtual elements. They utilised various gestures such as grasping (which involved actions like pinching, closing their hand, dragging, and then releasing) and indicating (including actions like touching and scrolling).
The HoloLens 2™ device is equipped with a system that recognises and responds to these specific hand movements. This feature enables users to manipulate 3D virtual models effortlessly through their hand gestures. As a result, the research team observed that the students found these haptic gestures both instinctive and natural (Mills & Brown, 2023). Figure 2 shows an example of a student using haptic gestures to select from the Hololens 2™ menu.
Figure 2. An example of a student using haptic gestures to select from the Hololens 2™ menu.
Conveying meaning in 3D visual mode
In the process of narrating holographic stories through smart glasses, the visual dimension took a fundamental role in conveying meaning. This involved superimposing a diverse array of still and animated 3D models to establish the story’s backdrop, bring the characters to life, and highlight key plot developments through various objects, all accompanied by audio narration.
As a result, our student participants harnessed 3D holographic models to guide the audience’s view towards the narrative’s setting, key characters, and crucial story points. The integration of animated holograms enhanced the storytelling experience (Mills & Brown, 2023).
MR creating new ways of writing and telling stories
As emphasised by Mills and Brown (2023) in this component of this study, one of the main objectives in exploring the potential of smart glasses for multimodal composition was to investigate how students’ prior experiences in school-related narrative creation could be compared. In addition to studying the transmedia stories students crafted using drawing, writing, and storytelling with smart glasses, researchers also prompted students to observe parallels with other narrative activities they had undertaken as part of their learning.
Students observed that, regardless of the medium involved, the essence of the activity remained grounded in the art of storytelling. When questioned about the differences in crafting narratives across diverse media, students highlighted the heightened level of interactivity facilitated by the blending of technology (Mills & Brown, 2023).
References
Mills, K. A., & Brown, A. (2023). Smart glasses for 3D multimodal composition. Learning Media & Technology, 1-22. [Published first online] http://dx.doi.org/10.1080/17439884.2023.2207142
Mills, K., Heck, E., Brown, A., & Funnell, P. (forthcoming). Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report. ARC Future Fellowship. Institute for Learning Sciences and Teacher Education Australian Catholic University.