Augmented reality: The tangibility of touch

In our previous blog post, we shared the experiences of some of our older students, specifically those in grades 9 and 10 (aged between 14 and 16 years), and their use of ARMaker as one of the two apps we explored in our recent Sensory Orchestration field work. The second of the two apps our students explored when using augmented reality is the Merge Cube™, a physical foam rubber cube approximately the size of a tennis ball. This unique cube serves as a solid foundation for the 3D visuals offered by the accompanying Merge Explorer™ app, granting students the power to ‘hold’ and manipulate digital 3D objects.

Figure 1. The physical Merge Cube™ and AR overlay, viewed via a smartphone or tablet device.

The Merge Cube™ introduces an innovative approach to digital interaction and learning, enabling students to delve into and uncover the multimodal possibilities of AR. To get a visual idea, Figure 1, displays a Merge Cube™, as well as some of the 3D images from one of the many educational activities found on the Explorer™ app. Within this dedicated app, there is a treasure trove of education-focused activities, covering a wide range of resources from studying the cosmos to biology and dissection. Each activity commences with an informative overview and subsequently prompts students to tap for related content that is projected onto the cube via an iPad/tablet or smart device. Following their exploration, a quiz is available to evaluate their comprehension of the AR module’s material, including factsheets and annotations (see Figure 2).

Figure 2. An example of some of the app content including factsheets and annotations.

What adds to the appeal of this experience is the students’ capacity to engage physically by touching, manipulating, and rotating the cube, while also having the option to control the size of the augmented reality projections using the iPad app. The cube’s tangible quality allows students to physically ‘zoom in’ or ‘zoom out’, which aligns with our previous observations in makerspaces and the integration of haptic feedback and other sensory interactions in virtual reality and mixed reality environments when working with digital and multimodal texts (Friend & Mills, 2021; Mills & Brown, 2022, 2023; Mills, Scholes & Brown, 2022; Mills, Heck, Brown & Funnell, forthcoming). Furthermore, this aligns with our earlier findings and the creation of a Touch Typology (Friend & Mills, 2021), as the Merge Cube™ allows students to physically handle, turn, and manipulate overlayed 3D models, engaging touch and texture, and explore the image’s annotations (see Figure 3).

Figure 3. Students could physically handle, turn, and manipulate overlayed 3D models and explore features and related annotations.

Merge EDU™ distinguishes itself by offering organised lesson plans and activities for teachers. Furthermore, a variety of other apps compatible with the Merge Cube™ are accessible, enabling teachers to devise their own lesson plans that cater to the unique needs of their classrooms. The Merge Cube™ website offers helpful resources, in addition to a Facebook community of practice (Wenger, 1998) where teachers can share advice and ideas.

This phase of our research explores how students can enhance their reading and comprehension skills through multimodal, sensory and augmented experiences, offering unexpected and engaging learning opportunities. Our upcoming report Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report (Mills, Heck, Brown & Funnell, forthcoming) further describes these outcomes.

References

Friend, L., & Mills, K. A. (2021). Towards a typology of touch in multisensory makerspaces. Learning, Media and Technology, 46(4), 465-482. https://doi.org/10.1080/17439884.2021.1928695

Mills, K. A., & Brown, A. (2022). Immersive virtual reality (VR) for digital media making: Transmediation is key. Learning, Media and Technology, 47(2), 179–200. https://doi.org/10.1080/17439884.2021.1952428

Mills, K. A., & Brown, A. (2023). Smart glasses for 3D multimodal composition. Learning Media & Technology, 1-22. [Published first online] http://dx.doi.org/10.1080/17439884.2023.2207142

Mills, K., Heck, E., Brown, A., & Funnell, P. (forthcoming). Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report. ARC Future Fellowship. Institute for Learning Sciences and Teacher Education. Australian Catholic University.

Mills, K. A., Scholes, L. & Brown, A. (2022). Virtual reality and embodiment in multimodal meaning making. Written Communication, 39(3), 335–369. https://doi.org/10.1177/07410883221083517

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge university press.