Augmented Reality: Enhancing Multimodal Literacy with Learning Activities

By Elizabeth Heck

Augmented Reality (AR) is a technology that enhances a user’s perception of the actual, physical environment. It involves overlaying virtual information, such as graphics, text, or audio, onto the real-world surroundings, seamlessly integrating them into the user’s field of vision. AR experiences can be accessed through various devices, including headsets, smart glasses, smartphones, or tablets (Akçayır & Akçayır, 2017; Wu et al., 2013).

In a previous blog post, we shared the experiences of some of our older students, in grades 9 and 10 (aged between 14 and 16 years), and their use of (AR) in reading and comprehension. We observed the students’ use of ARMaker, as one of the two apps we explored in our Sensory Orchestration field work. The second of the two apps, and as noted in the following post, is the Merge Cube™, a physical foam rubber cube approximately the size of a tennis ball (see Figure 1).

Figure 1. The Merge Cube

What made the Merge Cube particularly fascinating in regard to AR and multimodal capabilities, was its tangibility. Not only could students view such AR overlay via their iPads, but they could also move and manipulate these images with both the physical cube and iPad and engage in the content in a far more multimodal way. We found the students thought of this both engaging and fascinating. So, how can this be further applied in your classroom?

Teachers and schools interested in exploring the Merge Cube can do so by purchasing a class set of physical cubes. However, another option that may also be appealing is to make a classroom set of cubes. As a result, adding further multimodality in the creation of the cube itself. Both options can be found on the Merge Cube website, and an example of the paper Cube template can be seen in Figure 2.

Figure 2. An example of the Merge Paper Cube edition. A perfect way to encourage further multimodality in the learning process. This is also an ideal starter for schools wishing to explore this form of AR.

We have provided some suggested learning activities, where teachers may like to incorporate the Merge Cube in their classroom and explore the further possibilities of multimodal literacy learning.

Suggested Learning Activities: Exploring multimodal literacy with Merge Cube.

Learning Activity – Introducing Merge Cube and AR Storytelling: Genre and descriptive writing (Upper primary: Years 5-6 and can be extended to grades 7-10)

Learning Objectives:

• Introduce students to the Merge Cube and explore its use in augmented reality storytelling.
• Develop descriptive writing skills through the integration of AR technology.

Materials:

• Merge Cubes for each student or group
• Tablets/iPads or smart device for each student or group
• AR storytelling app compatible with Merge Cube (e.g., Merge EDU Explorer™). All Merge apps are available on Apple, Google Play and Windows Store.

Activity Overview:

Part 1: Introduction to Merge Cube:

Introduction:

1. • Briefly explain what the Merge Cube is and its basic features.
   • Emphasise its interactive elements and the potential for creative storytelling.

Explore Merge Cube:

1. • Allow students to explore the Merge Cube individually or in small groups.
   • Optional: Create a class set of paper Merge Cubes (see template above or via the Merge Website).
   • Encourage students to interact with the cube and become familiar with its capabilities.
   • Let students play and work through 1-2 of the activities with the Cube and the Explorer™ app. For example, they might want to explore the planets in ‘Galactic Explorer’, or the earth’s surface with ‘Transforming Earth’ and so forth. *Teacher note: Some of these activities are free and some are available via paid subscription.
   • What did they learn from the activity? Class discussion, and main points (e.g.: What did they learn from the space activity, or about the Earth’s surface and volcanos?)

Group Discussion:

1. • Facilitate a group discussion on potential uses for the Merge Cube in storytelling.
   • Encourage creative ideas and brainstorming. What did students learn in their allocated activities?
   • Thinking about what students learnt through one of the Explorer™ activities, what sort of story could they create? For example: A science fiction story set in outer space, or a story set in pre-historic times (with volcanos!).

Part 2: Augmented Reality (AR) Storytelling

 

Introduction to Augmented Reality (AR) Storytelling:

1. • After the group discission, for example, you may like to suggest students set a story as a suggested genre above, and they must incorporate some of the facts they learnt from the Explorer app activity. For example, a story about an astronaut and travelling to Mars.
   • Explain ‘Genre’ and what it means in storytelling.
   • Briefly explain how AR can enhance storytelling experiences. Can they capture any experiences or visual prompts to add to their story descriptions. What facts from the Explorer™ learning activities can they also incorporate into their story?

 

Story Planning:

1. • Have students brainstorm and outline a short story individually or in pairs.
   • Emphasise the importance of bright and colourful descriptions and engaging characters.

AR Implementation:

1. • Using the AR app and Merge Cube, guide students in creating an augmented reality scene that corresponds to a key point in their stories.

Conclusion

• Allow students to finish writing their stories, incorporating the ideas generated during the learning activities.
• At the conclusion of the activities, encourage students will share their written stories and present the corresponding AR scenes or visual stimulus using the Merge Cubes.
• Ask students what factual information they discovered about elements of their stories (e.g., Mars, the earth’s crust, and so forth).

Australian Curriculum Learning Areas for Years 8-10:

• English
• The Arts (including Media Arts and Visual Arts)
• Science

General Capabilities

• Critical and Creative Thinking
• Literacy
• Digital Literacy

References

ACARA. (2023). The Australian Curriculum (Version 9). https://v9.australiancurriculum.edu.au/

Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1–11. https://doi.org/10.1016/j.edurev.2016.11.002

MergeEDU. (n.d). Merge EDU. https://mergeedu.com/

Mills, K., Heck, E., Brown, A., & Funnell, P. (2023). Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report. ARC Future Fellowship. Institute for Learning Sciences and Teacher Education Australian Catholic University.

Wu, H. K., Lee, S. W.-Y., Chang, H.-Y., & Liang, J. C. (2013). Current status, opportunities and challenges of augmented reality in education. Computers & Education, 62, 41–49. https://doi.org/10.1016/j.compedu.2012.10.024