Blog
The Multimodal Capabilities Of AI: Opportunities And Challenges

The Multimodal Capabilities of AI: Opportunities and challenges

By Elizabeth Heck

Artificial Intelligence (AI) is a hot topic, although it is certainly not new. Yet, what has made AI of note more recently is the increasing ease of access and affordability. In a recent Conversation article, AI is now accessible to everyone: 3 things parents should teach their kids, Kathy Mills and Christian Moro highlighted three valuable points about this growing accessibility of AI technologies, and the opportunities and challenges this poses to young people and their parents/carers. Mills and Moro (2023) noted:

  1. Be critical

Generative AI cannot evaluate the credibility of sources, nor can it always find authoritative information to back up claims. The generative AI software is also trained on data from a specific time so recent events may not be included.

  1. Watch out for chatbots

Chatbots are computer programs designed to simulate conversations as if they were another human. For example, there were more than ten million Replika users as of 2022. Replika is a chatbot billed as a companion who cares. It acts like a friend but relationships with the chatbot can become romantic or sexual.

  1. Images, videos and audio also matter

With all the focus on text, be sure to remind your children images and video are also part of the generative AI landscape. Children may be careful about what text they enter online but careless with uploading images.

These points are important to consider, and so too are the opportunities of AI, and its potential in multimodal literacy and learning. In our final report, Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report (Mills, Heck, Brown & Funnell, 2023), we explored the possibilities of multimodal literacy, the senses and extended technologies, such as Augmented Reality, Virtual Reality and Mixed Reality (AR, VR and MR). Yet, there are also possibilities for AI in this space.

Below are suggested multimodal literacy possibilities of AI that are worth exploring in your classroom.

AI-Powered artistic expression and story creation

AI apps that produce artistic content can allow students to incorporate generative art elements into their stories and other projects. Examples include Imagine Art. With the incorporation of critical literacy, there are also opportunities for students to think about how they may attribute the AI to this creative work, and what sort of prompts are appropriate. It also provides many creative possibilities.

Image manipulation: In this example from Imagine Art (see Figure 1), we used a screen capture from the MR component of our field work from one of our students. In our original image, we see the augmented controls from the student’s point of view. This image was then added to Imagine Art with the following prompt, “Please create the ultimate superhero wrist control”. In the classroom, the student may then use this as a prompt to write their own story about a local superhero and add this as an illustration, and with any images they choose or create.

Figure 1: Student image with the prompt, “Please create the ultimate superhero wrist control”.

Story Prompts: Students may also like to use AI for descriptive story prompts. This can be done in Imagine Art by simply adding text prompts with no need to upload an image. For example, a student may ask Imagine Art, “I would like an image of the Australian bush with a mysterious Bunyip emerging from a creek” (Figure 2)

Figure 2. Student generated AI image, “Australian bush with a mysterious Bunyip emerging from a creek”.

Students may also like to see how they can manipulate the image with further prompts adding keywords such as ‘friendlier’, or ‘happier’ (Figure 3). They may then keep regenerating until they find a Bunyip they like or create a whole cast of characters.

Figure 3. Another example of a student generated AI image, with the further prompts including ‘friendlier’.

This activity can be extended to get students to think about the characters they are creating. Students could then also consider these images as prompts for a story, and new characters (see Figure 4). For example, ‘Are Bunyips friendly creatures?’, ‘What are the Bunyips’ names and who are they?’. Tapping into their senses, and to add to descriptive writing prompts, they may also ask, ‘would the Bunyip have fur or scales? Is it tree leaves?’ or ‘how would a Bunyip sound?’. From this, teachers and students could explore picture books, such as ‘The Bunyip of Berkeley’s Creek’ as an example of a children’s picture book or story example.

Figure 4. Three AI Generated Bunyips that could be characters in a descriptive story.

Critical Literacy: To extend this activity further, teachers may also want to discuss how AI works. For example, what or how may Imagine Art ‘understand’ the image it created as the “Australian Bush”, and how does it know what a “Bunyip” is? Students may also research themselves and ask, “What do I understand a Bunyip to be?”.  Where might the people (or ‘developers’) who created Imagine Art, program or teach AI to understand these things?

For example, as we regenerated the image three times, we have now three Bunyips. So, students may also ask:

  • “Are they three tree created Bunyips?”
  • “Did AI understand my questions?”
  • “Where did AI get it right?”
  • “Where did AI get it wrong?”
  • “What do I understand the Australian bush and a Bunyip to be?”
  • “How do I learn what these things are?”
  • “What prompts could I use to improve this outcome?”.

 

Additionally, an example of accessible AI terms can be found with this further Conversation article, AI to Z: all the terms you need to know to keep up in the AI hype age by Samar Fatima and Kok-Leong Ong.

As such, one activity in creative writing incorporating AI, can align with several key learning areas in the Australian Curriculum (years 5 – 6 in this instance):

This also taps into critical capabilities, including creative and critical thinking, literacy, ethical understanding, and intercultural understanding. Thus, allowing a multitude of cross-curriculum possibilities.

References:

Fatima, S. & Ong, K. (2023, April 25). AI to Z: all the terms you need to know to keep up in the AI hype age. The Conversation. https://theconversation.com/ai-to-z-all-the-terms-you-need-to-know-to-keep-up-in-the-ai-hype-age-203917

Imagine.Art. (2023). ImagineArt – Dashboard.  https://www.imagine.art/dashboard

Mills, K., Heck, E., Brown, A., & Funnell, P. (2023). Senses Together – Multimodal Literacy Learning in Primary Education: Final Project Report. ARC Future Fellowship. Institute for Learning Sciences and Teacher Education Australian Catholic University.

Mills, K.A, & Moro, C. (2023, November 20).  AI is now accessible to everyone: 3 things parents should teach their kids. The Conversation. https://theconversation.com/ai-is-now-accessible-to-everyone-3-things-parents-should-teach-their-kids-217797?utm_source=twitter&utm_mediu