Retention study using video plus AI-generated retrieval practice

Introduction

The aim of this study was to compare the effectiveness of traditional training videos versus the effectiveness of using a video with AI technology (Total Recall). The Total Recall option involved chunking the training video into easily digestible clips and punctuating those clips with AI generated retrieval questions. This required the trainee to effortfully recall information communicated in the video.

The two groups were compared. Both groups completed the same trivia questions after the course in order to establish which group had retained the most information.

Hypothesis

Our hypothesis was that the group who had completed the Total Recall version would score higher on the trivia questions, i.e. retain more information, than those who had only watched the videos.

The reason for this hypothesis was on the basis of the learning theory contained in Make it Stick by Brown, Roediger III and McDaniel, in particular the impact of active retrieval exercises on ones ability to recall information. In addition, Donald Clark ran a learning project with the AI-powered recall exercises for a global travel company and the results attributed a 36% increase in sales to the improvement in the ability of staff to recall useful travel information for customers.

Also, whilst video has become a popular tool in learning, little attention has been paid to the research that suggests the enhancement of video learning with active recall exercises. There has been research into the use of video for learning recommends several techniques to enhance the watching of video on its own (Reeves & Nass, 1996; Zhang, 2006; Mayer, 2008: Brame, 2016; Chaohua, 2019).

Method

Twenty-six participants were selected for this trial. Thirteen
of the participants watched the video only. The other thirteen completed the
Total Recall version; they watched the same video but it was broken into four
segments and was then infused with AI generated, relevant questions.

In the Total Recall version, the AI pulled key information
from the transcript of the video and created questions. The AI also offered the
participants:

  • the
    option to correct themselves if they make a mistake
  • the
    option to ‘go back’ and re-watch the segment of the video if they didn’t know
    the answer
  • forgiveness
    for spelling mistakes and accepted both British and American English.

After the video / Total Recall both groups completed an identical ‘trivia’ questionnaire to test their knowledge of the video content that had seen.

Results

Participants in the Total Recall trial group scored 61.5% higher than those who just watched the video. They also scored higher more frequently and consistently, making it more effective and long lasting.

  • Video + AI group had a 61.5% increase in mean retention, from a mean value of 9.00 to 14.54.

 


Bar chart comparing the total scores. The mean score for each set is shown with a red line.

Discussion

Combining video with artificial intelligence clearly had a positive impact on the learners ability to recall the information from the videos. The evidence suggests that this is because of the factors outlined below:

Strengths of video

Video has it’s strengths. As the core content type in the Total Recall formula, video is an engaging method for presenting the information to the learner. It’s notoriously easy to consume and it’s memorable.

Weaknesses of video and the ‘illusion of learning

But as a learning medium, video also has some weaknesses. When we watch a good documentary, we often have a powerful feeling that we are learning a lot, but when tested on the facts and figures and concepts from that film, we would generally be found lacking. 

Bjork (2013) and others uncovered the phenomenon of the illusion of learning; that learners can think that learning experiences have stuck when in fact they have disappeared from their memory, often as soon as within the first 20 minutes. This is the danger with video – it can create the ‘illusion of learning’. This isn’t surprising when we think of how engaging video content can be – it’s easy to imagine enjoying a piece of content and then when actually tested on detail, it being very difficult to recall the facts.

The common mistake that is made with the design and creation of video in learning is not taking learning theory into account. Either because designers forget about the limitations of working memory, or because they don’t understand those limitations.

Episodic AND semantic memory

Video taps into episodic memory. The AI active retrieval exercises impact semantic memory.

Put simply, episodic memory is our memory of experiences and of emotions. Whereas semantic memory is the recall of facts and figures. Episodic and semantic memory are processed in different ways by the brain.

With Total Recall, the combination of video and artificial intelligence means that episodic and semantic memory are both being called into action. And our hypothesis was that this would have a 2+2=5 effect.

Working memory

A person’s working memory lasts about 20 seconds and can only hold three or four things at one time. Without the time to encode, these things can be quickly forgotten through overload or the failure to consolidate into long-term memory (Sweller, 1988).

Chunking

Chunking is a method employed by Total Recall to combat the limitations associated with working memory. Chunking is the process of breaking the learning down into chunks.  Ideally, the chunks will be designed in a way that delivers the building blocks of learning in a rational order that builds towards a complete understanding of the topic. This model includes the idea that you need to absorb a chunk into long term memory before you can progress to the next chunk. 

Florella (2019) proposes that learning improves when there are “visual rests”. Also,  memory is enhanced when

people have a chance to stop and think about the information presented“.

Chunking video down to smaller, meaningful segments and providing the opportunity for active, effortful learning will both enhance learning by reducing cognitive load and increasing reinforcement, retention and recall.  

Active retrieval

Watching a video can be an enjoyable experience, but without an additional active, effortful learning, we simply forget.

MacHardy (2015) shows that the relationship between video and the active learning must be meaningful and closely related. In a large data mining exercise, they showed that if the two are too loosely related, it inhibits student attainment. To increase reinforcement, retention and recall, (Szpunar, 2013; Roediger, 2006; Vural, 2013) the suggestion is that retrieving key concepts is a powerful learning technique. This was the aim of this study, to test the hypothesis that chunked video with video plus AI-generated active retrieval practice increases reinforcement, retention and recall.

Practical applications

There are several possible applications of this form of
enhanced video learning:

  1. Existing video learning libraries can be made into far more effective learning experiences
  2. New videos for learning can be created with the intention of pairing it with the active retrieval exercises in order to create more effective learning experiences
  3. Different video treatments: as long as a clear transcript can be extracted from the video, the Total Recall application can be applied (e.g. talking heads video, animation with voiceover, live action footage of a process with voiceover).

Note that additional design recommendations identified
during the study include:

  1. Scripting the videos into a more ‘chaptered’ structure with ‘chunking’ in mind
  2. Clear edit points on visuals and audio at the end of each planned chunk of video
  3. Close relationship between the video and the retrieval practice

Summary

In summary, this trial provides evidence that the combined use of both chunked videos and AI-generated retrieval practice, significantly increases retention and recall and can be strongly recommended for both existing and new video learning content.

References

Bjork, R.A., Dunlosky, J. and Kornell, N., 2013. Self-regulated
learning: Beliefs, techniques, and illusions. Annual review of
psychology
64, pp.417-444.

Brame, C.J., 2016. Effective educational videos: Principles
and guidelines for maximizing student learning from video content. CBE—Life
Sciences Education
15(4), p.es6.

Chaohua, O, Joyner, D, Goel, A., 2019. Developing Videos for
Online Learning: A 7-Principle Model. Online Learning

Fiorella, L., van Gog, T., Hoogerheide, V. and Mayer, R.E., 2017.
It’s all a matter of perspective: Viewing first-person video modeling examples
promotes learning of an assembly task. Journal of Educational
Psychology
109(5), p.653.

Fiorella, L., Stull, A.T., Kuhlmann, S. and Mayer, R.E., 2019.
Fostering generative learning from video lessons: Benefits of
instructor-generated drawings and learner-generated explanations. Journal
of Educational Psychology
.

Guo PJ, Kim J, Robin R.  [email protected]’14 Proceedings of the First
ACM Conference on Learning at Scale.New York: ACM; 2014. How video production
affects student engagement: an empirical study of MOOC videos; pp. 41–50.

MacHardy Z, Pardos ZA., 2015 Evaluating the relevance of
educational videos using BKT and big data. In: Santos OC, Boticario JG, Romero
C, Pechenizkiy M, Merceron A, Mitros P, Luna JM, Mihaescu C, Moreno P,
Hershkovitz A, Ventura S, Desmarais M, editors. Proceedings of the 8th
International Conference on Educational Data Mining, Madrid, Spain.

Mayer, R.E. and Moreno, R., 2003. Nine ways to reduce cognitive
load in multimedia learning. Educational psychologist38(1),
pp.43-52.

Mayer, R.E., 2008. Applying the science of learning: Evidence-based
principles for the design of multimedia instruction. American
psychologist
63(8), p.760.

Reeves, B. and Nass, C.I., 1996. The media equation: How
people treat computers, television, and new media like real people and places
.
Cambridge university press.

Risko, E.F., Anderson, N., Sarwal, A., Engelhardt, M. and
Kingstone, A., 2012. Everyday attention: Variation in mind wandering and memory
in a lecture. Applied Cognitive Psychology26(2),
pp.234-242.

Roediger III, H.L. and Karpicke, J.D., 2006. The power of
testing memory: Basic research and implications for educational practice. Perspectives
on psychological science
1(3), pp.181-210.

Sweller, J., 1988. Cognitive load during problem solving: Effects
on learning. Cognitive science12(2), pp.257-285

Szpunar, K.K., Khan, N.Y. and Schacter, D.L., 2013. Interpolated
memory tests reduce mind wandering and improve learning of online
lectures. Proceedings of the National Academy of Sciences110(16),
pp.6313-6317.

Vural, O.F., 2013. The Impact of a Question-Embedded Video-based
Learning Tool on E-learning. Educational Sciences: Theory and Practice13(2),
pp.1315-1323.

WildFire www.wildfirlearning.co.uk

Zhang, D., Zhou, L., Briggs, R.O. and Nunamaker Jr, J.F., 2006.
Instructional video in e-learning: Assessing the impact of interactive video on
learning effectiveness. Information & management43(1),
pp.15-27.

Share on linkedin
Share on twitter
Share on facebook
Share on skype
Share on email

Want to make change happen together?​