Handbook of Learning Analytics
Chapter 6
Multimodal Learning Analytics: Rationale, Process, Examples, and Direction
Xavier Ochoa
Abstract
This chapter is an introduction to the use of multiple modalities of learning trace data to better understand and feedback learning processes that occur both in digital and face-to-face contexts. First, it will explain the rationale behind the emergence of this type of study, followed by a brief explanation of what Multimodal Learning Analytics (MmLA) is based on current conceptual understandings and current state-of-the-art implementations. The majority of this chapter is dedicated to describing the general process of MmLA from the mapping of learning constructs to low-level multimodal learning traces to the reciprocal implementation of multimedia recording, multimodal feature extraction, analysis, and fusion to detect behavioral markers and estimate the studied constructs. This process is illustrated by the detailed dissection of a real-world example. This chapter concludes with a discussion of the current challenges facing the field and the directions in which the field is moving to address them.
Alavi, Hamed S., and Pierre Dillenbourg. 2012. “An Ambient Awareness Tool for Supporting Supervised Collaborative Problem Solving.” IEEE Transactions on Learning Technologies 5 (3): 264–74. https://doi.org/10.1109/TLT.2012.7.
Andrade, Alejandro. 2017. “Understanding Student Learning Trajectories Using Multimodal Learning Analytics Within an Embodied-Interaction Learning Environment.” In Proceedings of the Seventh International Learning Analytics & Knowledge Conference, 70–79. ACM.
Andrist, Sean, Dan Bohus, and Ashley Feniello. 2019. “Demonstrating a Framework for Rapid Development of Physically Situated Interactive Systems.” In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 668–68. https://doi.org/10.1109/HRI.2019.8673067.
Bachour, Khaled, Frederic Kaplan, and Pierre Dillenbourg. 2010. “An Interactive Table for Supporting Participation Balance in Face-to-Face Collaborative Learning.” IEEE Transactions in Learning Technology 3 (3): 203–13. https://doi.org/10.1109/tlt.2010.18.
Blikstein, Paulo. 2011. “Using Learning Analytics to Assess Students’ Behavior in Open-Ended Programming Tasks.” In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, 110–16. Banff, Alberta, Canada: Association for Computing Machinery. https://doi.org/10.1145/2090116.2090132.
———. 2013. “Multimodal Learning Analytics.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 102–6. Leuven, Belgium: Association for Computing Machinery. https://doi.org/10.1145/2460296.2460316.
Cao, Zhe, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.” IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (1): 172–86.
Chen, Lei, Chee Wee Leong, Gary Feng, and Chong Min Lee. 2014. “Using Multimodal Cues to Analyze MLA’14 Oral Presentation Quality Corpus: Presentation Delivery and Slides Quality.” In Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge, 45–52. 2666640: ACM. https://doi.org/10.1145/2666633.2666640.
Cobb, Paul, Jere Confrey, Andrea diSessa, Richard Lehrer, and Leona Schauble. 2003. “Design Experiments in Educational Research.” Educational Researcher 32 (1): 9–13. https://doi.org/10.3102/0013189X032001009.
Cornide-Reyes, Hector, René Noël, Fabián Riquelme, Matías Gajardo, Cristian Cechinel, Roberto Mac Lean, Carlos Becerra, Rodolfo Villarroel, and Roberto Munoz. 2019. “Introducing Low-Cost Sensors into the Classroom Settings: Improving the Assessment in Agile Practices with Multimodal Learning Analytics.” Sensors 19 (15): 3291.
Cronbach, Lee J., and Paul E. Meehl. 1955. “Construct Validity in Psychological Tests.” Psychological Bulletin 52 (4): 281–302. https://doi.org/10.1037/h0040957.
Cukurova, Mutlu, Michail Giannakos, and Roberto Martinez-Maldonado. 2020. “The Promise and Challenges of Multimodal Learning Analytics.” British Journal of Educational Technology 51 (5): 1441–9. https://doi.org/10.1111/bjet.13015.
Di Mitri, Daniele, Jan Schneider, Marcus Specht, and Hendrik Drachsler. 2018. “From Signals to Knowledge: A Conceptual Model for Multimodal Learning Analytics.” Journal of Computer Assisted Learning 34 (4): 338–49. https://doi.org/10.1111/jcal.12288.
Echeverria, Vanessa. 2020. “Designing Feedback for Collocated Teams Using Multimodal Learning Analytics.” PhD thesis, University of Technology Sydney.
Echeverria, Vanessa, Roberto Martinez-Maldonado, Katherine Chiluiza, and Simon Buckingham Shum. 2017. “DBCollab: Automated Feedback for Face-to-Face Group Database Design.” In Proceedings of the 25th International Conference on Computers in Education, ICCE 2017-Main Conference Proceedings.
Eradze, Maka, Maria Jesus Rodriguez Triana, and Mart Laanpere. 2017. “How to Aggregate Lesson Observation Data into Learning Analytics Datasets?” In 6th Multimodal Learning Analytics (MMLA) Workshop and the 2nd Cross-LAK Workshop Co-Located with 7th International Learning Analytics and Knowledge Conference (LAK 2017).
Freedman, David H. 2010. “Why Scientific Studies Are so Often Wrong: The Streetlight Effect.” Discover Magazine 26.
Gall, Meredith Damien, Walter R. Borg, and Joyce P. Gall. 1996. Educational Research: An Introduction, 6th Ed. Educational Research: An Introduction, 6th Ed. White Plains, NY, England: Longman Publishing.
Giannakos, Michail N., Kshitij Sharma, Ilias O. Pappas, Vassilis Kostakos, and Eduardo Velloso. 2019. “Multimodal Data as a Means to Understand the Learning Experience.” International Journal of Information Management 48: 108–19.
Janning, Ruth, Carlotta Schatten, and Lars Schmidt-Thieme. 2014. “Multimodal Affect Recognition for Adaptive Intelligent Tutoring Systems.” In International Journal of Information Management.
Kizilcec, René F., Chris Piech, and Emily Schneider. 2013. “Deconstructing Disengagement: Analyzing Learner Sub-Populations in Massive Open Online Courses.” In Educational Data Mining Conference (Workshops), 170–79. Leuven, Belgium: Association for Computing Machinery. https://doi.org/10.1145/2460296.2460330.
Kluger, Avraham N., and Angelo DeNisi. 1996. “The Effects of Feedback Interventions on Performance: A Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory.” Psychological Bulletin 119 (2): 254.
Kress, Gunther R., and Theo Van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. An Arnold Publication Series. Arnold. https://books.google.com/books?id=R494tAEACAAJ.
Limbu, Bibeg Hang, Halszka Jarodzka, Roland Klemke, and Marcus Specht. 2019. “Can You Ink While You Blink? Assessing Mental Effort in a Sensor-Based Calligraphy Trainer.” Sensors 19 (14): 3244.
Lund, Kristine. 2007. “The Importance of Gaze and Gesture in Interactive Multimodal Explanation.” Language Resources and Evaluation 41 (3): 289–303. http://www.jstor.org/stable/30204707.
Martin, Taylor, Ani Aghababyan, Jay Pfaffman, Jenna Olsen, Stephanie Baker, Philip Janisiewicz, Rachel Phillips, and Carmen Petrick Smith. 2013. “Nanogenetic Learning Analytics: Illuminating Student Learning Pathways in an Online Fraction Game.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 165–69. Leuven, Belgium: Association for Computing Machinery. https://doi.org/10.1145/2460296.2460328.
Martinez-Maldonado, Roberto, Vanessa Echeverria, Doug Elliott, Carmen Axisa, Tamara Power, and Simon Buckingham Shum. 2019. “Making the Design of CSCL Analytics Interfaces a Co-Design Process: The Case of Multimodal Teamwork in Healthcare.” In Computer Supported Collaborative Learning 2019, 859–60. International Society of the Learning Sciences.
Ochoa, Xavier, and Federico Dominguez. 2020. “Controlled Evaluation of a Multimodal System to Improve Oral Presentation Skills in a Real Learning Setting.” British Journal of Educational Technology 51 (5): 1615–30. https://doi.org/10.1111/bjet.12987.
Ochoa, Xavier, Federico Domínguez, Bruno Guamán, Ricardo Maya, Gabriel Falcones, and Jaime Castells. 2018. “The RAP System: Automatic Feedback of Oral Presentation Skills Using Multimodal Analysis and Low-Cost Sensors.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 360–64. Sydney, New South Wales, Australia: Association for Computing Machinery. https://doi.org/10.1145/3170358.3170406.
Ochoa, Xavier, and Marcelo Worsley. 2016. “Augmenting Learning Analytics with Multimodal Sensory Data.” Journal of Learning Analytics 3 (2): 213–19.
Pardo, Abelardo, and George Siemens. 2014. “Ethical and Privacy Principles for Learning Analytics.” British Journal of Educational Technology 45 (3): 438–50. https://doi.org/10.1111/bjet.12152.
Prieto, Luis P., Kshitij Sharma, Pierre Dillenbourg, and María Jesús. 2016. “Teaching Analytics: Towards Automatic Extraction of Orchestration Graphs Using Wearable Sensors.” In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, 148–57. ACM.
Romano, Gianluca, Jan Schneider, and Hendrik Drachsler. 2019. “Dancing Salsa with Machines—Filling the Gap of Dancing Learning Solutions.” Sensors 19 (17): 3661.
Scherer, Stefan, Marcelo Worsley, and Louis-Philippe Morency. 2012. “1st International Workshop on Multimodal Learning Analytics: Extended Abstract.” In Proceedings of the 14th ACM International Conference on Multimodal Interaction, 609–10. Santa Monica, California, USA: Association for Computing Machinery. https://doi.org/10.1145/2388676.2388803.
Schneider, Jan, Dirk Börner, Peter Van Rosmalen, and Marcus Specht. 2015. “Presentation Trainer, Your Public Speaking Multimodal Coach.” In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 539–46. acm.
Selwyn, Neil. 2019. “What’s the Problem with Learning Analytics?” Journal of Learning Analytics 6 (3): 11–19. https://doi.org/10.18608/jla.2019.63.3.
Sharma, Kshitij, Zacharoula Papamitsiou, and Michail Giannakos. 2019. “Building Pipelines for Educational Data Using AI and Multimodal Analytics: A ‘Grey-Box’ Approach.” British Journal of Educational Technology 50 (6): 3004–31. https://doi.org/10.1111/bjet.12854.
Wagner, Johannes, Florian Lingenfelser, Tobias Baur, Ionut Damian, Felix Kistler, and Elisabeth Andr. 2013. “The Social Signal Interpretation (SSI) Framework: Multimodal Signal Processing and Recognition in Real-Time.” In Proceedings of the 21st ACM International Conference on Multimedia, 831–34. 2502223: ACM. https://doi.org/10.1145/2502081.2502223.
Wei, Shih-En, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. “Convolutional Pose Machines.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4724–32.
Worsley, Marcelo. 2018. “(Dis) Engagement Matters: Identifying Efficacious Learning Practices with Multimodal Learning Analytics.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 365–69. ACM.
Worsley, Marcelo, Dor Abrahamson, Paulo Blikstein, Shuchi Grover, Bertrand Schneider, and Mike Tissenbaum. 2016. “Situating Multimodal Learning Analytics.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 2:1346–9.
Worsley, Marcelo, and Paulo Blikstein. 2013. “Towards the Development of Multimodal Action Based Assessment.” In Proceedings of International Conference of the Learning Sciences, 94–101. Leuven, Belgium: Association for Computing Machinery. https://doi.org/10.1145/2460296.2460315.
No Supplementary Material Available
Title
Multimodal Learning Analytics: Rationale, Process, Examples, and Direction
Book Title
Handbook of Learning Analytics
Pages
pp. 54-65
Copyright
2022
DOI
10.18608/hla22.006
ISBN
978-0-9952408-3-4
Publisher
Society for Learning Analytics Research
Authors
Xavier Ochoa
Editors
Charles Lang
Alyssa Friend Wise
Agathe Merceron
Dragan Gašević
George Siemens