Handbook of Learning Analytics
Chapter 21
Human-centered Approaches to Data-informed Feedback
Yi-Shan Tsai & Roberto Martinez-Maldonado
Abstract
Learning analytics seeks to support and enhance learning through data-informed feedback practices. As learning analytics emphasizes an iterative loop from learner to data, metrics, and interventions, it is imperative that both teachers and learners play active roles in this process and contribute to the design and evaluation of enabling technologies. A key question that concerns us is: How can learning analytics tools enhance learners’ agency in the feedback process? We argue that the design and deployment of learning analytics need to recognize feedback as a dialogic process. In doing so, we emphasize that effective feedback is not just about providing information relevant to learning, but also about the practices of the people who carry out evaluations and produce or interpret information based on such evaluations. A human-centered approach is thus critical to the effectiveness of data-informed feedback. In this chapter we discuss key elements of feedback, current approaches to data-informed feedback and associated challenges; and propose a human-centered approach which facilitates collaborative learning and continuous learning among a network of actors and highlights the importance of developing data-informed feedback literacy among learners.
Aleven, Vincent, Ido Roll, Bruce M. McLaren, and Kenneth R. Koedinger. 2010. “Automated, Unobtrusive, Action-by-Action Assessment of Self-Regulation During Learning with an Intelligent Tutoring System.” Educational Psychologist 45 (4): 224–33. https://doi.org/10.1080/00461520.2010.517740.
An, Pengcheng, Kenneth Holstein, Bernice d’Anjou, Berry Eggen, and Saskia Bakker. 2020. “The TA Framework: Designing Real-Time Teaching Augmentation for K-12 Classrooms.” In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–17. CHI ’20. Honolulu, HI, USA: Association for Computing Machinery. https://doi.org/10.1145/3313831.3376277.
Arnold, Kimberly E., and Matthew D. Pistilli. 2012. “Course Signals at Purdue: Using Learning Analytics to Increase Student Success.” In Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge, 267–70. LAK ’12. New York, NY, USA: ACM. https://doi.org/10.1145/2330601.2330666.
Azcona, David, I.-Han Hsiao, and Alan F. Smeaton. 2018. “Personalizing Computer Science Education by Leveraging Multimodal Learning Analytics.” In Frontiers in Education Conference (FIE) 2018. San Jose, California, USA: IEEE. http://fie-conference.org/conferences/listing/past.
Biggs, John, and Catherine Tang. 2011. Teaching for Quality Learning at University. 4 edition. Maidenhead: Open University Press.
Boud, David, and Elizabeth Molloy. 2013. “Rethinking Models of Feedback for Learning: The Challenge of Design.” Assessment & Evaluation in Higher Education 38 (6): 698–712. https://doi.org/10.1080/02602938.2012.691462.
Broos, Tom, Maarten Pinxten, Margaux Delporte, Katrien Verbert, and Tinne De Laet. 2019. “Learning Dashboards at Scale: Early Warning and Overall First Year Experience.” Assessment & Evaluation in Higher Education 0 (0): 1–20. https://doi.org/10.1080/02602938.2019.1689546.
Buckingham Shum, Simon, Rebecca Ferguson, and Roberto Martinez-Maldonado. 2019. “Human-Centred Learning Analytics.” Journal of Learning Analytics 6 (2): 1–9–1–9. https://doi.org/10.18608/jla.2019.62.1.
Butler, Deborah L., and Philip H. Winne. 1995. “Feedback and Self-Regulated Learning: A Theoretical Synthesis.” Review of Educational Research 65 (3): 245–81. https://doi.org/10.3102/00346543065003245.
Caballé, Santi, and Jordi Conesa. 2019. “Conversational Agents in Support for Collaborative Learning in MOOCs: An Analytical Review.” In Advances in Intelligent Networking and Collaborative Systems, edited by Fatos Xhafa, Leonard Barolli, and Michal Greguš, 384–94. Lecture Notes on Data Engineering and Communications Technologies. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-98557-2_35.
Carless, David. 2020. “From Teacher Transmission of Information to Student Feedback Literacy: Activating the Learner Role in Feedback Processes.” Active Learning in Higher Education, July, 1469787420945845. https://doi.org/10.1177/1469787420945845.
Carter, Jeri. 1984. “Instructional Learner Feedback: A Literature Review with Implications for Software Development.” Computing Teacher 12 (2): 53–55.
Cha, Hyun-Jin, and Taejung Park. 2019. “Applying and Evaluating Visualization Design Guidelines for a MOOC Dashboard to Facilitate Self-Regulated Learning Based on Learning Analytics.” KSII Transactions on Internet and Information Systems 13 (6). https://doi.org/10.3837/tiis.2019.06.002.
Chen, Bodong, and Haiyi Zhu. 2019. “Towards Value-Sensitive Learning Analytics Design.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 343–52. LAK19. Tempe, AZ, USA: Association for Computing Machinery. https://doi.org/10.1145/3303772.3303798.
Clow, Doug. 2012. “The Learning Analytics Cycle: Closing the Loop Effectively.” In Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge, 134–38. LAK ’12. New York, NY, USA: ACM. https://doi.org/10.1145/2330601.2330636.
Cohen, Vicki Blum. 1985. “A Reexamination of Feedback in Computer-Based Instruction: Implications for Instructional Design.” Educational Technology 25 (1): 33–37. https://www.learntechlib.org/p/169679/.
Corrin, Linda, and Paula de Barba. 2015. “How Do Students Interpret Feedback Delivered via Dashboards?” In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, 430–31. LAK ’15. New York, NY, USA: ACM. https://doi.org/10.1145/2723576.2723662.
Corrin, Linda, Gregor Kennedy, and Raoul Mulder. 2013. “Enhancing Learning Analytics by Understanding the Needs of Teachers.” In, 201–5. Australasian Society for Computers in Learning in Tertiary Education. https://www.learntechlib.org/p/171128/.
Dawson, Phillip, Michael Henderson, Paige Mahoney, Michael Phillips, Tracii Ryan, David Boud, and Elizabeth Molloy. 2019. “What Makes for Effective Feedback: Staff and Student Perspectives.” Assessment & Evaluation in Higher Education 44 (1): 25–36. https://doi.org/10.1080/02602938.2018.1467877.
Dennis, Matt, Judith Masthoff, and Chris Mellish. 2016. “Adapting Progress Feedback and Emotional Support to Learner Personality.” International Journal of Artificial Intelligence in Education 26 (3): 877–931. https://doi.org/10.1007/s40593-015-0059-7.
Dollinger, Mollie, Danny Liu, Natasha Arthars, and Jason Lodge. 2019. “Working Together in Learning Analytics Towards the Co-Creation of Value.” Journal of Learning Analytics 6 (2): 10–26–10–26. https://doi.org/10.18608/jla.2019.62.2.
Dollinger, Mollie, and Jason Lodge. 2018. “Co-Creation Strategies for Learning Analytics.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 97–101. LAK ’18. New York, NY, USA: ACM. https://doi.org/10.1145/3170358.3170372.
Ellingsen, Matthew, Emile Fetscher, and Emma Saunders. n.d. “Design Thinking in a Day.” http://designthinking.co.nz/design-thinking-in-a-day/.
Gašević, Dragan, Shane Dawson, and George Siemens. 2015. “Let’s Not Forget: Learning Analytics Are About Learning.” TechTrends 59 (1): 64–71. https://doi.org/10.1007/s11528-014-0822-x.
Gašević, Dragan, Yi-Shan Tsai, Shane Dawson, and Abelardo Pardo. 2019. “How Do We Start? An Approach to Learning Analytics Adoption in Higher Education.” International Journal of Information and Learning Technology 36 (4): 342–53.
Gibson, Andrew, and Roberto Martinez-Maldonado. 2017. “That Dashboard Looks Nice, but What Does It Mean? Towards Making Meaning Explicit in Learning Analytics Design.” In Proceedings of the 29th Australian Conference on Computer-Human Interaction, 528–32. OZCHI ’17. Brisbane, Queensland, Australia: Association for Computing Machinery. https://doi.org/10.1145/3152771.3156171.
Hattie, John, and Helen Timperley. 2007. “The Power of Feedback.” Review of Educational Research 77 (1): 81–112. https://doi.org/10.3102/003465430298487.
Henderson, Michael, Michael Phillips, Tracii Ryan, David Boud, Phillip Dawson, Elizabeth Molloy, and Paige Mahoney. 2019. “Conditions That Enable Effective Feedback.” Higher Education Research & Development 38 (7): 1401–16. https://doi.org/10.1080/07294360.2019.1657807.
Holstein, Kenneth, Bruce M. McLaren, and Vincent Aleven. 2019. “Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity.” Journal of Learning Analytics 6 (2): 27–52–27–52. https://doi.org/10.18608/jla.2019.62.3.
Howell, Joel A., Lynne D. Roberts, and Vincent O. Mancini. 2018. “Learning Analytics Messages: Impact of Grade, Sender, Comparative Information and Message Style on Student Affect and Academic Resilience.” Computers in Human Behavior 89: 8–15. https://doi.org/10.1016/j.chb.2018.07.021.
Jivet, Ioana, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2017. “Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational Practice.” In Data Driven Approaches in Digital Education, edited by Élise Lavoué, Hendrik Drachsler, Katrien Verbert, Julien Broisin, and Mar Pérez-Sanagustín, 82–96. Lecture Notes in Computer Science. Springer International Publishing.
Jivet, Ioana, Maren Scheffel, Marcus Specht, and Hendrik Drachsler. 2018. “License to Evaluate: Preparing Learning Analytics Dashboards for Educational Practice.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 31–40. LAK ’18. New York, NY, USA: ACM. https://doi.org/10.1145/3170358.3170421.
Johnson, David W., and Roger T. Johnson. 1993. “Cooperative Learning and Feedback in Technology-Based Instruction.” In Interactive Instruction and Feedback, edited by John V. Dempsey and Gregory C. Sales, 133–57. Englewood Cliffs, NJ: Educational Technology Publications.
Kim, Jeonghyun, Il-Hyun Jo, and Yeonjeong Park. 2016. “Effects of Learning Analytics Dashboard: Analyzing the Relations Among Dashboard Utilization, Satisfaction, and Learning Achievement.” Asia Pacific Education Review 17 (1): 13–24. https://doi.org/10.1007/s12564-015-9403-8.
Kitto, Kirsty, Simon Buckingham Shum, and Andrew Gibson. 2018. “Embracing Imperfection in Learning Analytics.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 451–60. LAK ’18. New York, NY, USA: ACM. https://doi.org/10.1145/3170358.3170413.
Knight, Simon, Simon Buckingham Shum, and Karen Littleton. 2014. “Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space.” Journal of Learning Analytics 1 (2): 23–47–23‐47. https://doi.org/10.18608/jla.2014.12.3.
Kumar, Rohit, and Carolyn P. Rosé. 2011. “Architecture for Building Conversational Agents That Support Collaborative Learning.” IEEE Transactions on Learning Technologies 4 (1): 21–34. https://doi.org/10.1109/TLT.2010.41.
Lim, Lisa, Shane Dawson, Srecko Joksimovic, and Dragan Gašević. 2019. “Exploring Students’ Sensemaking of Learning Analytics Dashboards: Does Frame of Reference Make a Difference?” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 250–59. LAK19. Tempe, AZ, USA: Association for Computing Machinery. https://doi.org/10.1145/3303772.3303804.
Lim, Lisa, Sheridan Gentili, Abelardo Pardo, Vitomir Kovanović, Alexander Whitelock-Wainwright, Dragan Gašević, and Shane Dawson. 1BC. “What Changes, and for Whom? A Study of the Impact of Learning Analytics-Based Process Feedback in a Large Course.” Learning and Instruction. https://doi.org/10.1016/j.learninstruc.2019.04.003.
Lodge, Jason, and Linda Corrin. 2017. “What Data and Analytics Can and Do Say About Effective Learning.” NPJ Science of Learning 2 (1): 1–2. https://doi.org/10.1038/s41539-017-0006-5.
Lodge, Jason, and M. Lewis. 2012. “Pigeon Pecks and Mouse Clicks: Putting the Learning Back into Learning Analytics.” Future Challenges, Sustainable Futures. Proceedings Ascilite Wellington 2012. /scholarlywork/985857-pigeon-pecks-and-mouse-clicks--putting-the-learning-back-into-learning-analytics.
Lodge, Jason, Ernesto Panadero, Jaclyn Broadbent, and Paula G. de Barba. 2018. “Supporting Self-Regulated Learning with Learning Analytics.” In Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers, edited by Jason Lodge, Jared Cooney Horvath, and Linda Corrin. Abingdon-on-Thames, UK: Routledge.
Long, Phillip D., George Siemens, Grainne Conole, and Dragan Gašević, eds. 2011. Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK’11). Banff, AB, Canada: ACM. https://tekri.athabascau.ca/analytics/.
Martinez-Maldonado, Roberto, Vanessa Echeverria, Gloria Fernandez Nieto, and Simon Buckingham Shum. 2020. “From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics.” In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–15. CHI ’20. Honolulu, HI, USA: Association for Computing Machinery. https://doi.org/10.1145/3313831.3376148.
Matcha, Wannisa, Nora’ayu Ahmad Uzir, Dragan Gasevic, and Abelardo Pardo. 2019. “A Systematic Review of Empirical Studies on Learning Analytics Dashboards: A Self-Regulated Learning Perspective.” IEEE Transactions on Learning Technologies, 1–1. https://doi.org/10.1109/TLT.2019.2916802.
Milligan, Sandra. 2020. “Standards for Developing Assessments of Learning Using Process Data.” In Re-Imagining University Assessment in a Digital World, edited by Margaret Bearman, Phillip Dawson, Rola Ajjawi, Joanna Tai, and David Boud, 179–92. The Enabling Power of Assessment. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-41956-1_13.
Molenaar, Inge, and Carolien A. N. Knoop-van Campen. 2019. “How Teachers Make Dashboard Information Actionable.” IEEE Transactions on Learning Technologies. https://doi.org/10.1109/TLT.2018.2851585.
Nicol, David J., and Debra Macfarlane‐Dick. 2006. “Formative Assessment and Self‐regulated Learning: A Model and Seven Principles of Good Feedback Practice.” Studies in Higher Education 31 (2): 199–218. https://doi.org/10.1080/03075070600572090.
Pardo, Abelardo. 2018. “A Feedback Model for Data-Rich Learning Experiences.” Assessment & Evaluation in Higher Education 43 (3): 428–38. https://doi.org/10.1080/02602938.2017.1356905.
Pardo, Abelardo, Jelena Jovanovic, Shane Dawson, Dragan Gašević, and Negin Mirriahi. 2019. “Using Learning Analytics to Scale the Provision of Personalised Feedback.” British Journal of Educational Technology 50 (1): 128–38. https://doi.org/10.1111/bjet.12592.
Park, Yeonjeong, and Il-Hyun Jo. 2015. “Development of the Learning Analytics Dashboard to Support Students’ Learning Performance.” Journal of Universal Computer Science 21 (1): 110–33. http://www.jucs.org/jucs_21_1/development_of_the_learning.
Piattoeva, Nelli, and Rebecca Boden. 2020. “Escaping Numbers? The Ambiguities of the Governance of Education Through Data.” International Studies in Sociology of Education 0 (0): 1–18. https://doi.org/10.1080/09620214.2020.1725590.
Prieto, Luis P., María Jesús Rodríguez-Triana, Roberto Martínez-Maldonado, Yannis Dimitriadis, and Dragan Gašević. 2019. “Orchestrating Learning Analytics (OrLA): Supporting Inter-Stakeholder Communication About Adoption of Learning Analytics at the Classroom Level.” Australasian Journal of Educational Technology 35 (4). https://doi.org/10.14742/ajet.4314.
Prieto-Alvarez, Carlos G, Roberto Martinez-Maldonado, and Theresa Dirndorfer Anderson. 2018. “Co-Designing Learning Analytics Tools with Learners.” In Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers, edited by Jason Lodge, Jared Cooney Horvath, and Linda Corrin. Abingdon-on-Thames, UK: Routledge.
Quincey, Ed de, Chris Briggs, Theocharis Kyriacou, and Richard Waller. 2019. “Student Centred Design of a Learning Analytics System.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 353–62. LAK19. Tempe, AZ, USA: Association for Computing Machinery. https://doi.org/10.1145/3303772.3303793.
Ramos-Soto, Alejandro, Manuel Lama, Borja Vázquez-Barreiros, Alberto Bugarín, Manuel Mucientes, and Senén Barro. 2015. “Towards Textual Reporting in Learning Analytics Dashboards.” In Proceedings of the 2015 IEEE 15th International Conference on Advanced Learning Technologies, 260–64. ICALT ’15. USA: IEEE Computer Society. https://doi.org/10.1109/ICALT.2015.96.
Roberts, Lynne D., Joel A. Howell, and Kristen Seaman. 2017. “Give Me a Customizable Dashboard: Personalized Learning Analytics Dashboards in Higher Education.” Technology, Knowledge and Learning. https://doi.org/10.1007/s10758-017-9316-1.
Ryan, Tracii, Dragan Gaśević, and Michael Henderson. 2019. “Identifying the Impact of Feedback over Time and at Scale: Opportunities for Learning Analytics.” In. doi: 10.1007/978-3-030-25112-3.
Schmitz, Marcel, Evelien van Limbeek, Wolfgang Greller, Peter Sloep, and Hendrik Drachsler. 2017. “Opportunities and Challenges in Using Learning Analytics in Learning Design.” In Data Driven Approaches in Digital Education, edited by Élise Lavoué, Hendrik Drachsler, Katrien Verbert, Julien Broisin, and Mar Pérez-Sanagustín, 209–23. Lecture Notes in Computer Science. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-66610-5_16.
Schumacher, Clara, and Dirk Ifenthaler. 2018. “Features Students Really Expect from Learning Analytics.” Computers in Human Behavior 78 (Supplement C). https://doi.org/10.1016/j.chb.2017.06.030.
Schwendimann, Beat A., María Jesús Rodríguez-Triana, Andrii Vozniuk, Luis P. Prieto, Mina Shirvani Boroujeni, Adrian Holzer, Denis Gillet, and Pierre Dillenbourg. 2017. “Perceiving Learning at a Glance: A Systematic Literature Review of Learning Dashboard Research.” IEEE Transactions on Learning Technologies 10 (1): 30–41. https://dl.acm.org/doi/10.1109/TLT.2016.2599522.
Shimada, Atsushi, and Shin’ichi Konomi. 2017. “A Lecture Supporting System Based on Real-Time Learning Analytics.” In. International Association for the Development of the Information Society. https://eric.ed.gov/?id=ED579463.
Sigrist, Roland, Georg Rauter, Robert Riener, and Peter Wolf. 2013. “Augmented Visual, Auditory, Haptic, and Multimodal Feedback in Motor Learning: A Review.” Psychonomic Bulletin & Review 20 (1): 21–53. https://doi.org/10.3758/s13423-012-0333-8.
Soller, Amy, Alejandra Martinez, Patrick Jermann, and Martin Muehlenbrock. 2005. “From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning.” International Journal of Artificial Intelligence in Education (IJAIED) 15: 261–90. https://telearn.archives-ouvertes.fr/hal-00197378.
Stenalt, Maria Hvid. 2020. “Researching Student Agency in Digital Education as If the Social Aspects Matter: Students’ Experience of Participatory Dimensions of Online Peer Assessment.” Assessment & Evaluation in Higher Education 0 (0): 1–15. https://doi.org/10.1080/02602938.2020.1798355.
Tchounikine, Pierre. 2019. “Learners’ Agency and CSCL Technologies: Towards an Emancipatory Perspective.” International Journal of Computer-Supported Collaborative Learning 14 (2): 237–50. https://doi.org/10.1007/s11412-019-09302-5.
Teasley, Stephanie D. 2017. “Student Facing Dashboards: One Size Fits All?” Technology, Knowledge and Learning. https://doi.org/10.1007/s10758-017-9314-3.
Thompson, Kate, Sakinah S. J. Alhadad, Simon Buckingham Shum, Sarah Howard, Simon Knight, Roberto Martinez-Maldonado, and Abelardo Pardo. 2018. “Connecting Expert Knowledge in the Design of Classroom Learning Experiences.” In Learning Analytics in the Classroom, edited by Jason Lodge, Jared Cooney Horvath, and Linda Corrin, 111–28. Abingdon-on-Thames, UK: Routledge. http://www.taylorfrancis.com/.
Tsai, Yi-Shan, Carlo Perrotta, and Dragan Gašević. 2019. “Empowering Learners with Personalised Learning Approaches? Agency, Equity and Transparency in the Context of Learning Analytics.” Assessment & Evaluation in Higher Education 0 (0): 1–14. https://doi.org/10.1080/02602938.2019.1676396.
Tsai, Yi-Shan, Diego Rates, Pedro Manuel Moreno-Marcos, Pedro J. Muñoz-Merino, Ioana Jivet, Maren Scheffel, Hendrik Drachsler, Carlos Delgado Kloos, and Dragan Gašević. 2020. “Learning Analytics in European Higher Education: Trends and Barriers.” Computers & Education 155 (October): 103933. https://doi.org/10.1016/j.compedu.2020.103933.
Viberg, Olga, Mathias Hatakka, Olof Bälter, and Anna Mavroudi. 2018. “The Current Landscape of Learning Analytics in Higher Education.” Computers in Human Behavior 89 (December): 98–110. https://doi.org/10.1016/j.chb.2018.07.027.
Wager, Walter, and Susan Wager. 1985. “Presenting Questions, Processing Responses, and Providing Feedback in CAI.” Journal of Instructional Development 8 (4): 2–8. https://doi.org/10.1007/BF02906047.
Winstone, Naomi E., Robert A. Nash, James Rowntree, and Michael Parker. 2017. “‘It’d Be Useful, but I Wouldn’t Use It’: Barriers to University Students’ Feedback Seeking and Recipience.” Studies in Higher Education 42 (11): 2026–41. https://doi.org/10.1080/03075079.2015.1130032.
Wise, Alyssa Friend. 2019. “Learning Analytics: Using Data-Informed Decision-Making to Improve Teaching and Learning.” In Contemporary Technologies in Education: Maximizing Student Engagement, Motivation, and Learning, edited by Olusola O. Adesope and A.G. Rud, 119–43. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-89680-9_7.
No Supplementary Material Available
Title
Human-centered Approaches to Data-informed Feedback
Book Title
Handbook of Learning Analytics
Pages
pp. 213-222
Copyright
2022
DOI
10.18608/hla22.021
ISBN
978-0-9952408-3-4
Publisher
Society for Learning Analytics Research
Authors
Yi-Shan Tsai
Roberto Martinez-Maldonado
Editors
Charles Lang
Alyssa Friend Wise
Agathe Merceron
Dragan Gašević
George Siemens