Handbook of Learning Analytics
Chapter 2
A Practitioner's Guide to Measurement in Learning Analytics: Decisions, Opportunities, and Challenges
Geraldine Gray & Yoav Bergner
Abstract
What is our data measuring, why are we measuring it, and what can we infer from our measurements? These are key questions for models of learning, and the focus of this chapter. This chapter discusses the role of measurement in transitioning from predictive models of learning to models from which meaningful explanations about learning can be inferred. We consider how to associate latent constructs of learning with observable data from a variety of data sources relevant to learning contexts, illustrated with examples from recent LAK proceedings. We also review common sources of errors that arise with a variety of data collection instruments, and highlight the challenges and opportunities for progressing valid and reliable measurement of both learning itself and factors related to the learning process.
Aguilar, Stephen J., and Clare Baek. 2019. “Motivated Information Seeking and Graph Comprehension Among College Students.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 280–89. Tempe, Arizona: ACM.
Allen, Laura K., Caitlin Mills, Cecile Perret, and Danielle S. McNamara. 2019. “Are You Talking to Me? Multi-Dimensional Language Analysis of Explanations During Reading.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 116–20.
Azcona, David, Piyush Arora, I.-Han Hsiao, and Alan Smeaton. 2019. “User2code2vec: Embeddings for Profiling Students Based on Distributional Representations of Source Code.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 86–95.
Bergner, Yoav. 2017. “Measurement and Its Uses in Learning Analytics.” In Handbook of Learning Analytics, edited by Charles Lang, George Siemens, Alyssa Wise, and Dragan Gasevic, First, 35–48. Society for Learning Analytics Research (SoLAR). https://solaresearch.org/hla-17/hla17-chapter3.
Bergner, Yoav, Geraldine Gray, and Charles Lang. 2018. “What Does Methodology Mean for Learning Analytics?” Journal of Learning Analytics 5 (2): 1–8–1–8. https://doi.org/10.18608/jla.2018.52.1.
Bergner, Yoav, Charles Lang, and Geraldine Gray. 2017. “A Focus on Methodology in Learning Analytics: Building a Structurally Sound Bridge Discipline.” In Workshop on MLA/BLAC at the 7th International Conference on Learning Analytics & Knowledge.
Choi, Heeryung, Nia Dowell, Christopher Brooks, and Stephanie Teasley. 2019. “Social Comparison in MOOCs: Perceived SES, Opinion, and Message Formality.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 160–69.
Cukurova, Mutlu, Qi Zhou, Daniel Spikol, and Lorenzo Landolfi. 2020. “Modelling Collaborative Problem-Solving Competence with Transparent Learning Analytics: Is Video Data Enough?” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 270–75. Frankfurt Germany: ACM. https://doi.org/10.1145/3375462.3375484.
D’Mello, Sidney, Ed Dieterle, and Angela Duckworth. 2017. “Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning.” Educational Psychologist 52 (2): 104–23.
Dowell, Nia MM, Yiwen Lin, Andrew Godfrey, and Christopher Brooks. 2020. “Exploring the Relationship Between Emergent Sociocognitive Roles, Collaborative Problem-Solving Skills, and Outcomes: A Group Communication Analysis.” Journal of Learning Analytics 7 (1): 38–57.
Eagan, Brendan, Jais Brohinsky, Jingyi Wang, and David Williamson Shaffer. 2020. “Testing the Reliability of Inter-Rater Reliability.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 454–61.
Farrow, Elaine, Johanna Moore, and Dragan Gašević. 2019. “Analysing Discussion Forum Data: A Replication Study Avoiding Data Contamination.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 170–79.
Fiedler, Klaus, and Susanne Beier. 2014. “Affect and Cognitive Processes in Educational Contexts.” In International Handbook of Emotions in Education, 36–55. Educational Psychology Handbook Series. New York, NY, US: Routledge/Taylor & Francis Group.
Gasevic, Dragan, Shane Dawson, Tim Rogers, and Danijela Gasevic. 2016. “Learning Analytics Should Not Promote One Size Fits All: The Effects of Instructional Conditions in Predicting Academic Success.” The Internet and Higher Education 28 (January): 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002.
Geller, Shay A., Nicholas Hoernle, Kobi Gal, Avi Segal, Amy X. Zhang, David Karger, Marc T. Facciotti, and Michele Igo. 2020. “Confused and Beyond: Detecting Confusion in Course Forums Using Students’ Hashtags.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 589–94.
Harpstead, Erik, J. Elizabeth Richey, Huy Nguyen, and Bruce M. McLaren. 2019. “Exploring the Subtleties of Agency and Indirect Control in Digital Learning Games.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 121–29.
Hu, Xiao, Fanjie Li, and Runzhi Kong. 2019. “Can Background Music Facilitate Learning? Preliminary Results on Reading Comprehension.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 101–5.
Joksimovic, Srecko, George Siemens, Yuan Elle Wang, M. O. Z. San Pedro, and Jason Way. 2020. “Editorial: Beyond Cognitive Ability.” Journal of Learning Analytics 7 (1): 1–4–1–4. https://doi.org/10.18608/jla.2020.71.1.
Jung, Yeonji, and Alyssa Friend Wise. 2020. “How and How Well Do Students Reflect? Multi-Dimensional Automated Reflection Assessment in Health Professions Education.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 595–604.
Kitto, Kirsty, John Whitmer, Aaron E Silvers, and Michael Webb. 2020. “Creating Data for Learning Analytics Ecosystems.” https://www.solaresearch.org/core/creating-data-for-learning-analytics-ecosystems/.
Klebanov, Beata Beigman, Anastassia Loukina, John Lockwood, Van Rynald T. Liceralde, John Sabatini, Nitin Madnani, Binod Gyawali, Zuowei Wang, and Jennifer Lentini. 2020. “Detecting Learning in Noisy Data: The Case of Oral Reading Fluency.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 490–95. Frankfurt Germany: ACM. https://doi.org/10.1145/3375462.3375490.
Klebanov, Beata Beigman, Anastassia Loukina, Nitin Madnani, John Sabatini, and Jennifer Lentini. 2019. “Would You?: Could You? On a Tablet? Analytics of Children’s eBook Reading.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 106–10. Tempe AZ USA: ACM. https://doi.org/10.1145/3303772.3303833.
Knight, Simon, Simon Buckingham Shum, and Karen Littleton. 2013. “Epistemology, Pedagogy, Assessment and Learning Analytics.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 75–84. LAK ’13. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2460296.2460312.
Kohn, Alfie. 1999. Punished by Rewards:: The Trouble with Gold Stars, Incentive Plans, a’s, Praise, and Other Bribes. Houghton Mifflin Harcourt.
Kovanović, Vitomir, Srećko Joksimović, Negin Mirriahi, Ellen Blaine, Dragan Gašević, George Siemens, and Shane Dawson. 2018. “Understand Students’ Self-Reflections Through Learning Analytics.” In Proceedings of the Eight International Conference on Learning Analytics & Knowledge (LAK’18). LAK ’18. Sydney, NSW, Australia: ACM. https://doi.org/10.1145/3170358.3170374.
Kyllonen, Patrick C. 2012. “Measurement of 21st Century Skills Within the Common Core State Standards.” In Invitational Research Symposium on Technology Enhanced Assessments, 7–8.
Larmuseau, Charlotte, Pieter Vanneste, Piet Desmet, and Fien Depaepe. 2019. “Multichannel Data for Understanding Cognitive Affordances During Complex Problem Solving.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 61–70.
Lin, Yiwen, Nia Dowell, Andrew Godfrey, Heeryung Choi, and Christopher Brooks. 2019. “Modeling Gender Dynamics in Intra- and Interpersonal Interactions During Online Collaborative Learning.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 431–35.
MacCallum, Robert C., Shaobo Zhang, Kristopher J. Preacher, and Derek D. Rucker. 2002. “On the Practice of Dichotomization of Quantitative Variables.” Psychological Methods 7 (1): 19.
Matcha, Wannisa, Dragan Gašević, Nora’Ayu Ahmad Uzir, Jelena Jovanović, and Abelardo Pardo. 2019. “Analytics of Learning Strategies: Associations with Academic Performance and Feedback.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 461–70.
Milligan, Sandra. 2020. “Standards for Developing Assessments of Learning Using Process Data.” In Re-Imagining University Assessment in a Digital World, 179–92. Springer.
Motz, Benjamin A., Paulo F. Carvalho, Joshua R. de Leeuw, and Robert L. Goldstone. 2018. “Embedding Experiments: Staking Causal Inference in Authentic Educational Contexts.” Journal of Learning Analytics 5 (2): 47–59–47–59. https://doi.org/10.18608/jla.2018.52.4.
Motz, Benjamin, Joshua Quick, Noah Schroeder, Jordon Zook, and Matthew Gunkel. 2019. “The Validity and Utility of Activity Logs as a Measure of Student Engagement.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 300–309. LAK19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3303772.3303789.
Peffer, Melanie E., and Niloofar Ramezani. 2019. “Assessing Epistemological Beliefs of Experts and Novices via Practices in Authentic Science Inquiry.” International Journal of STEM Education 6 (1): 3. https://doi.org/10.1186/s40594-018-0157-9.
Peffer, Melanie, David Quigley, Liza Brusman, Jennifer Avena, and Jennifer Knight. 2020. “Trace Data from Student Solutions to Genetics Problems Reveals Variance in the Processes Related to Different Course Outcomes.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 47–52.
Poquet, Oleksandra, Liubov Tupikina, and Marc Santolini. 2020. “Are Forum Networks Social Networks? A Methodological Perspective.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 366–75.
Porter, Tenelle, Diego Catalán Molina, Lisa Blackwell, Sylvia Roberts, Abigail Quirk, Angela Lee Duckworth, and Kali Trzesniewski. 2020. “Measuring Mastery Behaviors at Scale: The Persistence, Effort, Resilience and Challenge-Seeking Task (PERC).” Journal of Learning Analytics 7 (1): 5–18.
Qiao, Chen, and Xiao Hu. 2019. “Measuring Knowledge Gaps in Student Responses by Mining Networked Representations of Texts.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 275–79.
Quick, Joshua, Benjamin Motz, Jamie Israel, and Jason Kaetzel. 2020. “What College Students Say, and What They Do: Aligning Self-Regulated Learning Theory with Behavioral Logs.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 534–43. LAK ’20. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3375462.3375516.
Reilly, Joseph M., and Chris Dede. 2019. “Differences in Student Trajectories via Filtered Time Series Analysis in an Immersive Virtual World.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 130–34.
Sharma, Kshitij, Zacharoula Papamitsiou, Jennifer Olsen, and Michail Giannakos. 2020. Predicting Learners’ Effortful Behaviour in Adaptive Assessment Using Multimodal Data.
Srivastava, Namrata, Sadia Nawaz, Jason M. Lodge, Eduardo Velloso, Sarah Erfani, and James Bailey. 2020. “Exploring the Usage of Thermal Imaging for Understanding Video Lecture Designs and Students’ Experiences.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 250–59.
Stone, Cathlyn, Abigail Quirk, Margo Gardener, Stephen Hutt, Angela L. Duckworth, and Sidney K. D’Mello. 2019. “Language as Thought: Using Natural Language Processing to Model Noncognitive Traits That Predict College Success.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 320–29.
Tempelaar, Dirk, Bart Rienties, and Quan Nguyen. 2020. “Subjective Data, Objective Data and the Role of Bias in Predictive Modelling: Lessons from a Dispositional Learning Analytics Application.” Edited by Vitomir Kovanovic. PLOS ONE 15 (6): e0233977. https://doi.org/10.1371/journal.pone.0233977.
Thaker, Khushboo, Paulo Carvalho, and Kenneth Koedinger. 2019. “Comprehension Factor Analysis: Modeling Student’s Reading Behaviour: Accounting for Reading Practice in Predicting Students’ Learning in MOOCs.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 111–15.
Vermunt, Jan D., Sonia Ilie, and Anna Vignoles. 2018. “Building the Foundations for Measuring Learning Gain in Higher Education: A Conceptual Framework and Measurement Instrument.” Higher Education Pedagogies 3 (1): 266–301. https://doi.org/10.1080/23752696.2018.1484672.
Vrzakova, Hana, Mary Jean Amon, Angela Stewart, Nicholas D. Duran, and Sidney K. D’Mello. 2020. “Focused or Stuck Together: Multimodal Patterns Reveal Triads’ Performance in Collaborative Problem Solving.” In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 295–304. LAK ’20. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3375462.3375467.
Williams, Betsy Anne, Catherine F. Brooks, and Yotam Shmargad. 2018. “How Algorithms Discriminate Based on Data They Lack: Challenges, Solutions, and Policy Implications.” Journal of Information Policy 8: 78. https://doi.org/10.5325/jinfopoli.8.2018.0078.
Wilson, Mark, Isaac Bejar, Kathleen Scalise, Jonathan Templin, Dylan Wiliam, and David Torres Irribarra. 2012. “Perspectives on Methodological Issues.” In Assessment and Teaching of 21st Century Skills, 67–141. Springer.
No Supplementary Material Available
Title
A Practitioner's Guide to Measurement in Learning Analytics: Decisions, Opportunities, and Challenges
Book Title
Handbook of Learning Analytics
Pages
pp. 20-28
Copyright
2022
DOI
10.18608/hla22.002
ISBN
978-0-9952408-3-4
Publisher
Society for Learning Analytics Research
Authors
Geraldine Gray
Yoav Bergner
Editors
Charles Lang
Alyssa Friend Wise
Agathe Merceron
Dragan Gašević
George Siemens