Reviewer Guidelines

REVIEW PROCESS

Papers that meet the conference requirements (in scope, proper length and format etc.) will go through a rigorous and robust review process. LAK25 will use double-blind peer review for all submissions except demos and the doctoral consortium (which each require elements that prevent blinding). In addition, full and short research paper submissions will have a rebuttal phase in which authors will be given seven days to flag any inaccuracies, omissions, or errors in the reviews in a maximum of 500 words. Rebuttals are optional, and there is no requirement to respond. Authors should keep in mind that papers are being evaluated as submitted and thus, responses should not propose new results or restructuring of the presentation. Therefore, rebuttals should focus on answering specific questions raised by reviewers (if any) and pointing out any misinterpretations or errors in the reviewers’ comments. Senior reviewers will read all reviews and authors’ rebuttal (if made) and make final recommendations for paper acceptance or rejection with justification to the program committee chairs. 

 

LAK25 Reviewer Guidelines

Preamble

The LAK conference involves a rigorous review process with two purposes:

  • Supporting fair and informed acceptance decisions based on robust assessments of submission quality
  • Providing authors with useful feedback about the research they have conducted and their communication of it to the interdisciplinary learning analytics community

To meet this dual purpose the following guidelines are provided to support the community in providing high quality reviews. The creation of these guidelines was informed by review guidelines used by related journals and conferences in the field such as the Journal of Learning Analytics (JLA) and the Educational Data Mining (EDM) conference. 

Guidelines

The review should include a total of 200-500 words of detailed feedback that gives a complete and well-justified assessment of a submission (the kind you would like to receive for your own work). Please avoid very short reviews, they are frustrating for authors and detrimental to the overall review process. 

More precisely, the review should include:

  • A brief summary of the paper itself (e.g., the question being addressed, the high level approach utilised, what was found).
  • A thorough assessment of the submission’s main strengths and weaknesses in making a substantial conceptual, technical or empirical contribution to learning analytics. 
  • Suggestions for improvement, whenever possible. 

When writing your review, please consider the following categories and questions:

    • Relevance: 
      • Is the submission trying to solve an important educational problem related to the design, development, implementation, or evaluation of learning analytics?
      • Does the submission attend to the real-world context?
    • Novelty: 
      • Is there a novel contribution in the submission in relation to previous work in the area?
      • If a replication study is reported, is it clear what is the contribution to knowledge in comparison to the original study?
  • Grounding:  
    • Is the work situated appropriately with respect to the current state of the field, including sufficient coverage of relevant literature and relevant theories of learning?
  • Methods: 
    • Are the methods used suitable, well-described, and justified with reference to the literature and from the perspective of the research objectives and questions?
    • Does the submission show accepted evidence of rigor in the tradition followed (statistical, computational, qualitative, design)?
  • Results: 
    • Do the claims made have appropriate empirical support?
    • If negative results are presented, have different explanations for the lack of findings been considered?
  • Discussion:
    • Are contributions to theory and/or practice outlined clearly? 
    • Are limitations with respect to data, analysis, and framing factors taken into account? 
    • Are potential issues of privacy, fairness, equity, and well-being considered?
  • Communication: 
    • Is the submission written clearly for understanding by an interdisciplinary audience?

In addition to the textual review, you should provide two numeric assessments of each submission that align with the textual comments you have made:

  • A 7-point scale numeric ranking (from -3 to 3) indicative of the overall evaluation of the quality of the paper that must be supported by your written appraisal
  • A 5-point scale numeric ranking (from -2 to 2) indicative of your confidence in assessing work in the area addressed by the paper

Senior reviewers will compare all reviews and numeric assessments of quality and confidence as well as author rebuttals (if made) and make final recommendations for paper acceptance or rejection with justification to the program committee chairs.

Additional Considerations

If the paper is not properly blinded (i.e. the identity of the authors is revealed in the paper) please review the paper as usual, but indicate this in the “Confidential remarks for the program committee” box on the reviewing form.

If there are issues with the English in the submission (e.g. grammatical mistakes, misspellings or unusual phrasings), this can be noted in the “Confidential remarks for the program committee” box on the reviewing form, but it should not affect the review and assessment of the submissions with respect to its scientific merit.

Society for Learning Analytics Research (SoLAR)