AcaWriter: designing automated feedback on writing that teachers and students trust
November 20, 2020
Keywords: Education, Feedback, NLP, Writing, Writing Analytics
Target readers: Educator; Learning Technologist; Instructional Designer; Researcher
Author: Antonette Shibani
Position: Lecturer & Researcher, Faculty of Transdisciplinary Innovation, University of Technology Sydney
AcaWriter: designing automated feedback on writing that teachers and students trust
The emergence of automated writing feedback
We know feedback is of paramount importance to help students learn to write and improve their writing skills. But providing good formative feedback on students’ drafts is a time-consuming and resource-intensive process for educators. We’re getting increasingly used to the idea that computers can understand our speech and writing, and this is now finding its way into education. In this blog post, I’ll share what we’re learning about the effective use of such tools in teaching, in particular, one we’ve been developing for the last 5 years, called AcaWriter. AcaWriter provides our students at the University of Technology Sydney (UTS) with instant feedback on their writing at any time of the day, as and when requested. It’s been evaluated here at UTS, grounded in peer-reviewed, published research, and released to all students and staff.
Automated feedback on writing is still pretty new, so it’s not surprising that some may be sceptical. After all, how can a machine possibly understand a complex piece of writing at university level? Would machine feedback lead to forumalic writing? There is of course an active debate (see the articles appended for details), and understanding the reactions of students and educators is exactly one of the aspects of the R&D underpinning AcaWriter. It is important to note that AcaWriter does not assess the overall quality of writing or provide grades — that’s a complex judgement, requiring human expertise — but aims to provoke student reflection to improve their writing using formative feedback about just one of the hallmarks of academic writing.
We like to ask students: “Are you being rhetorical?”
AcaWriter’s feedback complements conventional spelling and grammar checkers by providing feedback on sentence structure and construction. In particular, it looks for specific hallmarks of writing called ‘rhetorical moves’ that help convey the writer’s intent to the reader. For instance, in university-level written assignments, AcaWriter looks at cues such as “Previous work has shown” and “Recent studies indicate” as rhetorical moves presenting ‘Background’ work. Students learn that it’s important to signal their intent clearly, so that the reader can assess whether they’re simply reporting past work, actively agreeing or disagreeing with it, or presenting their own critical commentary.
AcaWriter’s feedback highlights such rhetorical moves and offers suggestions for improving how the core message can be clearly communicated in the writing. It is able to automatically identify rhetorical moves using the underlying technology of Natural Language Processing (NLP) that looks for specific sentence patterns, grammatical features and cues in written language.
AcaWriter’s automated writing feedback is available for two main types of writing and the genres defined within them:
- Analytical writing such as essays, research articles, formal reports;
- Reflective writing such as personal journals of learning experiences and informal reflective accounts.
What AcaWriter feedback looks like
AcaWriter’s main interface provides an editor (left pane) where students can paste and edit their writing. Clicking the central Get Feedback button generates feedback (right pane) within around 10-30 seconds depending on the text length.
Zooming in on this, here’s what Law students saw for an essay, as they received feedback organised into three tabs.
Tab 1 Analytical Report is where AcaWriter highlights sentences containing the important rhetorical moves that we want the students to understand, and demonstrate they can use:
Tab 2 offers more specific Feedback and suggestions for improving the writing tuned to the assessment brief, which is co-designed with the educator to be relevant to the assignment:
Tab 3 provides additional examples and links for students to learn more about rhetorical moves in the context of their course, and how they align with the assessment criteria:
AcaWriter has been tuned and tested for other types of academic writing such as research abstracts, business executive reports and reflective writing.
Reflective writing is distinctively different and known to be challenging to teach and learn. It’s written in the first person, in which the student makes sense of their experiences, talk about what they found challenging, and how they are changing. Here’s a great intro video from an academic who uses it to teach reflection in pharmacy, showing how different the feedback is:
Integrating AcaWriter into the student experience
AcaWriter is customizable, which means that the automated feedback can be re-designed to suit a teacher’s specific context. The feedback messages are tuned to assessments across different disciplines by working along with educators to provide their students with relevant feedback.
The ‘cogs’ diagram shows that AcaWriter is best introduced as part of (blue) activities teaching students different strategies to improve their writing with good learning design. AcaWriter’s textual features and user interface (orange cogs) are designed to be flexible so they can ‘synchronise’ with the teaching activities, driving everything in the same direction (learn more in this article). We work closely with educators so they are able to directly shape the way the feedback is given, building their trust in the tool.
What impact does it have?
AcaWriter has been evaluated across various authentic learning contexts in higher education with positive feedback. Students and educators have reported improvements in writing performance and an increased understanding of different components in writing. While students may be sceptical before using it, we’ve shown that this changes positively after experiencing it. All of the results and outcomes are documented in peer-reviewed publications and are publicly available (see this key article for details).
Considerations for using AcaWriter
AcaWriter supports English writing based on a specific linguistic feature (rhetorical moves). But we know that writing is complex and multi-dimensional, from simple mechanical aspects such as spelling and grammar, to more complex structural and argumentative features. Moreover, machines read text differently to humans as they do not understand the meaning of text like we do.
So we know the feedback provided by the tool can be imperfect, and we encourage students to reflect critically on its feedback, and ‘push back’ if they disagree. On the technical front, we are now testing if machine learning approaches can do a better job than the current set of rules for improved accuracy. Users of the tool should be aware of such limitations and use the tool effectively as an additional feedback mechanism, and not as the ground truth. Learners should still use other forms of writing support such as peer feedback, instructor feedback and self-assessment as required.
Can I try this out?
An open demo version of the tool is available, where you can paste in a small piece of writing and receive instant feedback based on the available genres of writing. Institutions can install their own copy of AcaWriter using the Open Source release. Students and staff from the University of Technology Sydney where AcaWriter was originally developed have full access to a hosted version of AcaWriter. You can also try the full version as part of this free interactive tutorial on writing better research abstracts.
To learn more…
AcaWriter orientation website for teachers and students: https://www.uts.edu.au/acawriter
Research publications: https://cic.uts.edu.au/tools/awa/