A research study at the University of North Texas spanning three semesters evaluated the effectiveness of Packback on online discussions. By analyzing 14,500 posts from over 800 students, the researchers studied how Packback’s AI-assigned “Curiosity Scores” compared with a subject-matter expert’s assessment of a post. They also examined how instructor feedback via private coaching, public praising, and public featuring of posts influenced a student’s effort in subsequent assignments.
The study presented three key takeaways:
- Packback’s Curiosity scores encouraged students to go more in-depth in their discussion questions and answers
- Instructor public praising and featuring of posts with high Curiosity scores proves that Packback’s AI identifies posts that subject-matter experts agree are of the highest quality
- When students received private coaching, their Curiosity scores significantly increased in following discussion posts
Curiosity Score
Every post on Packback is assigned a unique Curiosity score through a scoring algorithm which measures the presentation, credibility, and effort of the post. The researchers at UNT studied how the score affected the amount of effort a student gave on an assignment.
The study used word count as a gauge for student effort, theorizing that a submission with more words indicates a more complete, in-depth post. They concluded that this was the case, finding that every additional two words corresponded to a one-point Curiosity score increase. This suggests that students are encouraged to write more detailed posts to receive a higher Curiosity score.
Public Praising and Featuring
The research team also looked into how the AI-generated Curiosity scores compared to official grades given by subject-matter experts. By looking at the types of posts that instructors praised and featured, they concluded that “there was a high overlap between posts the AI flagged as high quality and posts the instructors elected to publicly praise.”
The posts that instructors choose to praise & feature have high Curiosity scores, which validates AI-assigned scores. The post pictured below is from one of the classes represented in the study. This post was featured by an instructor and exemplifies how high-quality posts chosen by an instructor also hold high Curiosity scores.
Another benefit of featured posts is giving other students a “blueprint” to follow for their own future posts. By showcasing a high-quality post, everyone in the course can see what the instructor expects for a successful submission. The previous week’s featured posts on Packback are automatically sent to students via the “Curious Reader Digest” email every Friday.
Private Coaching
The study also focused on how private instructor coaching impacted students’ efforts in subsequent posts. By looking at the four posts following instructor interaction, the researchers found a significant increase in quality.
“Our results suggest that targeted, one-on-one digital feedback with students can drive students who may be struggling in their online discussion to achieve at least an adequate (passing) level of effort in their posts, with this effect persisting over the student’s next several submissions.”
The data revealed an average increase of 15 points in Curiosity score and almost 40 words in a student’s four posts after receiving instructor coaching.
By using Packback, instructors are given the power to offer this highly impactful, individualized feedback at scale, saving hours of time in the process.
