A professor at the University of Georgia performed an experiment comparing the effectiveness of Packback, Blackboard and written reaction papers.
When Dr. Matt Goren was presented with an opportunity to teach Introduction to Personal Finance at the University of Georgia, he wanted to engage his students and equip them to make smart financial decisions during and after college.
According to the Financial Health of Young America report published by Young Invincibles, young adults today are in much worse financial health than 25 years ago, earning significantly lower incomes and having a dramatically lower net worth. Knowing this, Dr. Goren emphasized a personalized learning environment where each student worked on a semester-long finance project. By the end of the semester, every student had an actionable plan to improve and sustain their financial health.
Each week students were assigned an activity to relate a personal finance theory to their lives and create actionable steps to apply that theory. For example, Dr. Goren would start class by asking questions about a topic such as investing or budgeting to spark student discussions in class. After class, students would reflect on how they could apply the material to improve their own financial health and write a response. At the end of the semester, students compiled their weekly responses into a personal finance plan, which was the final exam.
Challenges of Increasing Course Size
When Dr. Goren started teaching at UGA, around 200 students were signed up for the course per semester. The first cohort of students saw the benefit of having a personal finance plan and word spread quickly. During the next year, 2,200 students signed up to take the course. Dr. Goren was excited about the growth but felt students were no longer getting the same valuable results from the finance project that he had seen with his first class.
“When we went to 2,200 students a year, we had two TAs,” says Dr. Goren. “Reaching all those people was a challenge. Trying to give somewhat personal feedback was a challenge. Assessing the work that is being done and show[ing] that people are taking it seriously, that is really hard to do. So we had to figure out a way to use technology to integrate all these students.”
Dr. Goren gave his course a critical review and determined the act of asking questions and having a healthy debate encouraged students to apply theories and resulted in a better personal finance plan. When the classes were smaller, Dr. Goren was able to weigh in on complicated subjects and give personal advice to each student. But as the class grew, there wasn’t enough time to give personalized feedback. At an enrollment of 2,200 students, just checking the weekly student submissions took 180 hours over the course of the semester. With only two TAs, Dr. Goren decided to test if technology could be used to facilitate these weekly discussions and scale the semester-long finance project.
A social psychologist by training, Dr. Goren set up an experiment to find the best method to increase engagement, decrease time spent grading and allow him to continue the personal finance project. Dr. Goren set up three different experiences for his three sections of Introduction to Personal Finance, each with 300 students. Every week, Dr. Goren posed a question to each section of the class that would later be adapted for students’ final finance project. In one section, questions were asked in class and students submitted written responses to Dr. Goren and his TAs. In another section, Dr. Goren posted the questions on the Blackboard discussion board and students would respond on the thread. In his third section, Dr. Goren used Packback where, like Blackboard, he posted questions online and the students were required to respond.
At the end of the semester, Dr. Goren surveyed his students and looked at how each platform affected student behavior. The results surprised him. Students liked using Packback and interactions on Packback were of a significantly higher quality and continued to improve throughout the semester. “If you looked at the discussion forum on Blackboard, as the semester went on, the responses got shorter, the quality of their responses went down and those were the responses that were being pushed up to the top based on how Blackboard posts responses”, explains Dr. Goren. “Meanwhile on Packback, the responses are getting longer, the quality of the responses are going up and the better responses were floating to the top. Students were seeing better responses when they logged in so they are learning from each other and the grades on the personal finance plan at the end of the class were better for the Packback students.”
As for the students who submitted written reaction papers, Dr. Goren says those students fell in the middle on almost every measure. The quality of responses, Dr. Goren explains, stayed the same throughout the term. He believes the consistent behavior was partly because the reaction papers were graded by hand by two TAs who didn’t have the time to give valuable feedback to 300 students each week. Each response took one minute per student to grade. With 300 students submitting a response each week for 12 weeks, the TAs were spending nearly 60 hours on grading, without giving personalized feedback. On Packback, posts are monitored by artificial intelligence which filters posts that do not follow Packback’s Community Guidelines. Professors can also download reports on how many times each student posts to make sure everyone is meeting the weekly requirements.
“With Packback, the amount of TA time that is required is extremely minimal,” says Dr. Goren. “It’s like a grand total of five hours of grading work over the entire semester versus, grading discussion posts or grading one of those reaction papers. It doesn’t even compare.”
Another reason Dr. Goren believes students who submitted written responses didn’t see improvement was because those students had no sense of social accountability. With online discussion boards, students felt social pressure to complete their weekly requirements. Plus, when they saw posts from other students that were of a higher quality, they were more likely to put effort into their questions and responses.
The Effects of Packback
One of the ways Packback pushed students to improve while the other courses saw no or poor results was Packback’s gamified components such as the Curiosity Points and Learner Leaderboard. “What I saw in the case of Packback, across all the semesters, is some students really cared about that Curiosity Score. Students saw their position on the leaderboard and then posted more and better responses to try and go up in the ranking,” says Dr. Goren. “What I noticed every single semester is the top 10% of any given section didn’t just do the required minimum. Some of those students post two or three times as often as they had to because they cared so much about their [Curiosity] Score. And because it’s a public discussion forum, that then improved the quality of the entire board. They’re giving better answers to more people which is then increasing the social pressure to give better answers and the whole community is better off.”
And students weren’t just improving online. The Packback students developed consistent behaviors of asking thought-provoking questions that carried into the classroom.
“The students that use Packback are more vocal in the classroom because they’ve thought about this material. They’ve seen other students think about the material. They’ve had discussions about it. By the time we get to class, they’ve already kind of tested their ideas somewhat anonymously in the online environment, now they’re more comfortable expressing those ideas in person, in the classroom,” says Dr. Goren.
At the end of the term, it was clear to Dr. Goren that Packback was the best tool for his class. It not only allowed him to keep the basic structure of his course, but it also led to greater success on the final personal finance project. Packback gave students a place to discuss financial theories. It pushed them to ask better questions and allowed them to receive valuable feedback from their peers, Dr. Goren and the TAs, while saving 175 hours on the administrative end.
Want to see a live Packback community and learn how you can increase student engagement and critical thinking in your course?