Ungrading and the Logic of Evaluation
Many of us who teach evaluation are familiar with the chocolate chip cookie exercise by Drs. Preskill and Russ-Eft in their book Building Evaluation Capacity. It’s a great introductory activity to help students understand and work through the logic of evaluation. Dr. Montrosse-Moorhead also wrote up a blog post detailing an adaptation of the activity suitable for online environments during the pandemic.
The basic premise of the activity is you pick something to be evaluated (e.g., chocolate chip cookies) and then go through the basic four steps of the logic of evaluation:
Develop the criteria for evaluating the objects
Set standards for performance
Rate the objects based on those criteria and standards
Synthesize an overall evaluative judgment (e.g., which is best)
When I began shifting some of my courses to ungrading (e.g., my evaluation courses and my interpersonal effectiveness course), one issue I had is that students had a difficult time determining what final grade they should get and documenting evidence for that final grade. I had some students who graded themselves too harshly on factors I didn’t deem too relevant (e.g., lateness wasn’t a big deal to me but many students penalized themselves even if they turned in things a few days late) and I had some students who graded themselves too leniently (e.g., did not turn in any reading reflections or journals, two of the major sets of assignments in the discussion based course and felt they had earned an A).
This difficulty in students grading themselves seemed to stem from not understanding how they have been graded in the past and therefore not knowing how to grade themselves, particularly when the criteria may be more amorphous. Students have previously only really been involved in step 3 of the logic of evaluation process: receiving their individual grades on assignments. They are rarely involved in developing criteria or setting standards, and the overall grade judgment can often feel mysterious or even unfair to students since they are also rarely involved in that process, either.
To combat this difficulty, I decided to apply the chocolate chip cookie evaluation activity to their mid-semester grade reflection letters. My university requires students know where they’re at grade-wise at Week 6 in the semester, presumably so they know whether they should consider dropping the course. Regardless, in an ungraded course this means that students need to practice determining what grade they are currently at and supporting that grade with evidence. This formative exercise helps them practice what they will do summatively at the end of the semester and allow them to not only document and defend their current grade but also reflect on what they will do differently for the rest of the semester to improve their performance.
This activity can be done in two ways, which I’ll document below.
Class-wide activity
In my interpersonal effectiveness course, I have mostly undergraduate students (n = 25) plus some graduate students in our program who had gone through the chocolate chip cookie exercise in our evaluation courses (n = 6). Thus, most of the students were unfamiliar with the activity but I had some students with familiarity who could support the other students. I clumped students into small groups to brainstorm ideas to bring back to the full class.
First, I had students in their small groups brainstorm criteria. I explained to them what criteria are and gave one example criteria (attendance). I had them pull up the syllabus again so they knew what activities, learning objectives, etc. we were doing throughout the semester. We then came together as a class, each group reporting their criteria out and clumping them together if folks came up with the same criteria. As a class, they decided on the following criteria: attendance, participation and engagement, journals, reading reflections, group project, individual project, and growth throughout the semester.
Second, I explained to students what standards are and some examples of what standards can look like (e.g., pass/fail, rubric). We worked together on the attendance criteria for developing standards as a class so they could see what that looked like. Then I assigned the remaining criteria to the groups so they had 1-2 criteria to set standards for performance. They then reported out to the class and we deliberated and discussed as a class until there was group consensus.
The third and fourth steps were then done individually in a mid-semester grade reflection letter. For the third step, I presented them the grading rubric (each criteria with their relevant standards) and had students rate their performance and document evidence and support for their ratings for each criteria. The fourth step involved them determining what final grade they would give themselves at this moment in the course and provide evidence and support for their overall judgment.
I also met individually with each student in short (usually 2-5 minutes) meetings to discuss their letter. This was an opportunity for us to chat 1-on-1, for students to practice those interpersonal skills we’d been learning throughout the semester, and for me to adjust their grading reflections as necessary. Thankfully, this process only involved increasing a few students’ grades because they were too harsh on themselves; I no longer had any students who were too lenient on grading themselves.
Individual activity
Although I have not yet done this, the other option would be to have students individually determine their own criteria and standards for performance, rate themselves, and then synthesize their overall evaluative judgment. This allows more flexibility for students and provides them more autonomy in the grading process.
I plan on doing this with my graduate students in our evaluation concentration moving forward because they will already have had practice doing the class-wide activity as part of the evaluation program. This will be an extension of their learning.
Regardless of the course I would implement this in, I would have students first submit their criteria and standards, perhaps doing a round of peer review or presentation to classmates to get feedback and more ideas of how they want to shape their criteria and standards and get my approval before finalizing them. Then I would have students do the third and fourth step the same as in the class-wide activity; the only difference would be that their criteria and standards would vary from student to student and would need to be presented in their letter clearly.
Conclusion
Overall, I am very pleased with how this activity turned out. Like I documented in my other blog post, I was increasingly concerned with how my teaching practices did not align with how I practiced evaluation. This process gives more power and control in the grading students to students in a way that supports them better in that process.
