Student Learning & AI: How to assess students’ understanding in an AI world

By Kate Cammack, CfT Co-Director

Unauthorized GenAI use is impacting our ability to accurately gauge our students’ learning

Recently, a Brown economics professor shared his experience shifting from take-home to in-person exams. Briefly, Prof. Roberto Serrano gave his 86 advanced microeconomic theory students a take-home midterm exam and was surprised to find a class average of 96%, when it had typically been 65-80%. Suspicious of unauthorized AI use, he gave the final exam in-person. While there were certainly many factors at play (see full article), there was a dramatic drop in exam performance:

(image source)

(While Student 22 has been the source of many educators’ knowing shrugs/eyerolls, it’s important to point out that their performance did improve over the course and their scores didn’t follow the pattern of purported AI use. Yes to you, S22!)

Like Prof. Serrano, many of us have been called to rethink how we’re assessing student learning.

What counts as “credible evidence” that students are learning what we want them to learn?
Are our existing assessments actually capturing the most meaningful aspects of students’ learning?
How is AI changing our ability to accurately gauge students’ actual understanding?

What GenAI is already capable of, even for novice learners

Regardless of your current AI stance, we need to acknowledge the GenAI landscape that we’re currently in. To limit the Debbie Downer-ness of this, you might choose one of these brief videos, made by academics at the University of Sydney.

Examples of how students can use/misuse GenAI for assessment:

Examples of how GenAI can produce reflective writing, including discussion board posts:

Examples of how GenAI can produce video, audio, and even students’ likenesses:

The same institution (University of Sydney) has developed what they’re calling a “two-lane approach” to rethinking assessments — this allows faculty to either lean in or lean out in a given type of assessment. As they summarized:

“If we do not change our assessments to align them with the capabilities of generative AI, our degrees will be easier to pass and we will have failed to prepare our students for their future.” (source)

So, what do we do?

Last year, I shared some of my struggles with grades. I’ve moved away from in-class exams for a variety of reasons, ranging from bad test-taker identity to compelling research on more equitable assessment strategies, and frankly I don’t want to return to them. I’m currently exploring multi-part portfolios that are mostly completed in class (also here, here), and I’m reading about new, alternative ways of evaluating student learning. I haven’t landed on any great solutions yet, but it’s been a valuable way for me to think deeply and critically about what aspects of student learning are actually most important to me.

I’d love to share our ideas, strategies, and approaches with each other. Please join us for one (or more!) of our (Re)Designing Assignments & Assessments for Real Thinking mini-workshops, where you’ll have protected time to:

Articulate the key goals (learning objectives) of one of your assignments/assessments!
Evaluate whether student work toward those goals (learning outcomes) is still being accurately represented/captured by that assignment
Revise/adjust that assignment/assessment to re-align with your objectives/outcomes, based on your current GenAI stance

Please join us! We’ll add more if needed:

Wed, Aug 12, 1-2pm (add to calendar)
Thurs, Aug 13, 9-10am (add to calendar)
Fri, Aug 21, 10-11am (add to calendar) and 11am-12pm (add to calendar)

For anyone looking for longer-term accountability and solidarity, we’ll also be facilitating two communities of practice (CoP) that will meet regularly throughout fall semester; each session will have time to:

have focused discussions on how/where/when GenAI is shifting our thinking about student learning (15-20min), and
update a specific assignment so that it is (re)aligned with your learning objectives and your current GenAI stance (30-40min)

Two options for this fall:

General CoP on (Re)designing assignments for real thinking: sign up HERE.
STEM/social sciences CoP on helping students read/critique scientific research papers: sign up HERE

Finally, if you’re looking for a thought-partner, you can always book a one-on-one consultation with us – we’d love to ideate and troubleshoot together.

In solidarity

Unauthorized GenAI use is impacting our ability to accurately gauge our students’ learning

What GenAI is already capable of, even for novice learners

So, what do we do?

Related readings: