How do we assess the 4Ds?

Assessing AI Fluency: Beyond Surface-Level Interaction

Hello, I'm Joe Feller. Let's look at one of the trickiest questions in teaching AI fluency: How do we actually assess it? How can we tell the difference between a student who's just figured out some clever prompts and one who's developing real collaborative skills?

We need to understand what AI fluency looks like in practice and distinguish between surface-level AI interaction and deep collaborative competency. We have found three general approaches that work well together.

Three General Approaches to Assessment

• Outcome-based assessments: These focus on what students produce through AI collaboration. We're assessing whether students achieve their stated goals through human-AI partnership.
• Process-based assessments: These examine how students work with AI over time. This captures iteration patterns, recovery from failed attempts, and methodological sophistication. We're interested in what happens during the collaboration, not just what it produces.
• Reflection-based assessments: These focus on metacognitive awareness—when students think about their own thinking process. We're interested in students' conclusions about why certain strategies worked or didn't, what they learned, and how they might apply that learning to future interactions.

We believe the most effective AI fluency assessments integrate all three approaches. They assess not just what students can produce with AI, but how they think and grow as AI collaborators.

Applying the Assessment Framework to the 4 Ds

Let's take a look at each of the "4 Ds" from these perspectives.

1. Delegation

Delegation is about setting goals and deciding whether, when, and how to engage with AI.

• Outcome-based: Did the delegation plan make sense? Were the goals realistic? Did they pick the right tool for the job?
• Process-based: This could involve reviewing annotated chat logs where students work through the delegation process with an AI assistant, showing how they explored options and made decisions.
• Reflection-based: Ask students to explain their choices. What else did they consider, and how did the delegation decisions shape everything that followed?

2. Description

Description is a student's ability to effectively communicate to prompt useful AI behaviors and outputs.

• Outcome-based: Examine the quality of prompts created and the AI response. Are the instructions clear? Do they provide good context? We might also assess the evolution of prompts from initial to final versions.
• Process-based: Involve rubric-supported analysis of conversation logs, showing iterative refinement, documentation of failed approaches, and pivots that hopefully lead to breakthroughs, as well as evidence of building shared context over time.
• Reflection-based: Explore students' analysis of which description techniques work best and why. We look for their recognition of how different tasks require different communication approaches and insights about the relationship between description quality and output quality. We might explicitly ask: "Which communication strategies work best and why?" or "How did you adapt your approach for different tasks?"

3. Discernment

Discernment focuses on the students' ability to accurately assess the usefulness of AI outputs and behaviors.

• Outcome-based: Students can annotate AI outputs to mark what's strong and what needs improvement. You could also have students keep decision logs, explaining which parts of the AI output were kept, modified, or discarded.
• Process-based: Look for in-line evaluation comments in chat logs, evidence of catching and correcting AI errors, and pattern recognition across multiple interactions. This allows you to see students develop their critical eye in real time.
• Reflection-based: Focus on students' analysis of their evaluation criteria and how it evolved, discussion of the issues that they initially missed, lessons learned, and a comparison of their discernment strategies across different task types.

4. Diligence

Diligence involves taking responsibility for what we do with AI and how we do it.

• Outcome-based: Evaluate the quality and completeness of diligence statements, appropriate attribution, transparency documentation, and evidence of fact-checking and verification processes. Ask: "How do they attribute the AI's role in their work?" or "What verification processes do they use, and were they effective at catching errors?"
• Process-based: Examine data handling practices visible in chat logs, ethical decision-making before, during, and after collaboration, intentional handling of sensitive information, and the documentation of permission-seeking or constraint checking.
• Reflection-based: Explore students' discussion of any ethical dilemmas they encountered. How do they understand their responsibilities within the bounds of their AI collaboration? What ethical challenges surprised them, and how do they plan on using these warnings to improve responsible AI use in future work?

Practical Example: The AI-Supported Essay

Let's bring together all of these concepts and look at how we might assess an assignment in practice. Imagine we ask students to write an essay on the impacts of GenAI on society with the support of an AI assistant. They are told to submit the final essay, a link to their interactions with the AI, a diligence statement, and a learning journal discussing their experience.

We assess the work from all three angles:

• Outcome-based perspective: We look at the essay itself and the diligence statement. Does the essay address the brief? Are there clear signs of strong human oversight in the final version? Is the process of AI collaboration transparently presented?
• Process-based perspective: We look at the AI interaction itself and what happened behind the scenes. Did the student effectively build shared context with the AI? How did the student refine their prompting over time? Is there evidence of pivoting and process correction in response to errors, misunderstandings, or lack of clarity?
• Reflection-based perspective: We look at the learning journal. Can the student explain what worked well and what didn't? Can they state lessons learned to improve future AI collaboration?

Core Principles for AI Fluency Assessment

Assessing AI fluency requires us to look beyond only final products to understand how students demonstrate collaborative competencies with AI. Here are some principles that work well:

1. Combine approaches: Use outcome, process, and reflection approaches to get a complete picture.
2. Focus on observables: Look for observable actions in concrete artifacts rather than making assumptions about understanding.
3. Tailor the approach: Remember that different competencies may benefit from different assessment methods.
4. Prioritize learning: Most importantly, make assessment a learning opportunity, not just a measurement. Help students recognize and find value in their own growth in AI collaboration.

Now that we've discussed how we might assess the 4 Ds, we'll next explore designing assignments based on the AI fluency framework.