Evaluation Shifts from Artifact Quality to User Behavior

AI tutors are evaluated on pedagogical quality—how good the feedback sounds—but not on whether students actually use it. A tutor can give perfect advice that students ignore.

Method: Analyzed 10,235 code submissions to measure whether students act on AI feedback and apply it correctly. This behavioral dimension—engagement patterns—correlated more strongly with student perception of helpfulness than pedagogical quality alone. Two deployed tutors with similar pedagogical scores showed substantial differences in engagement that pedagogy-only evaluation missed entirely.

Caveats: Tested only in introductory programming. Other domains may show different engagement patterns.

Reflections: Do behavioral engagement patterns predict long-term learning outcomes, or just immediate satisfaction? · Can tutors be optimized directly for engagement metrics, or does that create perverse incentives? · How do engagement patterns differ across disciplines beyond programming?

Existing AI assistants react to user queries or handle isolated short events. They don't track progress through multi-step procedural tasks or proactively intervene at the right moment.

Method: Pro²Assist uses AR glasses to continuously track fine-grained task progress and reason over evolving user state. It outperformed baselines by over 21% in procedural action understanding accuracy and achieved up to 2.29x the proactive timing accuracy. The system extracts step-oriented context from multi-scale temporal dynamics and task-specific expert knowledge to display timely assistance.

Caveats: Evaluated on curated datasets and testbed tasks. Real-world task diversity and interruption tolerance need validation.

Reflections: How do users adapt their task strategies when they know proactive assistance is monitoring them? · What's the optimal interruption threshold—when does proactive help become intrusive? · Can the system generalize to novel procedural tasks without task-specific expert knowledge?

Computer-use agents fail on unseen interfaces because implicit design assumptions—obvious to humans—create agent-specific failures. Improving agent capability alone won't solve this.

Method: Revisited Nielsen's 10 usability heuristics for agents, identifying which transfer and which need augmentation. Tested augmented heuristics in UI-Verse, controlled environments with functionally similar interfaces. Augmented heuristics consistently improved task completion and modestly improved efficiency, with combined heuristics yielding further gains. Human studies showed no observable usability regressions.

Caveats: Tested in controlled environments. Production interfaces with legacy constraints may face implementation barriers.

Reflections: Which specific heuristic augmentations provide the highest ROI for agent reliability? · Do agent-friendly designs inadvertently train humans to rely on explicit cues, degrading their adaptability? · How do these principles extend to mobile and voice interfaces?