Verification Becomes the Bottleneck When Generation Gets Cheap

Educators need to know which test questions are vulnerable to LLM cheating, but current benchmarks only provide aggregate scores—not item-level diagnostics that reveal where AI systematically outperforms or underperforms humans.

Method: Differential Item Functioning (DIF) analysis—borrowed from bias detection in psychometrics—flags test items where humans and chatbots show systematic response differences. Applied to a high school chemistry test and university entrance exam with six leading chatbots (ChatGPT-4o & 5.2, Gemini 1.5 & 3 Pro, Claude 3.5 & 4.5 Sonnet), the method reliably identified items where LLMs diverge from human learners, enabling subject-matter experts to characterize task dimensions that make problems particularly easy or difficult for AI.

Caveats: Tested on two STEM assessments. Transfer to humanities or open-ended tasks unverified.

Reflections: Do DIF-flagged items remain stable across LLM versions, or does each model update require re-analysis? · Can DIF patterns predict which item types will be vulnerable to future AI capabilities? · How do DIF results change when students use LLMs as assistants rather than direct answer generators?

GenAI adoption in programming education is treated as a binary choice, but longitudinal evidence on how student-AI relationships evolve as AI becomes routine in classrooms is missing.

Method: Tracking three successive cohorts (2023-2025) in an introductory Python course through questionnaires, coded student-AI dialogue logs, and assessment records revealed systematic shifts: AI familiarity and uptake became increasingly normative, and help-seeking practices evolved alongside growing AI literacy. Students' expectations of what the assistant should provide changed cohort-to-cohort, suggesting the central challenge is not whether students use AI but how courses redefine productive learning practices.

Caveats: Single course, single institution. Cohort effects may conflate with broader cultural shifts in AI adoption.

Reflections: Do interaction pattern shifts persist into advanced courses, or do students revert to non-AI strategies when problems get harder? · Which specific help-seeking behaviors correlate with better learning outcomes versus dependency? · How do instructor norms about AI use shape student interaction patterns across cohorts?

Mid-air VR interaction causes arm fatigue, but ergonomic UI design requires extensive human testing. Biomechanical models could serve as surrogate users, but their application to VR interface optimization remains underexplored.

Method: A hierarchical reinforcement learning framework trains a motion agent to perform button-press tasks in VR, estimating muscle-level effort via a validated three-compartment control with recovery (3CC-r) fatigue model. A UI agent then optimizes element layout using simulated fatigue as feedback. The RL-optimized layout produced significantly lower perceived fatigue in human validation compared to manually-designed centered and Bayesian-optimized baselines. Fatigue trends from the biomechanical model aligned with human user data.

Caveats: Validated only on button-press tasks. Complex gestures or continuous manipulation may require model extensions.

Reflections: Do fatigue-optimized layouts remain effective across users with different arm lengths and strength profiles? · Can the framework extend to seated versus standing VR contexts where fatigue patterns differ? · How do fatigue-optimized layouts trade off against other design goals like visual hierarchy or task completion speed?