Evaluation Protocols Fail Where Models Succeed

A Nature Medicine study claimed ChatGPT under-triages 51.6% of emergencies, suggesting consumer health AI poses safety risks. The evaluation used exam-style forced-choice formats that don't reflect how people actually use chatbots.

Method: Testing five frontier LLMs under constrained (exam-style, forced A/B/C/D output) versus naturalistic (patient-style messages) conditions revealed the evaluation format was the failure mechanism. Naturalistic interaction improved triage accuracy by 6.4 percentage points. Three models scored 0-24% with forced choice but 100% with free text (all p < 10^-8), consistently recommending emergency care in their own words while the forced-choice format registered under-triage. Diabetic ketoacidosis was correctly triaged in 100% of trials across all models and conditions.

Caveats: Tested on 17 scenarios from the original study. Full replication across all original cases not performed.

Reflections: What other high-stakes AI evaluations are contaminated by format artifacts that don't reflect deployment conditions? · How do clarifying questions versus forced-choice responses affect triage accuracy across different medical conditions? · What evaluation protocols would capture both safety risks and realistic usage patterns for consumer health AI?

Crowdsourced moderation like X's Community Notes could scale fact-checking, but no one has tested whether raters systematically avoid claims that require cognitive effort to evaluate.

Method: Analyzing 2,250 vaccine-related posts over eighteen months, researchers found claims perceived as more difficult to fact-check were significantly less likely to receive public/helpful notes. Survey data from 382 participants confirmed the pattern: obviously false, easy-to-refute claims got notes, while plausible misinformation requiring greater effort to debunk was systematically ignored. An LLM-assisted fact-checking pipeline ruled out alternative explanations, pointing to effort aversion as the mechanism.

Caveats: Focused on vaccine-related content. Effort aversion patterns may differ across topics or platforms.

Reflections: What incentive structures could overcome effort aversion without introducing gaming or quality degradation? · Does expertise level moderate the difficulty penalty—do domain experts avoid hard claims less than generalists? · How does the difficulty penalty interact with partisan bias in crowdsourced fact-checking?

Engineering analysis automation breaks when interfaces change—different data formats, units, naming conventions. Rigid scripts can't adapt as products evolve, forcing engineers back to manual work.

Method: DUCTILE separates adaptive orchestration (LLM agent interprets documented practices, inspects input data, adapts processing paths) from deterministic execution (verified engineering tools). Deployed on an industrial structural analysis task at an aerospace manufacturer, the agent handled input deviations in format, units, naming conventions, and methodology that would break traditional scripted pipelines. Evaluation against expert-defined acceptance criteria confirmed correct, methodologically compliant results across repeated independent runs.

Caveats: Single industrial deployment. Supervision burden and unintended effects on engineering work roles require further study.

Reflections: What happens to engineering skill development when orchestration is delegated but supervision remains? · How do you verify that LLM orchestration decisions remain methodologically sound as design practices evolve? · What's the right division of labor between adaptive agents and deterministic tools in other technical domains?