Evaluation Metrics Miss the Failures That Matter

AI decision support systems are evaluated on accuracy and reliance, but identical performance can mask how users actually form judgments. Do recommendation-driven interfaces change evidence standards even when outcomes stay the same?

Method: A 290-person experiment using Signal Detection Theory found that recommendation-driven designs lowered users' thresholds for sufficient evidence compared to hypothesis-driven designs, even when task performance remained identical. This created a "hidden bias"—a shifted distribution of errors. Experts showed the same threshold shifts as novices, meaning experience didn't protect against the systemic effect.

Caveats: Web-based experiment. Real-world stakes and repeated use might amplify or attenuate threshold shifts.

Reflections: Do threshold shifts persist or normalize with extended exposure to recommendation-driven systems? · Can interface design restore stable evidence standards while keeping recommendation convenience? · Which professional domains are most vulnerable to hidden bias from lowered thresholds?

LLMs increasingly mediate purchasing and information access for billions, but their systematic preferences for brands and cultures remain unquantified under realistic usage conditions.

Method: ChoiceEval framework generates persona-diverse prompts across psychographic profiles, converts free-form LLM outputs into normalized top-k choice sets, then quantifies preference and geographic bias. Applied to Gemini, GPT, and DeepSeek across 10 topics and over 2,000 questions, U.S.-developed models showed marked favoritism toward American entities, while China-developed DeepSeek exhibited more balanced but still detectable geographic preferences. Patterns persisted across user personas.

Caveats: Tested on three model families. Smaller or open-source models may show different bias patterns.

Reflections: Do users detect and discount geographic bias when it's made explicit? · Can fine-tuning or prompt engineering eliminate structural preferences without degrading output quality? · How do preference patterns shift when models are deployed in non-English languages?

AI systems for group interaction mediation fail with non-neurotypical users because detection models for turn-taking and eye contact are trained exclusively on neurotypical populations.

Method: Introduced MIDD dataset capturing atypical gaze and engagement patterns in people with Intellectual and Developmental Disabilities. Comparative analysis with neurotypical datasets revealed differences in class imbalance, speaking activity, gaze distribution, and interaction dynamics. Classifiers from SVMs to FSFNet showed improved performance when fine-tuned on MIDD, though notable limitations remained. Six therapists in a focus group interpreted findings, highlighting practical implications of atypical patterns.

Caveats: Dataset size and diversity of disabilities not specified. Generalization across IDD subtypes unclear.

Reflections: Which gaze features transfer across different IDD populations versus requiring condition-specific training? · Can hybrid models combine neurotypical and atypical data without degrading performance on either? · How do therapists want AI to surface atypical engagement patterns without pathologizing difference?