AI Systems Gain Influence While Losing Accountability

Screen reader users spend minutes traversing irrelevant elements before reaching desired information, while sighted users visually skim in seconds. Modern web interfaces overwhelm everyone with excessive text and visuals unrelated to current goals.

Method: Task Mode uses an LLM to dynamically filter web content based on user-specified goals. When a screen reader user states 'find the return policy,' the system strips out navigation menus, promotional banners, and unrelated product listings—leaving only policy-relevant content. The filtering happens in real-time as users navigate, not as a one-time preprocessing step. This reduces the cognitive load of sequential navigation by eliminating the noise before it reaches the user.

Caveats: Depends on LLM accuracy for relevance judgments. Overly aggressive filtering could hide critical context or navigation paths users didn't anticipate needing.

Reflections: How do users recover when the LLM filters out something they actually needed? · Can this approach generalize to mobile apps where DOM structure is less standardized? · What's the latency penalty for real-time LLM filtering on slower connections?

Text is everywhere in visualizations—titles, labels, annotations, captions—but no systematic framework exists for understanding what each element actually does. Designers wing it, leading to inconsistent or redundant text choices.

Method: The authors analyzed 120 real-world visualizations and built a framework categorizing text functions into five roles: orienting (titles, axis labels), highlighting (annotations pointing to specific data), explaining (captions describing patterns), instructing (how to interact), and contextualizing (sources, methodology notes). They found that 73% of visualizations use at least three text function types simultaneously, and annotations are most effective when they combine spatial proximity with explicit data references rather than vague directional language.

Caveats: Framework derived from static visualizations. Interactive dashboards may require additional text functions not captured here, especially for state management and feedback.

Reflections: How do text functions shift in animated or interactive visualizations where state changes over time? · What's the optimal text density before cognitive overload kicks in? · Can LLMs auto-generate contextualizing text from raw data sources without hallucinating?

Understanding team coordination in mixed reality requires manual video annotation and external observation systems. This doesn't scale and misses temporal dynamics that emerge during collaboration.

Method: MURMR passively captures collaboration data using two modules: a logging system that records gaze, hand positions, and object interactions from MR headsets, and a computer vision pipeline that analyzes spatial formations and movement patterns. The system automatically detects coordination events like joint attention (when two users look at the same object within 2 seconds) and parallel work (users manipulating different objects simultaneously). It achieved 87% accuracy in classifying collaboration phases without manual annotation.

Caveats: Requires MR hardware with gaze and hand tracking. Privacy concerns around continuous logging of user behavior need explicit consent frameworks.

Reflections: Can real-time coordination metrics be surfaced to users during collaboration without being distracting? · How do coordination patterns differ between expert and novice teams? · What's the minimum sensor fidelity needed to detect meaningful collaboration events?