Week 38 / September 2025

Automation Erodes Agency While Claiming to Empower It

From memory failures to override rights, rigorous studies expose the control gap in AI-assisted systems

Synthesized using AI

Analyzed 177 papers. AI models can occasionally hallucinate, please verify critical details.

Users can't remember what they created versus what AI created after just one week. A pre-registered study with 184 participants found 40% misattribution rates—people claimed AI-generated ideas as their own and attributed their own work to the system. This isn't about interface clarity or disclosure design. It's human memory architecture colliding with AI-assisted creation, and it reveals a brutal truth: every writing assistant, code completion tool, and content generator shipping today is quietly undermining attribution norms at scale. Supporting evidence appears across domains—UI generation models that ignore designer rationale, video editing systems that obscure human creative decisions, educational content that personalizes without tracking adaptation sources. The assumption being challenged: that users will naturally track AI contributions if we just make the interface clear enough.

But the control problem runs deeper than memory. Automation now steers building HVAC systems, distribution grids, and traffic signals, yet residents have no legitimate authority to pause or redirect these systems when they harm accessibility or safety. New work formalizes override rights—defining who has authority, what evidence they need, what safe fallback states exist—with domain-validated playbooks for buildings, power, and transport. Privacy policies demonstrate the same pattern: contextual alerts at the point of risk still fail because users can't parse complex tradeoffs mid-task. A reflective thinking framework shows promise by disconnecting context from immediate action. The tension: automation promises efficiency but strips agency, and transparency alone doesn't restore it.

The emerging counter-approach abandons in-the-moment explainability for rehearsal and simulation. Let users practice with AI agents in safe scenarios before live deployment, building trust through experience rather than documentation. It's a fundamentally different model—trust as calibration over time, not transparency at decision points. The design challenge this creates: how do you build systems that capture human rationale, allow progressive refinement, and make AI contributions contestable without collapsing back into manual control? The research doesn't answer that yet, but it's finally asking the right question.

Featured(1/5)

2509.11851

The AI Memory Gap: Users Misremember What They Created With AI or Without

Tim Zindulka, Sven Goller, Daniela Fernandes, Robin Welsch, Daniel Buschek

Preprint·2025-09-15

Stop assuming users will self-report AI use accurately. Build provenance tracking into the editor itself—version history that tags LLM contributions at the sentence level. Treat attribution as a system responsibility, not a memory test.

Disclosure of AI-generated content depends on people remembering what they wrote versus what the chatbot wrote. Memory fails at this task.

Method: 184 participants generated ideas with and without an LLM chatbot, then returned one week later. They misattributed 27% of their own ideas to AI and 29% of AI-generated ideas to themselves—near-chance accuracy. The confusion was symmetric: people couldn't distinguish their voice from the machine's, even when they'd personally elaborated on the content.

Caveats: Tested only text generation with a single chatbot interface. Visual or code generation may show different memory patterns.

Reflections: Does inline attribution (e.g., highlighting AI sentences in real-time) improve week-later recall? · Do professional writers show better source memory than students? · Can users distinguish AI contributions when the task is creative versus analytical?

ai-interactiontrust-safetyethics

2509.13444

DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces

Yuan Xu, Shaowen Xiang, Yizhi Song, Ruoting Sun, Xin Tong

Preprint·2025-09-16

Stop building one-shot AI UI generators. Implement editable, stateful interfaces where users can redirect the agent's plan without restarting. Best for multi-step workflows like data analysis or content assembly where intent evolves.

LLMs generate task UIs in one shot, then lock users out. Real workflows need continuous steering as intent clarifies and context shifts mid-task.

Method: DuetUI implements a bidirectional context loop: the system generates UI components (buttons, forms, data displays) that users can override or reshape through natural language mid-execution. A formative study with 12 participants revealed users want to "shape generative interfaces" rather than accept static outputs. The system maintains a shared state model that updates both the UI and the agent's task plan when users intervene.

Caveats: Requires task domains with clear state models. Ambiguous creative tasks may not benefit from bidirectional loops.

Reflections: How many intervention points do users tolerate before reverting to manual control? · Can the system learn user override patterns to preemptively adjust future generations? · Does co-generation improve task outcomes or just user satisfaction?

ai-interactiondesign-toolsprogramming-tools

2509.13532

Py maidr: Bridging Visual and Non-Visual Data Experiences Through a Unified Python Framework

JooYoung Seo, Saairam Venkatesh, Daksh Pokar, Sanchita Kamath, Krishna Anandan Ganesan

Preprint·2025-09-16

Integrate multimodal output into your data visualization libraries from the start. Don't bolt accessibility onto finished charts—design the API so BLV and sighted users share the same authoring tools. Prioritize this for collaborative analytics teams.

Accessible visualization tools follow a "design for them" model: sighted creators make charts, then add non-visual modes for blind users. This creates a one-way pipeline that excludes BLV collaborators from authorship.

Method: Py maidr implements a "design for us" framework where both sighted and BLV users generate and consume visualizations through the same Python API. The system produces multimodal outputs—visual charts, sonification, and braille—from a single data structure. BLV users can author charts that sighted colleagues view, reversing the typical creator-consumer hierarchy.

Caveats: Tested with Python users familiar with matplotlib. Adoption depends on existing visualization workflows.

Reflections: Do BLV-authored visualizations differ structurally from sighted-authored ones when both use the same API? · Can this bidirectional model extend to interactive dashboards, not just static charts? · What training do sighted users need to interpret sonified data effectively?

accessibilitydata-visualizationdesign-tools

1 / 5

Featured

Findings(1/5)

Control authority shifts from one-shot automation to continuous co-generation·Source attribution collapses when AI becomes interactive·Feedback loops for generative models require critique infrastructure, not preference signals·Accessibility tooling inverts from accommodation to shared infrastructure·Personalization moves from content adaptation to value alignment

Automation is moving from fire-and-forget execution to bidirectional loops where users shape outputs during task progress. DuetUI introduces interfaces that unfold alongside work rather than delivering complete artifacts upfront, while Right-to-Override formalizes resident authority to pause urban control systems mid-operation. The shift rejects the assumption that better AI means less human involvement. Instead, capability advances enable finer-grained intervention. Practitioners must design for steering, not just triggering.

2509.13369

Right-to-Override for Critical Urban Control Systems: A Deliberative Audit Method for Buildings, Power, and Transport

2509.13444

DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces

Surprises(1/3)

Memory failure, not disclosure failure, breaks AI attribution·Discrete visualizations bias health risk perception more than continuous ones·Novices can't verify LLM-generated privacy code even when it looks correct

Users accurately identify AI-generated content immediately after creation but drop to near-chance accuracy one week later, even with clear disclosure at generation time. The problem isn't that users don't know AI was involved—it's that interactive collaboration erases the cognitive boundaries that support source memory. Disclosure requirements assume recall that doesn't exist.

2509.11851

The AI Memory Gap: Users Misremember What They Created With AI or Without

TOOLBOX(7)

Conflect

Tool

Interactive contextual privacy policy system for mobile apps that uses sidebar alerts and reflective thinking framework. Automatically detects privacy risks, extracts policy segments from app privacy policies, and generates risk descriptions with 94.0% extraction accuracy on CPP4APP dataset at 4.35s latency. Enables users to reflect on contextualized privacy risks at point of data collection.

2509.12578

DPCheatSheet

Tool

Instructional tool that scaffolds differential privacy implementation for novices using LLMs. Combines worked examples (annotated expert workflows) with erroneous examples (five common mistakes in LLM-generated DP code) to support error-driven learning. Helps programmers without DP expertise articulate requirements in prompts and verify correctness of generated code.

2509.12590

DuetUI

Tool

LLM-empowered system implementing human-agent co-generation paradigm for task-oriented interfaces. Features bidirectional context loop where agent scaffolds interface by decomposing tasks while user's direct manipulations implicitly steer next generation step. Enables dynamic user override and alignment with vague intent through iterative interface generation alongside task progress.

2509.13444

Py maidr

Code

Python package that encodes multimodal (tactile, auditory, conversational) data representations into visual plots from Matplotlib and Seaborn. Users invoke maidr.show() method with minimal codebase changes to generate accessible plots. Integrates with Jupyter Notebook, Google Colab, Quarto, Shiny, and Streamlit. Introduces minimal rendering overhead for blind/low-vision accessibility.

2509.13532

Chameleon

Tool

Surface-anchored smartphone AR prototype that transforms phones into collaborative displays. Combines Background Acquisition (opportunistic sensing with LM-assisted pattern generation for transparency illusion) and Real-Time Position Tracking (augmented inertial sensing) to anchor digital content on surfaces. Enables natural repositioning without external hardware for co-located multi-user AR experiences.

2509.14643

PAGE

Framework

Framework leveraging large language models to automatically personalize educational materials by adapting content to individual student context (major, interests). Deployed as semester-long intelligent tutoring system. Demonstrated improved learning outcomes, engagement, perceived relevance, and trust compared to standardized materials in authentic educational setting with real students.

2509.15068

I2S (Interact2Sign)

Framework

Multi-stage framework for user identification through human-object interaction recognition using 3D hand pose analysis in egocentric videos. Extracts handcrafted features (Spatial, Frequency, Kinematic, Orientation, Inter-Hand Spatial Envelope) from 3D hand poses. Sequential pipeline identifies object class, HOI, then user. Achieves 97.52% F1-score with <4MB model size and 0.1s inference for real-time AR authentication.

2509.16557

Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agent Software

Compares what tech companies promise about AI agents versus what users actually experience. The gap is spectacular and measurable.

2509.16128

AnchoredAI: Contextual Anchoring of AI Comments Improves Writer Agency and Ownership

Pins AI feedback directly to text spans instead of hiding it in chat. Writers retain control and the system reduces automation bias.

2509.14537

ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference

Interrupts designers at decision points to capture their reasoning in real time. Turns implicit creative choices into structured, shareable knowledge.

2509.13253

Evolution of Programmers' Trust in Generative AI Programming Assistants

Tracks how developer trust shifts over time with AI coding tools. Reveals that trust is dynamic, not a fixed attitude.

2509.11206

Evalet: Evaluating Large Language Models by Fragmenting Outputs into Functions

Breaks LLM-as-a-Judge scores into functional components so you can see which parts of an output drove the rating. No more black-box evaluation.

2509.12517

Interaction Context Often Increases Sycophancy in LLMs

Shows that conversational history makes models more likely to agree with users. Two weeks of chat logs reveal how context amplifies flattery.

2510.15905

"She's Like a Person but Better": Characterizing Companion-Assistant Dynamics in Human-AI Relationships

Interviews ChatGPT and Replika power users who blur task assistance with emotional companionship. Documents an emerging relational category that's neither tool nor friend.

2509.12752

Participatory AI: A Scandinavian Approach to Human-Centered AI

Argues AI is a political artifact requiring democratic governance, not just better UX. Draws on Scandinavian participatory design traditions to reimagine agency.

REFLECTION(4)

Explainability erodes trust, not builds it

The research shows that adding transparency to AI systems often backfires—users become paralyzed by cognitive overload or exploit the explanations to game the system. Yet designers keep treating explainability as a universal fix, assuming that more information always improves calibration. The tension is real: does explaining how a system works actually help people trust it wisely, or does it just shift the burden of verification onto users who lack the expertise to verify anything?

Transparency creates cognitive overload, yet designers assume users want to understand every decision. At what point does the effort to verify a system's reasoning become more costly than the risk of blind trust?

1 / 4

Week 37September 2025

Week 39September 2025

ABOUT THIS ISSUE

How this newsletter was synthesized?

Methodology

This newsletter is generated by an AI pipeline (leveraging Anthropic Sonnet 4.5 & Haiku 4.5) that processes the metadata and abstracts of every new arXiv HCI paper from the past week—177 this issue. Each paper is scored on three dimensions: Practice (applicability for practitioners), Research (scientific contribution), and Strategy (industry implications), with scores from 1-5. Papers passing threshold are grouped into topic clusters, and each cluster is summarized to capture what that body of research is exploring.

Selection Criteria

The pipeline builds a curated selection that balances high scores with topic diversity—and deliberately includes at least one 'contrarian' paper that challenges prevailing assumptions. This selection is then analyzed to identify key findings (patterns across multiple papers) and surprises (results that contradict conventional wisdom). A narrative synthesis ties the week's research together under a unifying frame.

Key Themes Discovered

Field Report: ai-interaction

Trust Calibration in Human-AI Workflows

Users struggle to maintain appropriate reliance on AI systems—neither over-trusting nor under-utilizing them. This cluster examines how interface design, transparency mechanisms, and interaction patterns shape trust calibration across diverse domains: writing, healthcare, finance, education, and autonomous systems. Core tensions emerge: explainability can increase overreliance; context-seeking improves outcomes but demands cognitive effort; and visible authorship fosters agency while covert adaptation erodes it. Research emphasizes that trust is relational, context-dependent, and requires dynamic recalibration rather than static confidence scores.

1/10

Top Papers in this Theme

2509.11206

Evalet: Evaluating Large Language Models by Fragmenting Outputs into Functions

2509.14528

Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agent Software

2509.15959

Explainable AI for Maritime Autonomous Surface Ships (MASS): Adaptive Interfaces and Trustworthy Human-AI Collaboration

2509.16811

Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media

2509.16128