AI Agents Ship Without Delegation Protocols

AI agents that act autonomously—sending emails, editing files, making purchases—need to decide when to proceed and when to check with the user. Existing evaluation methods score models by showing them proposed actions and comparing their decisions to human labels.

Method: A 156-scenario benchmark with matched pairs that change single features (stakes, reversibility, who requested it) exposes three failures in current delegation scoring. A simple keyword rule outperforms eight of ten models on agreement scores, yet changes its decision in only 9 of 48 matched pairs. Equivalent phrasings shift how often models act by up to 52.5 percentage points. Every model asks less often when using tools than when judging proposed actions—despite identical stakes.

Caveats: Benchmark uses hypothetical scenarios, not real user tasks with actual consequences.

Reflections: Can models learn to maintain consistent delegation policies across equivalent phrasings without explicit rule statements? · Do tool-vs-judgment gaps persist when models have extended interaction history with a user? · What delegation consistency is acceptable for deployment—should matched pairs always yield identical decisions?

LLMs perform intermediate cognitive work while executing user requests, but it's unclear which steps users intended to delegate versus which they wanted to review or control themselves.

Method: 24 participants annotated 992 reasoning steps across three knowledge-work tasks, revealing that 48.6% of all steps were AI-initiated—work the AI took on without being asked. Desired involvement varied by cognitive work type and whether the AI's contribution matched intent. Even when the AI did what participants wanted, they often wished they'd been consulted first.

Caveats: Retrospective annotations may not reflect real-time preferences during task flow.

Reflections: Can users articulate delegation preferences upfront, or must systems learn them from interaction patterns? · Does reviewing AI-initiated steps impose enough burden to offset delegation benefits? · Which cognitive work types consistently require human involvement across users and domains?

LLMs disrupt the production-quality balance in knowledge commons, prompting many communities to ban them. Korean Wikipedia instead built WikiVault, an LLM-powered translation tool, and appropriated it for community needs.

Method: Platform-scale analysis, ten interviews, and quasi-experimental comparisons show WikiVault drastically amplified a small group of experienced editors, producing longer and more widely viewed articles. It shifted work toward reviewing and importing content, but also imported editorial judgments from English Wikipedia—not just facts. The community learned through use and adapted governance accordingly.

Caveats: Single-community case study. Other commons with different norms may appropriate LLMs differently.

Reflections: Do imported editorial judgments eventually diverge as local editors review and revise, or do they persist? · Can communities detect when LLM tools shift power dynamics toward experienced users? · What governance mechanisms help communities appropriate LLMs without losing editorial autonomy?