Capability Isn't the Bottleneck Anymore

Commercial moderation APIs flag legitimate speech while missing actual hate. Five million test cases reveal systematic failures across identity groups and dialects.

Method: The audit framework tests five APIs (Google Perspective, Azure Content Safety, OpenAI, AWS Comprehend, Hive) against controlled variations of hate speech targeting different groups. The system manipulates identity markers (e.g., swapping 'gay' for 'straight') and linguistic features (AAVE vs. Standard English) to expose bias patterns. APIs over-moderate reclaimed slurs and AAVE by 23-47% while under-detecting coded hate speech that substitutes euphemisms.

Caveats: Framework requires labeled hate speech datasets and doesn't address multimodal content (images, video) where most evasion happens.

Reflections: Can ensemble methods be optimized to minimize both over- and under-moderation simultaneously, or is there an irreducible tradeoff? · How do moderation APIs perform on emerging coded hate speech that evolves faster than training data? · What's the optimal human-in-the-loop intervention rate for different community contexts?

SQL queries on large datasets take seconds to minutes. Users wait, context-switch, and lose flow state during data exploration.

Method: SpeQL uses an LLM to predict and pre-execute likely queries before the user finishes typing. The system analyzes database schema, query history, and partial input to generate candidate completions, then speculatively runs them in parallel. When the user completes their query, results appear instantly if a match exists. The system maintains a cache of speculative results and uses query similarity metrics to determine which predictions to execute.

Caveats: Compute costs scale with prediction accuracy. Low hit rates waste resources. Works best for repetitive exploration patterns, not novel ad-hoc queries.

Reflections: What's the break-even point for speculative execution cost vs. user productivity gains? · Can user attention patterns (eye tracking, cursor position) improve prediction accuracy? · How does this approach scale to collaborative querying where multiple users explore the same dataset?

LLM agent teams generate thousand-message conversations. Developers can't locate where agents derailed or test configuration changes without full re-runs.

Method: AgentStudio provides three mechanisms: conversation summarization that collapses agent exchanges into decision trees, breakpoint-style debugging that pauses agent execution for inspection, and live configuration editing that modifies agent prompts mid-run. The system visualizes agent state, message history, and decision branches. Developers can fork conversations from any point to test alternative configurations without restarting from scratch.

Caveats: Tested with five developers on custom agent frameworks. Unclear how this generalizes to commercial agent platforms with locked-down architectures.

Reflections: Can automated fault localization identify which agent in a team caused cascading failures? · How do developers balance debugging granularity with cognitive load when inspecting complex agent interactions? · What metrics predict when an agent conversation is heading toward failure early enough to intervene?