AI Systems Work in Labs but Collapse at Deployment

CNNs are increasingly used to extract insights from data visualizations, but nobody knows if they actually perceive charts like humans do—or if they're just pattern-matching noise.

Method: The authors trained 800 CNN models on bar chart ratio estimation tasks and compared them against 113 human participants across 16 million trials. They tested three dimensions: sensitivity to training-test distribution shifts, stability with limited samples, and relative expertise versus humans. The finding: CNNs outperform humans in ratio estimation accuracy when trained on diverse bar chart distributions, but collapse when test data diverges from training distributions—humans maintain consistent performance across distribution shifts.

Caveats: Study focuses solely on bar chart ratio estimation. Doesn't address other chart types or visual reasoning tasks like trend detection.

Reflections: Do CNNs trained on diverse chart distributions develop human-like perceptual invariances, or do they rely on fundamentally different visual features? · Can we design training regimes that make CNNs robust to distribution shifts without sacrificing accuracy on in-distribution data? · What's the minimum training diversity needed for CNNs to match human-level generalization across chart styles?

Grading handwritten STEM responses in large courses consumes 30-40% of instructor time. Manual grading doesn't scale, and existing AI tools handle only transcription or rubric generation—not the full pipeline.

Method: Pensieve uses LLMs to transcribe handwritten work via OCR, then evaluates responses against instructor-defined rubrics, outputting scores, transcriptions, and confidence ratings. The system was deployed across 8 courses with 1,500+ students. Key mechanism: instructors review only low-confidence predictions (flagged by the model), reducing grading time by 68% while maintaining 85% agreement with human graders on rubric-aligned scores. The platform handles the entire workflow from upload to final grade export.

Caveats: Agreement drops to 75% for responses with non-standard notation or diagrams. Requires rubric structure upfront—doesn't work for holistic grading.

Reflections: Can confidence calibration be improved to reduce false positives in the low-confidence queue without sacrificing accuracy? · How does grading accuracy degrade with increasingly creative or unconventional student notation? · What's the optimal human-in-the-loop review threshold to balance time savings and grading quality?

Teachers in low-resource schools spend hours adapting generic lesson plans to local context, language, and student needs. Off-the-shelf AI tools generate plans that ignore regional curriculum and cultural nuances.

Method: Shiksha Copilot deployed a two-stage AI collaboration model across 1,043 government school teachers in Karnataka, India. First, curators and AI co-created English and Kannada lesson plans aligned to state curriculum. Then, teachers customized these curated plans using AI assistance for their specific classrooms. The system reduced lesson planning time by 47% while increasing alignment with state standards by 62% compared to teacher-only planning. Critical design choice: curators acted as quality gatekeepers before plans reached teachers, preventing hallucination propagation.

Caveats: Requires dedicated curator roles with domain expertise. Doesn't address real-time classroom adaptation needs—focuses on pre-class planning only.

Reflections: What's the optimal curator-to-teacher ratio for maintaining quality while scaling to thousands of teachers? · Can teacher customization patterns be fed back to improve curator-AI co-creation over time? · How does this model perform in subjects with less structured curricula, like arts or social studies?