Review: The Welch Labs Illustrated Guide to AI
A review of a rare AI book that uses mathematics to illuminate rather than intimidate, making difficult ideas feel genuinely learnable.
Evaluation
21 posts
Your AI Writes Tests That Pass. That Is the Problem.
AI-generated tests can look reassuring while proving very little, exposing a dangerous gap between green checkmarks and real verification.
From PDE Guarantees to LLM Inference: What BEACONS Gets Right About Reliability
BEACONS offers a model for reliability that AI systems badly need: explicit bounds, checkable guarantees, and less benchmark theater.
When an AI Can Act but Cannot Judge
LLMs may act impressively while still failing to know when they are capable, making self-assessment a core safety problem.
From Prompts to People: What OpenAI’s Usage Study Means for Model Leaderboards
OpenAI's usage study shifts attention from benchmark scores to how ordinary people actually use ChatGPT in daily life.
GPT-5 Rollout: The Aftermath
A follow-up on GPT-5's rocky rollout, user frustration, and OpenAI's attempts to tune expectations after launch.
Factual Recap: OpenAI's GPT-5 Keynote Event
A factual recap of OpenAI's GPT-5 keynote, collecting the main claims, demos, benchmarks, and availability details.
The Future of AI: How Self-Adapting Language Models Are Redefining Learning
SEAL points toward language models that rewrite their own training material, hinting at AI systems that learn after deployment.
Knowledge Graphs Won't Solve the LLM Crisis—Here's Why
Knowledge graphs are useful, but the post argues they are not a magic cure for LLM hallucination and reasoning failures.
The Rise of the AI Prophetes: Translating the Machine Oracle
As AI becomes an oracle, a new class of interpreters may emerge to translate machine outputs into human decisions.
When AI Masters Competitive Programming: Why Generalists Outperform Specialists
OpenAI's competitive-programming work suggests generalist reasoning models can outperform narrow specialists in demanding coding contests.
When Metrics Go Wrong: A Tale of Goodhart's Law and AI Misalignment
Goodhart's Law explains why AI alignment can fail when proxy metrics become targets and systems learn the wrong game.
Humanity’s Last Exam: The Ultimate Test for AI and the Future of Intelligence
Humanity's Last Exam is framed as a benchmark that tests not only models, but our assumptions about intelligence itself.
When AI Can't Count: A Hilarious Look at the Math Skills of Text-to-Image Models
Text-to-image models still struggle with counting, making their visual brilliance look surprisingly fragile at the level of basic numeracy.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 1/6
The opening part of a benchmark series asks what LLM evaluations really measure and why the numbers often mislead.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 2/6
Part two examines benchmark methods themselves, exposing the assumptions behind the scores used to compare language models.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 3/6
Part three moves from benchmark scores to application areas, asking where LLM performance actually matters in practice.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 4/6
Part four digs into the good, bad, and misleading sides of benchmark results and their interpretation.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 5/6
Part five steps beyond scores to consider real-world limitations, reliability, and practical model behavior.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 6/6
The final benchmark essay looks toward better evaluation methods that test usefulness rather than leaderboard theater.
Can You Spot the AI? The Turing Test and GPT-4's Sneaky Success
GPT-4's Turing-test performance revives the old question of whether fooling humans proves intelligence or just fluency.