Today's brief ·
Monday 2 November
Exploring the frontier, building what's next!
A daily AI brief and one deep dive a day, written for senior engineers and architects. New here? Start here →
Today's deep dive · Models & Agentic
A trillion-parameter open model lands. Your eval set decides whether it's yours to use
Mistral Large 4's open weights (1T total, about 52B active, 1M context) arrive with mostly self-reported benchmarks; the week's building block is the eval set, calibrated judge and trace that turn those claims into a go/no-go.
- Mistral Large 4 is about 1T parameters with 52B active and needs 8-way tensor parallelism; most of its published scores are Mistral's own.
- An LLM judge is ready only when its agreement with expert labels beats chance: at 80% raw agreement our toy set scored Cohen's kappa 0.47.
- Report pass^k, not pass@k, for agents users rely on: 75% per trial gives 42% over three runs.
This week
All briefs →- Mon2Models & AgenticDive: A trillion-parameter open model lands. Your eval set decides whether it's yours to useBrief: Claude Code's quiet new rule, and five AI coding tools with browser flaws
- Tue3Research·
- Wed4System·
- Thu5Interview·
- Fri6GitHub·
- Sat7Builds·
- Sun8Lab·
Recent deep dives
All deep dives →LabLab · Step 1, part 4
Lab, Step 1 (part 4): pull one piece out of a small GPT and see what you lose
Part 4 of Step 1: ablate a tiny GPT one component at a time, measure the damage against a seed-noise baseline, and learn which pieces are load-bearing and which are decoration at this size.
Builds
Build: a retrieval harness that tells you what each choice is worth, or a contract-aware chunker
Two three-hour builds from this week's retrieval theme: a harness that scores chunking, context lines and fusion on a labelled set (runnable, numpy only), or a clause-aware chunker for contracts.
GitHub
claude-mem: retrieval for an agent's own memory, in three calls that each cost less than the last
thedotmack/claude-mem stores what a coding agent did, then serves it back through search, timeline and get_observations; the interesting part is the retrieval contract and its size caps, which you can copy for any RAG system.
Interview
Principal/Staff Architect, project deep-dive: your retrieval stopped improving. Walk the panel through why
A practice project deep-dive in the style of a Meta Principal/Staff Architect loop: a RAG system you owned has plateaued, so how do you separate retrieval misses from insufficient context, and why can better recall make answers worse?