Measure Honestly, Try It Yourself (Series Finale)

About the characters

Miriam, Daniel, Jonas and the department head are fictional. In episode 1, Daniel stood in for the participants of the METR study; from episode 2 on, he is a fictional colleague on Miriam's team and did not take part in the study. The course and results of Miriam's experiment (the hard week two, the broken Thursday, Daniel's understanding check) are made up too; they show one possible interpretation of a pilot, not measured or typical values. A real experiment may also show no improvement or a deterioration.

The experiment to take with you

• Week 1: just watch. Nobody changes anything, the tools record what they can; work outside them is added, overlaps are not double counted, measurement gaps are disclosed. Result: a baseline.
• Weeks 2 and 3: switching sides (researchers call it a crossover design). One person (or half the team) works with the new rule in week 2, the other with the old one; in week 3 they swap. Like soccer teams switching sides at halftime: both rules are represented in both weeks, which reduces the influence of a randomly easy or hard week. Comparable tasks and shared review queues (conditions can affect each other through shared reviewers) still need to be accounted for. Limit: learning and carryover effects (after a week of doing less at once, you do not unlearn it overnight). The source recommends assigning the order of conditions randomly and in a balanced way.
• Week 4: look at the results. Set the success criterion and stop conditions in advance. Look at the distribution and the outliers, not just the median or the average.
• The full new rule (from episode 4): one complex task you actively supervise, at most one clearly bounded, quiet background job, and a firm limit on changes waiting for review. The old rule: as much at once as anyone wants. Everything else stays the same, including how reviews are handed over. Larger teams split in half.
• Stop conditions, for example a critical defect, a clear loss of understanding, a growing queue. There is no universal percentage for "success"; every team sets its own, but beforehand.
• Fair comparison: compare only tasks of similar size and risk.
• On your own: one week as usual, one week with a limit. If you kept the energy scale from episode 7, you already have the first week. At the office: one big document at a time, times from the calendar and version history.
• Talk to your works council or employee representatives and check data protection rules such as GDPR first; analyze only aggregated, at team level, one row per change instead of per name. Marking a change as "not understood yet" must not count as being unproductive. (Not legal advice.)

The dashboard: one measure per gauge

• Speedometer = minutes per change. The foundation of the series calls it "human attention minutes per accepted change" (HAMAC), explicitly a working hypothesis. Only human time counts (clarifying, explaining, stepping in, reviewing, fixing). Details left out of the audio: these are person minutes (10 minutes for two people = 20 minutes); discarded attempts count too, such as an agent run that went the wrong way, otherwise the most expensive detours disappear. "Finished" (accepted): criteria met, checks passed, a person takes responsibility. A change that belongs together stays one unit, even if it is split into several change requests.
• Fuel gauge = evening energy. Scale from 1 to 5, 5 = plenty of energy (as in episode 7). A local signal, not a diagnosis. NASA-TLX measures subjective workload across six dimensions. It complements the evening-energy scale, but measures a different construct (see sources).
• Warning light = the understanding check. Regularly discuss a randomly drawn change: what is the critical path? What must never happen? What happens in a failure no test covers? Feeling confident is not enough. A technically correct explanation is evidence of understanding; the conversation is a proposed check, not a validated or complete proof.
• Other useful signals (left out of the audio): rework, meaning how often a change reopens and what the repair costs. Since 2024, DORA counts something similar as its fifth metric, deployment rework rate (unplanned deployments that happen as a result of an incident in production); it starts later than internal rework. And quality: in Miriam's fictional experiment, the team assesses the serious incident against its predefined stopping rule before continuing; it remains part of the quality assessment. A pilot this short cannot reliably assess rare failures.
• Why several gauges: the SPACE framework (Microsoft Research, GitHub, University of Victoria) describes productivity along five dimensions (satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow) and recommends a handful of metrics across at least three dimensions, deliberately in tension, including at least one perceptual measure, reported only in aggregate at team level. DORA also warns against turning its metrics into targets or pitting teams against each other.
• Gut feeling and episode 1: a gut feeling is a poor speedometer (so times come from tool records, with work outside them added and measurement gaps disclosed), but a useful fuel gauge (so energy and understanding come from the people themselves).

Tool tip: collect times automatically

• Claude Code monitoring through the open standard OpenTelemetry: enable with the environment variable CLAUDE_CODE_ENABLE_TELEMETRY=1 plus an exporter configuration (off by default). The metric claude_code.active_time.total measures active time in seconds, with the attribute type = user (keyboard interactions) or cli (tool execution and AI responses). Further metrics include sessions, commits and pull requests. https://code.claude.com/docs/en/monitoring-usage
• Note: a timestamp measures time spent, not attention or strain. Claude Code records active use only within its own interface. Git provides commit timestamps; reviews are recorded by the pull-request platform (for example via gh). Together with calendars, these records do not yet give active human minutes: add work outside them, avoid double counting overlaps, disclose the remaining gaps. Errors can go in either direction.

Not fooling yourself (shortened in the audio)

• Being observed changes behavior. People who know they are measured may work more carefully. The famous factory story behind this (the Hawthorne effect) is shakier than its reputation, see Levitt and List.
• Small numbers. 20 tasks from one person are not 20 independent test subjects. The result is a learning signal for this team.
• Even large experiments stay cautious. At Google, 96 developers were randomly assigned to do the same demanding task with or without AI tools; with AI, by the best estimate, about a fifth less time, and the authors stress large uncertainty and limited transferability.
• Do not turn every saving straight into more output. With the same actual throughput and stable averages, more work in the system means a longer average lead time (Little's Law, episode 4).
• Goodhart in the original. Charles Goodhart wrote about monetary policy in the mid-1970s: a statistical regularity tends to collapse once you use it for control. The well-known short form is Marilyn Strathern's.

Sources

Basis of the series

• Healthy AI First Software Development, version 2 (2026-10-01), ch. 12 to 17: team practices, typical failure modes, measurement framework, experiments over 2 to 4 weeks, templates, open research questions. The metric "human attention minutes per accepted change" (HAMAC) is explicitly a working hypothesis there, not an established standard.

Metric and target

• Goodhart, C. (1975): Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics, Reserve Bank of Australia. Overview and original wording ("Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes"): https://en.wikipedia.org/wiki/Goodhart%27s_law
• Strathern, M. (1997): "Improving ratings": audit in the British University system. European Review 5(3), 305-321. Source of the well-known short form "When a measure becomes a target, it ceases to be a good measure." https://www.cambridge.org/core/journals/european-review/article/improving-ratings-audit-in-the-british-university-system/FC2EE640C0C44E3DB87C29FB666E9AAB

Measurement frameworks (no longer named in the audio)

• Forsgren, Storey, Maddila, Zimmermann, Houck, Butler (2021): The SPACE of Developer Productivity. ACM Queue 19(1). https://queue.acm.org/detail.cfm?id=3454124
• DORA: Software delivery performance metrics (five metrics including deployment rework rate; warning against turning them into targets with reference to Goodhart, no competition between teams). https://dora.dev/guides/dora-metrics/
• DORA: History of the metrics. Deployment rework rate was added in 2024 as the fifth metric. https://dora.dev/insights/dora-metrics-history/
• Further reading: Noda, Storey, Forsgren, Greiler (2023): DevEx: What Actually Drives Productivity. ACM Queue. https://queue.acm.org/detail.cfm?id=3595878

Measuring workload (NASA-TLX, subjective workload)

• Hart, S. G. (2006): NASA-Task Load Index (NASA-TLX); 20 Years Later. Six dimensions, Raw TLX as the unweighted variant. https://doi.org/10.1177/154193120605000909
• NASA Human Systems Integration Division: NASA TLX. https://www.nasa.gov/human-systems-integration-division/nasa-task-load-index-tlx/

Not fooling yourself

• Levitt, S. D. & List, J. A. (2011): Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments. American Economic Journal: Applied Economics 3(1). https://www.nber.org/papers/w15016
• Paradis, E. et al. (2024): How much does AI impact development speed? An enterprise-based randomized controlled trial. 96 Google software engineers, best estimate about 21 percent less time, large confidence interval. https://arxiv.org/abs/2410.12944

Employee representation and data protection

• Example from Germany: section 87(1) no. 6 of the Works Constitution Act (Betriebsverfassungsgesetz, BetrVG) gives the works council a say in introducing and using technical systems designed to monitor employees' behavior or performance (paraphrased). According to established case law of the Federal Labour Court, it is enough that a system is objectively suitable for monitoring. Rules differ by country; this is not legal advice. https://www.gesetze-im-internet.de/betrvg/__87.html

Open questions

• Warm, J. S., Parasuraman, R. & Matthews, G. (2008): Vigilance Requires Hard Mental Work and Is Stressful. Human Factors 50(3), 433-441. https://doi.org/10.1518/001872008X312152

Callback to episode 1

• Becker, Rush, Barnes, Rein / METR (2025): Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. 16 developers; expected to be 24 percent faster, measured 19 percent slower, estimated afterward 20 percent faster. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/