Update
Reports get analysis as a contract — and are written by your strongest configured model
TL;DR
The complaint was specific: reports read like a table with sentences around it. Three causes, all fixed here — nothing in the contract actually required analysis, the report was drafted by the analytics model rather than the strongest one configured, and comparing two periods was left to prose when it should be arithmetic.
What shipped
- A five-part report contract — a conclusion with the size of the change, a driver breakdown, the trend within the period, assumptions ranked by how much they'd move the answer, and a next step. The reviewer checks depth, not just accuracy.
- The report is drafted by the stronger configured model. Drafting ran on the analytics role, which is often a cheaper model than the one planning the work. It now picks the stronger of the two.
- A deterministic period comparison in the sandbox — a preloaded helper splits a change into volume and rate effects and reports the trend across each half, so "revenue rose because volume rose while rate fell" is computed rather than asserted.
- Monthly-average currency conversion — asking for a month converts at that month's average rate rather than one day's, which is what a monthly total actually needs.
- A device breakdown across the reporting surfaces that were missing it.
If both of your configured quality roles are weaker models, the deliverable now carries a short note saying a stronger model would go deeper — rather than silently handing you a thinner report.
Why the model choice mattered more than the prompt
Running the same task against a frontier model in a plain chat produced visibly deeper analysis, and the instinct is to blame the prompt. Measured against real traces, the prompt was only part of it: the drafting step was assigned to whichever model filled the analytics role, which on most configurations is the cheap one. Asking a cheap model for depth is a losing argument no matter how the request is worded.
Frequently asked questions
- Which model writes my reports now?
- The stronger of the models configured for your planning and analytics roles. Previously drafting always used the analytics role, which on most setups is a cheaper model than the one doing the planning.
- What does the report have to include?
- Five things: the conclusion with the size of the change, a breakdown of what drove it, the trend within the period, the assumptions ranked by how much they would move the answer, and a recommended next step. A reviewer checks for depth before the report is delivered.
- How are currencies converted for a monthly total?
- Asking for a month converts at that month's average rate rather than a single day's spot rate. A month of spend converted at one day's rate is a different number, and for reporting the average is the right one.