Did your AI agents actually save time, or did it just feel fast?

Your done column is full of agent tickets and your evenings are full of diffs. The board looks faster. Your week does not.

That gap is worth taking seriously and it proves nothing on its own, because nobody has measured it yet. Four weeks of ticket data can, and the design below is sized for three people and forty tickets, not a research group.

Before any of it, the limit. Taskfolk does not record when a pull request merged: our GitHub integration takes push and release events and nothing else, and no report computes the number this post cares about most. Everything here runs on the estimate field, one custom field, agent session rows, and an export you pull yourself.

I work on Taskfolk, so read the product section with the suspicion it deserves. The protocol works the same in Jira or Linear.

The agent's clock stops somewhere else

An agent session ends when the agent marks the ticket done. Your work starts there. The saving gets counted at one boundary, the cost lands at another, and the two show up on different people's calendars. A metric built that way flatters the first person and hides the second.

flowchart LR
  A["Ticket assigned, session runs"] --> C["Agent marks it done"]
  C --> D["Waits in the review queue"]
  D --> E["A human reads the diff"]
  E --> F["Merged"]
  C --> G["The saving is reported here"]
  F --> H["The cost lands here"]

Two 2026 surveys land on the same contradiction. Harness put the question to 700 engineers and managers in five countries and reported in May that 89% of engineering leaders say productivity improved since deploying AI, while 81% of those same leaders say code review time has gone up. Sonar surveyed 1,149 developers and found 75% agreeing that AI reduced their toil, while the share of the work week they said went to toil sat at 23 to 25% whether they used AI daily or occasionally. Sonar's own chapter heading for that is the illusion of toil savings. A survey contradicting itself inside one page is the cleanest version of this problem I have seen.

Why review absorbs the gain is a separate argument, made in the post on review becoming the bottleneck. This post is about measuring, not diagnosing.

You cannot run the ticket twice, so write the number down first

To know whether the agent saved time you need to know how long the ticket would have taken without it, and that ticket was only done once.

The standard workaround is to ask afterwards. The St. Louis Fed does exactly this: its Real Time Population Survey asks people how many additional hours they would need to work to get the same done without generative AI, and pooling the three 2025 waves lands on 1.6% of all work hours saved. Defensible for a national survey, because the Fed cannot see the work before it happens. You do.

So do not ask afterwards. Before an agent touches a ticket, a human writes one line on it: this would take me ninety minutes. Clinical trials solved a version of this by registering the outcome before the trial runs, and the logic transfers even if the stakes do not.

flowchart TD
  A["Ticket refined and ready"] --> B["Human writes would have taken X"]
  B --> C{"Who runs it"}
  C -->|"agent"| D["Assign to the agent"]
  C -->|"human"| E["Assign to a person"]
  D --> F["Done, record actual"]
  E --> F
  F --> G["After fourteen days, record reopens"]

Order matters more than accuracy. A pre-commitment that turns out badly wrong is still usable, because it was wrong before anyone knew the answer. The same number recalled after a smooth agent run is contaminated.

Four numbers, and the field each one lives in

Cost is deliberately absent from this list, because what an agent spends per task is its own question and we handle it separately.

Number	Where it comes from	Clean to export
Pre-commitment vs actual, agent and human split	Number custom field, estimate_minutes, spent_minutes	Yes, over the API
Done to merged gap	resolved_at here, merge time from GitHub	No, assembled by hand
Reopen or rework inside 14 days	Status re-entries in the activity log	Yes, per issue
Tickets touched more than twice	Count of activity rows by actor	Yes, per issue

The agent versus human split needs no discipline. Assigning an issue to a connected agent auto creates a pending session, so the row exists whether or not somebody remembered to tag the ticket. Sessions carry started_at, ended_at, a state and the pull request link in external_url. Taskfolk does not run the agent, it stores the record, and that record sorts tickets into two piles by fact instead of memory.

An issue with the Time card showing estimate against spent, and a live agent session below it carrying its state, start time and PR link.

Setting it up takes about twenty minutes

Add one custom field to the project. Kind number, named something blunt like "Would have taken (min)", applied to story, task and bug. Use the plain number kind, because you will average it later.

The project settings custom fields card with a number field defined, showing its kind, required flag and the work types it applies to.

Two honest notes. Marking it required does not stop anyone filing a ticket without it, it only stops the value being cleared once something is in there. And if you have a connected agent, drop estimate from its field policy so it cannot overwrite the number it is measured against. Setup detail is in custom fields in issue tracking and how to track time on issues.

Then the export, where the product is worse than it sounds. The list view CSV carries estimate_minutes but not spent_minutes, story_points, completion_pct or any custom field value, and it stops at 5,000 rows. Pull the rest from the REST API:

curl -s "https://taskfolk.ai/api/v1/workspaces/acme/projects/WEB/issues?status=done&limit=100" \
  -H "Authorization: Bearer tfk_live_a1b2..."

import os, requests
r = requests.get(
    "https://taskfolk.ai/api/v1/workspaces/acme/projects/WEB/issues",
    params={"status": "done", "limit": 100},
    headers={"Authorization": f"Bearer {os.environ['TASKFOLK_API_KEY']}"},
)

{
  "data": [
    {
      "key": "WEB-412",
      "status": "done",
      "assignee_id": "0199b1c4-7f3a-7c21-9d55-2b8e4a1f0c6d",
      "estimate_minutes": 90,
      "spent_minutes": 140,
      "resolved_at": "2026-07-14T09:21:44.000Z"
    }
  ],
  "pagination": { "next_cursor": "eyJjIjoiMjAyNi0wNy0xMVQxNjowMjoxMFoifQ" }
}

Custom field values come from the per issue custom fields endpoint, so it is two pulls. There is no date range filter on the issue list either, so a four week window means paging the cursor and filtering client side, or scoping the check to one sprint. I would take the sprint. More in how to use the REST API.

Four weeks, and why the rework window is fourteen days

Run it for four weeks, then stop. Longer is not better here, because the effect decays. He and colleagues at Carnegie Mellon matched 806 Cursor adopting repositories against projects that never adopted, and found a three to five times jump in lines added in the first adoption month that dissipates after two, alongside a persistent rise in static analysis warnings and code complexity. Measure at month six and you measure a different thing.

Fourteen days is not arbitrary either. GitClear's January 2026 analysis of 623 million code changes uses two week churn, code revised within a fortnight of being written, as its rework metric, and reports it up 15%. Borrowing that window keeps your number comparable to somebody else's.

The list view filtered to one agent assignee with the estimate column showing, which is how the agent run subset becomes a query rather than a spreadsheet tab.

Reading a result when n is forty tickets and three people

Forty tickets across three people is a small sample, and most of the standard advice does not survive it. Two rules keep it honest.

Compare paired, within person, only. Alice's pre-commitments against Alice's outcomes. Pooling three people into one bucket smuggles in the fact that Alice estimates optimistically and Raj does not, and that difference is bigger than the effect you are hunting.

Use a sign test, not a t test. You have paired directions, not a well behaved distribution, and forty tickets is nowhere near enough to pretend otherwise. Count the pairs where the agent beat its pre-commitment by more than your noise floor, count the pairs that went the other way, drop the ties, read the table.

Non tied pairs	Pairs needed one way for p below 0.05	p at that count
6	6	0.031
10	9	0.021
15	12	0.035
20	15	0.041
40	27	0.038

from math import comb

def sign_test(favor_agent, favor_human):
    n = favor_agent + favor_human       # ties dropped, never counted
    k = max(favor_agent, favor_human)
    tail = sum(comb(n, i) for i in range(k, n + 1))
    return min(1.0, 2 * tail / 2 ** n)

print(round(sign_test(17, 7), 3))   # 0.064
print(round(sign_test(19, 9), 3))   # 0.087

Both printed cases are ones you will hit: seventeen of twenty four favouring the agent, nineteen of twenty eight, and neither clears 0.05. Six pairs all pointing the same way is the smallest set that clears the bar, so fewer than six comparable pairs per person tells you nothing.

"No signal" is a real outcome rather than a failed experiment. It says the effect, if there is one, is smaller than your team can detect in four weeks. That rules out the enormous win and sends you to the review queue.

What the data cannot tell you

Conclusions this design does not support, however the numbers land:

• That agents are faster in general. You measured your agents, your codebase, your tickets, this month.
• That the effect will hold. The Carnegie Mellon work puts the gain in the first two months.
• That a null result means agents are useless. Equal time with less tedium is a real outcome, just not a time saving.
• That the agent caused it. You did not randomise, and the tickets handed to agents are the ones that looked agent shaped.
• Anything about code quality. Nothing here looks at the diff.

The thing that would be cheating: filling in a pre-commitment after the ticket closed, or adjusting one that turned out obviously wrong. Both turn the study back into a recollection.

What Taskfolk does not capture

No merge timestamp, as flagged at the top. The GitHub webhook takes push and release events only, and there is no merged column anywhere in the schema, so you get that side from GitHub yourself. Jira does surface it: its development panel marks a work item merged once a linked pull request lands, and a rule can transition the item on the same event. If done to merged is your headline number, Jira has it and we do not.

We do ship a cycle time report, which sounds like it solves this and does not. It measures the first transition into in progress through to done, falling back to created through done. The clock stops at done, the exact boundary this post argues you cannot trust.

The project cycle time report, useful for flow but stopping its clock at done, which is why the protocol needs a second source.

There is no per entry timesheet either, so spent_minutes is whatever a person typed. If you want cycle time and lead time defined properly, we wrote that up separately.

The instrument you already pay for

The measurement platforms are good products built for a different buyer.

Instrument	Published price, 27 July 2026	What it covers
Axify Starter	$100 a month flat, unlimited contributors	AI tool adoption and cost only
Axify Pro	$29 per active contributor, billed annually	Adds pull request, sprint and DORA metrics
DX	None published, contact sales	Quote after a conversation
Faros AI	None published, demo request	Quote after a conversation
Taskfolk	$3 or $6 per human editor, agents free	The four fields this protocol reads

The Axify Starter price is real and I would not talk anyone out of it, but that flat tier stops at tool adoption and cost, and the tier that reads pull requests starts at $29 a head on an annual commitment. DX and Faros publish a framework and no number. None of the three captures a counterfactual anyway, because none of them sees the ticket before the work starts. Your tracker does, and that is its only real advantage here.

Add the number field to your active project today and fill it in on the next ticket you refine. Four weeks of that beats another quarter of arguing about whether it feels faster. Tickets written well enough for an agent to finish give cleaner numbers too, which is its own piece of work.