Vector 1
AI Strategy & Economics

Document AI in Banking and Tax: The Time-Savings Math Before You Buy

Brian J. Curry
Brian J. Curry
November 3, 2026 · 24 min read

Nov 4, 2026 · @Brian Curry

A working model for bank and tax operations leaders who need to know how many hours document AI will actually return before they sign the contract, with Python you can run on your own document mix.

Brian Curry | Vector1 Research | Artificial Intelligence/Super Intelligence, Kansas City

The savings that never reach the P&L

Every document AI demo in banking and tax ends the same way. A W-2 or a credit agreement goes in, clean fields come out in two seconds, and someone multiplies those two seconds by last year’s volume. The resulting number is usually large, and it is usually wrong in a specific, predictable way.

The demo measures extraction, the one step of intelligent document processing that vendors benchmark. Your operation pays for everything around extraction: the documents that get routed to a person because one field looked uncertain, the errors that slip through and get fixed three weeks later by someone more expensive, the integration and validation work, and the uncomfortable fact that a freed hour only becomes a dollar if you stop paying for it or fill it with work that earns money. Banks and tax firms are full of pilots that reported 70 or 80 percent time savings and full of budgets where those savings never appeared.

The public sector shows the gap at scale. The IRS estimates that a paper individual return costs 43 times more to process than an electronic one, and paper returns were 6 percent of individual returns in the 2025 filing season but 72 percent of processing costs. The economic case for digitizing them is about as strong as a document automation case gets. Yet a February 2026 Treasury Inspector General report found that pilots from February 2023 through December 2024 digitized about 3.8 million of 53.3 million paper forms, roughly 7 percent, and an in-house scanning system was halted in April 2025 after about $61 million was spent with no returns scanned. The bottleneck was not the model. It was staffing, clearances and operations.

This piece gives you a way to estimate the real number before you buy. It starts with one equation that splits saved time into the parts vendors show you and the parts they don’t. Then it walks through why field accuracy compounds against you, why the human review queue is the actual product you’re designing, why per-page API prices are the smallest line in the budget, and why tax season changes the math in your favor. Each section has worked numbers, and the code is complete enough to run against a sample of your own documents. Whatever you believe about the race toward super intelligence, the back office will be won or lost on review minutes.

Document AI in banking and tax: one equation for net minutes saved

Net minutes saved per document is the baseline handling time minus everything the automated process still costs you, and the review and rework terms usually decide the answer. Here is the proposed model:

S = t₀ − (tₐ + p·tᵣ + e·tₑ)

Here t₀ is the baseline minutes to key, check and file one document by hand. The term tₐ is the residual minutes every automated document still costs: uploading, classifying the odd one, glancing at the output. The review rate p is the share of documents routed to a person, and tᵣ is the minutes each routed document takes. The escape rate e is the share of documents that leave the process with an uncaught error, and tₑ is the minutes it takes someone downstream to discover and fix it.

Annual dollar value then needs two more terms that most business cases leave out:

V = ρ · (N · S / 60) · w − N · g · c − F

N is documents per year, w is the loaded hourly labor cost, g is pages per document, c is the extraction price per page and F is fixed year-one cost (integration, validation, change management). The realization factor ρ is the share of freed hours that become dollars, either because you stop paying for them or because the people redeploy to work that earns revenue. A ρ of 1 means every saved hour shows up in the P&L. A ρ of 0 means everyone is still employed and doing slightly less.

A worked example. Take a regional tax practice that handles 120,000 source documents per season (W-2s, 1099s, K-1s, brokerage statements) at an average of 3 pages each. Assume 6 minutes of manual handling per document, 0.5 residual minutes after automation, 35 percent of documents routed to review at 3 minutes each, and 1.5 percent of documents escaping with an error that costs 30 minutes to fix later. Then S = 6 − (0.5 + 0.35 × 3 + 0.015 × 30) = 6 − 2.0 = 4.0 minutes. That is 8,000 hours a year.

For w, use the May 2025 OEWS median for loan interviewers and clerks, about $24.05 an hour, and gross it up by the BLS finding that wages were 70.0 percent of private-industry compensation in June 2026. That gives a loaded rate of about $34.36. The 8,000 hours are worth about $274,857. Extraction at $0.065 a page on 360,000 pages costs $23,400. Add $150,000 of year-one fixed cost and the net is about $101,457, if ρ is 1.

Now vary ρ. At 0.75 the net drops to about $32,743. At 0.5 it is about −$35,971. The break-even realization is 0.631: unless roughly 63 percent of the freed hours turn into real money, year one loses money even though the system works exactly as specified. That single number is the most useful thing to agree on before a purchase, and almost nobody computes it.

from dataclasses import dataclass

@dataclass
class DocFlow:
    n_docs: float          # documents per year
    pages_per_doc: float
    t_manual: float        # baseline minutes per document
    t_auto: float          # residual minutes on every automated document
    review_rate: float     # share of documents routed to a human
    t_review: float        # minutes per routed document
    escape_rate: float     # share of documents with an uncaught error
    t_rework: float        # minutes to find and fix an escaped error
    wage_loaded: float     # dollars per hour, wages plus benefits
    price_per_page: float  # dollars per page for extraction
    fixed_cost: float      # year-one integration, validation, change management
    realization: float     # share of freed hours that become dollars

    def minutes_saved_per_doc(self) -> float:
        return self.t_manual - (self.t_auto
                                + self.review_rate * self.t_review
                                + self.escape_rate * self.t_rework)

    def hours_saved(self) -> float:
        return self.n_docs * self.minutes_saved_per_doc() / 60

    def api_cost(self) -> float:
        return self.n_docs * self.pages_per_doc * self.price_per_page

    def net_value(self) -> float:
        gross = self.hours_saved() * self.wage_loaded * self.realization
        return gross - self.api_cost() - self.fixed_cost

    def breakeven_realization(self) -> float:
        gross = self.hours_saved() * self.wage_loaded
        return (self.api_cost() + self.fixed_cost) / gross

if __name__ == "__main__":
    wage = 24.05 / 0.70  # OEWS median, grossed up for benefits
    flow = DocFlow(120_000, 3, 6.0, 0.5, 0.35, 3.0, 0.015, 30.0,
                   wage, 0.065, 150_000, 1.0)
    print(f"minutes saved per doc: {flow.minutes_saved_per_doc():.2f}")
    print(f"hours saved per year:  {flow.hours_saved():,.0f}")
    print(f"break-even realization: {flow.breakeven_realization():.3f}")
    for rho in (1.0, 0.75, 0.5, 0.25):
        flow.realization = rho
        print(f"rho={rho:.2f}  net=${flow.net_value():,.0f}")

Every input here is an assumption until you measure it on your own documents, and the point of writing them down is that each one can be measured. The rest of this piece is about where those numbers come from and how they move.

Where the minutes actually go

The value of document AI scales with t₀, the minutes a person spends per document today, and t₀ varies by two orders of magnitude across bank and tax paperwork. Measuring it is the first step, and most teams skip it because they assume they know.

At one end sit standardized forms. A W-2 has a fixed federal layout and a small set of boxes. The IRS received more than 5 billion W-2s and other information returns in 2022, and the forms a tax practice sees from clients are mostly the same handful of types. Keying one takes a few minutes, extraction tools handle them well, and the savings per document are small but the volume is large.

In the middle sit semi-structured documents: bank statements, pay stubs, composite brokerage 1099s, appraisals. The fields are predictable but the layouts are not, since every bank and broker formats them differently. Here t₀ runs longer, and so does tᵣ, because a reviewer has to find the field on a page that looks different every time.

At the far end sit documents whose meaning lives in prose: commercial credit agreements, K-1 footnotes and statements, trust documents, loan covenants. The best-known public number in this category is old but still instructive. In 2017, JPMorgan said its COiN software handled review of about 12,000 commercial credit agreements a year, work that had taken lawyers and loan officers about 360,000 hours. That is 30 hours per agreement. At that t₀, even a modest percentage improvement is worth more than eliminating all keying on a stack of W-2s.

Mortgage origination shows how the minutes hide inside a larger cost. The Mortgage Bankers Association reported total loan production expenses of $11,094 per loan in 2025 for independent mortgage banks, against net production income of $785 per loan. Document handling is spread across processing, underwriting and closing in that $11,094, which is why a lender cannot read the savings off a single line. You have to time the document steps directly.

  • Standardized forms (W-2, 1098, 1099-INT, driver’s licenses): t₀ comes from keying and checking a fixed set of boxes, and automation nearly eliminates the keying.
  • Semi-structured documents (bank statements, pay stubs, composite brokerage 1099s): t₀ comes from finding fields in layouts that vary by issuer. Automation cuts search and keying, but review stays meaningful.
  • Prose-heavy documents (credit agreements, K-1 statements, trust documents): t₀ comes from reading and interpreting clauses. Automation locates and summarizes, and a person still decides.

The practical consequence is that one blended t₀ across all documents will mislead you. Measure it per class, because the equation above gives very different answers for each, and the class mix of your volume determines the total.

Field accuracy compounds against you

A vendor’s 99 percent accuracy figure is almost always a field-level number, and a document only skips review when every field on it is right. If each of k fields is correct with probability a and errors are independent, the share of documents that come out fully correct is:

P(document fully correct) = aᵏ

At a = 0.99 and k = 40, that is 0.9⁹⁴⁰ ≈ 0.669. A third of documents contain at least one wrong field. Raise field accuracy to 99.5 percent and a 40-field document is fully correct 81.8 percent of the time. Only at 99.9 percent does the document-level rate reach 96.1 percent. Independence is a simplifying assumption: in practice errors cluster on bad scans and unusual layouts, which helps a little (the bad pages fail together) and hurts a little (those are the pages a reviewer finds slowest).

This is why the review rate p in the equation is so much higher in production than in a demo. The document-level number is the one that sets p, and it falls quickly as forms get richer. A W-2 with a dozen boxes behaves very differently from a bank statement with 40 transactions to capture.

Published benchmarks show the same gap between clean and messy inputs. A 2025 Fraunhofer IAIS study tested eight multimodal models on invoices and receipts and found the best model, Gemini 2.5 Pro, reached 96.50 percent overall accuracy on clean synthetic invoices but 87.46 percent on scanned receipts. The same study found that feeding page images directly to the model beat converting them to markdown first, by a wide margin on scanned documents. A March 2026 SAP and Stanford study on real enterprise supply chain and finance documents reported F1 scores of roughly 70 to 75 percent for the strongest image-only configurations. Neither study used your documents, and that is the point: clean public datasets overstate what you will see, so the only accuracy number that belongs in your model is one measured on a labeled sample of your own volume, scored at the document level.

The review queue is the product

Once extraction is good enough, the design decision that moves the business case most is the confidence threshold that sends work to people, and the right threshold is the one that minimizes review time plus rework time, not the one that minimizes errors. Every document AI platform exposes some version of this dial. Few teams tune it with their own costs.

The trade-off is mechanical. Lower the threshold and fewer fields go to review, so p falls, but more wrong fields leave unchecked, so e rises. Each escaped error costs tₑ, which in banking and tax is far larger than tᵣ: a wrong figure caught at keying costs seconds, while the same figure caught in a reconciliation, a client query or a notice from a tax authority costs an analyst’s afternoon. With tₑ at 30 minutes and a review at a fraction of a minute per field, the optimum sits at a high threshold, though not the highest.

The simulation below makes that concrete. It generates 100,000 documents with 40 fields each. Most fields are easy, with confidence near 0.999, and 1.5 percent are hard, with confidence near 0.80. It assumes the confidences are calibrated, meaning a field reported at 0.90 is right 90 percent of the time. Review is field-level: opening a routed document costs 0.5 minutes and each flagged field costs 0.25 minutes.

The simulated field accuracy is 99.60 percent, yet only 85.15 percent of documents are fully correct, the compounding from the previous section at work. With no review, the expected cost is 4.454 minutes per document, almost all of it rework. A threshold of 0.98 routes 44.3 percent of documents, lets 3.92 percent escape, and brings the cost down to 1.542 minutes. Push the threshold to 0.999 and every document goes to review, escapes fall to 0.69 percent, and the cost rises to 3.802 minutes. The curve is flat near the bottom, so anything from 0.95 to 0.99 is close to optimal here, which is useful: you do not need a precise tₑ to land in the right region.

Two lessons generalize. First, field-level review changes the economics. If your reviewers must re-key or re-read an entire document whenever any field is flagged, tᵣ is several minutes rather than a fraction of one, and the optimal threshold moves down while total cost rises. Ask vendors to show you the reviewer screen, not just the extraction output. Second, the 3.92 percent of documents that escape at the optimum are errors the model was confident about. No threshold catches those. Cross-field validation rules do: Social Security tax withheld on a W-2 should be about 6.2 percent of Social Security wages up to the wage base, a 1099 composite’s totals should tie to its detail lines, a borrower’s name and income should match across the pay stub, the W-2 and the application. Rules like these are cheap, deterministic, and the main lever on e once the threshold is set.

"""Choose a confidence threshold for routing extracted fields to review.

Assumes calibrated confidences: a field reported at confidence c is correct
with probability c. Test calibration on a labeled sample of your own
documents before trusting the sweep.
"""
import numpy as np

def simulate(n_docs=100_000, n_fields=40, hard_share=0.015, seed=7):
    rng = np.random.default_rng(seed)
    hard = rng.random((n_docs, n_fields)) < hard_share
    easy_conf = rng.beta(400.0, 0.4, size=(n_docs, n_fields))  # mean ~0.999
    hard_conf = rng.beta(6.0, 1.5, size=(n_docs, n_fields))    # mean ~0.80
    conf = np.where(hard, hard_conf, easy_conf)
    correct = rng.random((n_docs, n_fields)) < conf
    return conf, correct

def sweep(conf, correct, thresholds, t_open=0.5, t_field=0.25, t_rework=30.0):
    rows = []
    for tau in thresholds:
        flagged = conf < tau
        routed = flagged.any(axis=1)
        escaped = (~flagged & ~correct).any(axis=1)  # a wrong field nobody saw
        minutes = (routed.mean() * t_open
                   + flagged.sum(axis=1).mean() * t_field
                   + escaped.mean() * t_rework)
        rows.append((tau, routed.mean(), escaped.mean(), minutes))
    return rows

if __name__ == "__main__":
    conf, correct = simulate()
    print(f"field accuracy {correct.mean():.4f}")
    print(f"documents fully correct {correct.all(axis=1).mean():.4f}")
    taus = [0.0, 0.80, 0.90, 0.95, 0.97, 0.98, 0.99, 0.995, 0.999]
    for tau, p, e, m in sweep(conf, correct, taus):
        print(f"tau={tau:<6} routed={p:.3f} escaped={e:.4f} min/doc={m:.3f}")

To use it on real data, replace simulate() with your platform’s field confidences and a hand-labeled correctness flag for a sample of a few hundred documents, and replace the three time constants with what you measure. Before you do, bin the confidences and check that observed accuracy in each bin matches the reported confidence. If it does not, the sweep will still run, but its answer will be wrong, and recalibrating (isotonic regression on the labeled sample is the usual first try) comes first.

The API bill is the smallest line

Per-page extraction prices are the number procurement negotiates hardest, and in most bank and tax workflows they are a rounding error next to review labor and fixed cost. Here are current list prices from the two cloud providers whose pricing pages state them plainly:

  • Amazon Textract AnalyzeDocument Forms: $0.05 per page, $0.04 above 1 million pages (US West, Oregon list price)
  • Amazon Textract AnalyzeDocument Tables: $0.015 per page, $0.010 above 1 million pages
  • Amazon Textract AnalyzeLending: $0.07 per page, $0.055 above 1 million pages
  • Google Document AI Form Parser or Custom Extractor: $30 per 1,000 pages, $20 per 1,000 above 1 million
  • Google Document AI W-2 parser: $0.30 per classified document
  • Google Document AI bank statement parser: $0.75 per classified document

Microsoft’s Azure Document Intelligence pricing page lists prebuilt W-2, 1098 and custom extraction models but did not display rates when I checked, so price it through the Azure calculator or your account team.

Now put the worked example’s numbers side by side. Keying a document by hand at 6 minutes and a loaded $34.36 an hour costs $3.44. After automation, the remaining labor is S’s subtracted terms, 2.0 minutes, or about $1.15. Extraction at Textract’s Forms plus Tables rate, $0.065 a page on a 3-page document, costs $0.195. The API is about 15 percent of the per-document cost that remains after automation, and about 6 percent of what the document cost by hand. A vendor that is 30 percent cheaper per page saves you less than a reviewer who is 10 percent faster.

Large language models used directly as extractors change the price per page but not that conclusion. Their cost is a token calculation: input tokens per page image times the input price, plus output tokens times the output price. At an illustrative 1,500 input and 300 output tokens per page, and illustrative prices of $3 and $15 per million tokens, that is $0.009 a page, or $3,240 for the 360,000 pages in the example. Check current prices and measure your own token counts, since page images vary widely in token cost by model and resolution. Either way, it is small next to the $274,857 of labor in play.

The number worth computing on the cost side is break-even volume, the documents per year at which the fixed cost pays back:

N* = F / (ρ · (S / 60) · w − g · c)

With the example’s inputs and ρ = 1, the denominator is 4/60 × 34.36 − 3 × 0.065 ≈ $2.10 per document, and N* is about 71,583 documents. At ρ = 0.75 it rises to about 98,499. At ρ = 0.5 it is about 157,855, more than the practice’s entire annual volume. A firm handling 30,000 documents a year with a $150,000 integration should not expect payback in year one under any realistic realization, whatever the per-page price.

def breakeven_docs(fixed_cost, minutes_saved, wage_loaded, pages_per_doc,
                   price_per_page, realization=1.0):
    per_doc = (realization * minutes_saved / 60 * wage_loaded
               - pages_per_doc * price_per_page)
    if per_doc <= 0:
        return float("inf")  # each document loses money; volume cannot fix it
    return fixed_cost / per_doc

def llm_cost_per_page(tokens_in, tokens_out, price_in_per_m, price_out_per_m):
    return (tokens_in * price_in_per_m + tokens_out * price_out_per_m) / 1e6

if __name__ == "__main__":
    wage = 24.05 / 0.70
    for rho in (1.0, 0.75, 0.5):
        n = breakeven_docs(150_000, 4.0, wage, 3, 0.065, rho)
        print(f"rho={rho:.2f}  break-even docs/year: {n:,.0f}")
    print(f"illustrative LLM $/page: {llm_cost_per_page(1500, 300, 3.0, 15.0):.4f}")

The fixed cost F deserves more scrutiny than the per-page price. It includes integration with the document management and core systems, building and maintaining the validation rules from the previous section, labeling a test set, the review UI, change management for the operations team, and in a bank, model risk work. I used $150,000 as a placeholder for a mid-size firm. Your number could be a third of that with a packaged tax product or several times it for a custom lending pipeline, and the equation is more sensitive to it than to anything on the price sheet.

Tax season is where realization comes easiest

Tax firms have a structural advantage over banks in turning saved minutes into dollars: their document work is seasonal, much of the peak is staffed with temporary or overtime labor, and cutting a seasonal hire is a clean, immediate saving. In the equation, that is a high ρ. A bank’s document staff are mostly salaried year-round, so its ρ depends on redeploying people, which is slower and harder to verify.

Peak sizing is a queueing problem, and Erlang C (the M/M/c model) is the standard tool for it. Keep the example practice’s 120,000 documents and assume 70 percent arrive in a 10-week peak, worked at 50 staffed hours a week. That is 8,400 documents a week, or 168 an hour. Keying all of them at 6 minutes is 840 staff-hours a week. Holding the average wait in the queue under 5 minutes takes 18 keyers.

With automation, only the 35 percent routed to review enter the human queue, at 3 minutes each: 147 staff-hours a week, and Erlang C says 4 reviewers hold the same 5-minute wait. Add the 0.5 residual minutes on every document, about 70 hours a week that preparers absorb as part of their normal work, and the peak team still shrinks by more than two thirds. Off peak, the remaining 30 percent of volume works out to about 857 documents a week, which the year-round team handles either way. Nearly all of the realizable saving sits in those 10 weeks.

import math

def erlang_c(arrivals_per_hr: float, service_min: float, servers: int) -> float:
    """Probability an arriving document has to wait (M/M/c)."""
    a = arrivals_per_hr * service_min / 60.0  # offered load in Erlangs
    if servers <= a:
        return 1.0
    head = sum(a ** k / math.factorial(k) for k in range(servers))
    tail = a ** servers / math.factorial(servers) * servers / (servers - a)
    return tail / (head + tail)

def mean_wait_min(arrivals_per_hr: float, service_min: float, servers: int) -> float:
    a = arrivals_per_hr * service_min / 60.0
    return erlang_c(arrivals_per_hr, service_min, servers) * service_min / (servers - a)

def staff_needed(arrivals_per_hr: float, service_min: float, max_wait_min: float) -> int:
    c = max(1, math.ceil(arrivals_per_hr * service_min / 60.0))
    while mean_wait_min(arrivals_per_hr, service_min, c) > max_wait_min:
        c += 1
    return c

if __name__ == "__main__":
    docs, peak_share, peak_weeks, hours_per_week = 120_000, 0.70, 10, 50
    per_hour = docs * peak_share / peak_weeks / hours_per_week
    print("manual keyers:", staff_needed(per_hour, 6.0, 5.0))
    print("reviewers:    ", staff_needed(per_hour * 0.35, 3.0, 5.0))

The queueing view also exposes a risk the averages hide. If peak arrivals run above plan and the review team is sized at the edge, waits grow sharply (the M/M/c wait climbs steeply as utilization approaches 1), and a backed-up queue invites reviewers to rush. Rushed review raises e, and every escaped error costs tₑ. Size the review team with headroom, and give the threshold a peak setting: lowering it a little during a surge trades a slightly higher escape rate for a queue that keeps moving, which is better than the same trade made silently by tired people.

For banks, the equivalent planning question is less about peaks and more about where the freed capacity goes. A lending operation that saves 8,000 hours and redeploys them to faster turn times only realizes value if faster turn times win more loans or cut fallout, which is measurable but has to be measured. Write down which it is before the purchase, because ρ is the term that turns a working system into a profitable one.

Where it breaks

The model above is a planning tool with known weak points, and the biggest is that every input comes from somewhere you have to trust.

Vendor accuracy figures are vendor claims. They are usually measured on curated sets, reported at the field level, and computed after the vendor chose which fields count. Treat them as an upper bound and replace them with document-level accuracy on your own labeled sample. The public research cited earlier points the same way: strong numbers on clean data, noticeably weaker numbers on real enterprise documents.

The threshold simulation assumes calibrated confidences and independent field errors. Neither holds automatically. Many extraction systems are overconfident on exactly the unusual layouts where they are wrong, and errors cluster on poor scans. If you skip the calibration check, the optimal threshold the code reports can be badly off, in either direction.

Documents drift. Tax forms are revised, banks and brokers redesign statements, and a new client brings a new layout. A threshold and a set of validation rules tuned in one season will degrade in the next unless someone owns monitoring. That ongoing cost belongs in F, or in a recurring line next to it, and it is the one most often left out.

Bank model risk rules changed this year, and the change cuts in an unexpected direction. On April 17, 2026, the Federal Reserve, OCC and FDIC issued revised model risk management guidance, SR 26–2, superseding SR 11–7 and the 2021 BSA/AML model statement, with the Fed describing it as most relevant to organizations with more than $30 billion in assets. The revised guidance takes a risk-based approach, excludes deterministic rule-based processes from the definition of a model, and places generative and agentic AI out of scope, while non-generative machine learning models stay in scope. Third-party models remain covered, with expectations to understand design and performance, monitor outcomes, and secure contractual rights to documentation and change notices. The agencies have also signaled a forthcoming request for information on AI-specific model risk.

Read literally, a classic machine learning extractor may sit inside model risk management while a large language model doing the same job sits outside it. That is not a reason to pick the LLM. Out of scope for this guidance is not out of scope for your examiners, your auditors or your own risk appetite, and the same lifecycle discipline (documented design, data lineage, performance testing, monitoring) is what makes the numbers in your business case trustworthy anyway. The validation rules from the review section, being deterministic, fall outside the model definition, which makes them a cheap control as well as a cheap accuracy lever.

Tax firms have their own constraint. Rules on the use and disclosure of tax return information by preparers, including consent requirements, apply to how client documents move through third-party and offshore processing. Settle that with counsel before the pilot, not after.

Finally, the evidence base is thin where it matters most. Public sources describe hours that automation can address, like the 360,000 hours behind COiN, or per-document cost gaps, like the IRS’s 43-to-1 ratio for paper returns. I did not find independent studies of realized, net-of-review savings for bank or tax document AI, measured against a credible counterfactual. The IRS experience is the cautionary version: a strong economic case, and a scanning program that digitized a small fraction of its target because staffing, contractor clearances and operations failed before any model did.

The path forward: measure before you buy

You can fill in every input of the equation in about six weeks, before a contract is signed, and the result will tell you whether to buy, what to negotiate, and which number to watch afterward.

Start with a time study. Pick 200 documents per class from last season’s volume, stratified so the hard ones are represented, and time how long they take today from intake to filed. That gives you t₀ by class and the class mix, which together matter more than any vendor feature.

Then run the candidate system in shadow mode for four weeks on live volume. People keep working as they do now; the system extracts in parallel, and a reviewer labels a random sample of its output field by field. From that sample you get document-level accuracy, a calibration check, the review rate at your chosen threshold, and a measured tᵣ from your own reviewers using the vendor’s actual review screen. Trace the errors that would have escaped to see what they would have cost downstream, which gives you tₑ.

Before the shadow run starts, write down how freed hours will be realized: fewer seasonal hires, fewer contractor hours, specific work the team will take on, or turn time that should convert into volume. Put a number on ρ and say how you will check it. This is the part teams skip and later argue about. Agreeing on it in advance turns the pilot from a demo into an experiment.

With those inputs, run the model, compute break-even realization and break-even volume, and compare them with what you wrote down. If the break-even realization is 0.63 and you can credibly realize 0.5, either negotiate F down, raise volume by adding document classes, or walk away. If it clears comfortably, sign, and keep the shadow-mode sample running as your monitoring.

The last step is proving after the fact that it worked, which is harder than it sounds, because volume, staffing and client mix all change at the same time as the tooling. Measuring whether an AI pilot actually moved the outcome, with a credible counterfactual rather than a before-and-after chart, is the subject of next Thursday’s piece, How to Measure Whether Your AI Pilot Worked. The time-savings model here gives you the prediction. That piece shows how to test it.

Related reading

On the same question from other angles: Fraud Is a Graph Problem Wearing a Classifier’s Clothes and Self-Hosting an LLM Costs $10.42 or $1.49 per Million Tokens. Utilization Decides. For the paid deep dives with full code packages, subscribe at brianjamescurry.substack.com.

About the author

Brian Curry is the founder of Vector1 Research in Kansas City. He has spent 30 years turning data into business decisions, first in search and digital media and for the past decade in machine learning and artificial intelligence/super intelligence. He has led data science and analytics work at enterprises from $4 billion to $125 billion, including Koch Industries, where he was Lead Data and Applied Scientist at Koch Ag & Energy Solutions, Tractor Supply, Vail Resorts, Hallmark, Garmin, AT&T and McClatchy. He founded KC AI Lab, which grew into a 1,500-member AI and machine learning community and business incubator, and created PyCausalSim and Papilon, open-source Python libraries for causal inference and simulation. Teams that want that experience applied to their own agentic AI, causal inference or forecasting work can reach him at vector1.ai.

Sources

Related research