OpenAI’s COBOL Push: Before You Replace Old Software

By Lazar Milicevic · Published October 28, 2026 · 9 min read

You have one piece of software nobody wants to touch: a billing app, an export script, a pricing module. Every Friday someone babysits it with a spreadsheet. OpenAI just reported big gains in migrating COBOL, and the headline sounds like "rewrite is now cheap." It isn't, and the gap between those two statements is where small businesses get hurt.

What OpenAI Actually Reported (and What It Didn't)

OpenAI published its low-resource-language results on October 19, 2026, covering two evaluations: MultiLangBench and a COBOL migration test. The numbers are large, but they come from OpenAI's own test suite, and only one of the two suites is public.

Here is what OpenAI reports:

MultiLangBench asks a model to build a SQLite-compatible database engine from scratch, and OpenAI released that setup and test suite on GitHub. The COBOL migration evaluation is different: OpenAI describes it as hidden parity tests and does not say that suite was released. That means outsiders can't inspect it or reproduce the COBOL numbers independently.

So read these as "OpenAI reports," not as "AI is 97.9% accurate on your code." A benchmark is a controlled set of programs. Your 25-year-old billing module, with its undocumented customer exceptions, is not in that set.

OpenAI itself is careful here. It says applying this capability to production software still requires expert review and testing against the system's business rules and dependencies. Secondhand reports also quote OpenAI calling legacy migration "far from a solved problem." Take the vendor at its word on that part.

Why "Byte-for-Byte Parity" Is the Interesting Part

The most useful detail in this story isn't the score. It's what the score measures. The COBOL migration benchmark asks models to migrate COBOL programs to C++, Java, Kotlin, Python, or Rust, and it requires byte-for-byte parity on hidden tests. The program has to produce the same output, not merely run.

That matters because of how COBOL handles numbers. OpenAI points to COBOL's fixed-point arithmetic: excess decimal digits are discarded unless rounding is explicitly requested. A modern language often rounds by default. So a rewrite can compile, run cleanly, pass a casual smoke test, and still produce different financial results.

Here is a minimal illustration, with made-up round numbers, of the class of bug that parity testing exists to catch:

from decimal import Decimal, ROUND_DOWN, ROUND_HALF_UP

# Example: a 15% discount on a $19.99 line item
price = Decimal("19.99")
discount = price * Decimal("0.15")          # 2.9985

# Legacy-style behavior: excess digits are truncated, no rounding requested
legacy_total = (price - discount).quantize(Decimal("0.01"), rounding=ROUND_DOWN)

# Typical "modern" rewrite: round to cents the way a developer would assume
rewrite_total = (price - discount).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

print(legacy_total, rewrite_total)   # 16.99 vs 17.00

One cent per invoice. Across a few thousand invoices a year, that is a reconciliation headache and, for a regulated customer, a potential audit question. The code in the second version is arguably cleaner. It is also wrong, because the old system's behavior was the business rule, even if nobody wrote it down.

OpenAI's hidden tests reportedly check exactly this sort of detail, including rounding and the handling of missing records. That's a sensible design. It also tells you what to test in your own system.

The Real Risk: Rules That Live Only in the Code

Legacy systems are hard to modernize because documentation is thin, the original developers are often gone, and important business rules may exist only in the code. Translation quality doesn't fix that. A perfect translator preserves the rules it can see; it can't tell you which odd behaviors were deliberate.

Consider the kinds of things hiding in an old billing application:

• A strange rounding rule that matches a contract signed years ago
• A discount tier tied to a customer who no longer exists, but whose rule still fires for others
• A blank field that silently routes an invoice down a different billing path
• A "temporary" fix from a long-gone developer that three downstream reports now depend on

Each of these is a business decision encoded as behavior. If a rewrite changes a charge, you need to explain why to the customer. "The model generated it" is not an explanation that survives an audit or an angry phone call.

For context on why the stakes are high at the large end: OpenAI cites IBM figures that COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions. You don't run a bank. But the lesson scales down: the systems that are scariest to touch are the ones that quietly move money.

A Behavior Sheet You Can Build This Week

The practical move is to document behavior before touching code. I call it a behavior sheet, and it's the cheapest artifact you can make that pays off whether you rewrite, wrap, or leave the system alone.

Pick one narrow workflow. Applying a customer discount is a good one. Then, using approved and sanitized records:

1. Collect a normal case, a missing-field case, and an exception.
2. For each, write down the inputs, the current output, and the reason that output is accepted.
3. If nobody knows the reason, mark it unresolved. Don't guess.

Keep it as plain structured data so a script can use it later:

# behavior_sheet.yaml  (sanitized, example values)
workflow: apply_customer_discount
cases:
  - id: normal-01
    inputs: {customer_tier: "B", subtotal: "199.90", discount_code: "SPRING"}
    current_output: {discount: "29.98", total: "169.92"}
    reason: "Tier B gets 15%, truncated to cents per finance"
    status: confirmed
  - id: missing-01
    inputs: {customer_tier: "", subtotal: "199.90", discount_code: "SPRING"}
    current_output: {discount: "0.00", total: "199.90"}
    reason: "Blank tier falls through to no discount"
    status: confirmed
  - id: exception-01
    inputs: {customer_tier: "C", subtotal: "500.00", discount_code: "LEGACY7"}
    current_output: {discount: "75.00", total: "425.00"}
    reason: ""
    status: unresolved   # nobody knows why LEGACY7 exists

Then use an AI assistant for what it's good at. Ask it to turn those examples into proposed test cases. If you have authorized access to the relevant code, ask it to identify the logic that could explain each result. Treat every explanation as a hypothesis to check, not a finding. This is the bounded, useful job that stronger code-understanding models make faster: locating where a rule lives, explaining a calculation, proposing tests.

Your first deliverable is a four-column table: input, expected result, observed result, reviewer decision.

Run Both Implementations Side by Side

If a replacement exists, even a prototype, run the same cases through both implementations in a test environment and diff the results. The harness doesn't need to be clever:

import yaml
from decimal import Decimal

def load_cases(path):
    with open(path) as f:
        return yaml.safe_load(f)["cases"]

def compare(legacy_fn, candidate_fn, cases):
    mismatches = []
    for case in cases:
        legacy = legacy_fn(case["inputs"])
        candidate = candidate_fn(case["inputs"])
        # Exact comparison on money fields. No tolerance, no rounding forgiveness.
        if {k: Decimal(v) for k, v in legacy.items()} != \
           {k: Decimal(v) for k, v in candidate.items()}:
            mismatches.append({"id": case["id"], "legacy": legacy, "candidate": candidate})
    return mismatches

if __name__ == "__main__":
    cases = load_cases("behavior_sheet.yaml")
    for m in compare(run_legacy_export, run_candidate, cases):
        print("MISMATCH", m)

Two rules for the diff. First, compare money exactly; a tolerance of "close enough" is how the one-cent bug survives. Second, investigate every mismatch, even when the new output looks more sensible. A "more sensible" answer that differs from what customers were charged yesterday is a change in behavior someone has to approve.

Be honest about what this proves. Agreement on three examples is an initial check, not proof that the whole system is safe. It tells you the harness works and the obvious paths line up. Coverage grows by adding cases, especially from real edge conditions: missing records, odd customer IDs, unusual quantities.

Start With the Handoff, Not the Rewrite

For a small team, my first move wouldn't be replacing the old application at all. It would be automating the handoff around it. Take the hypothetical agency whose Friday ritual is: export completed jobs, fix customer names in a spreadsheet, check discounts, type totals into accounting. The pain isn't the COBOL-or-whatever engine. It's the two to four hours of glue work around it.

A scoped version looks like this:

• The agent prepares records. It reads the old system's export, maps customer IDs, and drafts the invoices.
• A deterministic check validates them. Plain code compares totals against the legacy output and flags missing fields. No model judgment in this step.
• A person approves exceptions. Anything flagged goes to a human before it posts to accounting.

That gives the customer a concrete improvement, accurate invoices without the afternoon of copy-paste, while keeping the scope understandable and leaving the legacy engine untouched. It also generates exactly the data you need for a behavior sheet, because every flagged exception is a case someone had to explain.

If you do later evaluate a model-assisted migration, note what the vendor currently offers. GPT-6.1 Sol's standard API pricing is $2 per million input tokens, $0.10 per million cached input tokens, and $10 per million output tokens, with model ID gpt-6.1-sol. It's available in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise, and Edu users, but not yet in regular Chat. Per the official model page, prompts over 272K input tokens are billed at 2x input and 1.5x output for the whole request, and Batch and Flex run 50% below Standard. Check the official model page for current numbers before budgeting, since pricing changes.

Where bizflowai.io Fits

At bizflowai.io, the focus is practical AI automation for solopreneurs and small teams. For a legacy-system problem like this one, the pattern I'd start from is the same: the agent does the preparation, plain code verifies the numbers, and a human owns the exceptions. It's the same "show the action, show the checks, show who handles the exception" standard this whole story points toward, applied to workflows small teams already run every week.

The Standard to Carry Forward

OpenAI's reported gains make old systems easier to investigate, and that is the near-term opportunity. A rewrite becomes a better decision once you can describe the behavior you need to preserve, and a behavior sheet is how you describe it. Remember where the evidence stands: the strongest COBOL figures come from a hidden OpenAI suite that outsiders can't reproduce, so they tell you the direction of travel, not the safety of your migration. Build the sheet, build the diff, and let the evidence from your own cases decide.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows every week.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milićević, a senior engineer who builds AI automations.

Visit bizflowai.io for our services, case studies, and AI consulting.