Environment failures and real code bugs look identical on a red CI dashboard, but they leave different evidence once you rerun the same patch against the commit that existed before it.
- Environment failures and real code bugs can be separated by running the same patch twice, before and after, in an isolated sandbox.
- AdaptOrch produces a plain-language receipt that labels a failure as runner-caused or code-caused, not a score.
- A single CI red X does not tell you which bucket a failure belongs to; the pre-patch rerun does.
- Flaky tests, dependency drift, and network timeouts are the three most common disguises for a fake code bug in 2026.
Why this matters
AI coding agents, Cursor, Claude Code, Codex, turn out patches faster than any human reviewer can read them line by line, and in 2026 most teams merge on green CI, not on a careful diff review. The moment a test suite fails, someone has to answer one question fast: did the patch break the code, or did the runner have a bad day. Guess wrong in one direction and a real bug ships because CI is flaky anyway. Guess wrong in the other and a good patch dies because a package mirror timed out for ten seconds.
AdaptOrch exists to automatically flag environment failures before either mistake happens, by running the patch in an isolated sandbox instead of trusting the CI log at face value.
Can environment failures be separated from real code bugs?
Yes. The separation comes from comparing two runs, not from reading one failure message. Run the test suite on the commit before the patch is applied, then run it again after. If the pre-patch run also fails, the failure predates the patch and belongs to the runner or the environment, not the code.
| Signal | Environment failure | Real code bug |
|---|---|---|
| Runs without the patch (base commit) | Also fails | Passes clean |
| Error type | Timeout, connection refused, missing package, permission denied | Assertion error, wrong return value, exception in changed files |
| Reproducibility across identical reruns | Inconsistent, passes sometimes and fails other times | Consistent, fails every time on the same commit |
| Stack trace location | Points into the runner, network client, or package manager | Points into the files the patch touched |
| Fix | Retry the job, pin a dependency version, or fix runner config | Rewrite or revert the patch |
A stack trace that lands inside a package manager or a network client is runner territory. A stack trace that lands inside the files the patch actually touched is code territory. Read one CI log in isolation and both can look the same, a red exception with a scary trace. Read both runs side by side and the pattern shows up in under a minute, even in the noisier, more agent-heavy pipelines running in 2026.
AdaptOrch's receipt runs exactly this comparison automatically and prints the verdict in words, code failure or runner failure, instead of a pass or fail score that hides which bucket the exception belongs to. The receipt also carries a field called correctness_claim, and it is set to false by design: the tool tells you where the failure came from, not whether the patch is right.
“A verdict in words, not a score.”
Why the split breaks down
The mechanism above works cleanly in theory. In practice, a handful of conditions make the split harder to read at a glance:
- Shared CI runners under load introduce timeouts that have nothing to do with the diff.
- Dependency locks drift between an agent's sandbox and the CI runner, producing version mismatches that look like breakage.
- Non-deterministic test ordering surfaces state leaks that fail on one run and pass on the next; see the difference between flaky tests and real bugs for the reruns that catch this.
- Network calls to package registries or external APIs fail intermittently, especially in agent sandboxes with tighter egress rules than a developer laptop.
- Cached build artifacts from a prior run get reused incorrectly, producing stale failures unrelated to the current patch.
- Resource limits on ephemeral containers kill long test runs mid-execution, printing an error that reads exactly like a crash in the code.
Is a flaky test the same as an environment failure?
Not always. A flaky test can fail because of the code, a genuine race condition in the logic under test, or because of the runner, resource contention on a shared machine. The label environment failure only applies once a rerun on the base commit reproduces the same failure without the patch applied.
Does rerunning a test twice prove it's an environment issue?
Two identical reruns on the same commit that disagree with each other is a strong signal of a runner problem, but it is not proof by itself. The step that actually settles it is a clean run against the pre-patch commit, which is why teams that auto-merge AI-generated code without that second run inherit both kinds of mistakes at once.
Can CI logs alone separate a runner failure from a code failure?
CI logs alone rarely settle it, because a timeout or a stack trace can originate in either place and read the same way on a dashboard. The decision needs the pre-patch commit run for comparison, which is what a sandbox replay produces without anyone manually rerunning the suite.
See the verdict on your last patch
AdaptOrch runs it before and after in a sandbox, then writes the receipt.
FAQ
What's the fastest way to tell if a test failure is environment or code?
Rerun the same test suite against the commit before the patch was applied. If it fails there too, it's an environment failure; if the pre-patch run passes clean, the patch caused it.
Is a timeout always an environment failure?
A timeout is usually an environment failure, since code logic rarely produces a hard timeout on its own. Confirm it by rerunning on the base commit; if the timeout disappears, the patch is fine and the runner or network caused it.
Do AI coding agents cause more environment failures than human-written patches?
AI coding agents don't inherently cause more environment failures, but agent sandboxes often run under tighter network and dependency rules than a developer's own machine, which raises the odds of a runner-side failure landing on the same PR as an AI-generated patch.
Can a flaky test hide a real bug?
Yes: a test that fails intermittently can still be catching a real, intermittent bug in the code, not just a runner problem. Dismissing every intermittent failure as flaky is how real bugs from AI-generated patches make it to production in 2026.
Does AdaptOrch guarantee the code is correct?
No, AdaptOrch's receipt separates code failures from runner failures; it does not claim the code is correct. The correctness_claim field on the receipt is set to false by design.
How many reruns prove a failure is environment-caused?
There's no fixed rerun count that proves it. The reliable check is one clean run against the pre-patch commit, not repeated identical reruns of the same patched commit.
Should an environment failure block a merge?
No, once a failure reproduces on the pre-patch commit, it's a runner issue and shouldn't hold the PR. Only a failure that shows up exclusively after the patch should block the merge.
One last thing
The single highest-signal check costs nothing extra: rerun the failing suite on the untouched base commit before writing a single line of triage. Teams skip this step constantly in 2026 because it feels redundant when the CI dashboard is already screaming, and it is the one step that actually answers the question this page is titled after. AdaptOrch runs that comparison automatically on every patch, before it ever writes the receipt, so the environment-versus-code call is made from evidence instead of a guess about which mirror was slow that morning.
