Manually re-running a failed CI job every time an AI agent opens a pull request wastes an engineer's afternoon guessing whether the code broke or the runner did. Wire GitHub Actions to call a sandbox verification step on every pull request, and the check that shows up next to the merge button already tells you which one happened.
- GitHub Actions can trigger AdaptOrch's sandbox verification step on every pull_request event in 2026, separating real code failures from environment failures.
- Setup needs three pieces: a workflow file triggered on pull_request, a repository secret for your AdaptOrch key, and a required status check in branch protection.
- AdaptOrch's readable receipt distinguishes a broken patch from a broken runner, a distinction GitHub's raw pass/fail status does not make.
- A second workflow_dispatch trigger lets you re-verify one commit on demand instead of waiting for a new push.
- Best for teams merging AI-generated patches from Cursor, Claude Code, or Codex who need a merge gate that isn't a guess.
Why this matters
A GitHub Actions check run only reports three states: success, failure, or neutral. It doesn't say why. When an AI coding agent opens a PR and the check goes red, that red X could mean the patch broke a test, or it could mean the runner timed out, a dependency install flaked, or the sandbox lost network mid-job.
In 2026, teams running Cursor, Claude Code, and Codex against real repos hit this constantly: the agent iterates on a failing check that was never a code problem to begin with. AdaptOrch runs the patch in an isolated sandbox before and after applying it, then writes a receipt that names the failure type in words, not a score. That receipt is what gets wired into the GitHub Actions check instead of a bare pass/fail.
| Signal | What it means | What to do |
|---|---|---|
| Real code failure | The patch broke a test or introduced a regression | Send it back to the agent or a human reviewer |
| Environment/runner failure | The sandbox, install, or network failed, not the code | Re-run the job, don't touch the patch |
| Flaky test | Test fails intermittently regardless of the patch | Quarantine the test, don't block the merge |
Getting this distinction into the PR check is the entire point of the workflow below. Teams evaluating CI tools for AI-generated code verification run into the same gap: raw CI output treats every red X the same way, and that's exactly backwards for AI-authored code.
Before you start
- A GitHub repository with Actions enabled and admin access to Settings > Branches and Settings > Secrets and variables > Actions.
- An AdaptOrch account with an API key generated for sandbox authentication.
- The gotcha: GitHub does not expose repository secrets to workflows triggered by pull requests from forks, by default. If your AI agent opens PRs from a fork (common with bot accounts), the verification step will fail to authenticate silently and every check will read as a runner failure, not a code failure. Confirm your agent commits to branches in the same repo before you build branch protection around this check.
Set up the GitHub Actions trigger
- In your repo, open the Actions tab and click New workflow, then choose set up a workflow yourself.
- Name the file
.github/workflows/verify-ai-patch.yml. - Under
on:, set the trigger topull_requestwithtypes: [opened, synchronize]so every new AI-authored PR and every subsequent push to it re-triggers verification.
Expected result: the workflow file appears in the Actions tab and shows as a pending check the next time a pull request opens.
Add your AdaptOrch credentials as a secret
- Go to Settings > Secrets and variables > Actions.
- Click New repository secret.
- Name it
ADAPTORCH_API_KEY, paste the key value, and click Add secret.
Expected result: the secret is listed under Repository secrets, masked in every log line, and available to the workflow as a secrets reference.
Configure the sandbox verification step
- After your checkout step, add a job step that runs the AdaptOrch sandbox check against the PR diff, with the working directory pointed at the checked-out repo.
- Pass the
ADAPTORCH_API_KEYsecret into the step's environment block so the sandbox can authenticate. - Set the step to run on every push to the PR branch, not only when the PR first opens, so a follow-up commit re-triggers the sandbox rather than reusing a stale result.
Expected result: workflow logs show the sandbox spinning up an isolated environment, applying the patch, running the test suite before and after, and writing a receipt.
Surface the receipt in the PR
- Configure the step's output to post as a check run annotation, or as a PR comment if your workflow already uses one for other checks.
- Go to Settings > Branches, edit your branch protection rule, and enable Require status checks to pass before merging.
- Add the AdaptOrch verification check to the required list.
Expected result: every pull request shows a check labeled with the verdict, real failure, environment failure, or pass, before the merge button unlocks.
AdaptOrch's sandbox step is best for teams merging AI-generated patches from Cursor, Claude Code, or Codex who need to know before merge whether a red X means broken code or a broken runner.
Re-verify a single commit on demand
Sometimes you don't want to wait for a new push, you just want to re-run verification against the current head of a PR, say after a flaky runner failure. Add a second trigger for this:
- In the same workflow file, add
workflow_dispatch:under theon:key alongsidepull_request:. - Add an
inputs:block with apr_numberfield so you can target a specific pull request from the Actions tab. - In the job, use the input to check out the correct ref instead of the default trigger context.
Expected result: a Run workflow button appears on the workflow's Actions page, letting you re-verify one commit without pushing an empty commit to force a re-run.
Troubleshooting
- Check never appears on forked PRs. Secrets aren't passed to fork-triggered workflows by default. Either require agent commits to land on branches in the same repo, or switch the trigger to
pull_request_targetwith care around what code that trigger executes. - Secret shows as invalid even though it's set correctly. Secret names are case-sensitive and the reference in the workflow file must match exactly.
ADAPTORCH_API_KEYis not the same asAdaptOrch_Api_Key. - Every patch comes back as an environment failure. This usually means the sandbox can't reach a dependency registry or the working directory path is wrong, not that the AdaptOrch step is broken. Check the receipt's environment section before assuming the tool failed.
- Required check stays pending indefinitely. If your workflow has a
paths:filter and the PR only touches files outside that filter, GitHub never runs the job, so the required check never reports. Broaden the filter or remove it for this workflow. - Same test fails on some runs and passes on others with an identical patch. That's a flaky test, not a real code failure or an environment failure. Confirming the difference between the three is exactly what a tool that catches flaky tests vs real bugs is built to do, and it changes what you do next: quarantine the test, don't send the patch back to the agent.
Customize your workflow
Once the basic gate works, expand it. Add a second job that only runs the AdaptOrch step on PRs opened by your AI agent's bot account, using a condition on the actor, so human-authored PRs skip the extra sandbox run and keep CI fast. Add a scheduled nightly job that re-verifies any PR that's been open more than 24 hours, catching environment drift before someone tries to merge a stale check.
If your team also runs verification locally before code ever reaches GitHub, the flag-environment-failures workflow for Claude Code covers the pre-push side of this same problem.
Set up AdaptOrch verification
Get the sandbox step and receipt format for your GitHub Actions workflow.
FAQ
What is GitHub Actions AI patch verification?
It's a CI workflow that runs an AI-generated code patch in an isolated sandbox before merge and reports whether a failed check means the code is broken or the runner is. GitHub Actions provides the trigger; a tool like AdaptOrch provides the sandbox and the readable receipt.
Does GitHub Actions tell you if a failure is a real bug or an environment problem?
No. A native GitHub Actions check run only reports success, failure, or neutral, with no distinction between a code regression and a runner or dependency failure. That distinction has to come from a separate verification step wired into the workflow.
Can forked pull requests use repository secrets in this workflow?
Not by default. GitHub withholds repository secrets from workflows triggered by pull requests opened from forks, so an AI agent opening PRs from a fork will hit authentication failures on the verification step unless the trigger or permissions are changed.
How do you require the verification check before merging?
Add the check's name to the required status checks list under Settings > Branches, then enable Require status checks to pass before merging on the branch protection rule. Every PR is then blocked from merging until that check reports a pass.
What's the difference between a flaky test and a real code failure?
A real code failure reproduces consistently against the same patch; a flaky test fails intermittently regardless of the patch. Treating a flaky test as a real failure sends the AI agent chasing a bug that doesn't exist in the code it wrote.
How often should the verification step re-run on a pull request?
On every push to the PR branch, not just when it opens. Triggering on both opened and synchronize event types in the pull_request trigger catches follow-up commits from the AI agent without requiring a manual re-run.
Can you re-verify a single commit without pushing a new one?
Yes. Add a workflow_dispatch trigger with a pr_number input, which puts a Run workflow button on the Actions page so you can target a specific pull request on demand.
Does AdaptOrch replace GitHub Actions or run inside it?
It runs inside it, as a step in your existing workflow file. GitHub Actions handles the trigger and the check-run reporting; AdaptOrch handles the sandbox execution and the receipt that explains the result.
One last thing
The forked-PR secrets gap isn't an edge case for AI-agent workflows, it's the default failure mode for anyone whose bot account opens PRs from a fork instead of a branch. Fix the trigger permissions before you turn on required status checks, or you'll spend the first week of 2026 debugging a merge gate that's blocked on an authentication problem, not a code problem.
