An AI that tests its own work in a live cloud environment
outcome edit-to-verified loop with no human in the middle
Problem
AI-written code for cloud platforms (Microsoft Fabric notebooks driving data pipelines and email dispatch) can look correct and still fail in the real environment. The default working pattern most teams fall into makes this worse: the AI writes code, the developer runs it, hits an error, pastes the error back into the chat, and around again, with the human as the courier between the model and reality. Human verification was the bottleneck, and “the AI says it works” is not evidence.
What I built
The real project was access and environment design: sandbox workspaces inside Fabric, holding test data, that the agent is explicitly allowed to connect to and nothing else. Inside that boundary, an autonomous notebook-testing harness closes the loop. When a piece of work is finished, a tester agent leases an isolated non-production review environment, syncs the feature branch into it via the platform’s git integration API, executes the notebook’s guarded self-test in the actual cloud runtime, and reads back a strict machine-readable result. Never free-text output, which models are tempted to over-trust. The agent sees its own inputs, outputs, and errors, so it iterates on its own work instead of waiting for a human to ferry error messages.
It reacts only to real structured results: confirms with evidence, fixes small defects directly on the feature branch (only when it can reproduce the failure first), or raises a ticket for a human. Production is dry-run-only by design; the single sanctioned real side effect is an email to me.
Techniques that made it work
- Evidence before assertions: the agent must show the run result, not claim success
- Repro-gated fixes: no fix without first reproducing the failure
- A safety contract expressed in the environment itself: dry-run and fire run modes, fail-closed guards, parameter-guarded zero-call self-tests
- Dry-runs as a tripwire: every run first checks what it would fire and where; if anything falls outside the approved sandbox workspaces, the agent stops and revises instead of executing
Outcome
A full edit, commit, cloud-sync, execute, verify loop that runs without a human in the middle, with stronger guarantees than manual testing had. Most testing iterations that used to be manual now happen inside the loop; the developer still reads and owns the final code, but the back-and-forth is gone. Closing the loop, giving the agent the access and environment to check its own work, is the mental model I carry into every project.