Bitbucket Agentic Pipelines: Claude Test Fixes

Where does provider: claude go in Bitbucket Pipelines?

Set provider: claude on the named agent under definitions.agents, then call that agent from a pipeline step. Enable Agentic Pipelines and the Claude provider first, and store dedicated provider credentials as secured variables. The current setup guide covers those prerequisites.

This diagnostic example runs manually for an npm-based repository. It preserves failed-test output for analysis and produces a report; it does not push changes or open a pull request. Keep your normal required test pipeline as the merge gate.

image: atlassian/default-image:5

definitions:
  agents:
    investigate-flaky-test:
      provider: claude
      prompt: |
        Read test-output.log and the relevant tests.
        Explain likely sources of intermittent failure.
        Write evidence and proposed next steps to test-analysis.md.
        Do not modify source files or push changes.

pipelines:
  custom:
    investigate-flaky-test:
      - step:
          name: Investigate test failures
          script:
            - npm ci
            - npm test > test-output.log 2>&1 || true
            - agent: investigate-flaky-test
          artifacts:
            - test-analysis.md

For the separate fix-and-draft-PR workflow, use Atlassian's flaky-test remediation setup. A custom fixer additionally needs the documented trigger, repository and pull-request scopes, and required branch context.

Every engineering team has a ghost in the build. It's that one test that passes on Tuesday, fails on Wednesday, and passes again on Thursday for reasons no one can name. So we do the thing we all swore we'd never do: we hit "re-run" until the build goes green, mutter something about timing, and move on. The test never gets fixed. CI trust quietly erodes. And six weeks later, half the team treats a red build the way we treat a "check engine" light — technically important, practically ignored.

Atlassian just shipped something that goes straight for that ghost. As of mid-May, Bitbucket's Agentic Pipelines — the feature that hands repetitive engineering chores to AI agents — now supports Claude as a provider alongside Atlassian's own Rovo Dev, and teams already using Claude Code can plug it directly into Bitbucket Pipelines without extra infrastructure or integration glue. The provider selector belongs on the agent definition. If omitted, it defaults to Rovo Dev; workspace enablement and provider authentication still need to be configured. The example above shows the surrounding pipeline structure.

The Rovo Dev workflow guide provides the broader development context for comparing the tasks you want to delegate.

I'll be honest about why this caught my attention, and it isn't the headline. It's what it says about where Bitbucket is heading.

From "build and test" to "handle the boring stuff"

For years, Pipelines was a CI/CD engine: build, test, deploy, repeat. Agentic Pipelines reframes Bitbucket as an orchestration platform for AI agents that handle the low-value, high-effort, repetitive work surrounding code — things like updating READMEs, triaging security reports, cleaning up feature flags, and generating PR descriptions. The stuff that's too small to schedule and too annoying to enjoy.

The useful question is how much repetitive investigation your own engineers could delegate. Measure time spent diagnosing failures, reviewing the proposed fixes, and cleaning up mistakes. That gives you a business case grounded in your workflow rather than a broad percentage about how developers spend their day.

The flaky-test agent, and the part that matters

Atlassian's flagship example is a flaky-test remediation agent, and the workflow is worth walking through because the final step is the whole point.

You point it at a flaky test. The agent pulls the test's full execution history, failure patterns, and surrounding code context, runs the test to observe the failure firsthand, hypothesizes a likely root cause — timing issues, shared state, environment dependencies, test ordering — proposes a plan, makes targeted changes, and re-runs the test to verify the fix holds.

And then it stops. It opens a draft pull request and tags the person who triggered the fix as the reviewer, who then reviews the change and merges it.

Read that again, because it's the difference between a tool I'd recommend to a client and one I'd quietly steer them away from. The agent does the archaeology nobody wants to do. The human keeps the judgment and the merge button. That's not a limitation Atlassian forgot to remove — that's good design. Agents are extraordinary at tedious investigation. They are not your release manager.

Review the Bitbucket permission boundaries for deployment and packages alongside the repository and pipeline access used by the agent.

The fine print I'd make every client read first

Here's where the trusted-advisor hat goes on. Running Claude Code through Agentic Pipelines falls under Atlassian's third-party product terms, which means your source code, prompts, and logs are sent to Claude Code, and Atlassian states it is not responsible for the privacy, security, or costs involved on that side.

For a lot of teams, that's a perfectly reasonable trade. For a regulated client, a defense contractor, or anyone with strict data-residency commitments, it's a conversation to have before anyone types provider: claude — not after. The capability is excellent. Knowing exactly what leaves your boundary, and getting sign-off to send it, is just due diligence. Rovo Dev is the Atlassian-managed option, but provider choice alone is not proof of data residency or compliance. Review the applicable processing terms and your organization's requirements for either option.

Our guide to the AI context gap explains why selecting useful, approved context belongs in that design.

The Avaratak Take

If you want to try this without turning your CI into a science experiment, here's how we'd roll it out:

• Start with exactly one chore. Resist the urge to automate everything you've ever resented. Pick a single repetitive task — flaky-test cleanup is a great first candidate — and let the agent earn trust there before you expand its job description.
• Keep the human in the loop on purpose. The draft-PR-and-reviewer pattern isn't training wheels; it's the operating model. Treat agent output like a sharp junior engineer's first pass: frequently right, occasionally confidently wrong, always worth a read.
• Settle data governance up front. Decide whether provider: claude or the Rovo Dev default fits your compliance posture, and write that decision down where your team can actually find it.
• Measure the thing. Track how many flaky tests actually get retired, and how many agent PRs you merge versus reject. Good numbers make the case for expanding. Bad numbers mean you spent one line of config learning that — cheaply.

The flaky-test remediation feature remains documented as beta. Current Agentic Pipelines documentation also lists Codex alongside Claude Code as a supported third-party provider. Translation: the orchestration layer is opening up, and the teams that get comfortable now will have a head start when the catalog grows.

The exciting part isn't that your pipeline can suddenly think. It's that it can finally take the chores off your plate so your people can do the work only people should be doing. The agent fixes the flaky test. You decide whether the fix is right.

Always with your best interests first. That part doesn't get automated.

Weighing where AI agents fit in your Atlassian stack — and which guardrails belong around them? That's exactly what we do at Avaratak.

Related reading

• Bitbucket and the Atlassian Software Collection
• Jira Planner: Specs Before Agent Execution