Chunk - FAQs
Does CircleCI use my data to train the models?
Your source code is not stored nor used for training purposes by CircleCI
What data does Chunk access?
Chunk accesses historical build data and repository contents to identify and fix flaky tests. This is the same information that CircleCI already has access to through your existing CircleCI configuration.
Will my test results be shared with other customers?
Your usage data, including test results, will not be used for any other customer. Each customer’s data remains isolated and is only used to support their own Chunk tasks.
How long are agent logs stored?
We store agent logs for 90 days. This is a fixed retention period that applies to all organizations, regardless of your plan’s standard data retention policy. After 90 days, logs are automatically deleted to keep your workspace at optimal performance.
OpenAI organization verification required. Please verify your organization at…
When encountering the message:
OpenAI organization verification required. Please verify your organization at https://platform.openai.com/settings/organization/generaland see ourcommunity forum for more debugging help
inside an agent task, it indicates that your OpenAI organization verification is still pending.
To fix this: In OpenAI Platform navigate to General > Organization settings and click ‘Verify Organization’ to follow the necessary steps to have your organization verified.
Additional help: OpenAI Organization Verification Guide
What if I can’t get my organization verified when using OpenAI?
If organization verification isn’t possible, you can bypass this requirement by adding an environment variable:
-
Go to Organization Settings > Contexts > circleci-agents
-
Add new Environment Variable:
-
Name: CCI_AGENT_OPENAI_MODEL
-
Value: gpt-5-nano
Invalid OpenAI model specified. Please check the model name and ensure it is available for your account.
When encountering the message:
Invalid OpenAI model specified. Please check the model name and ensure it is available for your account.
you need to make sure your organization has gpt-5 model access.
To verify this: In OpenAI Platform
-
Switch to the project you want to check (top-left dropdown).
-
Go to Settings → Limits in the left-hand menu.
-
This page shows the models and rate limits for your project.
-
If gpt-5 is listed, you have access. If not, that project doesn’t.
-
You can also check in the Playground:
Action required - agent execution error
When encountering the message:
Action required - agent execution error
The agent ran into an error while executing this task. See our community forum for how to solve this error.
Email us at sebastian@circleci.com and we’ll help you figure it out.
Unable to run verification tests
Chunk runs in a Linux Machine VM with basic software installed by default. To verify that a proposed fix resolves flakiness, it re-runs the affected test several times. To do this, the agent may install additional software needed to set up the test environment, using clues from your circleci/config.yml to determine how to run the tests.
You can view these attempts in the CircleCI web app by opening the Chunk Tasks → Select a task logs → Expand All, then searching for “run the command for each attempt.” This will take you to the sections where the agent is trying to run the tests.
Improving verification success
Create an “agent environment” CircleCI YML file. This file lets you copy the environment-setup parts of your existing CircleCI config into a dedicated file for Chunk. Name the file cci-agent-setup.yml and ensure that it is present in your .circleci directory and on the default branch.
Chunk supports all standard CircleCI configuration options. This includes executors, resource classes, caching, contexts, environment variables, service containers, orbs, and everything else you’d use in a normal CircleCI pipeline. If it works in your .circleci/config.yml, it works in cci-agent-setup.yml. For a complete reference of available configuration options, see the CircleCI Configuration Reference.
Example cci-agent-setup.yml files:
Basic Python setup
version: 2.1
workflows:
cci-agent-setup:
jobs:
- cci-agent-setup
jobs:
cci-agent-setup:
docker:
- image: cimg/python:3.12
- image: cimg/postgres:15.3
steps:
- checkout
- run:
name: Install dependencies
command: |
pip install -r requirements.txt
With Caching and Contexts
version: 2.1
workflows:
cci-agent-setup:
jobs:
- cci-agent-setup:
context:
- my-team-context # Includes any secrets/env vars from this context
jobs:
cci-agent-setup:
docker:
- image: cimg/node:18.0
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "package-lock.json" }}
- run:
name: Install dependencies
command: npm install
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ checksum "package-lock.json" }}
With multiple services
version: 2.1
workflows:
cci-agent-setup:
jobs:
- cci-agent-setup
jobs:
cci-agent-setup:
docker:
- image: cimg/ruby:3.2
- image: cimg/postgres:15.3
environment:
POSTGRES_USER: circleci
POSTGRES_DB: test_db
- image: redis:7.0
steps:
- checkout
- run:
name: Wait for DB
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Install dependencies
command: bundle install
- run:
name: Setup database
command: bundle exec rake db:setup
With custom resource class and machine executor
version: 2.1
workflows:
cci-agent-setup:
jobs:
- cci-agent-setup
jobs:
cci-agent-setup:
machine:
image: ubuntu-2204:2024.01.2
resource_class: large
steps:
- checkout
- run:
name: Install dependencies
command: |
sudo apt-get update
sudo apt-get install -y build-essential
Environment Variables & Contexts
Project environment variables: Chunk automatically has access to any environment variables you’ve configured at the project level in CircleCI. You don’t need to recreate or reference these, they’re already available.
Contexts: If you’re using CircleCI contexts to manage secrets or environment variables, simply include the context in your cci-agent-setup job (as shown in the caching example above). Chunk will have access to all variables from that context, no need to manually recreate them.
Testing Your Environment Setup
To build & iterate on Chunk’s environment, navigate to Organization Settings → Chunk Tasks → Identify desired Agent Task → Select [ … ] → Select [ Chunk Environment ]. This page lets you run the contents of your cci-agent-setup.yml file on a specific branch and immediately see the results from those ad-hoc tasks. Use the Custom button to submit a task to Chunk and see the results.
Merge the cci-agent-setup.yml file to your default branch when the results on the environment setup page are satisfactory.
Additional Guidance for Chunk
To improve Chunk’s ability to run tests & produce fixes that are aligned with stylistic/architectural preferences, many users also include a markdown file (claude.md or agents.md) in the root of their repo with instructions for running tests. Chunk should pick this up automatically.
Changing Chunk’s model provider
Currently, Chunk can only have one model provider installed at a time. To change your model provider:
-
Navigate to Organization settings → Chunk tasks from the left navigation
-
Click Edit in Contexts
-
Select circleci-agents, scroll down to Environment variables and delete your current model provider API key
-
Click on Add environment variable and input your new API key information:
-
Environment variable name: Enter OPENAI_API_KEY or ANTHROPIC_API_KEY (depending on your model provider)
-
Value: Enter your model provider API key
-
Click Add environment variable
Task Summary or Pull Request Body Too Long or Poorly Formatted
If you’re noticing that your Chunk Task responses appear incomplete or poorly formatted, this may indicate that your API key needs to be configured for a more capable model.
Identifying the Issue
When viewing a Chunk Task through the CircleCI UI → Chunk Tasks, you might observe these indicators of suboptimal model performance:
-
Inconsistent formatting: The task body lacks properly bolded section headers for Run Summary, Root Cause, Proposed Fix, and Verification
-
Interactive prompts: Chunk Task body ends with open-ended questions like:
“Would you like me to implement the robust wait pattern in the test now, and add a small helper for future tests? If yes, I’ll apply the changes and run the targeted tests.”
To guide you, here are some examples of how a Chunk Task body should look:
Root Cause
These formatting and content issues typically occur when Chunk is using a less powerful language model to analyze and propose fixes for flaky tests.
Resolution
To fix this, you’ll need to ensure your organization has access to gpt-5 and it’s properly verified.
For details on verification requirements, see “OpenAI organization verification required. Please verify your organization at…” in our FAQs.
Important: If your team previously overrode the model used by Chunk, you’ll need to remove that configuration to prevent using a lower-performance model:
-
Navigate to CircleCI web app > Organization Settings > Contexts > circleci-agents
-
Remove the CCI_AGENT_OPENAI_MODEL environment variable from Environment variables section
Start Task button disabled
We’ve noticed some users experiencing an issue where the Start Task button remains disabled in Chunk Tasks > Assign new task, even after filling in all required inputs.Resolution
- Look for a callout in the top navigation bar that says “Your GitHub identity is not verified” and click Authorize.
- Select “Authorize CircleCI App” when asked3
- Try assigning a new Chunk Task again
This should resolve the issue and allow you to proceed with assigning a Chunk task to a project.
