Using CircleCI to generate GitHub repos within a job

Helpful Resources Support Solutions
1

Using CircleCI to generate GitHub repos within a job

I found myself needing to generate new repositories a lot, usually around testing. This required the creation of the repo, an initial commit including a valid CircleCI config, and then setting up the project in CircleCI.

With a little legwork I setup a CircleCI project that will take a pipeline create API call, with optional parameters, that then:

  1. Creates a new GitHub repository
  2. Based on optional parameters adds proper files and commits them to the repo (i.e. a README and basic CircleCI config.yml)
  3. Kicks off the first build in CircleCI
  4. Invites collaborators to the repository

The end result is a completely new test repository all correctly setup and ready to go – all from a single quick API call.

Requirements

  1. This guide utilizes Pipeline parameters, so you will need to be on at least version 2.1
  2. You will need a personal access GitHub token with the “repo” scope
  3. You will need a personal CircleCI API token

Configure Environment Variables

The commands that are run in the job that generates the repo require some environment variables to be set. These can be done at the project level, or, if you plan to use it across projects – set these up in a context.

Variable Name Description
CIRCLE_TOKEN This will be a personal API token.
ACCESS_TOKEN A generated GitHub personal access token that has the ‘repo’ scope.

Setup the repository that will generate the new testing repos

  1. Add the config.yml file you are going to add to the newly created repository, this will be added at the root, I am calling mine create-config.yml with the following contents:
version: 2.1

jobs:
  setup:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "hi"
  test:
    docker:
      - image: cimg/base:stable
    steps:
      - run: echo "hi"

workflows:
  version: 2
  setup-and-test:
    jobs:
      - setup
      - test:
          requires:
            - setup
  1. Add a README.md file you are going to add to the newly created repository, I am adding mine to the root with the following contents:
### This is a repo for testing

See the [documentation](https://circleci.com/docs/2.0/) on how to utilize CircleCI!
  1. Create a .circleci directory at the root.
  2. Within .circleci directory create a config.yml file, within that file save the following contents:
version: 2.1

parameters:
  repo-name:
    type: string
    default: testing-demo
  collaborator:
    type: string
    default: none
  api-call:
    type: boolean
    default: false

jobs:
  generate-repo:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - run:
          name: Create new repo
          command: |
            curl -H "Authorization: token $ACCESS_TOKEN" --data '{"name":"<< pipeline.parameters.repo-name >>"}' https://api.github.com/user/repos
      - run:
          name: Add and commit config
          command: |
            git clone https://$ACCESS_TOKEN@github.com/nbialostosky/<< pipeline.parameters.repo-name >>.git
            cd << pipeline.parameters.repo-name >>
            git config --global user.email "test@example.com"
            git config --global user.name "nick bialostosky"
            touch README.md
            cp ~/project/README.md README.md
            mkdir .circleci
            cd .circleci
            touch config.yml
            cp ~/project/create-config.yml config.yml
            cd ..
            git add .
            git commit -m "add init config"
            git config push.default current
            git push
      - run:
          name: Setup project in circleci
          command: |
            curl --location --request POST 'https://circleci.com/api/v1.1/project/github/nbialostosky/<< pipeline.parameters.repo-name >>/follow' \
            --header 'Content-Type: application/json' \
            -u "$CIRCLE_TOKEN:" \
            --data-raw '{
              "first_build": "true"
            }'
      - run:
          name: Optionally add colaborator
          command: |
            if [ "<< pipeline.parameters.collaborator >>" != "none" ]; then
              curl -H "Authorization: token $ACCESS_TOKEN" "https://api.github.com/repos/nbialostosky/<< pipeline.parameters.repo-name >>/collaborators/<< pipeline.parameters.collaborator >>" -X PUT -d '{"permission":"push"}'
            else
              echo "No collaborator passed in API call, skipping"
            fi
            
workflows:
  generate-repo:
    when: << pipeline.parameters.api-call >>
    jobs:
      - generate-repo:
          context:
            - repo-create

With the above, you may need to make some adjustments. First will be to set the proper GitHub user information, so modify the following lines appropriately:

git config --global user.email "test@example.com"
git config --global user.name "nick bialostosky"

In addition, I am using a context (as mentioned previously) to store the variables. If you are using project-level variables you can remove:

context:
  - repo-create

Otherwise, rename the context to the context where you are storing your variables.

:tada: You should now be setup correctly, the above will only build and kick off repoistories when sending an API call, so to test:

curl --location --request POST 'https://circleci.com/api/v2/project/github/<org>/<project>/pipeline' \
--header 'Content-Type: application/json' \
-u '<PERSONAL_API_KEY>:' \
--data-raw '{
  "branch": "main",
    "parameters": {
      "api-call": true,
      "collaborator": "username-here",
      "repo-name": "this-should-work"
  }
}'

(The above API call will need to be updated to have your project, organization, and personal API token)

Additional uses

While the above is fairly straightforward, you can expand upon the idea:

  1. Have multiple config.yml stored at the root and use a pipeline parameter to specify which one to add and commit to the new repo
  2. Since the repo creation is triggered via the API, you can actually leverage the API call in other automations or CircleCI builds

Resources

Here is an example project setup following the above steps:

Here is one of the repositories created by sending an API call to the above project: