Working with git submodules on CircleCI and Bitbucket

Build Environment
1

Good afternoon folks

We’re using CircleCI with BitBucket repositories.

In repo-one, I have added a git submodule of repo-two (as a folder name docker). I added this using command git submodule add git@bitbucket.org:myteam/repo-two.git docker then git add, git push etc.

My CircleCI pipeline looks (abridged form) like:

my_service_test:
  machine:
    image: ubuntu-1604:202010-01
  steps:
    - checkout
    - run: echo "do some other stuff using the docker folder"

This pipeline is run automatically using the CircleCI API by a “service account” user in CircleCI (just a normal user whose tokens etc. we store in contexts) rather than the individual who pushes code.

When it runs, the docker folder doesn’t exist - the pipeline can’t access resources in it. If I remove the submodule and just add a folder called docker with these resources in, the pipeline works fine.

When I add the line:

  - run: git submodule sync --recursive && git submodule update --recursive --init

I get this error:

Submodule 'docker' (git@bitbucket.org:myteam/repo-two.git) registered for path 'docker'
Cloning into '/home/circleci/project/docker'...
Unauthorized
fatal: Could not read from remote repository.

The service account user running this pipeline has write access in BitBucket on both repo-one and repo-two.

What am I missing to get my submodules working?

2

I’m also running into this issue if there is any helpful feedback.

3

I do not know git in enough detail to fully understand submodules, but my first thought is - does the standard checkout command even consider submodules when it fetches the code base?

If you have a good understanding of git the place to check is within the logs of a pipeline job as the script used to do the git fetch and so you can check what it is truly doing.

There is a thread on this issue from 2015, but I have no idea what if anything has changed in the last few years.

 https://discuss.circleci.com/t/automatically-load-git-submodules/159/8

This link shows a workaround and provides a bit of background regarding SSH key management as that is how the details are retrieved.