Not able to push image to google artifact registry

as google container registry is deprecated soon so we now need to use google artifact registry , so for this we are updating circleci workflow currently its working fine with GCR but i have updated google artifact registry URL in workflow but its able to build the image but not able to push the image to google artifact registry. here is error i am getting
Pushing Docker image: /lender-deployment/lender-deployment:1.1.25
The push refers to repository [
/lender-deployment/lender-deployment]
86f1c9869e07: Preparing
3509119ad1d1: Preparing
b0375484cefc: Preparing
62a918fd9bad: Preparing
7aa78da3e61e: Preparing
9d19eb5c7836: Waiting
e262df620e78: Waiting
a1771f126974: Waiting
19a51c156a1b: Waiting
401d40a3e0d6: Waiting
3ce18771576c: Waiting
e81bff2725db: Waiting
unauthorized: You don’t have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: Authentication methods  |  Container Registry documentation  |  Google Cloud

Exited with code exit status 1

we already granted all permission to service account .
any thing we are missing here or any different authentication method for google artifact registry?

here my code for authentication
steps:

  • checkout
  • run:
    name: Authenticate gcloud CLI
    command: |
    echo $GCP_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
    gcloud auth list
    gcloud --quiet config set project ${<<parameters.gcp-project-id>>}
    # check to see if cluster parameters are defined
    if [ ! -z ${<<parameters.gke-cluster-name>>} ]; then
    if [ $(gcloud container clusters list --format=json | jq “. |
    select(.name == "$<<parameters.gke-cluster-name>>") | .locations | length”) -gt 1 ];
    then
    gcloud --quiet config set
    compute/region ${<<parameters.gke-cluster-compute-region>>}
    else
    gcloud --quiet config set
    compute/zone ${<<parameters.gke-cluster-compute-zone>>}
    fi
    fi
  • run:
    name: Configure Docker with gcloud CLI Credential Helper
    command: |
    gcloud auth configure-docker europe-docker.pkg.dev

build and push command we are using
docker build --build-arg NPM_TOKEN=${<<parameters.npm-token-var>>} -t
$<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG
-t $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:latest
<<parameters.path-to-dockerfile>>

docker push
$<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG
docker push
$<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:latest

Can you please post your script using the </> option in the editor. This will retain all the white spacing and stop the line wrapping.

One thing to note, each run command executes in a separate shell, so none of the setup you do in the first ‘run’ is available to the cloud auth command in the second ‘run’ if the gcloud command is using the shell as a store.

---
description: Build a docker image
parameters:
  gcp-container-registry-url:
    description: The env var containing our container image registry URL
    type: env_var_name
    default: GCP_CONTAINER_REGISTRY_URL
  gcp-project-id:
    description: The env var containing the Google project ID to connect with via the gcloud CLI
    type: env_var_name
    default: GCP_PROJECT_ID
  gke-cluster-compute-zone:
    description: The compute zone of the GKE cluster to deploy to
    type: env_var_name
    default: GKE_CLUSTER_COMPUTE_ZONE
  gke-cluster-compute-region:
    description: The compute region of the GKE cluster to deploy to
    type: env_var_name
    default: GKE_CLUSTER_COMPUTE_REGION
  gke-cluster-name:
    description: The name of the GKE cluster to deploy to
    type: env_var_name
    default: GKE_CLUSTER_NAME
  path-to-dockerfile:
    description: The relative path to the Dockerfile to use when building image
    type: string
    default: "."
  npm-token-var:
    description: NPM token for logging into a private registry
    type: env_var_name
    default: GITHUB_TOKEN
  docker-image-suffix:
    description: The string used to distingish between build-and-push-images
    type: env_var_name
    default: PHALANX
steps:
  - checkout
  - run:
      name: Authenticate gcloud CLI
      command: |
        echo $GCP_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
        gcloud auth list
        gcloud --quiet config set project ${<<parameters.gcp-project-id>>}
        # check to see if cluster parameters are defined
        if [ ! -z ${<<parameters.gke-cluster-name>>} ]; then
          if [ $(gcloud container clusters list --format=json | jq ".[] | \
          select(.name == \"$<<parameters.gke-cluster-name>>\") | .locations | length") -gt 1 ];
          then
              gcloud --quiet config set \
              compute/region ${<<parameters.gke-cluster-compute-region>>}
          else
              gcloud --quiet config set \
              compute/zone ${<<parameters.gke-cluster-compute-zone>>}
          fi
        fi
  - run:
      name: Configure Docker with gcloud CLI Credential Helper
      command: |
        gcloud auth configure-docker europe-docker.pkg.dev
  - run:
      name: Authenticate Docker with Artifact Registry
      command: |
        gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin europe-docker.pkg.dev     
          
  - setup_remote_docker
  - run:
      name: Set build variables from build.json
      command: |
        SERVICE_NAME=$(jq -r '.name' build.json)
        VERSION=$(jq -r '.version' build.json)
        DOCKER_TAG=$VERSION
        if [ "${<< parameters.docker-image-suffix >>}" != "" ]; then
            DOCKER_TAG=$DOCKER_TAG-${<< parameters.docker-image-suffix >>} >> $BASH_ENV
        fi
        if [ "$CIRCLE_BRANCH" = "develop" ] && [[ ! $VERSION == *"-develop" ]]; then
            DOCKER_TAG=$DOCKER_TAG-develop >> $BASH_ENV
        fi
        echo "Service name is $SERVICE_NAME, node package version is" \
          "$VERSION and docker tag will be $DOCKER_TAG"
        TAG_COUNT=$(gcloud container images list-tags \
        ${<<parameters.gcp-container-registry-url>>}\/${SERVICE_NAME} \
        --filter="tags=${DOCKER_TAG}" --format=json --quiet | jq ". | length")
        echo GCR image tag Count is $TAG_COUNT
        if [ $TAG_COUNT != 0 ]; then
            echo "ERROR: image tag exists.  Found total of $TAG_COUNT image " \
            "tags at: $<<parameters.gcp-container-registry-url>>" \
            "/$SERVICE_NAME:$DOCKER_TAG"
            exit 1
        fi
        # Debug: Print Docker Tag and Image URL
        echo "Docker Tag: $DOCKER_TAG"
        echo "Image URL: $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG"
        gcloud auth list
        # tag with generated unique tag and update latest
        docker build --build-arg NPM_TOKEN=${<<parameters.npm-token-var>>} -t \
        $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG \
        -t $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:latest \
        <<parameters.path-to-dockerfile>>
        docker images
        # push all tags
        echo "Pushing Docker image: $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG"
        docker push \
        $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:$DOCKER_TAG
        echo "Pushing latest Docker image: $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:latest"
        docker push \
        $<<parameters.gcp-container-registry-url>>/$SERVICE_NAME/$SERVICE_NAME:latest

error

Pushing Docker image: *******************************/lender-deployment/lender-deployment:1.1.25
The push refers to repository [*******************************/lender-deployment/lender-deployment]
86f1c9869e07: Preparing 
3509119ad1d1: Preparing 
b0375484cefc: Preparing 
62a918fd9bad: Preparing 
7aa78da3e61e: Preparing 
9d19eb5c7836: Waiting 
e262df620e78: Waiting 
a1771f126974: Waiting 
19a51c156a1b: Waiting 
401d40a3e0d6: Waiting 
3ce18771576c: Waiting 
e81bff2725db: Waiting 
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Exited with code exit status 1

Thanks, there is a limit to what I can comment on, the CircleCI environment is able to run the config.yml file without any reported issues.

The error indicates that something is not correct between the gcloud setup and the docker setup.

One thing that stands out is that you are using

gcloud auth configure-docker europe-docker.pkg.dev

to hardcode the docker configuration, while everywhere else you are using a parameter. It will be worth checking that these 2 values are consistent. Sadly CircleCI’s process of ***** out any parameter value sent to the console logs can not be overridden so it makes it harder to debug issues like this.

Yes tried with

  • run:
    name: Configure Docker with gcloud CLI Credential Helper
    command: |
    gcloud auth configure-docker $<<parameters.gcp-container-registry-url>> --quiet --project $<<parameters.gcp-project-id>>

but same error.
alsoi observed one thing when i tried hardcode the value in docker build and push command its able to build and push the image to GAR so it means no permission issue with service account
here is command i used in docker build and push command - working
docker build --build-arg NPM_TOKEN=${<<parameters.npm-token-var>>} -t
europe-docker.pkg.dev/project-id/repo-name/imageName:1.1.26
-t europe-docker.pkg.dev/project-id/repo-name/imagename:latest
.

docker push
europe-docker.pkg.dev/project-id/repoName/imageN:latest

but if we use parameters values its faling

OK, that example makes what you are trying to do clearer and simplifies the issue. It also I think provides an answer.

Parameter values are not shell environment variables, they are instead substituted at the time that the config.yml is pre-processed by the CircleCI system.

So if parameters.gcp-project-id has the value of ‘hello’ when you place $<<parameters.gcp-project-id>> within a shell command what ends up being processed by the shell will be $hello. So the shell ends up referencing the environment variable ‘hello’.

Try the following

run:
name: Configure Docker with gcloud CLI Credential Helper
command: |
      gcloud auth configure-docker <<parameters.gcp-container-registry-url>> --quiet --project <<parameters.gcp-project-id>>

Without the $ character the substitution should place the parameter value into the shell script.

It seems that the docs here

Jump into an example that uses this environment variable substitution ability without any example of just using the parameter values. Instead, simple examples are provided here