"npm install" downgrades package-lock.json links from HTTPS to HTTP

[kevinburke]

I check in and push a package-lock.json file that contains links to e.g. “https colon slash slash registry dot npmjs dot com.” When I push this file and run “npm install” using NPM 6, the links in the file update to use “http colon slash slash registry dot npmjs dot com.” (I’m not allowed to post links on this forum; infer what I meant to type)

Do you have any idea why this would happen? It does not happen on my laptop or in any other test environment.

Here for example is a public project that works around this problem: https://github.com/cBioPortal/cbioportal-frontend/blob/master/.circleci/config.yml#L29

[eddiewebb]

Hi @kevinburke - I don’t think I have the full picture, as I am not able to recreate this using the provided CircleCI image.

docker run -it circleci/node:8.9.4-browsers bash
cd ~ && git clone https://github.com/cBioPortal/cbioportal-frontend.git
cd cbioportal-frontend
npm install
cat package-lock.json | grep 'http:'   #no results

The package-lock.json file is still using https for all the registry links. I also ran this on my circleci aws instance and did not have the URLs replaced.

Do you know if this happens consistently? Is there anything else about the environment that you suspect might alter behavior?

[pamo]

Hey Eddie, thanks for looking into the issue. We’re actually using the node 10 image for our repo and can’t think of what would cause the flakey protocol used to resolve dependencies other than the registry being set differently than what we are using on our dev environments (using the default HTTPS).

Here’s our configuration (including the http --> https fix)

references:
  defaults: &defaults
    working_directory: ~/app
    docker:
      - image: circleci/node:10.0.0-browsers

  npm_permissions: &npm_permissions
    run:
      name: set-user-permissions
      command: sudo chown -R $(whoami) $(npm config get prefix)/{lib/node_modules,bin,share}
  npm_restore_cache: &npm_restore_cache
    restore_cache:
      key: dependency-cache-{{ checksum "package.json" }}
  npm_versions: &npm_versions
    run:
      name: "Checking Versions"
      command: |
        npm install --global npm@6
        node --version
        npm --version
  npm_install: &npm_install
    run:
      name: install-npm-dependencies
      command: npm install --silent
  npm_save_cache: &npm_save_cache
    save_cache:
        key: dependency-cache-{{ checksum "package.json" }}
        paths:
          - ./node_modules
  prepare_lockfile: &prepare_lockfile
    run:
      name: "Ignore https to http switches in lockfile due to registry used by circleci"
      command: "git diff --exit-code package-lock.json || sed -i 's/http:/https:/g' package-lock.json"
  diff_lockfile: &diff_lockfile
    run:
      name: "Validate package-lock.json"
      command: git diff --exit-code package-lock.json || (echo -e "Untracked or modified lockfile present after running npm install. See the changes at $(cat package-lock.json | curl -F c=@- https://ptpb.pw | grep url) and include this file in your PR to fix this test" && exit 1)
  repo_restore_cache: &repo_restore_cache
    restore_cache:
      keys:
        - v1-repo-{{ .Environment.CIRCLE_SHA1 }}

version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - << : *npm_permissions
      - << : *npm_versions
      - checkout
      - << : *npm_restore_cache
      - << : *npm_install
      - << : *prepare_lockfile
      - << : *diff_lockfile
      - << : *npm_save_cache
      - run:
          name: gulp-build
          command: npx gulp build
      - save_cache:
          key: v1-repo-{{ .Environment.CIRCLE_SHA1 }}
          paths:
            - ~/app/public
            - ~/app/build
  test:
    <<: *defaults
    steps:
      - checkout
      - << : *npm_restore_cache
      - << : *repo_restore_cache
      - run:
          name: test
          command: npm run test-ci
  deploy:
    <<: *defaults
    steps:
      - checkout
      - << : *npm_restore_cache
      - << : *repo_restore_cache
      - run:
          name: deploy
          command: |
            DEPLOY_ENV=$(echo $CIRCLE_BRANCH | sed -e "s/^integration\///")
            npx gulp deploy --env=$DEPLOY_ENV

workflows:
  version: 2
  build_test_or_deploy:
    jobs:
      - build
      - test:
          requires:
            - build
          filters:
            branches:
              ignore: /integration\/.*/
      - deploy:
          requires:
            - build
          filters:
            branches:
              only: /integration\/.*/

[kevinburke]

Could it have to do with the specific NPM package being installed? That would be weird.

[eddiewebb]

I checked the node 10 image and registry still set to use https globally.

circleci@ed5b84b1dd62:~/repo$ npm config ls -l | grep registry
metrics-registry = "https://registry.npmjs.org/"
; metrics-registry = null (overridden)
registry = "https://registry.npmjs.org/"

I don’t know if upping the provided npm 5.6 to 6.1 would have any impact, but I’m still not able to re-create the behavior. Based on some searching though it seems npm 6+ changes the behavior of package resolution (stricter hash matching) and package-lock.json format. It might be worth taking your ask to a specific npm community, or someone here might have other thoughts.

[kevinburke]

I’ve opened a ticket with NPM as I believe this may be a security issue.