I am using Google Cloud’s gcloud/gsutil included within Circle CI build environment. When I use “gsutil rsync” it appears like it’s complaining about the slowness of checksum
$ time gsutil -m rsync -r -c -d output gs://
WARNING: You have requested checksumming but your crcmod installation isn’t
using the module’s C extension, so checksumming will run very slowly. For help
installing the extension, please see “gsutil help crcmod”.Building synchronization state…
Starting synchronization
Done.
This operation takes 65 seconds, as opposed to 5 seconds on my laptop. (In both cases, no files are copied, so it’s a noop command just doing checksums.)
As https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod article suggests, I added the following to my dependency section:
sudo apt-get install gcc python-dev python-setuptools && sudo easy_install -U pip && sudo pip install -U crcmod
It successfully installs crcmod-1.7. However I am still getting the slowness warning from gsutil and it is still slow. Any ideas on how to fix this? Could this possibly be because of how Python is installed/compiled?