*TL; DR *
Check if CircleCI have change the image you are building with, if they have, your nodejs version or other parts of the toolchain will probably have changed and this can severely mess up the React native build process
Okay, we just wasted two days of debugging why our React Native 0.59.0 build failed. We hadn’t changed any code, we just rerun the last successful one, but it didn’t work any more.
The error displayed was: (As you can se it was truncated in the output but shown in its entirety below it.)
Running script 'Bundle React Native code and images'
the transform cache was reset.
❌ error: File /Users/distiller/Library/Developer/Xcode/DerivedData/xxx/BuildProductsPath/Release-iphoneos/xxx/main.jsbundle does not exist. This must be a bug with
** ARCHIVE FAILED **
...
[05:25:02]: ▸ + echo 'error: File /Users/distiller/Library/Developer/Xcode/DerivedData/xxx/Build/Intermediates.noindex/ArchiveIntermediates/xxx/BuildProductsPath/Release-iphoneos/xxx.app/main.jsbundle does not exist. This must be a bug with'
[05:25:02]: ▸ error: File /Users/distiller/Library/Developer/Xcode/DerivedData/xxx/Build/Intermediates.noindex/ArchiveIntermediates/xxx/BuildProductsPath/Release-iphoneos/xxx.app/main.jsbundle does not exist. This must be a bug with
[05:25:02]: ▸ + echo 'React Native, please report it here: https://github.com/facebook/react-native/issues'
[05:25:02]: ▸ React Native, please report it here: https://github.com/facebook/react-native/issues
[05:25:02]: ▸ + exit 2
It was almost 3 months since we did a build of our app last time, so we started looking at the dependencies and circleci release log.
We checked for dependency changes/issues in:
- Yarn with yarn.lock
- Gem with Gemfile.lock
- CocoaPods
- React Native, native dependencies
- And more, so much more!
We tested a lot but didn’t find anything there.
Then, we started looking at the Docker images & xcode images. Our configuration:
Android (yeah, it’s an old version):
`image: circleci/android:api-28-node
IOS:
xcode: '10.2.0'
https://circle-macos-docs.s3.amazonaws.com/image-manifest/build-534/index.html
Since there was not “tags” for them, as for other docker hub images (e.g. python 3.7-stretch) we were crazy to think that they would not change without getting a new version number.
By parsing and using diff on old success logs vs the current failed builds we figured out that the image for circleci/android:api-28-node had changed nodejs version and that the image for xcode: '10.2.0' also had change nodejs version.
After SSH into the server, and running the fastlane manually, it was clear that the main.jsbundle was not generated with the nodejs version being used in the image. To understand more we looked at the xcodebuild log, a log that is linked in the output of the react native build:
[05:25:02]: ⬆️ Check out the few lines of raw `xcodebuild` output above for potential hints on how to solve this error
[05:25:02]: 📋 For the complete and more detailed error log, check the full log at:
[05:25:02]: 📋 /Users/distiller/xxx/ios/output/buildlogs/gym/xxx.log
In that log we saw a new error:
app/node_modules/react-native/cli.js bundle --entry-file index.js --platform ios --dev false --reset-cache --bundle-output /Users/distiller/Library/Developer/Xcode/DerivedData/xxx/Build/Intermediates.noindex/ArchiveIntermediates/xxx/BuildProductsPath/Release-iphoneos/xxx.app/main.jsbundle --assets-dest /Users/distiller/Library/Developer/Xcode/DerivedData/xxx/Build/Intermediates.noindex/ArchiveIntermediates/xxx/BuildProductsPath/Release-iphoneos/xxx.app
warning: the transform cache was reset.
error `fsevents` unavailable (this watcher can only be used on Darwin). Run CLI with --verbose flag for more details.
The error was: “error fsevents unavailable”. Strange, since this was working before and we were still on Darwin. We started to log the nodejs version that was used to compare it with historical git commits (by CircleCI) to the CircleCI image Dockerfiles.
We saw that three months ago the nodejs version had been bumped from v10 to v12 together with some other potentially scary changes.
For the Android build we could revert back to the previous image by using the exact hash (why no real tags?), circleci/android@sha256:89f51dc9d6fdfb7006fe8d8713c2d0d135da823485827150df081b95273a73d3 however for MacOS there is no such option as the MacOS images are not versioned or tagged.
The change worked for Android, so we tried to find a way to accomplish the same thing in the iOS build. But we couldn’t find a way to specify or change node version for a specific Xcode image, but in the process found numerous threads asking how to do it along with some hacks to put in the circleci/config.yaml.
We ended up using the following hack as a starting point:
- run:
name: Force specific nodejs version
command: |
export NODE_VERSION=$(cat .nvmrc)
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.35.1/install.sh | bash -s -- --no-use
echo 'export NVM_DIR=$HOME/.nvm' >> $BASH_ENV
echo 'source $NVM_DIR/nvm.sh' >> $BASH_ENV
echo 'nvm install v10.2.0' >> $BASH_ENV
echo 'nvm alias default v10.2.0' >> $BASH_ENV
And it worked!
We are happy that it’s now up and running again but we’re very unhappy that images used for existing builds can change with very little information and no way of easily summarising any differences in the build environment since last time.
We also have a hard time understanding why images are not better versioned and tagged, and why you can’t specify node version (or specify an older image at least) in the iOS build flow.