If we build multiple times the same code on our local system, it takes very less time as xcode caches the ouput from previous build command. It could be a good optimization for circle ci cache if it can provide additional handling for sharing xcode cache i.e. Derived data.
In current options available, at max we can try to include the derived data path too while saving cache. But! this wont work, as xcodebuild command also notices the file modification date, which changes while pulling the cache.