Severe performance problems with Xcode 15

Screenshot 2023-11-13 at 19.56.37

This is the same workflow running three times against the exact same code.

It even affects our coverage job which runs after all build/test jobs, which gathers coverage data and uploads it to Coveralls - this job downloads and compiles a standalone Swift tool and even this has has significantly different compile times - sometimes it takes 30s, sometimes it takes double that.

Is there some other issue with these images?