The tests in my R package used to be noticeably faster than running them on Travis-CI and pretty much as fast as on my local machine. However, recently they are much, much slower (around 10x slower).
Could you try switching to the Machine executor in a new branch, to see if that helps? I think some new CPUs went in recently, and they were meant to be faster .