CircleCI 2.0 is moving from AUFS to BTRFS


#1

CircleCI 2.0 will be switching file systems on Monday, December 11, 2017 from AUFS to BTRFS. After extensively testing performance we have found that any projects using heavy I/O operations should see a performance improvement without any action required. The underlying CPU is not changing and therefore your cache will not be impacted. We do not expect any projects to be negatively impacted by this change.

EDIT: CircleCI 2.0 will be switching file systems back from BTRFS to AUFS. We have not seen the performance gains we were hoping to see when fully rolled out in production. As a result, the change back to AUFS will be made tomorrow morning PST, Wednesday, December 20, 2017.


#2

Interesting. Out of curiosity, are you using it for its advanced features, as well as the performance gain? I should imagine things like de-duplication would be extremely useful where packages/binaries are going to be installed by many users in an identical fashion. Not sure what de-dup does to performance though… :smiley_cat:


#3

Not at this time.


#4

Hi! I’ve been experiencing a lot of random failures in my project since the BTRFS upgrade.

Here’s my master branch workflow list: https://circleci.com/gh/decidim/workflows/decidim/tree/master

My project is a rails engine that generates a rails application (on the first step of the workflow) and then runs unit tests and also acceptance tests with headless chrome (via a custom docker image).

Rails and chrome do use the filesystem extensively, so it makes sense they both could be correlated. Errors seems to happen in the acceptance tests (chrome) but I’ve also seen some happening in unit tests, mostly random, but some of them tend to happen in similar places.

Have you seen any other projects behaving like this since the upgrade? Thanks!


#5

No, I have not. It might be a coincidence. Maybe your containers need more resources.


#6

Is it safe to say that this change definitely does not affect Circle 1.0?


#7

That is 100% correct.


#8

CircleCI 2.0 is moving back from BTRFS to AUFS. The original post has been updated with more information.


#9

We have indeed been seeing similar issues:

We also have a Rails app with acceptance tests (Firefox and PhantomJS). The graph shows that we have some pre-existing flaky tests, but it’s been very noticeably worse recently.

I’m hoping that the AUFS rollback will improve the situation!


#10

Wow that’s brutal.


#12

where do you see charts like that?


#13

Build Insights


#14

awesome thanks. i had no idea that page existed (been using circleci 1.0 for 2.5 years and just switched to 2.0 :slight_smile:


#15

Curious what errors you were getting while CircleCI 2.0 was using BTRFS?


#16

Why not use the Docker overlay2 driver?