CircleCI became aware this morning of a blog post written by Kevin Burke about CircleCI’s use of 3rd-party analytics tools. We take security at CircleCI very seriously, and want to share a response with our community here. For more details on our security policies and instructions on how to submit a flaw, please visit: https://circleci.com/security/
To be clear: He did not expose a vulnerability.
CircleCI spends a considerable amount of money to hire third-party penetration test firms on a quarterly basis for external security validation. The test conducted in May 2017 tested the security of our web application and API, which is the subject of Kevin’s post.
In his post, Kevin highlighted tools that have access to user data on CircleCI. Like many companies, we use third-party tools to help drive our business, including other tools not mentioned such as Slack, Gmail, Zoom, Zendesk, AWS and Google Cloud. This can be in the form of handling our company email to understanding if user interaction with our Documentation page frequently results in them submitting a Support ticket. If Support tickets are generated frequently after customers visit a certain section of our Documentation, then we know we need to rewrite that documentation.
Part of our philosophy at CircleCI is that we should focus on building features and functionality that drive value to our end users, and not build tools or solutions that already exist (see more in our blog post titled “Write Less Code, Use More Tools”). That being said, we take pains to make sure that our 3rd party tools are vetted and reviewed, and only have access to the data they need for their use case.
What follows is a list of services Kevin mentioned and how we use them. Note: We refer to our outer site (our marketing site, e.g. all the pages you see before signup or log in) and our inner site (e.g. what an authenticated user sees in the build dashboard and settings pages).
Pusher: We use Pusher to enable real-time updates to our dashboards and notifications. If you watch your build status on CircleCI, the stream of build output you see is sent by and made possible by Pusher.
LaunchDarkly: We use LaunchDarkly for feature flagging so we can do contained rollouts and testing of new beta features. LaunchDarkly allows us to try new things quickly, get real user data about that beta feature and is a core enabler of our ability to be continuous.
Amplitude: We use Amplitude (along with Looker and Segment, more on our blog here) for analytics. Amplitude helps us understand user behavior on CircleCI so we can make a better, more useful product.
Appcues: We use Appcues to highlight new features and product updates to customers in-app. We actively chose this because it is a better user experience than sending email or using pop-up modals to alert customers of new features.
Quora: We advertise on Quora and incorporated their tracking here to help us understand whether those advertising dollars are effective (drive people to sign up) or not. We recently created a plan to remove marketing tracking from our inner site and have been removing advertisers like Quora from our inner dashboard. We plan to change Quora tracking to stop after recording sign-up events.
Elev.io: We use elev.io to help surface our documentation and customer support resources to users. We’ve recently moved most of this to Zendesk, and we already had plans to phase this out.
Optimizely: We use Optimizely for A/B testing, primarily for the marketing portion of our website. Prior to this post going live, we had been in discussion to remove Optimizely from the inner part of our site.
Intercom: We previously used Intercom for customer communication, but have since moved to other tools and were already in the process of removing it from the site.
It’s clear that we can do a better job of removing the cruft of tools that we are no longer using for business reasons. This was already a topic of ongoing discussion for a few weeks now, which has since been (understandably) reprioritized.
The reason we have tracking on the inner part of our site is because of the way we create accounts. We currently redirect the user to GitHub/Bitbucket/Google to complete the OAuth process, after which they land directly on our inner dashboard. Using tracking on our inner site helps us determine if a click on the signup button actually results in a successful signup. Going forward, we are planning to change our signup flow by setting up a special page on our outer marketing site that simply records a signup event and then redirects users to our inner dashboard, allowing us to remove tracking from the inner site.
Kevin’s post included bullet points of suggestions for CircleCI and its users. We’ve addressed each one here:
Put all domains used by the above services in /etc/hosts, for each machine and each person on your team that accesses CircleCI.
- This is an option for users and it would not affect the performance of the site. Putting all domains in /etc/hosts will prevent some functionality from working. However, it would impede our ability to provide users with the best experience possible, specifically related to new features. A good example would be the loss of streaming build output that comes from Pusher.
Turn off Test Commands setting.
- Our Product and Engineering teams developed the Test Commands setting to help developers test functionality without actually having to rewrite their circle.yml configuration files. To date, the page only receives about 200 views and only about 20-25 test commands run per day. Gaining access to that page requires gaining access to a user’s secured sessions. We’ve already taken steps to pare down third-party access and initiate work on a tool that would allow org-wide disabling of the feature.
Don’t put any sensitive keys in CircleCI, or in source code; inject them directly into production runtime.
-
CircleCI is designed to store sensitive keys and users should never put sensitive keys in source code.
-
All sensitive data (such as environment variables) are encrypted at rest, transmitted via secure protocols such as HTTPS and only the last four characters are viewable.
-
No one can read the full environment variable via the API after it has been written.
-
All environment variable values returned to an authenticated user via the API are handled in the backend (not with Javascript) and obfuscated to show only the last 4 characters.
Demand that CircleCI take steps to make their dashboard and your source code more secure.
- Our understanding from Kevin is that he wants to create a discussion around industry practices that could be improved versus a specific vulnerability at circleci.com. We appreciate his scrutiny and kicking off this discussion. In addition to taking some short-term steps to reduce our use of third-party services, we are regularly reviewing our site for vulnerabilities.
Don’t load analytics scripts from eight different companies on pages that contain sensitive content, or that have access to the CircleCI “create token” API. Host the dashboard and API on a separate domain from the marketing page.
- We utilize these 3rd-party services to help us create a rich user experience for our customers, as well as inform what features we should be iterating on to continuously improve that experience. We are actively working on segmenting the tracking footprint of our web application from the outer marketing site.
Send an email any time an API access token is created. Add a setting to allow org-wide disabling of API token creation.
- This is a great idea, and is on our product backlog.
Add an option to delete old logs.
- Customers have always been able to create a Support ticket and have logs deleted.
Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain.
- We appreciate Kevin bringing this to our attention. We’re discussing it internally.
Thanks again to Kevin and all of the other researchers in our Security Hall of Fame who have worked with us to keep CircleCI safe for our users. If you have questions about this post or Kevin’s, spot a potential flaw, or just want to talk to us about security, please follow the reporting instructions in our security policy here: https://circleci.com/security/