Compatibility with Phabricator

shipped

#1

We use Phabricator for code reviews but are having a hard time integrating with CircleCI because of this problem:

CircleCI does not git fetch --tags, and does not fetch the tag explicitly when building a tag. It does resolve the tag, but never actually downloads the corresponding commit. This means that tags and other refs which are not ancestors of a branch head (which is to say: everything in the staging area) are not buildable. I don’t think I can work around this because we need the commit to be fetched in order to make it to the checkout: post: step.
https://secure.phabricator.com/T9456#140502

Is there a timeline to resolve this?


#2

At Bloc, we are also eager to see better compatibility between these two great services!


#3

Thanks for the feedback @dpaola2, can you make sure to “like” this request by clicking on the little heart. This is how we gauge demand for issues :slight_smile:


#4

I am currently working on this. My current idea is to subscribe to git tags on github and make each phabricator tag a push --force to a dummy branch so circleci can build. Will let you know how it works out.


#5

My current idea is to subscribe to git tags on github and make each phabricator tag a push --force to a dummy branch so circleci can build.

One minor concern is that this approach has the potential to race if Harbormaster triggers the build faster than you push the dummy branch.


#6

Here’s the specific problem we’re running into:

REPRODUCTION STEPS

  • Make a local commit.
  • Push it into the remote to a tag ONLY (say, tag1). Do not push it to a branch. So tag1 exists in the remote, but is not the ancestor of any branch head in the remote.
  • Call the /project/:username/:project/tree/:branch API method, intending to build the tag.
    • Pass the tag name (refs/tags/tag1) as the revision parameter.
    • Pass any :branch (for example, master). This value appears to be ignored when revision is present.

EXPECTED BEHAVIOR

CircleCI builds the tag (desired, allows Phabricator integration).

OR

CircleCI fails with an explicit error like “building tags is not supported”.

ACTUAL BEHAVIOR

CircleCI accepts, schedules, and executes the build. It successfully resolves the tag to the correct commit during setup, executes partially, then fails in the “Checkout using deploy key…” step because the tag was never fetched.

EXAMPLE

This is an example of a build which has failed in this manner:

https://circleci.com/gh/epriestley/poems/28

Here’s what it looks like in the UI:

Note that the commit has been resolved correctly (upper left) but the build has failed because it is unable to find the commit (bottom center). The hashes are the same. The commit exists in the repository with a tag, but is not the ancestor of any branch head:


#7

Explicit fix/feature request: Be able to run the command (or do it for me) git fetch --tags before you resolve things.

ubuntu@box69:~/myrepo$ git fetch -av
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
giFrom github.com:signalfuse/myrepo
 = [up to date]      master     -> origin/master
 = [up to date]      release    -> origin/release
tubuntu@box69:~/myrepo$ git show phabricator/diff/13592
fatal: ambiguous argument 'phabricator/diff/13592': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
ubuntu@box69:~/myrepo$ git version
git version 1.8.5.6
ubuntu@box69:~/myrepo$ git fetch --tags
remote: Counting objects: 19, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 19 (delta 9), reused 11 (delta 1), pack-reused 0
Unpacking objects: 100% (19/19), done.
From github.com:signalfuse/myrepo
 * [new tag]         phabricator/diff/13590 -> phabricator/diff/13590
 * [new tag]         phabricator/diff/13591 -> phabricator/diff/13591
 * [new tag]         phabricator/diff/13592 -> phabricator/diff/13592
ubuntu@box69:~/myrepo$ git show phabricator/diff/13592
# See stuff

#8

Because everything is a hammer when I have a nail, I’ve figured this out and actually have it working. It was so easy.

First, modify your circle.yml file like so

general:
  branches:
    ignore:
      - /phabricator_diff_branch_.*/

notify:
  webhooks:
    - url: https://me.execute-api.us-east-1.amazonaws.com/prod/me
  1. Turn off phabricator’s autoclose commits feature
  2. Enable phabricator’s staging area feature
  3. Enable github’s SQS integration
  4. Write daemon that listens for SQS github messages that are the creation of a tag like “phabricator/diff/%d”
  5. Push a branch to github that looks like “phabricator_diff_branch_%d” when you see the tag “phabricator/diff/%d”
  6. curl -H "Content-Type: application/json" -d '{"revision":"refs/tags/phabricator/diff/13587", "build_parameters": {"diffID":"13587"}}}' -X POST https://circleci.com/api/v1/project/signalfuse/ourproject/tree/phabricator?circle-token=XYZ
  7. Create an AWS lambda function that adds a message to an SQS queue. Put it in your webhook and enable it as a API gateway.
  8. Have a daemon watch this SQS queue, and when a build result comes in
  9. Remove the phabricator tag
  10. Remove the phabricator branch
  11. Update the diff
  12. Make AWS permissions work correctly for all of this.

Bonus points for correct usage of robot accounts.

A lot of this was because there is an inbound network restrictions on our phabricator instance. It would be great if

  1. CirlceCi had shell script notifications so I could delete the lambda endpoint and talk to SQS directly
  2. CircleCi supported fetching/building tags


#9

We got an out-of-band notification that this should work now. I’m still having various kinds of trouble with it. Here’s what I did, and one problem I encountered:

The API build resolves the tag correctly but fails to fetch the commit:

I was earlier able to get somewhat further (I don’t understand why) but builds hung indefinitely when running git show, while apparently equivalent builds of branches worked correctly:


#10

@epriestley: Isn’t it expected for git show to hang? Won’t it pipe it to less or something like that if the output doesn’t fit on the terminal?


#11

@cep21: Ah, yeah – there might be some pager-related things happening there which could explain that particular issue, and the succeeding cases may have just had short git show text. That would explain the spooky behavior and leave things with well-defined tag failures.


#12

I am happy to say that first-class tag support has been shipped. Please check out this doc section for information on triggering a tag build.

The checkouts in cases where a commit is only present on a tag and not on a branch, which are very common with Phabricator, will now work correctly.

:tada:


#13

Thanks, tag builds work great.

However, I’m having difficulty getting build_parameters to work with tag builds. Are they expected to work? I notice they aren’t mentioned in the documentation. We need both tag builds and build_parameters.


#14

Thanks for pointing this out. Could you please give me an example of a request that you are making to the API? Please omit the sensitive information.


#15

Sure. Here’s an example request:

$ curl -X POST --header "Content-Type: application/json" -d '{
>   "tag": "phabricator/diff/402",
>   "build_parameters": {
>     "EXAMPLE_PARAMETER": "example_value"
>   }
> }' https://circleci.com/api/v1/project/epriestley/poems?circle-token=<REDACTED>

Here’s the first part of the response:

{
  "compare" : null,
  "previous_successful_build" : {
    "build_num" : 54,
    "status" : "success",
    "build_time_millis" : 43824
  },
  "build_parameters" : null,
  "oss" : true,
  "committer_date" : null,
  "body" : null,
  "usage_queued_at" : "2016-03-20T13:58:53.461Z",
  "retry_of" : null,
  "reponame" : "poems",
  "ssh_users" : [ ],
  "build_url" : "https://circleci.com/gh/epriestley/poems/59",
  "parallel" : 1,
  "failed" : null,
  "branch" : null,
  "username" : "epriestley",
  "author_date" : null,
  "why" : "api",
  "user" : {
    "is_user" : true,
    "login" : "epriestley",
    "name" : "Evan Priestley",
    "email" : "epriestley@javelinjs.com",
    "avatar_url" : "https://avatars.githubusercontent.com/u/102631?v=3"
  },
  "vcs_revision" : "1ef0e21e517a29d5b09ec9ecd89cfda19e95d5a7",
  "vcs_tag" : "phabricator/diff/402",
...

Note that build_parameters is empty.

If I use an otherwise similar “revision” request instead:

$ curl -X POST --header "Content-Type: application/json" -d '{
>   "revision": "4dcd4c492b31d82f0fec6f447cf3719b9cc079e0",
>   "build_parameters": {
>     "EXAMPLE_PARAMETER": "example_value"
>   }
> }' https://circleci.com/api/v1/project/epriestley/poems/tree/master?circle-token=<REDACTED>

…the build_parameters are populated:

{
  "compare" : null,
  "previous_successful_build" : {
    "build_num" : 58,
    "status" : "success",
    "build_time_millis" : 41098
  },
  "build_parameters" : {
    "EXAMPLE_PARAMETER" : "example_value"
  },
  "oss" : true,
  "committer_date" : null,
  "body" : null,
  "usage_queued_at" : "2016-03-20T14:01:23.514Z",
  "retry_of" : null,
  "reponame" : "poems",
  "ssh_users" : [ ],
  "build_url" : "https://circleci.com/gh/epriestley/poems/60",
  "parallel" : 1,
  "failed" : null,
  "branch" : "master",
  "username" : "epriestley",
  "author_date" : null,
  "why" : "api",
...

#16

This should be fixed now. Could you please try it out again?


#17

Thanks, things look good now. We’ve landed CircleCI support.


#18

Thanks @epriestley and @alexey . Great work! Now I can replace my hacked together custom solution with something a bit more stable.


#19

Added a doc on configuring Phabricator to run builds on CircleCI: https://circleci.com/docs/1.0/phabricator/


#20

Is it only possible to use CircleCI with Phabricator if the git repo is hosted on GitHub?

We are using a Phabricator hosted repo, so we get an url like: ssh://test-abc123@vault.phacility.com/diffusion/2/validation.git

I tried to follow the guide but got stuck at step 1. :slight_smile: