GoLang 1.5.1 w/ vendor experiment

go

#1

I’ve been attempting to get a Go lang project (1.5.1) going in CircleCI and then do continuous deployment to Heroku. We are using the vendor experiment and have the packages properly vendored.

I’ve had some assistance in getting 1.5.1 installed but am running in to trouble attempting to run the tests.

This is a sample of the errors:

cd ~/.go_workspace; go test -v $(go list ./... | grep -v /vendor/) -race -p=1
src/github.com/onsi/gomega/ghttp/handlers.go:12:2: cannot find package "github.com/golang/protobuf/proto" in any of:
	/usr/local/go/src/github.com/golang/protobuf/proto (from $GOROOT)
	/home/ubuntu/.go_workspace/src/github.com/golang/protobuf/proto (from $GOPATH)
	/usr/local/go_workspace/src/github.com/golang/protobuf/proto
src/github.com/stretchr/testify/mock/mock.go:12:2: cannot find package "github.com/stretchr/objx" in any of:
	/usr/local/go/src/github.com/stretchr/objx (from $GOROOT)
	/home/ubuntu/.go_workspace/src/github.com/stretchr/objx (from $GOPATH)
	/usr/local/go_workspace/src/github.com/stretchr/objx

My circle.yml looks like this:

machine:
  post:
    - sudo rm -rf /usr/local/go
    - if [ ! -e go1.5.linux-amd64.tar.gz ]; then curl -o go1.5.linux-amd64.tar.gz https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz; fi
    - sudo tar -C /usr/local -xzf go1.5.linux-amd64.tar.gz

dependencies:
  cache_directories:
    - ~/go1.5.linux-amd64.tar.gz
  override:
    - echo "Skipping defaults (go get)"
test:
  override:
    - cd ~/.go_workspace; go test -v $(go list ./... | grep -v /vendor/) -race -p=1

Note that I skip the “go get” because the packages are vendor and because it otherwise tries to get the project package itself.

Also the test command is overridden to facilitate the -p=1 option (to force main_test.go to run first to set up the database).

Finally here is my GoDeps.

{
	"ImportPath": "hw-distribution",
	"GoVersion": "go1.5.1",
	"Packages": [
		"hw-distribution",
		"hw-distribution/brands",
		"hw-distribution/models",
		"hw-distribution/orders",
		"hw-distribution/products",
		"hw-distribution/shipments",
		"hw-distribution/util",
		"hw-distribution/webhooks"
	],
	"Deps": [
		{
			"ImportPath": "github.com/gin-gonic/gin",
			"Comment": "v1.0rc1-136-gfd5d429",
			"Rev": "fd5d4294a5d5223d55fc76d7840f4b0ad91647bb"
		},
		{
			"ImportPath": "github.com/go-sql-driver/mysql",
			"Comment": "v1.2-37-g8afc3be",
			"Rev": "8afc3be42072f568b4868b4afe97bff228a1fb32"
		},
		{
			"ImportPath": "github.com/jmoiron/sqlx",
			"Comment": "sqlx-v1.1-42-g344b1e9",
			"Rev": "344b1e96d6e410a093b7bce40287731a49e253f6"
		},
		{
			"ImportPath": "github.com/manucorporat/sse",
			"Rev": "fe6ea2c8e398672518ef204bf0fbd9af858d0e15"
		},
		{
			"ImportPath": "github.com/mattn/go-colorable",
			"Rev": "40e4aedc8fabf8c23e040057540867186712faa5"
		},
		{
			"ImportPath": "github.com/stretchr/testify/assert",
			"Comment": "v1.0-40-g7c2b1e5",
			"Rev": "7c2b1e5640dcf2631213ca962d892bffa1e08860"
		},
		{
			"ImportPath": "golang.org/x/net/context",
			"Rev": "b026840ad5cb594d38a2b783be1b6644fe5325fb"
		},
		{
			"ImportPath": "gopkg.in/bluesuncorp/validator.v5",
			"Comment": "v5.11",
			"Rev": "98121ac23ff3d764f525d18d8de944d150e3c0fe"
		},
		{
			"ImportPath": "gopkg.in/bsm/ratelimit.v1",
			"Rev": "bda20d5067a03094fc6762f7ead53027afac5f28"
		},
		{
			"ImportPath": "gopkg.in/guregu/null.v2",
			"Comment": "v2.1.1",
			"Rev": "5bc3c32da7d04df1bcea52382b239d6dcf4e6ded"
		},
		{
			"ImportPath": "gopkg.in/redis.v3",
			"Comment": "v3.2.10",
			"Rev": "eef3fd78ef0f78e8406339c27fd46c6af93f24ba"
		}
	]
}

I’m at a loss at this point where to proceed. There is not enough documentation about how the environment is set up for Go on the container.


#2

Hi @cwise,

It looks like this article might be relevant to your issue, since you’re overriding the dependencies step.

Can you please try making the dependencies and test sections of your circle.yml look like this?

dependencies:
  cache_directories:
    - ~/go1.5.linux-amd64.tar.gz
  override:
    - mkdir -p $HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME
    - ln -fs $HOME/$CIRCLE_PROJECT_REPONAME $HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
    - |
      echo 'export GOPATH=$HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME:$GOPATH' >> ~/.circlerc
test:
  override:
    - cd $HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME && go test -v $(go list ./... | grep -v /vendor/) -race -p=1 

-Frank


#3

I basically got the same issue.

can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/brands
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/models
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/orders
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/products
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/shipments
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util
	/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/util
can't load package: package _/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks: cannot find package "_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks" in any of:
	/usr/local/go/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks (from $GOROOT)
	/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks (from $GOPATH)
	/home/ubuntu/.go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks

/usr/local/go_workspace/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/webhooks cd $HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME && go test -v $(go list ./... | grep -v /vendor/) -race -p=1 returned exit code 1

#4

It looks like I made a typo with the last line. Could you try making the test section look like this?

test:
  override:
    - cd $HOME/.go_project/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME && go test -v $(go list ./... | grep -v /vendor/) -race -p=1

Unless you already caught that typo. Do you have any idea why there’s an underscore (after .go_project) in the GOPATH as shown here?

/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution/src/_/home/ubuntu/.go_project/src/github.com/HealthWave/hw-distribution (from $GOPATH)

#6

Here’s what I had to do to get it to work:
Pre-dependency commands
go get github.com/tools/godep

Dependency overrides
echo "Nothing"
(to disable go get ./...)
YMMV. Basically if your github user/org is “XYZ” then you want
$HOME/.go_workspace/src/github.com/XYZ/$CIRCLE_PROJECT_USERNAME

mkdir -p $HOME/.go_workspace/src/github.com/$CIRCLE_PROJECT_USERNAME/ ;

cd $HOME/.go_workspace/src/github.com/$CIRCLE_PROJECT_USERNAME && mv $HOME/$CIRCLE_PROJECT_REPONAME ./ ;

Test Command
cd $HOME/$CIRCLE_PROJECT_REPONAME && godep go test -v $(go list ./... | grep -v /vendor/)

Hope that saves someone the dozens of builds it took me to figure out :stuck_out_tongue:


#7

In order to get my build process working with Go 1.5.3 I created the following circle.yml file:

machine:
  timezone:
    America/Denver
  services:
    - docker
  environment:
    GO15VENDOREXPERIMENT: 1
    GOPATH: /home/ubuntu
    HOME: /home/ubuntu/src/github.com/rmulley

checkout:
  post:
    - cd /home/ubuntu && mkdir -p src/github.com/rmulley && mv project-name /home/ubuntu/src/github.com/rmulley
    - cd /home/ubuntu/src/github.com/rmulley/project-name && git submodule sync
    - cd /home/ubuntu/src/github.com/rmulley/project-name && git submodule update --init

test:
  override:
    - cd /home/ubuntu/src/github.com/rmulley/project-name && make build
    - cd /home/ubuntu/src/github.com/rmulley/project-name && make test

Notice I’m setting the GO15VENDOREXPERIMENT environment variable and moving my project underneath the src directory. I think the only drawback of this current yml file is if CircleCI ever moves to another OS (not Ubuntu) it may break. There may be a way to improve upon that.


#8

Can you show your make build and make test commands?


#9

My Makefile is below. It’s pretty basic, but is nice to ensure everyone is running the same build and test commands. I have various internal packages underneath project-name/src, but if your code lives at the top level where your vendor folder exists you may want to adjust the test command to avoid that folder.

GO ?= go
export PATH := $(PATH):/usr/local/go/bin
export GOPATH := $(GOPATH):/home/ubuntu/project-name
export GO15VENDOREXPERIMENT := 1

all: build

build:
	$(GO) fmt ./src/...
	$(GO) tool vet -shadow=true ./src
	$(GO) install ./src/...

clean:
	rm -rf pkg/darwin_amd64/
	rm -rf vendor/pkg/darwin_amd64/
	rm -rf pkg/linux_amd64/
	rm -rf vendor/pkg/linux_amd64/

test:
	$(GO) test ./src/... -v -cover -race

#10

Thats a great circle.yml.

I took some cues from https://robots.thoughtbot.com/configure-circleci-for-go and put much of the directory repetition into env vars, made it more readable:

machine:
  environment:
    REPO: "$GOPATH/src/github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME"

#11