Nov 9, 2017

Go 1.10 cmd/go: build cache, test cache, go install, go vet, test vet

Email sent to golang-dev by Russ Cox (rsc) on Nov 3

It's been a busy week. After about a month of debugging and fixing prerequisites for content-based staleness, getting that submitted on Tuesday unblocked a bunch of other work that has now landed and will no doubt need fine tuning based on feedback from all of you.

Build cache

The go command now maintains a cache of built packages and other small metadata (CL 68116 and CL 75473). The cache defaults to the operating system-defined user cache directory but can be moved by setting $GOCACHE. Run "go env GOCACHE" to see the current effective setting. Right now the go command never deletes anything from the cache. If the cache gets too big, run "go clean -cache" instead of deleting the directory. That command will preserve the cache's log.txt file. In a few weeks I'll ask people to post their log.txt files to a Github issue so that we can evaluate cache size management approaches.

The main effect of the build cache is that commands like "go test" and "go build" run fast and do incremental builds always, reusing past build steps as aggressively as possible. You do not have to use "go test -i" or "go build -i" or "go install" just to get fast incremental builds. We will not have to teach new users those workarounds anymore. Everything will just be fast.

Test cache

The go command now also maintains a cache of passing test results (CL 75631). The cache only applies to "go test package-list", not to "go test" without a package list. So if you're working in a directory and just keep running "go test", none of those are cached. But if you test an explicit list of packages ("go test all", "go test mydir/...", "go test math", "go test ."), then caching kicks in. (See "go help test" for more about that command-line distinction.) The cache also only applies when all the test arguments are recognized as cacheable, which is defined as -cpu, -list, -parallel, -run, -short, and -v. If these conditions are met and the cache has a result from a previous run of an identical test binary with identical arguments, go test will redisplay that result instead of rerunning the test. When doing so, it replaces the usual time elapsed "0.123s" in the final summary line with "(cached)".

For example, if you run "go test -short std" and then you modify a file in math/big and run "go test -short std" again, all tests that don't import math/big (transitively) will show cached results without even being rebuilt. Only the tests that do depend in some way on math/big are even rebuilt. And then only the ones that produce a different test binary from last time will be rerun. We expect test caching to be a significant win for people working in large code bases.

Go install + dependencies

The "go install" command no longer installs dependencies of the named packages (CL 75850). If you run "go install foo", the only thing installed is foo. Before, it varied. If dependencies were out-of-date, "go install" also installed any dependencies. The implicit installation of dependencies during "go install" caused a lot of confusion and headaches for users, but it was previously necessary to enable incremental builds. Not anymore. We think that the new "install what I said" semantics will be much more understandable, especially since it's clear from bug reports that many users already expected them. To force installation of dependencies during "go install", use the new "go install -i", by analogy with "go build -i" and "go test -i".

The fact that "go install" used to install any rebuilt dependencies caused confusion most often in conjunction with -a, which means "force rebuild of all dependencies". Now, "go install -a myprog" will force a complete rebuild of all dependencies of myprog, as well as myprog itself, but only myprog will get installed. (All the rebuilt dependencies will still be saved in the build cache, of course.) Making this case work more understandably is especially important in conjunction with the new content-based staleness analysis, because it sees good reasons to rebuild dependencies more often than before, which would have increased the amount of "why did my dependencies get installed" confusion. For example, if you run "go install -gcflags=-N myprog", that installs a myprog built with no compiler optimizations, but it no longer also reinstalls the packages myprog uses from the standard library without compiler optimizations.

Ideally, the very idea of "installing dependencies" would be an implementation detail completely hidden from users. We're not there yet. The main reason you might still need "go install -i" is if you are using a code analysis tool that expects to read package .a files from the $GOPATH/pkg directory. We plan to fix this problem, to update the tools to find what they need more dynamically, but we have not done that yet. It's unclear whether those tools will be updated in time for Go 1.10. One exception is "go vet".

Go vet

The "go vet" command works best with full type information for the packages it is analyzing. Historically that's been problematic in a variety of situations: packages using cgo, packages using vendoring, and packages that don't have up-to-date installed dependencies have all been things that kept vet from running with full type information. Not anymore. The "go vet" command now passes to vet full information about all these things, building new .a files if needed (CL 74355 and CL 74750). Now "go vet" is guaranteed to have up-to-date type information when it analyzes a package, which will improve the accuracy of its analyses. Having the build cache to amortize the cost of building those .a files was the final piece needed to make this happen.

Only "go vet" has this guarantee. Do not use "go tool vet", which is essentially only useful to people working on vet (just like you don't typically run "go tool compile"). Previously you needed to use "go tool vet" if you wanted control over vet flags, but "go vet" now accepts all the flags that "go tool vet" does. (See "go help vet".)

Test Vet

Now that "go vet" can be guaranteed proper type information, we're able to do something Rob and I started talking about at least three years ago: run "go vet" automatically during "go test" (CL 74356). This has two key benefits. First, it avoids needing to teach users a new manual workflow step. Just as we encourage users to set up editor hooks to run gofmt automatically, go test can run go vet automatically. Second, it broadens the set of useful checks. Before, it really only made sense to add a check to vet if that mistake would have made it through routine testing. Now, if there's something that would probably cause a mysterious test failure but that vet can pinpoint much precisely and save the user a debugging session, we can do that.

It only makes sense to run the checks that are effectively 100% reliable. The exact list for Go 1.10 is not yet decided (that's #18085). Right now the list is atomic, bool, buildtags, nilfunc, and printf. You can change the list by (for example) "go test -vet=printf,shift", and you can disable vet entirely by "go test -vet=off". My hope is that being able to enable them in go test will encourage raising the bar for vet check accuracy. One eventual goal would be to retire the misc-vetall builder, because all the vet checks we want are being run during test (without any kind of whitelist of ignored reports).

We expect no perceptible slowdown to "go test" from running vet, because we run vet in parallel with linking the test binary. (There may be a ~2% slowdown when running a very large number of tests.)

Debugging

If you run into a problem that you think is being caused by the cache, you can retry the command you're running with GOCACHE=off to avoid looking at or updating the cache. Even better, you can leave GOCACHE as is and instead set GODEBUG=gocacheverify=1, which will not refuse to read from the cache but then will double-check that any writes back to the cache of data already present match bit-for-bit. If you ever see a failure with GODEBUG=gocacheverify=1 set, please file a bug. On my laptop right now, GODEBUG=gocacheverify=1 ./all.bash passes.

If you do run into problems, please file them on Github and cc @rsc. If you have questions about anything, this email thread is a good place for them.

Thanks.
Russ