Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate .pb.go files #564

Merged
merged 3 commits into from
Jan 15, 2018
Merged

Generate .pb.go files #564

merged 3 commits into from
Jan 15, 2018

Conversation

ianthehat
Copy link

This uses the gogo repository and new proto rules to generate the .pb.go
files on demand.

Note that this has to use my fork of gogo/protobuf (the bazel rules are not in a good enough shape to ask them to accept them yet, just good enough to use the plugins) and my fork of rules go (although the changes are already in a pull request so that won't be for long)

@jml
Copy link
Contributor

jml commented Sep 17, 2017

Thanks!

@ianthehat Just for reference…

@jml
Copy link
Contributor

jml commented Sep 17, 2017

Testing locally, build seems to work…

$ bazel build //cmd/...
INFO: Found 16 targets...
INFO: From Compiling external/com_google_protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/com_google_protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
           ^
1 warning generated.
INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/arenastring.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/stubs/atomicops_internals_x86_msvc.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/stubs/io_win32.o has no symbols
INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/external/com_google_protobuf/src/google/protobuf/io/gzip_stream.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/external/com_google_protobuf/src/google/protobuf/util/internal/error_listener.o has no symbols
INFO: From Linking external/com_github_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf/external/com_github_google_protobuf/src/google/protobuf/io/gzip_stream.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf/external/com_github_google_protobuf/src/google/protobuf/util/internal/error_listener.o has no symbols
INFO: From Linking external/com_github_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/arenastring.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/stubs/atomicops_internals_x86_msvc.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/stubs/io_win32.o has no symbols
INFO: From Compiling external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
           ^
1 warning generated.
INFO: From Linking vendor/golang.org/x/sys/unix/libgo_default_library.cgo_c_lib.lo:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/_cgo_export.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_env_unset.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_constants.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_dirent.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_endian_little.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_env_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_flock.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_race0.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_sockcmsg_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_str.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall_bsd.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall_darwin.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_syscall_unix_gc.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_zerrors_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_zsyscall_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_zsysnum_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo.dir/vendor_golang.org_x_sys_unix_ztypes_darwin_amd64.cgo2.pic.o has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: warning for library: bazel-out/local-fastbuild/bin/vendor/golang.org/x/sys/unix/libgo_default_library.cgo_c_lib.lo the table of contents is empty (no object file members in the library define global symbols)
INFO: From Linking vendor/golang.org/x/sys/unix/go_default_library._cgo_.o:
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
INFO: Elapsed time: 319.015s, Critical Path: 36.08s

But tests don't…

$ bazel test //pkg/...
INFO: Found 25 targets and 9 test targets...
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/ingester/client/BUILD.bazel:11:1: GoCompile pkg/ingester/client/~normal~cortex_go_proto~/github.com/weaveworks/cortex/pkg/ingester/client.a failed (Exit 1).
2017/09/17 10:05:35 missing strict dependencies:
	bazel-out/local-fastbuild/bin/pkg/ingester/client/github.com/weaveworks/cortex/pkg/ingester/client/cortex.pb.go: import of github.com/gogo/protobuf/proto, which is not a direct dependency
	bazel-out/local-fastbuild/bin/pkg/ingester/client/github.com/weaveworks/cortex/pkg/ingester/client/cortex.pb.go: import of github.com/weaveworks/cortex/pkg/util/wire, which is not a direct dependency
	bazel-out/local-fastbuild/bin/pkg/ingester/client/github.com/weaveworks/cortex/pkg/ingester/client/cortex.pb.go: import of golang.org/x/net/context, which is not a direct dependency
	bazel-out/local-fastbuild/bin/pkg/ingester/client/github.com/weaveworks/cortex/pkg/ingester/client/cortex.pb.go: import of google.golang.org/grpc, which is not a direct dependency
INFO: Elapsed time: 1.703s, Critical Path: 1.06s
//pkg/chunk:go_default_test                                           NO STATUS
//pkg/configs/api:go_default_xtest                                    NO STATUS
//pkg/configs/client:go_default_test                                  NO STATUS
//pkg/distributor:go_default_test                                     NO STATUS
//pkg/ingester:go_default_test                                        NO STATUS
//pkg/querier:go_default_test                                         NO STATUS
//pkg/ring:go_default_test                                            NO STATUS
//pkg/ruler:go_default_test                                           NO STATUS
//pkg/util:go_default_test                                            NO STATUS

Executed 0 out of 9 tests: 9 were skipped.

Investigating.

@tomwilkie
Copy link
Contributor

This is indeed most excellent! A few questions / comments:

  • Changes in the vendor dir shouldn't be made directly, you should use the dep tool to update it. If you did then you're missing changes to Gopkg.toml and Gopkg.lock.
  • Have you open a PR / spoke to the gogo guys about upstreaming the changes? I think its fine using a fork for a little while, but its not something we should depend on long term, so I'd be keen to see if this is something they'd accept in principle. If not, there are alternative way, less desirable ways to do this (using gen_rules for instance).
  • In https://github.com/gogo/protobuf/compare/master...ianthehat:bazel?expand=1 I see you're using checked-in generated .pb.go files - is this intentional? Now that bazel can generate them with your other PR, I'd have thought it would be best to generate them here too?

Otherwise I think this change looks excellent - I particularly like the embed = [":cortex_go_proto"], bits, I've been looking for a way to do that for a month or so!

@jml
Copy link
Contributor

jml commented Sep 17, 2017

Ran out of time to investigate.

Copy link
Contributor

@jml jml left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bazel test //pkg/... needs to pass before this can be merged.

@jml jml mentioned this pull request Sep 17, 2017
5 tasks
@ianthehat
Copy link
Author

I wasn't really expecting this pr to get submitted, I was just putting it up so you could try it and comment :)

Yes that is my gogo fork, and yes, that was the rules_go fork that is already merged.

The changes in vendor are legacy from when I was seeing what it would take to make the gogo repository play nice, the changes are all in my fork which is pulled directly from the workspace now.
The test problems look like I did not add the deps to the gogo toolchain, that's a fairly trivial fix.

The changes in the gogo repository were generated by gazelle and then hand adjusted and added to only in places where I needed to to make the plugins work and add the toolchains rules. They are far from correct, but I am waiting for the full changes to gazelle to arrive so I can regenerated them before I consider putting it up as a pull request to the main gogo repository. This is also why they are doing things like using the checked in .pb.go files...

If they won't accept the changes it's not a big deal, we can do it all with an external flat mode build file. I am just trying to be a good citizen by contributing an alternative that works better than make to their project while I am looking at it anyway :)

Note that I know nothing about weaveworks, I have no idea what this project does, and I have not run or tested the results in any way at all, so you are going to have to make sure I have not broken something before you start using this.

I am also not happy that the vendor directory is an unbuildable mess, I was originally trying to do

bazel test //...

Which is my basic mode of operation on all bazel projects, but it does not stand a hope of building the vendor dir right now, so I dropped back to

bazel build //cmd/...

because I assume that had to build, but it has no tests.
What I might try to do is write a tool to update a workspace from a lock file, and then not use the vendor dir at all, just exclude it from the build, but that's not going to happen in a hurry, I have too many other things to make work first (like windows)

@jml
Copy link
Contributor

jml commented Sep 18, 2017

I wasn't really expecting this pr to get submitted, I was just putting it up so you could try it and comment :)

@ianthehat, thank you for that! It's extremely helpful. Certainly didn't mean to imply that you were on the hook for fixing everything—I'm grateful for what you've done already.

The test problems look like I did not add the deps to the gogo toolchain, that's a fairly trivial fix.

Not 100% sure what this means, but maybe that's because I'm still not precise on Bazel terminology (or because I'm not awake yet).

Naively, one of the missing deps is github.com/weaveworks/cortex/pkg/util/wire, which is specific to Cortex and not a part of the toolchain, so it probably shouldn't be added there?

@ianthehat ianthehat force-pushed the bazel-0.5.4 branch 2 times, most recently from c17bc67 to c63bdeb Compare September 18, 2017 17:39
@ianthehat
Copy link
Author

I am happy to do it, I just didn't want you thinking it was my normal habit to put up such shoddy pull requests with any expectation they would be accepted :)

I have pushed a new version now that no longer requires any forks, the toolchains are locally declared, and it passes all tests.

bazel test //cmd/... //pkg/...

It does however require two very small changes to the vendor folder, which makes me sad. One is something that we can (and will) do something about, which is add the proto_library rules to the gazelle generated build files, the other is to add a missing go_package option to the gogo proto file, which would need to be pushed upstream.

This uses the gogo repository and new proto rules to generate the .pb.go
files on demand.
@ianthehat
Copy link
Author

This is now an actual viable change.
It requires no forked repositories and no vendor folder changes.
It uses a vile custom repository rule to manage this, and has a bunch of stuff that if we manage to upstream changes to the gogo repository can be removed again, but for now I think this approach is your best bet.

@tomwilkie
Copy link
Contributor

Nice one; one nit - tools is a subtree of weaveworks/build-tools, so you might want to find a different place for those files.

I'll do a thorough review tomorrow evening, but a quick first pass looks good.

@jml
Copy link
Contributor

jml commented Sep 22, 2017

@tomwilkie Happy for you to do the review, but LMK if you can't get to it soon & I'll get onto it next week.

@ianthehat
Copy link
Author

I am changing the proto rules again after feedback on the current system.
see bazel-contrib/rules_go#1098 for the first change.
Would you like me to update this to use the new system with the gogo plugin?

@jml
Copy link
Contributor

jml commented Dec 4, 2017 via email

@ianthehat
Copy link
Author

This now works with the new system (you also need the as yet not submitted bazel-contrib/rules_go#1102)

@jml
Copy link
Contributor

jml commented Dec 5, 2017 via email

@jml
Copy link
Contributor

jml commented Dec 5, 2017

I just tried this out.

Before trying anything I made the following local change:

--- a/WORKSPACE
+++ b/WORKSPACE
@@ -2,8 +2,8 @@ workspace(name = "com_github_weaveworks_cortex")

 http_archive(
     name = "io_bazel_rules_go",
-    url = "https://codeload.github.com/bazelbuild/rules_go/zip/0fb90c43c5fab2a0b2d7a8684f26f6995d9aa212",
-    strip_prefix = "rules_go-0fb90c43c5fab2a0b2d7a8684f26f6995d9aa212",
+    url = "https://codeload.github.com/bazelbuild/rules_go/zip/43e3cadb0469e3ada11ac0371f924004c77ea01a",
+    strip_prefix = "rules_go-43e3cadb0469e3ada11ac0371f924004c77ea01a",
     type = "zip",
 )

In order to pull in bazel-contrib/rules_go#1102 as mentioned.

First time I got errors like:

error: could not expand include path '~/.gitcinclude'
fatal: bad config line 49 in file /usr/local/git/etc/gitconfig

Which is because brew installs a global git config with ~ in paths, and Bazel unsets $HOME.

After working around that, bazel build //cmd/... failed again with:

ERROR: /private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/com_github_gogo_protobuf/gogoproto/BUILD.bazel:19:1: GoCompile external/com_github_gogo_protobuf/gogoproto/darwin_amd64_stripped/go_default_library~/github.com/gogo/protobuf/gogoproto.a failed (Exit 1)
external/com_github_gogo_protobuf/gogoproto/helper.go:35:37: cannot use E_Embed (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension

Full output:

$ bazel build //cmd/...
INFO: Analysed 16 targets (6 packages loaded).
INFO: Found 16 targets...
INFO: From Compiling external/com_google_protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/com_google_protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
           ^
1 warning generated.
INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/arenastring.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/stubs/atomicops_internals_x86_msvc.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/external/com_google_protobuf/src/google/protobuf/stubs/io_win32.o has no symbols
INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/external/com_google_protobuf/src/google/protobuf/io/gzip_stream.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/external/com_google_protobuf/src/google/protobuf/util/internal/error_listener.o has no symbols
INFO: From Linking external/com_github_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/arenastring.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/stubs/atomicops_internals_x86_msvc.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf_lite/external/com_github_google_protobuf/src/google/protobuf/stubs/io_win32.o has no symbols
INFO: From Linking external/com_github_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf/external/com_github_google_protobuf/src/google/protobuf/io/gzip_stream.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_github_google_protobuf/_objs/protobuf/external/com_github_google_protobuf/src/google/protobuf/util/internal/error_listener.o has no symbols
INFO: From Compiling external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc [for host]:
external/com_github_google_protobuf/src/google/protobuf/compiler/js/embed.cc:37:12: warning: unused variable 'output_file' [-Wunused-const-variable]
const char output_file[] = "well_known_types_embed.cc";
           ^
1 warning generated.
INFO: From GoStdlib external/go_stdlib_darwin_amd64_cgo/bin/go [for host]:
# runtime/cgo
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
# runtime/cgo
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
# os/user
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
# os/user
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
# net
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
# net
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
# plugin
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
# plugin
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
# crypto/x509
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
# crypto/x509
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-gno-record-gcc-switches' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
INFO: From Linking vendor/golang.org/x/sys/unix/libgo_default_library.cgo_c_lib.lo:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/_cgo_export.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_env_unset.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_constants.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_dirent.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_endian_little.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_env_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_flock.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_race0.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_sockcmsg_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_str.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall_bsd.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall_darwin.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall_unix.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_syscall_unix_gc.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_zerrors_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_zsyscall_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_zsysnum_darwin_amd64.cgo2.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/_objs/go_default_library.cgo_c_lib/vendor/golang.org/x/sys/unix/go_default_library.cgo_codegen~/vendor_golang.org_x_sys_unix_ztypes_darwin_amd64.cgo2.pic.o has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: warning for library: bazel-out/darwin-fastbuild/bin/vendor/golang.org/x/sys/unix/libgo_default_library.cgo_c_lib.lo the table of contents is empty (no object file members in the library define global symbols)
ERROR: /private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/com_github_gogo_protobuf/gogoproto/BUILD.bazel:19:1: GoCompile external/com_github_gogo_protobuf/gogoproto/darwin_amd64_stripped/go_default_library~/github.com/gogo/protobuf/gogoproto.a failed (Exit 1)
external/com_github_gogo_protobuf/gogoproto/helper.go:35:37: cannot use E_Embed (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:39:37: cannot use E_Nullable (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:43:37: cannot use E_Stdtime (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:47:37: cannot use E_Stdduration (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:94:36: cannot use E_Enumdecl (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:94:85: cannot use E_EnumdeclAll (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:98:39: cannot use E_Typedecl (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:98:88: cannot use E_TypedeclAll (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetBoolExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:106:37: cannot use E_Customtype (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:119:37: cannot use E_Casttype (type *"github.com/golang/protobuf/proto".ExtensionDesc) as type *"github.com/gogo/protobuf/proto".ExtensionDesc in argument to "github.com/gogo/protobuf/proto".GetExtension
external/com_github_gogo_protobuf/gogoproto/helper.go:119:37: too many errors
2017/12/05 18:20:55 error running compiler: exit status 1
INFO: Elapsed time: 275.347s, Critical Path: 27.61s
FAILED: Build did NOT complete successfully

@ianthehat
Copy link
Author

Looks like your bazel version is old, you need at least 0.8

@jml
Copy link
Contributor

jml commented Dec 5, 2017

I'm running 0.8. Still see those errors in external/com_github_gogo_protobuf/gogoproto/helper.go

$ bazel version
Build label: 0.8.0-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Nov 27 20:38:20 2017 (1511815100)
Build timestamp: 1511815100
Build timestamp as int: 1511815100

@jml
Copy link
Contributor

jml commented Dec 5, 2017

I should mention that I only recently upgraded from 0.5.4 (brew update; brew upgrade bazel). Perhaps there's some extra step required?

@jml
Copy link
Contributor

jml commented Dec 6, 2017

Still failing with same errors with bazel 0.8.1

$ bazel version
Build label: 0.8.1-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Dec 5 19:29:04 2017 (1512502144)
Build timestamp: 1512502144
Build timestamp as int: 1512502144

@ianthehat
Copy link
Author

Sorry for the delay, was waiting for everything to be submitted in rules_go so I could update the dependency.
I have pushed a new change, It all works with this cl at the moment for me.

@jml
Copy link
Contributor

jml commented Dec 7, 2017

Great! Works for me too, modulo #527.

As @tomwilkie says, we need to move tools/build/bazel/ to somewhere other than tools/ before we can merge. No strong preference on name, perhaps bazel-build-tools/?

Then I think we can merge this, hook it up to our CI (separate PR), and then it's time for me to sit down & figure out how I can build the docker images w/ Linux binaries by cross-compiling from my Mac. Exciting times!

Thanks so much, @ianthehat!

@jml
Copy link
Contributor

jml commented Dec 8, 2017

@ianthehat Where does this leave us wrt Gazelle?

I had a quick play around to see what bazel run //:gazelle would do. It succeeds, generates a bunch of warnings (either gazelle: multiple rules (//pkg/ingester/client:cortex_go_proto and //pkg/ingester/client:go_default_library) may be imported with "github.com/weaveworks/cortex/pkg/ingester/client" or warning: file contains rules whose structure is out of date. Consider running 'gazelle fix'.), and adds new proto_library and go_grpc_library targets to pkg/ingester/client/BUILD.bazel. Needless to say the build fails with this change.

Naïvely adding a # gazelle:proto disable directive to pkg/ingester/client/BUILD.bazel before running gazelle works a little better, but introduces changes like:

diff --git a/pkg/util/BUILD.bazel b/pkg/util/BUILD.bazel
index 2d24cc8..b86e274 100644
--- a/pkg/util/BUILD.bazel
+++ b/pkg/util/BUILD.bazel
@@ -19,7 +19,6 @@ go_library(
     importpath = "github.com/weaveworks/cortex/pkg/util",
     visibility = ["//visibility:public"],
     deps = [
-        "//pkg/ingester/client:go_default_library",
         "//vendor/github.com/blang/semver:go_default_library",
         "//vendor/github.com/gogo/protobuf/proto:go_default_library",
         "//vendor/github.com/golang/snappy:go_default_library",
@@ -45,8 +44,8 @@ go_test(
         "priority_queue_test.go",
         "validate_test.go",
     ],
+    embed = [":go_default_library"],
     importpath = "github.com/weaveworks/cortex/pkg/util",
-    library = ":go_default_library",
     deps = [
         "//vendor/github.com/prometheus/common/model:go_default_library",
         "//vendor/github.com/prometheus/prometheus/storage/local:go_default_library",

Which leads to build failures like:

$ bazel build //pkg/...
INFO: Analysed 34 targets (0 packages loaded).
INFO: Found 34 targets...
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/util/BUILD.bazel:3:1: GoCompile pkg/util/darwin_amd64_stripped/go_default_library~/github.com/weaveworks/cortex/pkg/util.a failed (Exit 1)
2017/12/08 11:25:57 missing strict dependencies:
	pkg/util/compat.go: import of github.com/weaveworks/cortex/pkg/ingester/client, which is not a direct dependency
INFO: Elapsed time: 0.733s, Critical Path: 0.31s
FAILED: Build did NOT complete successfully

I'll try to dig more deeply into this later today. Reporting my notes now in case you (or others) have more insight.

@jml jml self-assigned this Jan 15, 2018
jml added a commit to weaveworks/build-tools that referenced this pull request Jan 15, 2018
jml added a commit to weaveworks/build-tools that referenced this pull request Jan 15, 2018
jml added a commit that referenced this pull request Jan 15, 2018
…ools

Update to get latest weaveworks/build-tools

Needed to merge #564 changes
@jml jml mentioned this pull request Jan 15, 2018
@jml jml merged commit abf7723 into cortexproject:bazel-0.5.4 Jan 15, 2018
@jml
Copy link
Contributor

jml commented Jan 15, 2018

Thanks so much @ianthehat. I'll try to get this merged into master as soon as I can.

@jml jml mentioned this pull request Jan 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants