Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: bazel changes to build on Windows #3786

Closed
wants to merge 11 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ SOURCE_VERSION
.cache
.vimrc
.vscode
.vs
9 changes: 9 additions & 0 deletions api/bazel/api_build_system.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ load("@io_bazel_rules_go//go:def.bzl", "go_test")

_PY_SUFFIX="_py"
_CC_SUFFIX="_cc"
_CC_NATIVE_SUFFIX="_cc_native"
_GO_PROTO_SUFFIX="_go_proto"
_GO_GRPC_SUFFIX="_go_grpc"
_GO_IMPORTPATH_PREFIX="github.com/envoyproxy/data-plane-api/api/"
Expand Down Expand Up @@ -131,6 +132,14 @@ def api_proto_library(name, visibility = ["//visibility:private"], srcs = [], de
],
visibility = ["//visibility:public"],
)

native.cc_proto_library(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, without the PGV validation, there are two things we lose:

  1. The ability to reject configurations that are somewhat nonsensical.
  2. The guarantee that various preconditions on Envoy config processing code are met.

I think (1) is fine for an interim solution, but (2) is positively dangerous, since configuration can easily crash or pose security hazards to an Envoy instance. E.g. the config processing code might access an element in an array that has not been allocated, because it believes that due to PGV validation that the array is at least N elements long. If we want to accept (2) and the non-PGV variant for Windows, we probably want to slap a bunch of TODOs and warnings, together with tracking bugs, to ensure we bridge this gap.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When building with the PGV validation, we ran into a few issues:

  1. rules_go wasn't properly handling Windows paths. This appears to be fixed in 1c7fd7e58310b832decc04963ec13db204f79150, but this commit has not been released
  2. in PGV, _go_proto_library_gen_impl: https://github.com/lyft/protoc-gen-validate/blob/master/bazel/go_proto_library.bzl#L130 was not generating an OS agnostic script
  3. The cc_proto_library rule from @com_google_protobuf//:protobuf.bzl did not de-duplicate command line arguments, causing it to hit the 32,768 character command line limit. The native Bazel cc_proto_library rule does not have this issue.

So it looks like the way forward would be something like:

  1. Update project to latest rules_go once it is released
  2. Fix up PGV _go_proto_library_gen_impl so it generates valid scripts for both Linux and Windows
  3. PGV uses native Bazel cc_proto_library rules.

From the comment in api/bazel/api_build_system.bzl, it looks like the third step might not be possible at the moment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think there's going to be some non-trivial changes to get us to native cc_proto_library. I don't think we can do release binaries for Windows or consider Windows supported beyond experimental until this is fixed though. We should add TODOs and tracking issues for now.

name = _Suffix(name, _CC_NATIVE_SUFFIX),
deps = [name],
tags = ["manual"],
visibility = ["//visibility:public"],
)

if (require_py == 1):
api_py_proto_library(name, srcs, deps, has_services)

Expand Down
16 changes: 12 additions & 4 deletions api/bazel/repositories.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def api_dependencies():
strip_prefix = "googleapis-" + GOOGLEAPIS_SHA,
url = "https://github.com/googleapis/googleapis/archive/" + GOOGLEAPIS_SHA + ".tar.gz",
build_file_content = """
load("@com_google_protobuf//:protobuf.bzl", "cc_proto_library", "py_proto_library")
load("@com_google_protobuf//:protobuf.bzl", "py_proto_library", pb_cc_proto_library="cc_proto_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

filegroup(
Expand All @@ -45,6 +45,13 @@ proto_library(
)

cc_proto_library(
name = "http_api_protos_native",
deps = [":http_api_protos_proto"],
tags=["manual"],
visibility = ["//visibility:public"],
)

pb_cc_proto_library(
name = "http_api_protos",
srcs = [
"google/api/annotations.proto",
Expand Down Expand Up @@ -93,7 +100,8 @@ proto_library(
deps = ["@com_google_protobuf//:any_proto"],
visibility = ["//visibility:public"],
)
cc_proto_library(

pb_cc_proto_library(
name = "rpc_status_protos",
srcs = ["google/rpc/status.proto"],
default_runtime = "@com_google_protobuf//:protobuf",
Expand Down Expand Up @@ -133,7 +141,7 @@ py_proto_library(
strip_prefix = "protobuf-" + GOGOPROTO_SHA,
url = "https://github.com/gogo/protobuf/archive/" + GOGOPROTO_SHA + ".tar.gz",
build_file_content = """
load("@com_google_protobuf//:protobuf.bzl", "cc_proto_library", "py_proto_library")
load("@com_google_protobuf//:protobuf.bzl", "py_proto_library", pb_cc_proto_library="cc_proto_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

proto_library(
Expand All @@ -154,7 +162,7 @@ go_proto_library(
visibility = ["//visibility:public"],
)

cc_proto_library(
pb_cc_proto_library(
name = "gogo_proto_cc",
srcs = [
"gogoproto/gogo.proto",
Expand Down
29 changes: 29 additions & 0 deletions bazel/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,35 @@ genrule(
stamp = 1,
)

config_setting(
name = "windows_x86_64",
values = {"cpu": "x64_windows"},
)

config_setting(
name = "windows_opt_build",
values = {
"cpu": "x64_windows",
"compilation_mode": "opt",
},
)

config_setting(
name = "windows_dbg_build",
values = {
"cpu": "x64_windows",
"compilation_mode": "dbg",
},
)

config_setting(
name = "windows_fastbuild_build",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not that familiar with Bazel's platforms/toolchains support, but is this something we should be using to make the Windows vs. Linux vs. Mac OS X situation cleaner? In particular, I think we have multiple orthogonal build concerns:

  1. Target OS (Windows (native vs. Linux compat?), Linux, Mac OS X)
  2. Target runtime (libc++ vs. libstdc++)
  3. Target toolchain (clang, gcc, Visual Studio)
  4. Etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We added the windows_*_build config settings because there is no way to perform a select on multiple conditions. This allows envoy_copts to conditionally append --ggdb3 on Linux and never to append it on Windows: https://github.com/greenhouse-org/envoy/blob/windows-build-pr/bazel/envoy_build_system.bzl#L36-L41

With respect to the more general question, we are open to suggestions and what the community decides as the best approach.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@laszlocsomor @jmillikin-stripe is there a sensible way to handle this with https://docs.bazel.build/versions/master/platforms.html?

pinging @katre to answer

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding a cflag on one platform should be a property of the cc_toolchain, if I understand it properly (given how confusing C++ is, I possibly don't). Are you defining your own cc_toolchain targets, or using the ones from auto-configuration?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, on Windows we are just using the ones from auto-configuration, though on Linux there's a wrapper script: https://github.com/envoyproxy/envoy/blob/master/bazel/cc_configure.bzl

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, the existing code is probably the best way to do things.

values = {
"cpu": "x64_windows",
"compilation_mode": "fastbuild",
},
)

config_setting(
name = "opt_build",
values = {"compilation_mode": "opt"},
Expand Down
80 changes: 69 additions & 11 deletions bazel/envoy_build_system.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,40 @@ def envoy_package():

# Compute the final copts based on various options.
def envoy_copts(repository, test = False):
return [
"-Wall",
"-Wextra",
"-Werror",
"-Wnon-virtual-dtor",
"-Woverloaded-virtual",
"-Wold-style-cast",
"-std=c++14",
] + select({
posix_options = [
"-Wall",
"-Wextra",
"-Werror",
"-Wnon-virtual-dtor",
"-Woverloaded-virtual",
"-Wold-style-cast",
"-std=c++14",
]

msvc_options = [
"-WX",
"-DWIN32",
"-DDISABLE_PROTO_VALIDATE",
"-DWIN32_LEAN_AND_MEAN",
# need win8 for ntohll
# https://msdn.microsoft.com/en-us/library/windows/desktop/aa383745(v=vs.85).aspx
"-D_WIN32_WINNT=0x0602",
"-DNTDDI_VERSION=0x06020000",
"-DCARES_STATICLIB",
"-DNGHTTP2_STATICLIB",
]

return select({
"//bazel:windows_x86_64": msvc_options,
"//conditions:default": posix_options,
}) + select({
# Bazel adds an implicit -DNDEBUG for opt.
repository + "//bazel:opt_build": [] if test else ["-ggdb3"],
repository + "//bazel:fastbuild_build": [],
repository + "//bazel:dbg_build": ["-ggdb3"],
repository + "//bazel:windows_opt_build": [],
repository + "//bazel:windows_fastbuild_build": [],
repository + "//bazel:windows_dbg_build": [],
}) + select({
repository + "//bazel:disable_tcmalloc": ["-DABSL_MALLOC_HOOK_MMAP_DISABLE"],
"//conditions:default": ["-DTCMALLOC"],
Expand All @@ -44,6 +65,9 @@ def envoy_linkopts():
# See note here: http://luajit.org/install.html
"-pagezero_size 10000", "-image_base 100000000",
],
"//bazel:windows_x86_64": [
"-DEFAULTLIB:advapi32.lib",
],
"//conditions:default": [
"-pthread",
"-lrt",
Expand All @@ -66,6 +90,8 @@ def _envoy_stamped_linkopts():
"-sectcreate __TEXT __build_id", "$(location @envoy//bazel:raw_build_id.ldscript)"
],

"//bazel:windows_x86_64": [],

# Note: assumes GNU GCC (or compatible) handling of `--build-id` flag.
"//conditions:default": [
"-Wl,@$(location @envoy//bazel:gnu_build_id.ldscript)",
Expand Down Expand Up @@ -115,6 +141,16 @@ def tcmalloc_external_deps(repository):
"//conditions:default": [envoy_external_dep_path("tcmalloc_and_profiler")],
})

# Dependencies on libevent should be wrapped with this function.
def libevent_external_deps(repository):
return [envoy_external_dep_path("event")] + select({
repository + "//bazel:windows_x86_64": [],
"//conditions:default": [envoy_external_dep_path("event_pthreads")],
})

def http_api_protos_external_deps(repository):
return envoy_protobuf_cc_select(repository, [envoy_external_dep_path("http_api_protos")])

# Transform the package path (e.g. include/envoy/common) into a path for
# exporting the package headers at (e.g. envoy/common). Source files can then
# include using this path scheme (e.g. #include "envoy/common/time.h").
Expand All @@ -130,21 +166,39 @@ def envoy_include_prefix(path):
def envoy_basic_cc_library(name, **kargs):
native.cc_library(name = name, **kargs)

def envoy_windows_protobuf_cc_dep_path(dep):
return dep + "_native"

def envoy_protobuf_cc_select(repository, deps = []):
windows_deps = [envoy_windows_protobuf_cc_dep_path(dep) for dep in deps]
return select({
repository + "//bazel:windows_x86_64": windows_deps,
"//conditions:default": deps,
})

# Envoy C++ library targets should be specified with this function.
def envoy_cc_library(name,
srcs = [],
hdrs = [],
copts = [],
visibility = None,
protobuf_cc_deps = [],
external_deps = [],
tcmalloc_dep = None,
libevent_dep = None,
http_api_protos_dep = None,
repository = "",
linkstamp = None,
tags = [],
deps = [],
strip_include_prefix = None):
if tcmalloc_dep:
deps += tcmalloc_external_deps(repository)
if libevent_dep:
deps += libevent_external_deps(repository)
if http_api_protos_dep:
deps += http_api_protos_external_deps(repository)

native.cc_library(
name = name,
srcs = srcs,
Expand All @@ -158,11 +212,14 @@ def envoy_cc_library(name,
envoy_external_dep_path('abseil_strings'),
envoy_external_dep_path('spdlog'),
envoy_external_dep_path('fmtlib'),
],
] + envoy_protobuf_cc_select(repository, protobuf_cc_deps),
include_prefix = envoy_include_prefix(PACKAGE_NAME),
alwayslink = 1,
linkstatic = 1,
linkstamp = linkstamp,
linkstamp = select({
repository + "//bazel:windows_x86_64": None,
"//conditions:default": linkstamp,
}),
strip_include_prefix = strip_include_prefix,
)

Expand Down Expand Up @@ -458,5 +515,6 @@ def envoy_select_force_libcpp(if_libcpp, default = None):
return select({
"@envoy//bazel:force_libcpp": if_libcpp,
"@bazel_tools//tools/osx:darwin": [],
"//bazel:windows_x86_64": [],
"//conditions:default": default or [],
})
4 changes: 4 additions & 0 deletions bazel/external/libcircllhist.BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,8 @@ cc_library(
],
includes = ["src"],
visibility = ["//visibility:public"],
copts = select({
"@envoy//bazel:windows_x86_64": ["-DWIN32"],
"//conditions:default": [],
}),
)
4 changes: 4 additions & 0 deletions bazel/repositories.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
echo "Start"
@ECHO OFF
%BAZEL_SH% -c "./repositories.sh %*"
exit %ERRORLEVEL%
30 changes: 27 additions & 3 deletions bazel/repositories.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ load(":genrule_repository.bzl", "genrule_repository")
load(":patched_http_archive.bzl", "patched_http_archive")
load(":repository_locations.bzl", "REPOSITORY_LOCATIONS")
load(":target_recipes.bzl", "TARGET_RECIPES")
load("@bazel_tools//tools/cpp:windows_cc_configure.bzl",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@meteorcloudy: do you think it's OK to use windows_cc_configure.bzl? As far as I know it has no public interface. If Envoy starts depending on it, we won't be able to change it without potentially breaking them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine. windows_cc_configure.bzl does expose find_vc_path, setup_vc_env_vars, find_msvc_tool.
I believe they are quite common functions that're needed when setting up MSVC related repo. The function interface didn't change since the beginning, so I think it's stable enough to let others to depend on it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is why do we need to setup the MSVC environment variables? @sesmith177

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dependencies that are built with scripts (e.g. https://github.com/envoyproxy/envoy/blob/75909a7d16f0291ccd0bbe29517ac387a34701ea/ci/build_container/build_recipes/cares.sh) need to have the MSVC environment variables set. So we set the variables up when calling the repositories.bat script that ends up building them

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explanation, then this looks good to me!

"find_vc_path",
"setup_vc_env_vars",
)
load("@bazel_tools//tools/cpp:lib_cc_configure.bzl", "get_env_var")

def _repository_impl(name, **kwargs):
# `existing_rule_keys` contains the names of repositories that have already
Expand Down Expand Up @@ -62,6 +67,7 @@ def _repository_impl(name, **kwargs):
def _build_recipe_repository_impl(ctxt):
# Setup the build directory with links to the relevant files.
ctxt.symlink(Label("//bazel:repositories.sh"), "repositories.sh")
ctxt.symlink(Label("//bazel:repositories.bat"), "repositories.bat")
ctxt.symlink(Label("//ci/build_container:build_and_install_deps.sh"),
"build_and_install_deps.sh")
ctxt.symlink(Label("//ci/build_container:recipe_wrapper.sh"), "recipe_wrapper.sh")
Expand All @@ -72,11 +78,25 @@ def _build_recipe_repository_impl(ctxt):
ctxt.symlink(Label("//ci/prebuilt:BUILD"), "BUILD")

# Run the build script.
environment = {}
command = []
env = {}
if ctxt.os.name.upper().startswith("WINDOWS"):
vc_path = find_vc_path(ctxt)
current_path = get_env_var(ctxt, "PATH", None, False)
env = setup_vc_env_vars(ctxt, vc_path)
env["PATH"]+=(";%s" % current_path)
env["CC"]="cl"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you merge master? We just merged a PR that applied Buildifier on Skylark files that should clean a lot of this up for consistency.

env["CXX"]="cl"
env["CXXFLAGS"] = "-DNDEBUG"
env["CFLAGS"] = "-DNDEBUG"
command = ["./repositories.bat"] + ctxt.attr.recipes
else:
command = ["./repositories.sh"] + ctxt.attr.recipes

print("Fetching external dependencies...")
result = ctxt.execute(
["./repositories.sh"] + ctxt.attr.recipes,
environment = environment,
command,
environment = env,
quiet = False,
)
print(result.stdout)
Expand Down Expand Up @@ -186,6 +206,10 @@ def _envoy_api_deps():
name = "http_api_protos",
actual = "@googleapis//:http_api_protos",
)
native.bind(
name = "http_api_protos_native",
actual = "@googleapis//:http_api_protos_native",
)
_repository_impl(
name = "six_archive",
build_file = "@com_google_protobuf//:six.BUILD",
Expand Down
2 changes: 1 addition & 1 deletion bazel/repository_locations.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ REPOSITORY_LOCATIONS = dict(
remote = "https://github.com/bombela/backward-cpp",
),
com_github_circonus_labs_libcircllhist = dict(
commit = "476687ac9cc636fc92ac3070246d757ae6854547", # 2018-05-08
commit = "050da53a44dede7bda136b93a9aeef47bd91fa12", # 2018-07-02
remote = "https://github.com/circonus-labs/libcircllhist",
),
com_github_cyan4973_xxhash = dict(
Expand Down
18 changes: 14 additions & 4 deletions ci/build_container/build_recipes/benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,23 @@ export COMMIT="e1c3a83b8197cf02e794f61228461c27d4e78cfb" # benchmark @ Jan 11,
git clone https://github.com/google/benchmark.git
(cd benchmark; git reset --hard "$COMMIT")
mkdir build

cd build
cmake -G "Unix Makefiles" ../benchmark \

cmake_generator="Unix Makefiles"
make_cmd=make
benchmark_lib="libbenchmark.a"

if [[ "${OS}" == "Windows_NT" ]]; then

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$OS is defined under MSYS Bash and Cygwin Bash, but not under WSL Bash (Ubuntu 14.04 on Windows Subsystem for Linux).

Which Bash do you expect to run this script under?

(Same question goes for other shell scripts that use $OS.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We expect this to run under MSYS2 Bash

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sesmith177 can you add some build documentation to https://github.com/envoyproxy/envoy/blob/master/bazel/README.md that capture the build flow and also these assumptions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to PR the build documentation when we PR source code changes? As it stands, Envoy won't actually build

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to land the documentation either here or in a distinct PR for docs alone; it doesn't really make sense to make the source delta PR larger with this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, we'll follow up the source PR with a docs PR

cmake_generator="Ninja"
make_cmd=ninja
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Ninja? Is there no way to do GNU make on Win?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We decided to use Ninja because that is what Microsoft uses in their VCPKG tool for managing/building C/C++ libraries. For benchmark specifically, see: https://github.com/Microsoft/vcpkg/blob/d977ac231e995250c169ac1778b7de34f7f57ead/ports/benchmark/portfile.cmake#L22

and https://github.com/Microsoft/vcpkg/blob/master/docs/maintainers/vcpkg_configure_cmake.md#prefer_ninja

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, does it fail if you use GNU make? I'm curious what the Bazel folks who are working on external dependency integration with cmake plan on Windows, CC @iirina.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For benchmark at least, we were not able to get it to work with GNU make

benchmark_lib="benchmark.lib"
fi

cmake -G "$cmake_generator" ../benchmark \
-DCMAKE_BUILD_TYPE=RELEASE \
-DBENCHMARK_ENABLE_GTEST_TESTS=OFF
make
cp src/libbenchmark.a "$THIRDPARTY_BUILD"/lib
"$make_cmd"
cp "src/$benchmark_lib" "$THIRDPARTY_BUILD"/lib
cd ../benchmark

INCLUDE_DIR="$THIRDPARTY_BUILD/include/testing/base/public"
Expand Down
Loading