Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 #11023

Closed
wants to merge 3 commits into from

Conversation

adamkotwasinski
Copy link
Contributor

@adamkotwasinski adamkotwasinski commented Apr 30, 2020

Commit Message: Jinja dependency upgrade (2.10.3 -> 2.11.2)

Additional Description:
The archive we are downloading changed its structure (.py files moved from jinja2 to src/jinja2).
Trivial globbing src/jinja2/*.py causes problems, as the result library is provided as src.jinja2.
Even if we accept this (despite looking ugly), then it looks like jinja invokes code reflectively (via python exec) to generate templates, and that fails because the generated code appears to invoke module jinja2 again.
So I did the trivial thing - moved all the .py files from src/jinja2 to jinja2, so the library would look okay. Unfortunately couldn't figure out how to list output files as function of input files, but I'm not a Bazel wizard.
misc commentary: if http_archive (what makes envoy_http_archive work) could strip multiple segments with stip_prefix we could have just stripped jinja2-2.11.2/src, as we do not care about the rest of repository.

Risk Level: Low
Testing: Build machine & local; Kafka integration tests pass too
Docs Changes: some txt files got changed too
Release Notes: N/A

Signed-off-by: Michael Payne <michael@sooper.org>
genrule(
name = "move_sources_up_one_level",
srcs = ["//:jinja2_sources"],
outs = [
Copy link
Contributor Author

@adamkotwasinski adamkotwasinski May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't figure this out, I wish I knew how to do something like
outs = [f.replace('src/jinja2', 'jinja2') for f in //:jinja2_sources]

@adamkotwasinski adamkotwasinski changed the title [WORK IN PROGRESS DO NOT REVIEW] Jinja2 dependency upgrade Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 May 1, 2020
@adamkotwasinski adamkotwasinski marked this pull request as ready for review May 1, 2020 21:22
@adamkotwasinski
Copy link
Contributor Author

/review @htuch @moderation

py_library(
name = "jinja2",
srcs = glob(["jinja2/**/*.py"]),
srcs = ["//:move_sources_up_one_level"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use https://docs.bazel.build/versions/master/be/python.html#py_binary.imports to enable the glob to still work?

@adamkotwasinski
Copy link
Contributor Author

Converting back to draft - looks like at least one other lib was doing something similar e.g. https://github.com/envoyproxy/envoy/blob/master/bazel/external/apache_thrift.BUILD#L4
so I can reuse their pretty code

@htuch had an great idea of using pip_import but I believe it would be larger build engineering effort to move all python libs (thrift, jinja, twitter, etc.) there.

@adamkotwasinski adamkotwasinski changed the title Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 [DRAFT] Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 May 1, 2020
@adamkotwasinski adamkotwasinski marked this pull request as draft May 1, 2020 22:25
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
…e python package is still 'jinja2'

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
Copy link
Contributor

@moderation moderation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All checks passed! LGTM

@adamkotwasinski adamkotwasinski marked this pull request as ready for review May 3, 2020 05:32
@adamkotwasinski adamkotwasinski changed the title [DRAFT] Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 Jinja2 dependency upgrade: 2.10.3 -> 2.11.2 May 3, 2020
) for f in src_files],
),
visibility = ["//visibility:private"],
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed on Slack, I reckon https://github.com/bazelbuild/rules_python would be a better option and probably more maintainable. I don't think it would be a ton of work to get happening, but famous last words. If you prefer you can file an issue and put a TODO here.

Out of curiosity, would https://docs.bazel.build/versions/master/be/python.html#py_binary.imports have been any help?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't get imports to work, but I've had only limited exposure to bazel.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure; I'm actually doing some of the required plumbing here in #11058, so in the near future we might have pip3 support in the repository.

@adamkotwasinski
Copy link
Contributor Author

I'll close this issue for now, as I think we got into a bit of XY problem, where python dependencies should be managed in a bit different way, without all the black magic that's touched in this PR.
All in all, it's not a Kafka issue per se - @moderation feel free to reuse these commits if you want to go the way of amending what is present now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants