Support dict select union #15075

AlessandroPatti · 2022-03-18T14:59:04Z

Starlark has recentely added support for union operations over dictionaries (bazelbuild/starlark#215). The syntax is already supported in bazel as of #14540, but the same operation with selects of dictionaries is still usupported.

Related issue: #12457

fmeum · 2022-04-26T14:09:02Z

@tetromino Would you be able to review this? It's the reason why I worked on introducing the dict union operator in the first place.

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java

src/main/java/com/google/devtools/build/lib/packages/Type.java

tetromino

Please add tests for type/operator mismatches: dict | list, list | dict, dict + list, list + dict.

fmeum · 2022-05-21T17:29:45Z

@comius AFAIK tetromino won't be available until July. Is there anybody else who could take over the review?

adonovan · 2022-05-22T15:20:58Z

src/main/java/com/google/devtools/build/lib/packages/Type.java

@@ -559,6 +560,15 @@ public Map<KeyT, ValueT> convert(Object x, Object what, Object context)
      return ImmutableMap.copyOf(result);
    }

+    @Override
+    public Map<KeyT, ValueT> concat(Iterable<Map<KeyT, ValueT>> iterable) {
+      LinkedHashMap<KeyT, ValueT> output = new LinkedHashMap<>();


Presumably LinkedHashMap is used instead of ImmutableMap.Builder (at the cost of a copy) to avoid an exception when the keys intersect. A comment to this effect would be good.

Better still: use Dict.Builder, which avoids the exception, and will take advantage of the exception-free builder semantics (buildKeepingLast) when they become available.

buildKeepingLast is in fact already available: https://cs.opensource.google/bazel/bazel/+/4eb62cd07393e495ab9ead49b771871d1050ee51:src/main/java/com/google/devtools/build/lib/actions/ActionEnvironment.java;l=111

Updated to use buildKeepingLast, although from the docs:

In the current implementation, all values associated with a given key are stored in the Builder object, even though only one of them will be used in the built map. If there can be many repeated keys, it may be more space-efficient to use a LinkedHashMap and ImmutableMap.copyOf(Map) rather than ImmutableMap.Builder.

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java

AlessandroPatti · 2022-06-29T11:50:01Z

@fmeum @tetromino @comius Any update on this? I'd be awesome to have it in the next release

fmeum · 2022-07-07T08:00:05Z

@AlessandroPatti Not a Googler, so I can't do more than stating that this LGTM.

@brandjon Do you happen to have cycles to get this into 5.3.0?

fmeum · 2022-08-03T18:29:02Z

@comius @tetromino Friendly ping. This has been in review for almost half a year now, it would be great to finally get it merged.

AlessandroPatti · 2022-08-18T08:52:05Z

@comius @tetromino @gregestren Friendly ping, is there any blocker for merging this?

tetromino · 2022-08-18T18:07:40Z

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java

-   * @throws EvalException if all values don't have the same underlying type
+   * @throws IllegalStateException if all values don't have the same underlying type
   */
-  static SelectorList of(Iterable<?> values) throws EvalException {
+  static SelectorList of(Iterable<?> values) {


I disagree with this change. Unchecked exceptions in the Starlark interpreter are dangerous: at worst, they could lead to a Bazel server process crash; at best, they might be caught somewhere in the depths of skyframe where we would be in no position to print a useful error message. (In particular, what worries me is the possibility of the unchecked exception in of being thrown as a result of a native.existing_rule() call - see StarlarkNativeModule.) Instead, we genuinely do want to throw an EvalException, because that's what gets caught at the boundary between Bazel and the Starlark interpreter and results in an error message with a beautiful Starlark stack trace using which a rule author can use to fix their rules.

(Also, for future reference: if we wanted to use an unchecked exception, we would signal an invalid argument to a function by throwing an IllegalArgumentException, not an IllegalStateException. See e.g. Guava's https://github.com/google/guava/wiki/PreconditionsExplained guide.)

From my understanding, the other two places where this function is called are not performing any evaluation, but rather exposing this data structure in a human-friendly way. This is why I assumed that this should never happen (assuming the validation during the creation of the SelectorList is correct) and it is thus an "illegal state" if it happens. Regardless, reverted to throw an EvalException if this happens to be on the safe side.

src/main/java/com/google/devtools/build/lib/packages/SelectorValue.java

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java

tetromino · 2022-08-18T19:22:55Z

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java

+    if (!canConcatenate(getNativeType(this), getNativeType(that))) {
+      throw Starlark.errorf(
+          "'%s' operator applied to incompatible types (%s, %s)",
+          op.toString(), getTypeName(thisLeft ? this : that), getTypeName(thisLeft ? that : this));
+    }
+    if (getNativeType(that).equals(Dict.class)) {
+      if (op == TokenKind.PIPE) {
+        return thisLeft ? concat(this, that) : concat(that, this);
+      }
+    }


This means we are pointlessly checking canConcatenate twice: in binaryOp and in of().

A solution, IMHO, would be to make concat a private method which simply does return new SelectorList(getNativeType(x), ImmutableList.of(x, y)) without performing any canConcatenate check. It should be easy to do since we're no longer calling concat from SelectorValue.

Tried making concat private and return a new SelectorList directly but

A few tests use concat, so the visibility cannot be private, it would require using @VisibileForTesting

of flattens the SelectorLists elements, which we would not do here instantiating the SelectorList directly, causing errors like expected type <type> for attribute <attr>, but got select

Instead I removed the check entirely so we rely on of to fail if the types cannot be combined.

tetromino · 2022-08-18T19:27:10Z

My apologies for the very late review!

tetromino

Thank you, looks good now!

Starlark has recentely added support for union operations over dictionaries (bazelbuild/starlark#215). The syntax is already supported in bazel as of bazelbuild#14540, but the same operation with selects of dictionaries is still usupported. Related issue: bazelbuild#12457 Closes bazelbuild#15075. PiperOrigin-RevId: 469827107 Change-Id: If82cfaf577db41efc2e9b55af47b0b1710badc10

sgowroji added team-Starlark-Interpreter Issues involving the Starlark interpreter used by Bazel team-Build-Language labels Mar 19, 2022

AlessandroPatti force-pushed the apatti/select-dict branch 2 times, most recently from 3c29422 to 7ff757c Compare March 30, 2022 19:41

sgowroji requested a review from brandjon April 21, 2022 04:51

sgowroji added the awaiting-review PR is awaiting review from an assigned reviewer label Apr 21, 2022

tetromino reviewed Apr 27, 2022

View reviewed changes

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java Outdated Show resolved Hide resolved

tetromino reviewed Apr 27, 2022

View reviewed changes

src/main/java/com/google/devtools/build/lib/packages/SelectorList.java Outdated Show resolved Hide resolved

tetromino reviewed Apr 27, 2022

View reviewed changes

src/main/java/com/google/devtools/build/lib/packages/Type.java Show resolved Hide resolved

tetromino requested changes Apr 27, 2022

View reviewed changes

AlessandroPatti force-pushed the apatti/select-dict branch 3 times, most recently from aa0c989 to 13675fc Compare April 30, 2022 15:33

AlessandroPatti requested a review from tetromino May 11, 2022 17:58

adonovan reviewed May 22, 2022

View reviewed changes

AlessandroPatti force-pushed the apatti/select-dict branch from 13675fc to 462f796 Compare May 25, 2022 21:00

tetromino requested changes Aug 18, 2022

View reviewed changes

AlessandroPatti added 6 commits August 21, 2022 13:49

Support dict select union

006b744

Fix dicts with same key

a20ddbf

Review feedback

2f89dcc

ImmutableMap.Builder

003fcb9

EvalException

b117fcb

Feedback

5d3d76d

Dict

f6ff54b

AlessandroPatti force-pushed the apatti/select-dict branch from 462f796 to f6ff54b Compare August 21, 2022 11:49

AlessandroPatti requested review from tetromino and removed request for brandjon August 21, 2022 11:50

tetromino approved these changes Aug 22, 2022

View reviewed changes

copybara-service bot closed this in ebae486 Aug 24, 2022

ShreeM01 removed the awaiting-review PR is awaiting review from an assigned reviewer label Sep 15, 2022

ankit-agarwal-ai mentioned this pull request Dec 1, 2022

Support dict select union AppliedIntuition/bazel#4

Merged

tetromino mentioned this pull request Aug 28, 2024

Convert values to Starlark values when concating string_list_dict type #23424

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support dict select union #15075

Support dict select union #15075

AlessandroPatti commented Mar 18, 2022 •

edited

Loading

fmeum commented Apr 26, 2022

tetromino left a comment

fmeum commented May 21, 2022

adonovan May 22, 2022

fmeum May 22, 2022

AlessandroPatti May 25, 2022

AlessandroPatti commented Jun 29, 2022

fmeum commented Jul 7, 2022

fmeum commented Aug 3, 2022

AlessandroPatti commented Aug 18, 2022 •

edited

Loading

tetromino Aug 18, 2022

AlessandroPatti Aug 21, 2022 •

edited

Loading

tetromino Aug 18, 2022

AlessandroPatti Aug 21, 2022

tetromino commented Aug 18, 2022

tetromino left a comment

Support dict select union #15075

Support dict select union #15075

Conversation

AlessandroPatti commented Mar 18, 2022 • edited Loading

fmeum commented Apr 26, 2022

tetromino left a comment

Choose a reason for hiding this comment

fmeum commented May 21, 2022

adonovan May 22, 2022

Choose a reason for hiding this comment

fmeum May 22, 2022

Choose a reason for hiding this comment

AlessandroPatti May 25, 2022

Choose a reason for hiding this comment

AlessandroPatti commented Jun 29, 2022

fmeum commented Jul 7, 2022

fmeum commented Aug 3, 2022

AlessandroPatti commented Aug 18, 2022 • edited Loading

tetromino Aug 18, 2022

Choose a reason for hiding this comment

AlessandroPatti Aug 21, 2022 • edited Loading

Choose a reason for hiding this comment

tetromino Aug 18, 2022

Choose a reason for hiding this comment

AlessandroPatti Aug 21, 2022

Choose a reason for hiding this comment

tetromino commented Aug 18, 2022

tetromino left a comment

Choose a reason for hiding this comment

AlessandroPatti commented Mar 18, 2022 •

edited

Loading

AlessandroPatti commented Aug 18, 2022 •

edited

Loading

AlessandroPatti Aug 21, 2022 •

edited

Loading