feat: Add support for range mappings proposal #77

kdy1 · 2024-01-25T06:56:59Z

See https://github.com/tc39/source-map-rfc/blob/main/proposals/range-mappings.md

src/decoder.rs

loewenheim · 2024-01-25T14:15:53Z

Hi, thank you for the contribution!

I think there is a fundamental misunderstanding here: it seems to me that the both decode_rmi function and the loop in decode_regular in which it is used assume that there is at most one range mapping per line. This is the only way I can see

            if rmi != 0 && (line_index as u32) == rmi - 1 {
                range_tokens.push(tokens.len() as u32)
            }

making sense—if the current index within the line is the number of the range token, we push the current token index to the list of range tokens.

But I think the proposal is meant to work differently: there can be many range mappings per line, and the ith bit in the rangeMappings for that line (modulo bit/byte endianness, I actually find this slightly confusing) tells you whether the ith mapping is one of them.

Consider this example: currently, decode_rmi("AAB") == 13. This is correct insofar as "AAB" makes the 13th mapping a range mapping, but the function shouldn't return 13, it should return the number with only the 13th bit set ¹. The current implementation clearly doesn't generalize to more than one range mapping per line.

Correctness aside, I'm really excited about this. These range mappings would've saved me a lot of pain in the past.

¹ Or better yet, as @Swatinem says, a bit vector.

src/types.rs

src/decoder.rs

src/types.rs

src/decoder.rs

src/encoder.rs

loewenheim · 2024-01-31T12:21:19Z

I believe converting between 0- and 1-based indices causes some unnecessary confusion here.

In decoder.rs line 219, you push a 0-based index to range_tokens.
In SourceMap::get_token you binary search range_tokens for the token's index, which is 0-based.
In test_decode_rmi, you convert to 1-based indices, I assume for purposes of clarity?
In encoder.rs line 81, you add 1 to the index to make it 1-based.
In encoder.rs lines 43-45, you assert that the indices are all greater than 0 (i.e. 1-based), but then subtract 1 to make them 0-based.

I would say remove the 1-based indexing in (4) and (5) (and arguably (3) as well) and just assume 0-based indices everywhere.

loewenheim · 2024-02-22T11:11:31Z

Just a heads-up—I haven't forgotten about this, I just haven't had the time to look at it.

src/decoder.rs

src/encoder.rs

Swatinem · 2024-02-27T12:42:27Z

src/errors.rs

+    /// Range mapping index is invalid
+    InvalidRangeMappingIndex(data_encoding::DecodeError),


Just a NOTE: this is a breaking change as Error is not (yet) marked as #[non_exhaustive]

I would say if we're ok with making a breaking change here, then we should also put the information whether a token is a range token directly on RawToken.

Swatinem

lgtm.

We have a very limited number of unit tests that just test the en/decoding of the range mappings.
Can we construct a more e2e test that shows how these are actually used?

src/encoder.rs

src/types.rs

kdy1 · 2024-03-13T02:53:27Z

@Swatinem I added some tests for lookup_token

Swatinem · 2024-03-13T15:34:16Z

The test now makes it a bit clearer what this proposal is trying to achieve.

So if I understand it correctly, new bitvec mapping adds this is_range flag.
And the is_range flag determines if the mapping is using the "offset" from the initial binary search lookup to then adjust the output line/column numbers.

With a more visual example:

   /* foo */ if (some_other_code) { return; }
//           ^ - original token
// v - output token
   if (some_other_code) { return; }
//     ^ - lookup position

So the offset between lookup position and output token is added to original token, so that you end up exactly at the same position in relative code if it is merely shifted a bit by the sourcemap?

Swatinem

code lgtm!

thanks for working on this and implementing all the feedback ❤️

Swatinem · 2024-03-13T15:38:20Z

src/builder.rs

        };
        self.tokens.push(raw);
        raw
    }

    /// Adds a new mapping to the builder.
+    #[allow(clippy::too_many_arguments)]


TBH, clippy is absolutely right to complain here. Trying to review the tests for this on github, I completely lose track of what all these parameters mean (without inlay hints).

Yeah, but I'm not sure how should I fix it.

Yes, me neither :-( I think this is fine this way, just wanted to call it out as a future improvement opportunity.

Yeah this is definitely on the list for an eventual rework. These signatures can't stay like this.

kdy1 · 2024-03-14T04:11:58Z

Yeap, exactly. This will be useful for the minifier, which does not have full input source locations.

This is the case because no tools emit a perfect source map while transpiling, and the mappings in inputSourceMap is not enough. With this RFC, it will have much more information about the input.

loewenheim

Looks good to me. Thank you for working on this!

loewenheim · 2024-03-20T09:54:22Z

src/builder.rs

        };
        self.tokens.push(raw);
        raw
    }

    /// Adds a new mapping to the builder.
+    #[allow(clippy::too_many_arguments)]


Yeah this is definitely on the list for an eventual rework. These signatures can't stay like this.

**Related issue:** - getsentry/rust-sourcemap#77

### Description Update SWC crates. 1. To keep in sync 2. Prepare usage of source map range mappings. getsentry/rust-sourcemap#77 ### Testing Instructions See next.js counterpart. vercel/next.js#63790 Closes PACK-2861

# Turbopack * vercel/turborepo#7409  * vercel/turborepo#7920  * vercel/turborepo#7929  * vercel/turborepo#7926  * vercel/turborepo#7925  * vercel/turborepo#7928  * vercel/turborepo#7856  ### What? Update SWC crates. ### Why? 1. To keep in sync 2. Prepare usage of source map range mappings. getsentry/rust-sourcemap#77 ### How? Closes PACK-2860

### Description Update SWC crates. 1. To keep in sync 2. Prepare usage of source map range mappings. getsentry/rust-sourcemap#77 ### Testing Instructions See next.js counterpart. #63790 Closes PACK-2861

sokra reviewed Jan 25, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

sokra reviewed Jan 25, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

sokra reviewed Jan 25, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

sokra reviewed Jan 25, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

Swatinem reviewed Jan 25, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

src/decoder.rs Outdated Show resolved Hide resolved

src/decoder.rs Outdated Show resolved Hide resolved

loewenheim reviewed Jan 25, 2024

View reviewed changes

src/types.rs Outdated Show resolved Hide resolved

kdy1 marked this pull request as ready for review January 29, 2024 09:13

loewenheim requested changes Jan 29, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

src/decoder.rs Outdated Show resolved Hide resolved

src/types.rs Outdated Show resolved Hide resolved

src/types.rs Outdated Show resolved Hide resolved

kdy1 marked this pull request as draft January 30, 2024 04:22

kdy1 marked this pull request as ready for review January 30, 2024 04:31

loewenheim reviewed Jan 30, 2024

View reviewed changes

src/decoder.rs Outdated Show resolved Hide resolved

src/encoder.rs Outdated Show resolved Hide resolved

src/encoder.rs Outdated Show resolved Hide resolved

src/encoder.rs Show resolved Hide resolved

kdy1 requested review from loewenheim, sokra and Swatinem February 1, 2024 02:10

Swatinem reviewed Feb 27, 2024

View reviewed changes

kdy1 requested a review from Swatinem February 28, 2024 06:24

Swatinem approved these changes Feb 28, 2024

View reviewed changes

sokra reviewed Feb 29, 2024

View reviewed changes

src/encoder.rs Outdated Show resolved Hide resolved

sokra reviewed Feb 29, 2024

View reviewed changes

src/encoder.rs Outdated Show resolved Hide resolved

sokra reviewed Feb 29, 2024

View reviewed changes

src/types.rs Outdated Show resolved Hide resolved

kdy1 requested a review from sokra March 4, 2024 07:42

kdy1 added 6 commits March 13, 2024 11:11

rsm

327c2af

range_mapping

879fb1b

range_mapping

9de6692

error variant

b13e00c

rmi_result

7558335

fix compilation

502e9a9

fix clipy

ca6d0c5

kdy1 force-pushed the range-mapping branch from b1a8d0b to ca6d0c5 Compare March 13, 2024 02:12

kdy1 changed the title ~~feat: Add support for range mappings propsal~~ feat: Add support for range mappings proposal Mar 13, 2024

kdy1 added 7 commits March 13, 2024 11:16

Update test refs

b6fd204

Add a test

d8e5bbe

fixup

2fedd9f

offset

8538bd6

saturate

a7f3356

Update test

95ff8b8

Fix test & logic

b5c4ca8

kdy1 requested a review from Swatinem March 13, 2024 02:53

Swatinem approved these changes Mar 13, 2024

View reviewed changes

Merge branch 'master' into range-mapping

df6cc81

Swatinem mentioned this pull request Mar 19, 2024

perf: using Arc<str> instead of String #84

Merged

loewenheim approved these changes Mar 20, 2024

View reviewed changes

Swatinem merged commit c95386a into getsentry:master Mar 20, 2024
4 checks passed

kdy1 deleted the range-mapping branch March 25, 2024 03:03

kdy1 mentioned this pull request Mar 25, 2024

chore(deps): Update sourcemap to v8.0.0 swc-project/swc#8771

Merged

kdy1 added a commit to swc-project/swc that referenced this pull request Mar 25, 2024

chore(deps): Update sourcemap to v8.0.0 (#8771)

f8681c9

**Related issue:** - getsentry/rust-sourcemap#77

This was referenced Mar 28, 2024

build: Update swc_core to v0.90.30 vercel/next.js#63790

Merged

build: Update swc_core to v0.90.30 vercel/turborepo#7856

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support for range mappings proposal #77

feat: Add support for range mappings proposal #77

kdy1 commented Jan 25, 2024 •

edited

Loading

loewenheim commented Jan 25, 2024 •

edited

Loading

loewenheim commented Jan 31, 2024

loewenheim commented Feb 22, 2024

Swatinem Feb 27, 2024

loewenheim Feb 28, 2024

Swatinem left a comment

kdy1 commented Mar 13, 2024

Swatinem commented Mar 13, 2024

Swatinem left a comment

Swatinem Mar 13, 2024

kdy1 Mar 14, 2024

Swatinem Mar 14, 2024

loewenheim Mar 20, 2024

kdy1 commented Mar 14, 2024

loewenheim left a comment

loewenheim Mar 20, 2024

		/// Range mapping index is invalid
		InvalidRangeMappingIndex(data_encoding::DecodeError),

feat: Add support for range mappings proposal #77

feat: Add support for range mappings proposal #77

Conversation

kdy1 commented Jan 25, 2024 • edited Loading

loewenheim commented Jan 25, 2024 • edited Loading

loewenheim commented Jan 31, 2024

loewenheim commented Feb 22, 2024

Swatinem Feb 27, 2024

Choose a reason for hiding this comment

loewenheim Feb 28, 2024

Choose a reason for hiding this comment

Swatinem left a comment

Choose a reason for hiding this comment

kdy1 commented Mar 13, 2024

Swatinem commented Mar 13, 2024

Swatinem left a comment

Choose a reason for hiding this comment

Swatinem Mar 13, 2024

Choose a reason for hiding this comment

kdy1 Mar 14, 2024

Choose a reason for hiding this comment

Swatinem Mar 14, 2024

Choose a reason for hiding this comment

loewenheim Mar 20, 2024

Choose a reason for hiding this comment

kdy1 commented Mar 14, 2024

loewenheim left a comment

Choose a reason for hiding this comment

loewenheim Mar 20, 2024

Choose a reason for hiding this comment

kdy1 commented Jan 25, 2024 •

edited

Loading

loewenheim commented Jan 25, 2024 •

edited

Loading