Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for gzipped xrefmap that stored as local file #9966

Merged
merged 2 commits into from
Jun 5, 2024

Conversation

filzrev
Copy link
Contributor

@filzrev filzrev commented May 30, 2024

This PR add support for gzip compressed xrefmap file.

Background

xrefmap files can exceed 100 MB in size in some cases.
For example. .NET API's xrefmap takes 132MB. (And 8.5MB when using gzip compression)

And when specifying xrefmap with URL schema.
docfx download xrefmap file on every build.

So it's recommended to use pre downloaded local file for large xrefmap.
But in this case, xrefmap can't be stored on GitHub repository. (By default it's limited to50MB and It requires git LFS for larger file)

And currently there is zipped xrefmap support that compressed by zip.
But it support only yml format that is created by dotnet download command.
And yml file processing run 4x slower than json.

This PR add support for gzipped xrefmap file to resolve described above issue.

Copy link

codecov bot commented May 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.91%. Comparing base (fe673ec) to head (72e686c).
Report is 189 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9966      +/-   ##
==========================================
+ Coverage   74.31%   78.91%   +4.59%     
==========================================
  Files         536      540       +4     
  Lines       23189    23439     +250     
  Branches     4056     4060       +4     
==========================================
+ Hits        17234    18497    +1263     
+ Misses       4853     3804    -1049     
- Partials     1102     1138      +36     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yufeih
Copy link
Contributor

yufeih commented Jun 4, 2024

will a .xrefmap.json.gz be uploaded in a separate PR?

@yufeih yufeih added the performance Makes the pull request appear in "Performance" section of the next release note label Jun 4, 2024
@filzrev filzrev changed the title feat: Add support for gzipped xrefmap file feat: Add support for gzipped xrefmap that stored as local file Jun 4, 2024
@filzrev
Copy link
Contributor Author

filzrev commented Jun 4, 2024

will a .xrefmap.json.gz be uploaded in a separate PR?

This PR intended to support gzipped files that saved on local files.
For network files. gzip support is not implemented.
Because it should use HTTP Compression if available.

For https://github.com/dotnet/docfx/raw/main/.xrefmap.json file.
It's stored on Git LFS. and don't support HTTP Compression. (#9659 (comment))

In this case.
It need manually download xrefmap and compress as gzip.

Alternatively. It can use xrefmap provided by learn.microsoft.com (Though It might not officially supported)

https://learn.microsoft.com/en-us/dotnet/.xrefmap.json

Copy link
Contributor

@yufeih yufeih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@yufeih yufeih added new-feature Makes the pull request to appear in "New Features" section of the next release note and removed performance Makes the pull request appear in "Performance" section of the next release note labels Jun 5, 2024
@yufeih yufeih merged commit 168b704 into dotnet:main Jun 5, 2024
7 checks passed
p-kostov pushed a commit to ErpNetDocs/docfx that referenced this pull request Jun 28, 2024
renovate bot referenced this pull request in buehler/dotnet-operator-sdk Jul 1, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [docfx](https://github.com/dotnet/docfx) | `2.76.0` -> `2.77.0` |
[![age](https://developer.mend.io/api/mc/badges/age/nuget/docfx/2.77.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/nuget/docfx/2.77.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/nuget/docfx/2.76.0/2.77.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/nuget/docfx/2.76.0/2.77.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>dotnet/docfx (docfx)</summary>

### [`v2.77.0`](https://github.com/dotnet/docfx/releases/tag/v2.77.0)

<!-- Release notes generated using configuration in .github/release.yml
at main -->

#### What's Changed

##### 💥 Breaking Changes

- chore: Drop .NET 7 SDK supports by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9911](https://github.com/dotnet/docfx/pull/9911)

##### 🎉 New Features

- feat: Enable PreferCSSPageSize option for PDF generation by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9850](https://github.com/dotnet/docfx/pull/9850)
- feat: Add docfx JSON Schema files and related tests by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9852](https://github.com/dotnet/docfx/pull/9852)
- feat: Add `toc.json` transform logics using `toc.extension.js` by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9953](https://github.com/dotnet/docfx/pull/9953)
- feat: Better error when link not found by
[@&#8203;Patrick8639](https://github.com/Patrick8639) in
[https://github.com/dotnet/docfx/pull/9957](https://github.com/dotnet/docfx/pull/9957)
- feat: Add `categoryLayout` option for metadata generation by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9965](https://github.com/dotnet/docfx/pull/9965)
- feat: Permits to specify the placement of overwrites by
[@&#8203;Patrick8639](https://github.com/Patrick8639) in
[https://github.com/dotnet/docfx/pull/9937](https://github.com/dotnet/docfx/pull/9937)
- feat: Support private symbols by
[@&#8203;Patrick8639](https://github.com/Patrick8639) in
[https://github.com/dotnet/docfx/pull/9972](https://github.com/dotnet/docfx/pull/9972)
- feat: Add support for gzipped xrefmap that stored as local file by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9966](https://github.com/dotnet/docfx/pull/9966)
- feat: Enable view transitions feature by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9909](https://github.com/dotnet/docfx/pull/9909)

##### 🐞 Bug Fixes

- fix: PDF `Producer` document information by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9851](https://github.com/dotnet/docfx/pull/9851)
- fix: Xrefmap baseUrl problem reported at
[#&#8203;9866](https://github.com/dotnet/docfx/issues/9866) by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9869](https://github.com/dotnet/docfx/pull/9869)
- fix: Xref links are not resolved on docs site by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9880](https://github.com/dotnet/docfx/pull/9880)
- fix: Change same URL check logic to case invariant by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9883](https://github.com/dotnet/docfx/pull/9883)
- fix: Improve unresolved xref messages by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9884](https://github.com/dotnet/docfx/pull/9884)
- fix: Fix nightly build errors by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9913](https://github.com/dotnet/docfx/pull/9913)
- fix: TOC filter value is not shared between pages by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9912](https://github.com/dotnet/docfx/pull/9912)
- fix: Build problems when running .NET 6 version of docfx on some
environment by [@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9922](https://github.com/dotnet/docfx/pull/9922)
- fix: `docfx metadata` command throw `ArgumentException` when
referencing empty namespace by doc comment by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/10023](https://github.com/dotnet/docfx/pull/10023)
- fix: serve url link by
[@&#8203;WeihanLi](https://github.com/WeihanLi) in
[https://github.com/dotnet/docfx/pull/10035](https://github.com/dotnet/docfx/pull/10035)

##### 🚀 Performance Improvements

- perf: Change serializer for XrefMap from NewtonsoftJson to
System.Text.Json by [@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9872](https://github.com/dotnet/docfx/pull/9872)
- perf: Remove some enum boxing in GlobMatcher by
[@&#8203;lahma](https://github.com/lahma) in
[https://github.com/dotnet/docfx/pull/10051](https://github.com/dotnet/docfx/pull/10051)
- perf: Optimize CountWordInText by
[@&#8203;lahma](https://github.com/lahma) in
[https://github.com/dotnet/docfx/pull/10050](https://github.com/dotnet/docfx/pull/10050)

##### 🔧 Engineering

- chore: Skip unstable SVG content check that returned from PlantUML
Online Server by [@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9858](https://github.com/dotnet/docfx/pull/9858)
- deps: Update Spectre.Console package versions by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9894](https://github.com/dotnet/docfx/pull/9894)
- chore: fix NU5129 warning on `dotnet pack` command by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9942](https://github.com/dotnet/docfx/pull/9942)
- chore: Add PolySharp libarary to use latest C# syntax by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9960](https://github.com/dotnet/docfx/pull/9960)
- chore: Add snapshot update workflow by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9969](https://github.com/dotnet/docfx/pull/9969)
- chore: Remove unused workflow settings by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/10030](https://github.com/dotnet/docfx/pull/10030)

##### 📄 Documentation

- docs: Fix typo by
[@&#8203;carlos-regis](https://github.com/carlos-regis) in
[https://github.com/dotnet/docfx/pull/9871](https://github.com/dotnet/docfx/pull/9871)
- docs: Fix URL in markdown and match to html example by
[@&#8203;si618](https://github.com/si618) in
[https://github.com/dotnet/docfx/pull/9881](https://github.com/dotnet/docfx/pull/9881)
- docs: Fix documentation site build warnings by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9895](https://github.com/dotnet/docfx/pull/9895)
- docs: Fix missing docfx.json config docs by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9935](https://github.com/dotnet/docfx/pull/9935)
- docs: Fix Docfx.App nuget package usage document by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/9994](https://github.com/dotnet/docfx/pull/9994)
- docs: Fix plugin related docs and logs by
[@&#8203;filzrev](https://github.com/filzrev) in
[https://github.com/dotnet/docfx/pull/10029](https://github.com/dotnet/docfx/pull/10029)

#### New Contributors

- [@&#8203;carlos-regis](https://github.com/carlos-regis) made their
first contribution in
[https://github.com/dotnet/docfx/pull/9871](https://github.com/dotnet/docfx/pull/9871)
- [@&#8203;si618](https://github.com/si618) made their first
contribution in
[https://github.com/dotnet/docfx/pull/9881](https://github.com/dotnet/docfx/pull/9881)
- [@&#8203;Patrick8639](https://github.com/Patrick8639) made their
first contribution in
[https://github.com/dotnet/docfx/pull/9957](https://github.com/dotnet/docfx/pull/9957)

**Full Changelog**:
dotnet/docfx@v2.76.0...v2.77.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 9pm,before 6am" in timezone
Europe/Zurich, Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/buehler/dotnet-operator-sdk).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjQyMS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@filzrev filzrev deleted the feat-add-gzipped-xrefmap-support branch September 15, 2024 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-feature Makes the pull request to appear in "New Features" section of the next release note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants