Adds Visual Studio Filters & Sorts #12704
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Visual Studio Filters & Sorting Options | ||
|
|
||
| - [Jon Douglas](https://github.com/JonDouglas) | ||
| - 2023-06-23 | ||
|
|
||
| ## Summary | ||
|
|
||
| <!-- One-paragraph description of the proposal. --> | ||
| This proposal represents a common ask for more filters and sorting options in the NuGet Package Manager experience in Visual Studio. | ||
|
|
||
| ## Motivation | ||
|
|
||
| <!-- Why are we doing this? What pain points does this solve? What is the expected outcome? --> | ||
| As NuGet continues to expand as a package ecosystem, there has become a dire need for being able to filter and sort packages through one's daily management in their IDE of choice. While NuGet.org supports a plethora of filters and sorts on the search experience today, we'd like to bring these similar experiences to the various tabs(browse, installed, etc) in Visual Studio to help aid various jobs-to-be-done when it comes down to package management. | ||
|
|
||
| A recent survey in May 2023 suggests that over 31% of people do not feel like there are adequate sorting and filtering options in Visual Studio when asked. Common free-response to this question relate around the following: | ||
|
|
||
| - "There are no filtering options" | ||
| - "More search options would make this better" | ||
| - "Would like to find packages by tags, authors, licenses, and more" | ||
| - "There are no way to sort packages outside of creation date" | ||
| - "I'm not aware of any filters but prerelease on/off" | ||
| - "Often I'm looking for a widely-used and respected package and I have to filter and sort through random niche-market packages" | ||
| - "I should be able to filter based on an official package source and license of a package" | ||
| - "I manually have to use google or the console because desired results are too hard to find without filters" | ||
| - "Filters and sorts do not match the website" | ||
|
|
||
| ## Explanation | ||
|
|
||
| ### Functional explanation | ||
|
|
||
| <!-- Explain the proposal as if it were already implemented and you're teaching it to another person. --> | ||
| <!-- Introduce new concepts, functional designs with real life examples, and low-fidelity mockups or pseudocode to show how this proposal would look. --> | ||
|
|
||
| When a developer is using Visual Studio to search and manage NuGet packages, they will see a new filter and sort experience. | ||
|
|
||
| The browse filter experience will allow the developer to filter results based on common concepts such as "frameworks", "prerelease", "owners", "download count", "license", and "created on". | ||
|
There was a problem hiding this comment. Do we know what will happen or what customers expect to happen for embedded licenses? There was a problem hiding this comment. license support will only be for properly expressed licenses. anything else would likely not work with the filter. data suggests this is a very rare case too. There was a problem hiding this comment. This is good rationale as to why: https://github.com/aaronpowell/dotnet-delice#undetermined-licenses and follows my thinking. There was a problem hiding this comment. Would having tags as a keyword filter be useful? I haven't thought through it, but wondering if customers mentioned tags at all, or if we understand how tags are impacting those google searches they referred to as being useful. There was a problem hiding this comment. Tags are not in scope right now as they are hardly used for even the top 100 packages in helpful ways. The list provided here comes directly from survey results. Tags was not a popular request. There was a problem hiding this comment. The owners filter does not make sense on many non-NuGet.org feeds. I assume this is explicitly NOT the authors field which is metadata and not validated. The There was a problem hiding this comment. owners specifically represents the entity that currently owns the package on the registry/source. Not to be confused with the owners/authors nuspec fields. this mostly applies to the central registry (nuget.org). There was a problem hiding this comment. If there are concepts that apply to nuget.org only, I think we should explicitly call them out in the spec and make sure those are hidden when a source does not support them. There was a problem hiding this comment. Do these filters apply only to the latest package version per package ID? This is how search works today except for the This is a nuanced space, so let's take a real example: If you search for If you uncheck prerelease, Polly disappears entirely. That's the "added metadata" case meaning the latest version matches a filter but the non-latest does not. It's a similar story for the "removed metadata" case. If you search All of this is because currently the search service only operates on the latest package version's metadata w.r.t to the "include prerelease" and (this is a small detail) the include SemVer 2.0.0 flag (which all recent clients set to true). So we index up to 4 package versions per package ID. 4 because there are 4 combinations of a For some filters, this behavior is probably fine. But what about frameworks? If the latest version of a package does not support I think the spec should call out whether the filtering applies only to the latest package version or to all package versions. If it's the latter then it will be a much bigger ask for the server side to implement since we'll need to index a lot more data (400K package IDs vs. 6 million package versions). There was a problem hiding this comment. This is my Kusto query for reference NiPackageVersions
| where IsLatestSemVer2 or IsLatestStableSemVer2
| join kind=inner NiPackageManifests on Identity
| summarize
LatestVersion = take_anyif(Version, IsLatestSemVer2),
LatestStableVersion = take_anyif(Version, IsLatestStableSemVer2),
LatestTags = take_anyif(array_sort_asc(SplitTags), IsLatestSemVer2),
LatestSplitTags = take_anyif(array_sort_asc(SplitTags), IsLatestStableSemVer2)
by LowerId
| where isnotempty(LatestVersion) and isnotempty(LatestStableVersion)
| where LatestVersion != LatestStableVersion
| extend Added = set_difference(LatestTags, LatestSplitTags)
| extend Removed = set_difference(LatestSplitTags, LatestTags)
| where array_length(Added) > 0 or array_length(Removed) > 0
| join kind=inner (NiPackageDownloads | distinct LowerId, TotalDownloads) on LowerId
| order by TotalDownloads desc
| project LowerId, LatestStableVersion, LatestVersion, Added, Removed, TotalDownloadsThere was a problem hiding this comment. It would be consistent with the current experiences as seen on screen. No experience is changing here outside the concept that you can apply additional filters and quickly sort results. So that would be latest version of a package from both the source (browse tab) and locally installed (installed/updates tab) Frameworks would also behave this way for now. If it isn't found on stable or pre-rel channels, then the functionality just doesn't show it. Mirroring search by tfm behavior on nuget.org. There was a problem hiding this comment.
Wow! That makes a lot of sense as to what behavior I have seen in the past. Unlike some, I have to maintain a lot of legacy code and often feel like Microsoft is almost deliberately tanking my experience to somehow coerce me to upgrade (as if I have not tried - it's taken 6 years to get off Framework and we're finally going to be there in a few months). I can't tell from your comments if this is something you plan to improve, but I greatly appreciate you explaining it as a limitation. There was a problem hiding this comment. In the case of NuGet.org, we're not deliberately limiting the search of older version metadata to coerce users. We do it as a way to scope the amount of metadata that we index. And, perhaps most significantly, it was a decision made long ago that resulted in the UI and protocol working a specific way. The UI and protocol all center around ID-level results, not version-level results. This does lead to smaller and perhaps more understandable results. I don't think we'd want to crowd the top results with a bunch of Newtonsoft.Json versions and I also think it's reasonable to respect the most recent wishes of the package author on what their package metadata is (i.e. the latest version metadata). Packages are immutable so there's no other way to change your metadata than to release a new version. Perhaps there's a middle ground where we only index packages when certain metadata fields change or we somehow allow users to opt into an "all version" search, but that's just not implemented today and we haven't had many complaints about it (or any in recent years that I know of). The cost would be big to change the decision at this point so we'd need a compelling reason for the user experience. Based on @JonDouglas's comment, it seems like this spec isn't the time to change course on the "indexing all vs. latest version" decision. We have a million things on our backlog so we have to be careful about the big changes we pick up so we don't overwhelm the team. If you feel that this is something we should change, please feel free to open a proposal and gather support with upvotes (perhaps starting with a bug report on NuGet/NuGetGallery). |
||
|
|
||
|  | ||
|
|
||
| The installed filter experience will allow the developer to filter results based on all the **browse filter** and additional package status filters such as "deprecated", "vulnerable", and "update". | ||
|
|
||
|  | ||
|
There was a problem hiding this comment. If we're adding a has updates filter, I think we should consider removing the updates tab. The only thing that the updates tab now has is that it has checkboxes that allow you to start the operation on multiple packages. There was a problem hiding this comment. yeah or we can just omit updates as a filter and bring the filtering/sort experiences to updates tab too. There was a problem hiding this comment. Fair. Given the amount of work that'd go into this, I'd love for us to consider removing the updates tab as a parallel workstream. It's one of those, if you do them one after the other, it might take 3 weeks each for a total of 6, but doing them together is probably just 4 weeks. There was a problem hiding this comment. May need to look again in the actual usage of update tab, but this makes sense to me. There was a problem hiding this comment. Updates tab could also be thought of as a shortcut to quickly filter. It's essentially enabling the "show updates" filter, without the option to disable it. There was a problem hiding this comment. Not part of this spec discussion, but it may be worthwhile to discuss the future of "Updates" tab - removing it or overloading it for quick "attention seeker" tab for all the 3 filters together? There was a problem hiding this comment. Btw, UI for package status seems like "enable" buttons and less of filters. I am assuming these will go through UX scrub. |
||
|
|
||
| To apply filters, the developer will click the new filter button next to the search box. This will open up a new window of the possible filtering options to choose from. | ||
|
|
||
| When a filter is selected, the filter will show up as a pill/badge affordance in the package list which can be removed by clicking on the "X" associated with the pill/badge. For some filters, they will support multiple values such as frameworks, owners, and licenses. For other filters, they will only support one value such as prerelease, download count, created on, etc. For filters that have popular values but also have custom values, a input box will be provided. For example the license filter will have popular OSS licenses, but with the ability to insert the name of a not well known license as well. | ||
|
|
||
|  | ||
|
|
||
| Whenever a filter is applied, it will additionally show up in the Visual Studio search box so that developers get used to the search prefix syntax and can apply filters more quickly in the future. | ||
|
|
||
| The sorting experience will allow the developer to sort these options on common concepts such as "alphabetical", "version", "relevance", "downloads", and "recently updated". | ||
|
There was a problem hiding this comment. I'm guessing this is jsut examples, rather than a concrete proposal? There was a problem hiding this comment. these are concrete examples 🥲 can go into more details on them if needed. There was a problem hiding this comment. Ah, I see :D My feedback on the ones I have concerns with.
Another side question, are the filters sticky? Within same session? There was a problem hiding this comment. version would sort by the package version in asc/desc order. relevance would be the default search experience today just as a filter to get back to (like most search experiences) downloads / recently updated - could we not cache the package details and use that locally? At the time of writing, filters are not sticky as typical filtering and sorting experiences are not sticky outside the context of the same session. In other words, if a user exits the PM UI, the defaults will be set again. Further customer feedback may change this behavior. There was a problem hiding this comment. another thought might be that version could possibly be helpful if you can specify a range to filter as well. that might be helpful for the .NET libraries case and upgrading those in bulk. There was a problem hiding this comment. I think versions should allow package versions and assembly versions. The reason is that if there is an assembly not found exception or similar issue, the error returns the actual assembly version rather than the package version. Right now my process is to click Show All Files in Solution Explorer, search for the problematic dll by name, and then look at the Dependencies tree to find the concrete dll version used by clicking into it. There was a problem hiding this comment. Right now our concept would only support package metadata which would be a package version. If an actual assembly version does not match the package version, that may be better resolved with the package author. There was a problem hiding this comment.
That is impractical in the case when an author ships a hotfix for an older framework. In any case, completely disagree as standard practice among many top 100 non Microsoft packages is major.0.0.0 precisely because this is a PITA to deal with. Microsoft however refuses this convention for OOB packages. Please consider this filter. There was a problem hiding this comment. I think the assembly filter is a reasonable ask, but having it as a later phase might be more practical, based on upvotes etc. I haven't heard this ask on the NuGet.org side recently. Additionally, there can be multiple assemblies per package (both via cross-targeting and with multiple assembly names). These all have different assembly versions. So a filtering model would need to handle that 1-to-many relationship between package version and assembly version. It's not a simple thing unless we take some very heavy handed concessions. There was a problem hiding this comment. Understood. I think some people don't know to ask for it, though. It's like the Steve Jobs principle of listening to your customers (we need better searching), but then figuring out what they should really have help searching for, rather than just building the ones they ask for. When you consider that .NET Tree Shaking of assemblies removes classes/methods, the possibility of version conflicts grows and the ultimate debugging details lie in what assembly you are using, not a nuget package version. - When .NET throws an assembly-related exception, the error message is always from the point of view of the assembly, not the package. I have a pretty convoluted process to work around these problems, personally, and I've mastered it, so it doesn't really affect me any more since I have now become an expert at it, but boy, do I notice a lot of people struggle with this. And the messaging that .NET Core took care of this stuff with deps.json generation is just wrong - netstandard2.0 has an unbounded upper cardinality on what .NET 5+ version it could accept, etc. Anyway, will table this aspect of the review for now - assuming there is a separate later phase documented so this doesnt get lost. There was a problem hiding this comment. I think we're missing the clarity on what exact filters we want to do and the feasibility of them. We discussed some of those concerns in https://github.com/NuGet/Home/pull/12704/files#r1244465668, so it'd be great to incorporate that here. I think it might make it easier to review if there are different sections for installed vs updates. |
||
|
|
||
| The sorting options will show up as a dropdown in the package list. | ||
|
|
||
| ### Technical explanation | ||
|
|
||
| <!-- Explain the proposal in sufficient detail with implementation details, interaction models, and clarification of corner cases. --> | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| <!-- Why should we not do this? --> | ||
|
|
||
| The NuGet package manager experience in Visual Studio today somewhat has these filtering concepts built into the default tabs such as "Browse", "Installed", "Update", and "Consolidate". There is however limited sorting options in Visual Studio for package management. By implementing this work, we may be re-inventing a small portion of work such as the current concept of the "Update" and "Consolidate" tabs such as the "update available" package status filter. | ||
|
|
||
| ## Rationale and alternatives | ||
|
|
||
| <!-- Why is this the best design compared to other designs? --> | ||
| <!-- What other designs have been considered and why weren't they chosen? --> | ||
| <!-- What is the impact of not doing this? --> | ||
|
|
||
| One major challenge with search filtering experiences today is knowing the [supported search prefixes](https://learn.microsoft.com/en-us/nuget/consume-packages/finding-and-choosing-packages#search-syntax). Technically one who knows or finds the documentation on this topic could use these in Visual Studio, but it should be more visual and easy to find in the product itself through the means of a user interface that one can interact with. | ||
|
There was a problem hiding this comment. Maybe this is too technical for a high level spec, but I'm confused if this proposal is saying NuGet client should filter on the client side, or if we're going to start making nuget.org style "advanced search" requests? If we're filtering on the client side, I guess this is an advantage of using an infinite scroll list, as it might keep making more requests to fill the screen with packages. However, if I have some very specific client side filters that filter out everything (say, I request download count over 100 billion), then PM UI is going to keep asking for page after page, until the server says there are no more search results. It obviously depends a bit on the search service technology the server uses, but high page numbers are typically more computationally expensive than low page numbers, and in this scenario PM UI will be automatically requesting these in a very short time period. On the other hand, if the proposal is to change the search query NuGet sends to servers, then are we're effective imposing a change on the server protocol. I assume we should make servers report to us which filters they support, and then disable the filter controls when a filter is not supported? When the "all" sources is selected, we should use the lowest common denominator? Or is the proposal to just always send an "advanced query", and show the customer low quality results, whatever the server happens to return? There was a problem hiding this comment.
+1 There was a problem hiding this comment. The aim would be to bring browse parity to NuGet.org search experiences first. There are plenty of experiences that work today in VS that users simply don't know about and it would be very helpful to expose that through this proposal. If another registry does not support our search syntax/etc, then this feature will not work for them full stop. If they support it, then they would see the functionality light up. For local packages (i.e. installed), we want to bring quality of life filters and sorts based on whatever we have cached / available. Some of the browse filters/sorts may just simply not apply if they cost too much to retrieve metadata about / the package does not have an equivalent on nuget.org. There was a problem hiding this comment. Okay, so this is a request for protocol changes. This can be done, certainly, but we'd need to plan them carefully so that a) other package sources could reasonably implement them in part or whole (e.g. support sort by published date but not filter by owners) and b) the experience is reasonable when other sources do not support the new protocol at all which will be the case forever for some sources, and for a good amount of time for all sources (e.g. waiting for ADO to ship support). This is just expanding on @zivkan's point a bit. This also brings in the question of upstreaming again. If a lot of folks use ADO upstreaming, is reduces the impact of nuget.org implementing the protocol so perhaps a parallel activity would be getting early buy-in from ADO or even GPR. |
||
|
|
||
| This concept plays off of inspiration for new package management UI concepts we've designed in the past while also taking on contemporary UI/UX concepts found in popular applications such as Etsy, Spotify, AirBnB, and more. | ||
|
|
||
| Additionally, the advanced search experience helps users learn the current prefixes that are supported while also being consistent with other platforms such as how GitHub implemented advanced search. | ||
|
|
||
| ## Prior Art | ||
|
|
||
| <!-- What prior art, both good and bad are related to this proposal? --> | ||
| <!-- Do other features exist in other ecosystems and what experience have their community had? --> | ||
| <!-- What lessons from other communities can we learn from? --> | ||
| <!-- Are there any resources that are relevant to this proposal? --> | ||
| - [NuGet Visual Studio Package Manager UI vNext](https://github.com/microsoft/vscode-dotnettools/issues/62#issuecomment-1582471743) | ||
| - [GitHub](https://github.com/search) and [GitHub Advanced Search](https://github.com/search/advanced) | ||
| - [Etsy](https://www.etsy.com/search) | ||
| - [Airbnb](https://www.airbnb.com/?adults=2&search_mode=flex_destinations_search) | ||
| - [Spotify](https://open.spotify.com/search/rock) | ||
| - [Google](https://www.google.com/search) | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| <!-- What parts of the proposal do you expect to resolve before this gets accepted? --> | ||
| <!-- What parts of the proposal need to be resolved before the proposal is stabilized? --> | ||
| <!-- What related issues would you consider out of scope for this proposal but can be addressed in the future? --> | ||
|
|
||
| ## Future Possibilities | ||
|
|
||
| <!-- What future possibilities can you think of that this proposal would help with? --> | ||
| - There is additionally a concept of "dependency sorting" which would sort dependencies based on the correct order to maintain compatibility such as updating versions. This could be a helpful feature if feasible / done correctly. | ||
| - Ability for Visual Studio experiences to deep link specific filters and sorts for different development experiences. For example finding packages that are only supported for Blazor or another platform. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm guessing for licenses we'd only support SPDX?
created on=>published on.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Licenses most likely would need to be SPDX expressed, yes. Maybe a custom textbox for anything outside that but that is a edge case imo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
worth clarifying that.
I'm not so convinced files are an edge case though.
says
It's close to 10%. 31% have license url as well which also need to fall in that group.
Granted, I'm looking at all packages, but I think the Expression vs File ratio still applies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I was referring to the SPDX expressed name as those are typically the same as the file expression. For the edge case, I was referring to atypical licenses. For the sake of this feature, if it is too difficult to parse a file / licenseURL, we should only support expression to encourage best practices.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't mind having the filter, I was mostly just trying to clarify behavior for the places where it won't make sense.
It was a very technical approach to reading the spec from my side.