Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SemVer2 - Missing db index on Packages table #498 #4073

Merged
merged 2 commits into from
Jun 9, 2017
Merged

Conversation

xavierdecoster
Copy link
Member

Noticed an increase in DTU consumption on the gallery db after deploying semver2, and it's caused by a missing index on the Packages table.

The Query Processor estimates that implementing the following index could improve the query cost by 99.5235%.

Affected EF query:

(@p__linq__0 int, @p__linq__4 int,@p__linq__5 int)

SELECT	[GroupBy1].[A1] AS [C1]
FROM 
( 
	SELECT	COUNT(1) AS [A1]
    FROM	[dbo].[Packages] AS [Extent1]
    WHERE	([Extent1].[Deleted] <> 1) 
		AND (([Extent1].[SemVerLevelKey] = @p__linq__0) 
			OR (([Extent1].[SemVerLevelKey] IS NULL) AND (@p__linq__0 IS NULL))
			) 
		AND ((
			CASE 
			WHEN (@p__linq__4 = @p__linq__5) 
			THEN [Extent1].[IsLatestSemVer2] 
			ELSE [Extent1].[IsLatest] 
			END) = 1)
)  AS [GroupBy1]

@joelverhagen
Copy link
Member

What are the parameters in this case? What code path does this primarily affect?

{
public override void Up()
{
Sql("IF NOT EXISTS(SELECT * FROM sys.indexes WHERE name = 'nci_wi_Packages_SemVerLevelKey_PackageRegistrationKey' AND object_id = OBJECT_ID('Packages')) CREATE NONCLUSTERED INDEX [nci_wi_Packages_SemVerLevelKey_PackageRegistrationKey] ON [dbo].[Packages] ([SemVerLevelKey],[IsPrerelease],[Deleted]) INCLUDE ([PackageRegistrationKey],[Description],[Tags])");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for these not to be in a single migration?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they were separate PRs, and have been evaluated separately to find out whether the change was for the better or not.

@xavierdecoster
Copy link
Member Author

Want to point out these indices have already been applied (see offline discussion), and monitoring has clearly shown they had a positive effect on gallery db DTU consumption.

@xavierdecoster xavierdecoster merged commit b590859 into master Jun 9, 2017
@xavierdecoster xavierdecoster deleted the dev-eng-498 branch June 9, 2017 06:47
joelverhagen added a commit that referenced this pull request Jun 9, 2017
* Fixed Report Abuse Page's Accessibility (#4001)

Fixes #4002. Relevant to [VSTS #395879](https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems?id=395879&_a=edit). This help-text bug was for the sign-in page, but this also affects the Report Abuse page.

* Fixed accessibility of email field in report abuse page
* Fixed signature accessibility

* Use ServerCommon's Response Code Processor (#3999)

* Added Nuget.Services.Logging; moved to TelemetryResponseCodeProcessor
* Added binding redirect for AI

* Removing the "WITH(ONLINE=ON)" from the index creation as per #3952 (#4004)

* Removing the "WITH(ONLINE=ON)" from the index creation as per #3952 (#4004) (#4030)

* Remove validation rules that block uploading valid semver2 versions #3645 (#3757)

* Add support for semVerLevel query parameter to V2 endpoints (#3714)

* adding new optional semVerLevel query parameter to v2 odata endpoints

* adding new optional semVerLevel query parameter to v2 autocomplete endpoints

* Applying semVerLevel filter on v2 OData endpoints

* Use [FromUri] attribute on semVerLevel
(avoids having single quotes in the parameter value)

* Ensure navigation links on v2 feeds use normalized version

* Clarifying comment on Get(Id=,Version=) v2 API

* Properly default to semver2 inclusion on Get(Id=,Version=)

* Compare NormalizedVersion to be able to retrieve matching SemVer2 package versions for a given normalized version string.

* Code review feedback

* Update and fix broken test data

* Keep legacy version compliance checks in place for non-SemVer2 versions (#3761)

* Keep legacy version compatibility checks in place for non-semver2 versions

* Added comment to clarify the reasons behind the legacy version check.

* Fix typo

* Rename test for clarity

* code review feedback

* Set SemVerLevelKey after setting Dependencies

* #3861 V2 NuGetEntityTypeSerializer Id link patcher must retain curated feed name (#3864)

* Support Is(Absolute)Latest for SemVer2 + semVerLevel for SearchService (#3842)

* LuceneIndexingService in Gallery should take into account IsLatest(Stable)SemVer2 (#3863)

* POST VerifyPackage version validation should use ToFullString comparison

* ODataV2CuratedFeedController should support semver2 by default when requesting a specific version, and compare on NormalizedVersion

* Refactor NuGetEntityTypeSerializer + unit test coverage (#3879)

* Use NormalizedVersion in URLs contained in PackageAddedNotice (#3886)

* Add nullcheck and use TryParse. (#3890)

* Show full version on package details page (#3887)

* Highlight semver2 packages on package details view (#3893)

* Fix bug in IsLatest(Stable)SemVer2 (#3895)

* VerifyPackage on ApiController should treat version as optional parameter (#3903)

* UpdateIsLatest not resetting IsLatest(Stable)SemVer2 on previous latest versions (#3909)

* Fix malformed URL in redirect after package upload (#3915)

* Minor fix for search results package URLs (when to use version or not in the URL)

* UrlHelper extension for Package should use NormalizedVersion (#3925)

* Default to latest stable semver2 on package details page (#3930)

* User profile page does not show SemVer 2.0.0 packages #3911 (#3933)

* Fix functional test failure SearchMicrosoftDotNetCuratedFeed #3941 (#3942)

* Fix System.NotSupportedException on User profile page (#3943)

* Fix Functional Test failure for ODataFeeds.V2FeedExtendedTests.FindPackagesByIdTest #3947 (#3948)

* Fix load test failure due to incorrect test setup (#3957)

* Hijack IsLatest(Stable)Version OData filter when semVerLevel=2.0.0 (#3966)

* Detect if package only differ by metadata and show optimal user-facing error message (#3970)

* Update Semver2 package details message with final nuget client version #3897 (#3988)

* On package validation failure an actionable error message should be displayed. #3916 (#4031)

* Make downloads link on home page a proper link (#4052)

* Fix the date format on stats page (#4057)

* Update telemetry processors (#4059)

* Reorder SemVer2Latest migration to match deployment history (#4062)

* Average download shown incorrectly when its 1.x #4039 (#4040)

* Average download shown incorrectly when its 1.x #4039

* Moved logic to viewmodel and added UTs

* Port latest changes made to package details page

* SemVer2 - Missing db index on Packages table #498 (#4073)

* SemVer2 - Missing db index on Packages table #498

* SemVer2 - Missing db index on Packages table for partial search #499 (#4074)

* Package-Versions autocomplete endpoint does not properly handle semVerLevel when using the db #4086 (#4087)

* v2 package-versions auto-complete endpoint should exclude deleted versions #4092 (#4093)

* Remove auto-refresh AJAX call for total stats on home page #4090 (#4091)

* Fix merge conflicts
joelverhagen added a commit that referenced this pull request Jun 19, 2017
* Fixed Report Abuse Page's Accessibility (#4001)

Fixes #4002. Relevant to [VSTS #395879](https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems?id=395879&_a=edit). This help-text bug was for the sign-in page, but this also affects the Report Abuse page.

* Fixed accessibility of email field in report abuse page
* Fixed signature accessibility

* Use ServerCommon's Response Code Processor (#3999)

* Added Nuget.Services.Logging; moved to TelemetryResponseCodeProcessor
* Added binding redirect for AI

* Removing the "WITH(ONLINE=ON)" from the index creation as per #3952 (#4004)

* Removing the "WITH(ONLINE=ON)" from the index creation as per #3952 (#4004) (#4030)

* Remove validation rules that block uploading valid semver2 versions #3645 (#3757)

* Add support for semVerLevel query parameter to V2 endpoints (#3714)

* adding new optional semVerLevel query parameter to v2 odata endpoints

* adding new optional semVerLevel query parameter to v2 autocomplete endpoints

* Applying semVerLevel filter on v2 OData endpoints

* Use [FromUri] attribute on semVerLevel
(avoids having single quotes in the parameter value)

* Ensure navigation links on v2 feeds use normalized version

* Clarifying comment on Get(Id=,Version=) v2 API

* Properly default to semver2 inclusion on Get(Id=,Version=)

* Compare NormalizedVersion to be able to retrieve matching SemVer2 package versions for a given normalized version string.

* Code review feedback

* Update and fix broken test data

* Keep legacy version compliance checks in place for non-SemVer2 versions (#3761)

* Keep legacy version compatibility checks in place for non-semver2 versions

* Added comment to clarify the reasons behind the legacy version check.

* Fix typo

* Rename test for clarity

* code review feedback

* Set SemVerLevelKey after setting Dependencies

* #3861 V2 NuGetEntityTypeSerializer Id link patcher must retain curated feed name (#3864)

* Support Is(Absolute)Latest for SemVer2 + semVerLevel for SearchService (#3842)

* LuceneIndexingService in Gallery should take into account IsLatest(Stable)SemVer2 (#3863)

* POST VerifyPackage version validation should use ToFullString comparison

* ODataV2CuratedFeedController should support semver2 by default when requesting a specific version, and compare on NormalizedVersion

* Refactor NuGetEntityTypeSerializer + unit test coverage (#3879)

* Use NormalizedVersion in URLs contained in PackageAddedNotice (#3886)

* Add nullcheck and use TryParse. (#3890)

* Show full version on package details page (#3887)

* Highlight semver2 packages on package details view (#3893)

* Fix bug in IsLatest(Stable)SemVer2 (#3895)

* VerifyPackage on ApiController should treat version as optional parameter (#3903)

* UpdateIsLatest not resetting IsLatest(Stable)SemVer2 on previous latest versions (#3909)

* Fix malformed URL in redirect after package upload (#3915)

* Minor fix for search results package URLs (when to use version or not in the URL)

* UrlHelper extension for Package should use NormalizedVersion (#3925)

* Default to latest stable semver2 on package details page (#3930)

* User profile page does not show SemVer 2.0.0 packages #3911 (#3933)

* Fix functional test failure SearchMicrosoftDotNetCuratedFeed #3941 (#3942)

* Fix System.NotSupportedException on User profile page (#3943)

* Fix Functional Test failure for ODataFeeds.V2FeedExtendedTests.FindPackagesByIdTest #3947 (#3948)

* Fix load test failure due to incorrect test setup (#3957)

* Hijack IsLatest(Stable)Version OData filter when semVerLevel=2.0.0 (#3966)

* Detect if package only differ by metadata and show optimal user-facing error message (#3970)

* Update Semver2 package details message with final nuget client version #3897 (#3988)

* On package validation failure an actionable error message should be displayed. #3916 (#4031)

* Make downloads link on home page a proper link (#4052)

* Fix the date format on stats page (#4057)

* Update telemetry processors (#4059)

* Reorder SemVer2Latest migration to match deployment history (#4062)

* Average download shown incorrectly when its 1.x #4039 (#4040)

* Average download shown incorrectly when its 1.x #4039

* Moved logic to viewmodel and added UTs

* SemVer2 - Missing db index on Packages table #498 (#4073)

* SemVer2 - Missing db index on Packages table #498

* SemVer2 - Missing db index on Packages table for partial search #499 (#4074)

* Package-Versions autocomplete endpoint does not properly handle semVerLevel when using the db #4086 (#4087)

* Package-Versions autocomplete endpoint does not properly handle semVerLevel when using the db #4086 (#4087)

* v2 package-versions auto-complete endpoint should exclude deleted versions #4092 (#4093)

* Remove auto-refresh AJAX call for total stats on home page #4090 (#4091)

* v2 package-versions auto-complete endpoint should exclude unlisted versions #4092 (#4099)

* Support for policy propagation (#4061)

- Added inline confirmation when adding new package owner
- Added package URL link to package owner request emails
- Added new notification to co-owners when package owner request is confirmed
- Added secure push policy messaging to communication above (confirmation, request, and notification)
- Added secure push policy messaging to package view for owners and admins
- Fixed bug on security policy admin view where toggle all broken if multiple subscriptions
- Updated security policy admin view to not reload page on update postback

* Preserve original 409 exception in AI logs (#4136)

* System.ArgumentNullException GET packages/DisplayPackage #4204 (#4210)

* Resolve merge conflicts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants