track total content length and files indexed #3318

esmadau · 2024-01-22T17:47:13Z

Description

add index on content length to prep for use with getting totals
added cron function to get total sum of content length and total files indexed

Here's an example of the run logs

[2024-01-26T00:05:00.022Z] Executing 'IndexMetricsCollectionFunction' (Reason='Timer fired at 2024-01-25T16:05:00.0095051-08:00', Id=a307874e-8d4d-4d23-95af-8b8bc2f8a9ef)
[2024-01-26T00:05:00.026Z] Collecting a daily summation starting. At: 01/26/2024 00:05:00 +00:00
[2024-01-26T00:05:00.031Z] Collecting a daily summation time taken: 3 ms with ExternalStoreEnabled: True and DataPartitionsEnabled: False
[2024-01-26T00:05:00.032Z] DICOM telemetry - total files indexed: 0 with ExternalStoreEnabled: True and DataPartitionsEnabled: False
[2024-01-26T00:05:00.033Z] DICOM telemetry - total content length indexed: 0 with ExternalStoreEnabled: True and DataPartitionsEnabled: False
[2024-01-26T00:05:00.034Z] Collecting a daily summation completed. with ExternalStoreEnabled: True and DataPartitionsEnabled: False
[2024-01-26T00:05:00.035Z] Executed 'IndexMetricsCollectionFunction' (Succeeded, Id=a307874e-8d4d-4d23-95af-8b8bc2f8a9ef, Duration=25ms)

Related issues

Addresses [AB#105725].

Testing

tested manually, see design doc for execution plans and runtime

src/Microsoft.Health.Dicom.Functions/IndexMetricsCollection/IndexMetricsCollectionFunction.cs

test/Microsoft.Health.Dicom.Tests.Integration/Persistence/IndexedFilePropertiesTests.cs

...alth.Dicom.Functions.UnitTests/IndexMetricsCollection/IndexMetricsCollectionFunctionTests.cs

src/Microsoft.Health.Dicom.Core/Features/Common/IndexedFileProperties.cs

src/Microsoft.Health.Dicom.Functions/MetricsCollection/IndexMetricsCollectionFunction.cs

src/Microsoft.Health.Dicom.Core/Features/Common/IndexedFileProperties.cs

...th.Dicom.SqlServer/Features/Schema/Sql/Sprocs/GetTotalAndSumContentLengthIndexedAsyncV57.sql

src/Microsoft.Health.Dicom.SqlServer/Features/Store/SqlIndexDataStoreV57.cs

src/Microsoft.Health.Dicom.SqlServer/Features/Schema/SchemaVersionConstants.cs

src/Microsoft.Health.Dicom.Functions/MetricsCollection/Telemetry/IndexMetricsCollectionMeter.cs

src/Microsoft.Health.Dicom.SqlServer/Features/Schema/Sql/Tables/FileProperty.sql

...th.Dicom.SqlServer/Features/Schema/Sql/Sprocs/GetTotalAndSumContentLengthIndexedAsyncV57.sql

src/Microsoft.Health.Dicom.Functions/MetricsCollection/IndexMetricsCollectionFunction.cs

src/Microsoft.Health.Dicom.Core/Features/Common/IndexedFileProperties.cs

src/Microsoft.Health.Dicom.Tests.Common/Telemetry/MeterValidationHelper.cs

...ealth.Dicom.SqlServer/Features/Schema/Sql/Sprocs/GetTotalAndSumContentLengthIndexedAsync.sql

src/Microsoft.Health.Dicom.SqlServer/Features/Store/SqlIndexDataStoreV57.cs

…tensions instead

src/Microsoft.Health.Dicom.Core/Features/Common/IndexedFileProperties.cs

src/Microsoft.Health.Dicom.Core/Features/Telemetry/MetricPointExtensions.cs

src/Microsoft.Health.Dicom.Tests.Common/Microsoft.Health.Dicom.Tests.Common.csproj

src/Microsoft.Health.Dicom.SqlServer/Features/Store/SqlIndexDataStoreV57.cs

This reverts commit 1310460.

This reverts commit 8a0247d.

#3349) * Reapply "track total content length and files indexed (#3318)" (#3346) This reverts commit 8a0247d. * add FunctionsPreservedDependencies so openTelemetry can be used / DiagnosticSource loaded as needed by functions host

add index on content length to prep for use with getting totals

0e3140c

esmadau marked this pull request as ready for review January 22, 2024 17:47

esmadau requested a review from a team as a code owner January 22, 2024 17:47

fix name

bbb4c10

esmadau changed the title ~~add index on content length to prep for use with getting totals~~ track total content length and files indexed Jan 25, 2024

esmadau force-pushed the users/esmadau/clindex branch from 518f9f2 to e7b5a49 Compare January 26, 2024 00:02

esmadau commented Jan 26, 2024

View reviewed changes

src/Microsoft.Health.Dicom.Functions/IndexMetricsCollection/IndexMetricsCollectionFunction.cs Outdated Show resolved Hide resolved

add function to get sum and total indexed

fe0431c

esmadau force-pushed the users/esmadau/clindex branch from d2ad972 to fe0431c Compare January 26, 2024 00:11

wsugarman requested changes Jan 26, 2024

View reviewed changes

esmadau added 11 commits January 25, 2024 16:58

move frequency to options

c887257

only run if idp

831bf7b

make readonly struct

9e1d480

move to ctor

1c98fd0

add unique name to meter to avoid collisions

05dceec

fix nits

d0c90d5

rename method

0ddb0a5

other fixes

902def8

rename namespace

bb85d7d

rename namespace

cfbf899

move frequency to every midnight

42d7f6f

jnlycklama reviewed Jan 27, 2024

View reviewed changes

src/Microsoft.Health.Dicom.Functions/MetricsCollection/IndexMetricsCollectionFunction.cs Show resolved Hide resolved

wsugarman reviewed Jan 29, 2024

View reviewed changes

src/Microsoft.Health.Dicom.Core/Features/Common/IndexedFileProperties.cs Outdated Show resolved Hide resolved

...th.Dicom.SqlServer/Features/Schema/Sql/Sprocs/GetTotalAndSumContentLengthIndexedAsyncV57.sql Outdated Show resolved Hide resolved