Reduce snapshot file path length #1696

andrewlock · 2021-08-23T10:30:04Z

We're running into maxpath issues cloning the repo on Windows. This PR reduces the length of the snapshot file names to try and avoid that somewhat. The contents of the snapshot files are unchanged

numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames

macrogreg · 2021-08-23T15:45:28Z

Thanks for doing this, @andrewlock !

Fixes crank pipeline on PR merge commits. (DataDog/dd-trace-dotnet#1669) * show folder content * changes. * Copies the managed profiler to the crank runners. Removes Datadog.Trace.ClrProfiler.Managed from the throughput project. * fix pipeline. * Fix app. * list loaded assembly. * Enable debug mode. * Force loader injection in calltarget scenarios. * Fixes assembly name. Disable CallSite scenario from Throughput tests. (DataDog/dd-trace-dotnet#1674) Add instrumentation for Begin/EndGetResponse to WebRequest (DataDog/dd-trace-dotnet#1658) * Add overload to scope factory that allows providing a start time * Add HttpWebRequest integration from BeginGetRequestStream, BeginGetResponse, and EndGetResponse BeginGetRequestStream can be used in both .NET Core and .NET Framework, but BeginGetResponse and EndGetResponse are .NET Framework only, because the .NET Core implementation uses HttpClient under the covers (which we already instrument) * Update WebRequestTests to handle additional scenarios * Fix rename service tests * Fix WebRequest20Tests to reflect additional instrumentation * Add a test for BeginGetResponse when Task.Factory.FromAsync is used * Simplify Task.Factory.FromAsync implementation * Update integrations json after rebase The v0 version of App Sec (DataDog/dd-trace-dotnet#1647) This PR implements a first version of App Sec. Please forgive the message history, it will be squashed away :) The App Sec module introduces two new environment variables to activate App Sec features: `DD_APPSEC_ENABLED` – enables the App Sec feature, any HTTP requests will be examined by the WAF (Web Application Firewall). By default, this feature is disabled. `DD_APPSEC_BLOCKING_ENABLED` – this controls whether the App Sec feature will block suspicious requests. By default, this feature is disabled. The app sec uses the ‘call target’ instrumentation to gather the data about the HTTP request that will be passed to the WAF. It is has not been implemented for ‘call site’ instrumentation and currently there is not plan to support this style of instrumentation. Data is gathered in the instrumentations for ASP.NET and ASP.NET Core and then passed to the ‘instrumentation gateway’ which starts the security flow. Currently the instrumentation gateway simply creates and event with the incoming data. In future version this component will be used to route the data to different security components based on the addresses associated with the data, but this functionality is not necessary for the first version. After the instrumentation gateway the data flow is as follows: ``` InstrumentationGateway -> Security -> PowerWaf -> Native -> Sqreen.dll ``` `Security` – this class coordinates the reaction to security events `PowerWaf` – this class provides .NET idiomatic wrapper to the WAF, as the WAF is implemented in C++ with C style interface `Native` – this class is responsible for finding, loading and providing low level bindings with the WAF `[lib]Sqreen.[dll|so|dylib]` – this are the native binaries containing the WAF implementation. Security events are reported to the agent and backend by `AgentWriter` class, which implements a basic queuing system. Two new samples are provided, that form the basis for the security integration tests. These are `Samples.AspNetMvc5` and `Samples.AspNetCore5` which are used to test ASP.NET and ASP.NET Core respectively. Integration tests based on similar principals to `Datadog.Trace.ClrProfiler.IntegrationTests` have been provided in `Datadog.Trace.Security.IntegrationTests` and are run as part of the integration tests in the `consolidated-pipeline` CI. ARM based Linux systems are not supported yet, but all other environment supported by the .NET libraries are. ASP.NET Core running on .NET Framework is not supported. The instrumentation is a no-op in these scenarios. This is because it though this unlikely to work, since we rely on referencing several `Microsoft.AspNetCore.*` dlls to be able instrument the ASP.NET Core pipeline. Co-authored-by: Anna Yafi <anna.yafi@datadoghq.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Native logger error fix (DataDog/dd-trace-dotnet#1677) Adds NGEN scenario on Throughput test. (DataDog/dd-trace-dotnet#1678) Removes code for Callsite scenario from the throughput tests. (DataDog/dd-trace-dotnet#1679) Small build improvements (DataDog/dd-trace-dotnet#1646) * Allow running an arbitrary dotnet sample app from Nuke i.e. it doesn't have to be part of the solution * Automatically build the "multiPackageProject" list from PackageVersionsGeneratorDefinitions.json There's no point building these twice, so previously we had a list of sample projects to not build. This simplifies things somewhat by loading the list dynamically * Try using a unique docker network per build to avoid issues with concurrent builds on arm64 * Upload artifacts to azure (for use in AAS extension automation) The latest file list can be downloaded from: https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/index.txt and the latest commit sha of the uploads can be downloaded from https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/sha.txt The blobs are stored in a subdirectory * Only push assets to Azure from master * Pull Azure storage location from variable Update CI Visibility specification (DataDog/dd-trace-dotnet#1682) * Update CI Visibility specification * refactor and fixes. replace "minimal" solution with a solution filter (DataDog/dd-trace-dotnet#1631) Add a custom test framework to monitor execution of unit tests in ducktyping library (DataDog/dd-trace-dotnet#1680) Add tests for changes to Datadog.Trace's Public API (DataDog/dd-trace-dotnet#1681) * Add tests for public API and public browsable (intellisense) API Note that we can't use Verify in Datadog.Trace.Tests because it uses v5.0.0 of the Logging abstractions library, which is unfortunately incompatible with v2.x of the aspnetcore libraries. Resorted to a poor man's version instead. * Add public API snapshots based on v1.28.2 (pre assembly merge) * Update snapshots _after_ assembly merge. This adds a _lot_ of surface area * Update snapshots for master * Make Security class internal * Add [Browsable(false)] and [EditorBrowsable(EditorBrowsableState.Never)] to all public APIs in the ClrProfiler library * Exclude non-browsable APIs from "public API" * Try making some things internal Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * More updates * Fix test visibility Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> [Version Bump] 1.28.3-prerelease (DataDog/dd-trace-dotnet#1688) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> Adds Execution time benchmarks (DataDog/dd-trace-dotnet#1687) * Adds initial execution benchmarks * fixes * Adds sleep to the end of the pipeline. Produce NuGet package to deploy automatic instrumentation (DataDog/dd-trace-dotnet#1661) Produces a NuGet package that contains all of the automatic instrumentation assets that will be deployed to the application output directory. The current name is "Datadog.Instrumentation" Details: - TargetFrameworks: net45, net461, netstandard2.0, netcoreapp3.1 (same as Datadog.Trace) - Dependencies: Datadog.Trace - This means customers who try to restore an earlier version of `Datadog.Trace` will face a build-time NU1605 error (https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu1605) - Contents: Publishes the following to application output |-- datadog | |-- linux-arm64 | | |-- Datadog.Trace.ClrProfiler.Native.so | |-- linux-musl-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- linux-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- net45 | | |-- Managed assemblies | |-- net461 | | |-- Managed assemblies | |-- netcoreapp3.1 | | |-- Managed assemblies | |-- netstandard2.0 | | |-- Managed assemblies | |-- osx-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dylib | | |-- libSqreen.dylib | |-- win-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- win-x86 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- createLogPath.sh | |-- integrations.json Additional changes: - Nuke changes: build BuildTracerHome [CreateInstrumentationHome] BuildInstrumentationNuget - CreateInstrumentationHomebuild target: Copies automatic instrumentation binaries from tracer home to the src/Datadog.Instrumentation/home directory, to be used by the Instrumentation, RunnerTool, and StandaloneTool projects - BuildInstrumentationNuget build target: Builds the Datadog.Instrumentation NuGet package with the binaries in the src/Datadog.Instrumentation/home directory - Removes src/Datadog.Trace.Tools.Runner/Datadog.Trace.Tools.Runner.proj - Add a sample application to samples/NugetDeployment directory with a docker-compose file to run the application on each platform supported by automatic instrumentation Appsec/anna/v0 integration/crankscenarios (DataDog/dd-trace-dotnet#1684) * crank scenarios for appsec * Reset sln configs * Include new yaml in sln fix json filename fix name of file for linux profile Scenarios according to acceptance criterias increase timeout limit logging rate restore ultimate pipeline increase timeout change order of tests to check increase timeout for linux Sharing profiles in another file to import Proxy calls to dl (DataDog/dd-trace-dotnet#1686) * Proxy calls to dl Some Linux distros don't have a version of libdl.so without a version prefix. Call from manage code will fail to find the library on these distros. Proxying via a C++ library should cure this, because the C libraries will handle loading in this case. * Add linker command for libdil * Rename exported fuctions to avoid strange issue * Rewrite PInvoke tables for the native methods used by App Sec too * Correct build error on windows Co-authored-by: Robert Pickering <robertfpickering@fastmail.com> Add end-to-end integration tests for logs injection (DataDog/dd-trace-dotnet#1637) New Tests - `LogsInjection.Log4Net`, `LogsInjection.NLog`, `LogsInjection.Serilog` - Behaviors - Logs injection occurs with raw logs and JSON logs - After logs injection setup, a method is invoked another AppDomain to test LogicalCallContext storage - Libraries tested - log4net - .NET Framework: 1.2 - 2.x - .NET Core: 2.0.6+ - NLog - .NET Framework: 1.x - 4.x - .NET Core: 4.5+ - Serilog - .NET Framework: 1.x - 2.x - .NET Core: 2.x - Caveats - **Note**: Windows testing currently does not test multiple package versions. That will need to be enabled at a later point in time - All log4net versions are still vulnerable to serialization issues when crossing AppDomains, so the cross-AppDomain scenario is skipped in the log4net test application Deleted Tests - `Log4Net.SerializationException` regression test (and helper library ApplicationWithLog4Net) - The regression test was added in DataDog/dd-trace-dotnet#551 but is now solved by adding new sample applications that directly cause calls to cross AppDomain boundaries - `NLog10LogsInjection.NullReferenceException` regression test - The regression test was added in DataDog/dd-trace-dotnet#614 but is now solved by the new type FallbackNLogLogProvider Fixes - Disable Serilog logs injection support when the detected Serilog version is earlier than 2.0. This is enforced by adding a new `NoOpSerilogLogProvider` log provider to perform no-ops in this scenario - Add better exception handling in `CustomLog4NetLogProvider.ExtJsonAssemblySupported()` Add Microsoft.Extensions.Logging.ILogger instrumentation (DataDog/dd-trace-dotnet#1663) * Add LogsInjectionTestBase from open-telemetry#1637 * Add ILogger integration When a logger iterates the scopes for a log message we insert ourselves as the first entry. Note that we don't _really_ add a scope (by instrumenting `BeginScope()` or similar) as that would insert our scope in the "scope stack", and make us responsible for removing it at the appropriate time. The approach here is much simpler, and works for all types of instrumentation that are using the standard ILogger infrastructure This essentially works the same way as the "activity" ILogger integration does in the runtime (https://github.com/dotnet/runtime/blob/v5.0.0/src/libraries/Microsoft.Extensions.Logging/src/LoggerFactoryScopeProvider.cs#L36-L52) * Update profiler skipped assembly prefixes We need to instrument Microsoft.Extensions.Logging.Abstractions now, so can't skip the Microsoft.Extensions prefix This adds most of them to the skip list, but there may be a better approach, e.g. an additional "don't skip" list? * Add LogsInjection aspnetcore sample * Add tests for ILogger logs injection * Update ILogger integration to inject dd_service etc even if we don't have a trace_id * Inject dd_span when a span is available Update tests to verify span_id is injected correctly * Update integrations.json after rebase * Convert DatadogLoggingScope indexer to switch expression * Add required [Browsable(false)] and [EditorBrowsable(Never)] attributes to ilogger integration Reduce snapshot file path length (DataDog/dd-trace-dotnet#1696) * Use numeric status code in snapshots + a shorter name numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames * Remove method name from aspnetcore snapshots to reduce file length * Rename snapshots Merging repos: 'dd-shared-components-dotnet' into 'dd-trace-dotnet'. (DataDog/dd-trace-dotnet#1694) This is the first of several self-contained PRs targeted at moving the Shared Code (as in _shared between the Tracer and the Profiler_) from [dd-shared-components-dotnet](https://github.com/DataDog/dd-shared-components-dotnet) into [dd-trace-dotnet](https://github.com/DataDog/dd-trace-dotnet). For this PR, I created a branch `permanent/MoveFrom_dd-shared-components-dotnet` and moved all files from the `dd-shared-components-dotnet` into there, including their complete change history. The branch will remain permanently as a history record, and we will squash-merge this PR into the `master` branch. The files are all inside the `/shared` folder, so that they can be naturally merged into the planned joint directory structure. In addition, this PR also makes the following small changes: * Make whatever tweaks necessary so that _all_ the Shared Code projects build inside `dd-trace-dotnet`. * Set up placeholder folders for the `/tracer` and the `/profiler` products to facilitate the directory refactoring in the subsequent PRs. * Update ReadMe files to describe the changes. Note that this PR keeps the directory structure within the `/shared` folder exactly as it was within the original repo. Subsequent changed will adjust this structure to joint conventions as required. This will require coordination with the profiler build configuration which makes some assumptions about the layout of the relevant folders. This change is self-contained. It will not affect any existing Tracer components, and it will allow the overall repo merge project to proceed in several steps that can be validated independently. This PR will be followed up with additional PR to address the following: * Move Tracer-specific files into the `/tracer` folder. * Ensure that changes contained to shared files that are not (yet) used by the Tracer do not require the Tracer CI validation to pass (this will be updated in the future if and when the Tracer uses more shared components) * Joint build of components used by several products. * Joint release package. * Joint conventions for folder structure. * Etc... Sending more relevant data to backend (DataDog/dd-trace-dotnet#1695) * Adding more data and filtering cookie from the no cookie header waf address Remove usage of non-builtin types in Attribute named arguments (DataDog/dd-trace-dotnet#1601) To workaround an issue discovered in Azure Functions, remove all usages of named arguments in Attributes that are not built-in types. This means removing arguments that use our own custom enums, but keeping arguments that use `string`, etc. While rare, the failing scenario occurs when the corelib attempts to create the `Attribute` object from the metadata but the assembly that gets resolved from the default load context (or one of its handlers) does not resolve to the same assembly as the one we expected, causing type equality to fail. Note: This workaround is only required for attributes that will be instantiated from metadata, like `DuckAttribute`, but it is easier to maintain a broader rule that we can always revisit later. Specific changes include: - Replacing all usages of `[Duck(Kind = DuckKind.Field)]` with a new attribute `[DuckField]` - Replacing the one usage of `[InterceptMethod(MethodReplacementAction = MethodReplacementActionType.InsertFirst)]` with `[InsertFirstInterceptMethod]` - Adding a unit test to continuously maintain this coding guideline Don't cache the process instance for runtime metrics (DataDog/dd-trace-dotnet#1698) Add tracer metrics to startup logs (DataDog/dd-trace-dotnet#1689) Use AppDomain.CurrentDomain.BaseDirectory to get current directory for configuration (DataDog/dd-trace-dotnet#1700) Environment.CurrentDirectory does not necessarily point to the directory containing the application. The most obvious example is Windows Services, where Environment.CurrentDirectory points to C:\Windows\System32 Using AppDomain.CurrentDomain.BaseDirectory seems to be a better (and safer) alternative. Tested the following scenarios, and they behave as expected * asp.net sites, (iis express + iis) * asp.net in a virtual directory ((iis express + iis) * Owin self hosted * Owin hosted in a windows service * ASP.NET Core app hosted in iis * `dotnet run` on an asp.net core app (returns the bin directory, whereas Environment.CurrentDirectory returns the app directory/content root) * `dotnet dll`/running .exe directly. Returns app directory instead of the console's current directory (e.g. if entering full path to exe) Prepare the `/shared` folder for consumption by the Profiler. (DataDog/dd-trace-dotnet#1697) This change refactors the file structure within the top-level `/shared` folder into its long-term state. Specifically: * Modify the directory names inside of the top-level shared directories according to team design discussions (lower case). * Simplify references to project-external (shared) `.cs` files. * Clean up relative directory references. * Update T4 auto-generated files. * Add ReadMe information. * Ensure that all shared projects build cleanly. This is the [key ReadMe file describing the shared sources](https://github.com/DataDog/dd-trace-dotnet/tree/macrogreg/master/shared/src#readme). It explains things in detail. It is located under `/shared/src/README.md`. The original PR has a screen shot the folder, drilled in up to the individual libraries: DataDog/dd-trace-dotnet#1697 New version of the WAF (DataDog/dd-trace-dotnet#1692) New version of the WAF The latest version of the C++ binary that analyses the incoming web requests. The configs have moved to a yaml, I have included a test rule, but there will be another later PR that will add yaml parsing and a real default rule set. Add fix for Samples.MultiDomainHost.App.NuGetHttpWithRedirects to send test HTTP traffic to a local HTTP listener instead of contoso.com (DataDog/dd-trace-dotnet#1691) Add PreserveContext attribute for async integrations (DataDog/dd-trace-dotnet#1702) Don't trigger Tracer-specific CI for changes to shared assets not used by the Tracer (DataDog/dd-trace-dotnet#1701) [Version Bump] 1.28.5-prerelease (DataDog/dd-trace-dotnet#1709) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> In the integration_tests_arm64 stage, replace COMPOSE_PROJECT_NAME with the projectName input on the DockerCompose Azure DevOps task (DataDog/dd-trace-dotnet#1715) This should correctly set the project name because the task's projectName input takes precedence over the environment variable with the -p option, and the task will always set a projectName input even if not specified. Don't fail the build if uploading core dumps fails (DataDog/dd-trace-dotnet#1722) By adding `continueOnError: true` to the log upload stage Also removed some "duplicate" uploads for consistency Create new docker-compose services only for arm64 that exposes the ports only via linked services (DataDog/dd-trace-dotnet#1719) This allows multiple stacks to be hosted on one machine, as is the case in the arm64 integration test runs Fix flakiness in Datadog.Trace.ClrProfiler.IntegrationTests.WebRequestTests.SubmitsTraces (DataDog/dd-trace-dotnet#1718) As part of the test, the trace ID and span ID of `spans.First()` is compared against the first distributed tracing headers seen by the in-memory HttpListener. In the most recent test failure, the trace ID comparison failed, and I suspect it's because the in-memory list of spans was not ordered by start time. Ordering the spans should solve this. Enable code coverage for security tests (DataDog/dd-trace-dotnet#1699) * Enable code coverage for security tests * Enable aspnetcore5 tests * Try fixing RabbitMQ coverage Add support for Aerospike (DataDog/dd-trace-dotnet#1717) Update name of NuGet package carrying automatic instrumentation assets (DataDog/dd-trace-dotnet#1720) * Rename Datadog.Instrumentation to Datadog.Monitoring.Distribution, and rename all "instrumentation" references to "distribution" * Update NuGet Deployment test to not expose the Datadog agent to the host, only to the running containers Add FmtLib debug files that were previously forgotten during import of shared files. (DataDog/dd-trace-dotnet#1725) The recent merge of the `dd-shared-components-dotnet` repo included the native [fmt library](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native). During the merge, we included the release version of that library, but we forgot the [debug files](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native/fmt_x64-windows-static/debug/lib). Now, we need to have the Profiler use the shared files from their new location in `dd-trace-dotnet`. However, that is currently not building correctly, because the fmt debug files are missing. This change adds the missing files. There is no other functional change, beyond unblocking the Profiler build. Define 'SharedSrcBaseDir' consistently across projects. (DataDog/dd-trace-dotnet#1724) Several projects in `/shared` and in `dd-continuous-profiler-dotnet` define the `SharedSrcBaseDir` MSBuild property to refer to the folder that has recently been moved to `dd-trace-dotnet\shared\src\managed-src\`. Previously, projects were using different ways to refer to that same folder, which was fragile. This change switches to a consistent reference based on the existing `EnlistmentRoot` MSBuild variable. Add a beta suffix to the Datadog.Monitoring.Distribution NuGet package (DataDog/dd-trace-dotnet#1728) Add a beta01 suffix to the Datadog.Monitoring.Distribution while we ensure it's production-ready Implement shutdown to help fix app sec test flakiness (DataDog/dd-trace-dotnet#1713) Package windows native symbols in a separate archive (DataDog/dd-trace-dotnet#1727) * Package windows native symbols in a separate archive * Only publish symbols in gitlabs Native memory improvements (DataDog/dd-trace-dotnet#1723) * changes * Skip Dynamic and Resource modules from CallTarget. Other improvements. * Fix linux build * Revert RequestRejitForNGenInliners call move. * Move string allocations to contants * Changes based in the review. * Simplify the ModuleLoadFinished callback on CallTarget * use constant * fix test project. * Changes based in the review. Add a previously forgotten comment to a shared editorconfig file. (DataDog/dd-trace-dotnet#1735) Downgrade severity of a shared editoconfig rule to match ongoing Profiler work. (DataDog/dd-trace-dotnet#1736) (The modified `.editorconfig` currently only applies to files used by the Profiler.) Disable AppSec crank till a new runner machine can be created (DataDog/dd-trace-dotnet#1739) Implement 1.0.7 of libddwaf (DataDog/dd-trace-dotnet#1732) * Implement 1.0.7 of libddwaf Our C++ WAF implementation, libddwaf - previously PowerWaf / libSqreen, has been though a big clean up of their APIs and the format of configuration file. This commit synchronizes the API signatures and naming of methods / objects. The details of the new API can be found here: https://github.com/DataDog/libddwaf/blob/master/include/ddwaf.h Disable ILogger integration by default (DataDog/dd-trace-dotnet#1738) * Disable ILogger integration by default In order to better verify the performance impact of the ILogger integration, disable the integration by default, by adding an additional feature flag. * Fix configuration tests broken by disabling ilogger by default Correct memory management for interactions with libddwaf (DataDog/dd-trace-dotnet#1742) I had assumed that libddwaf took ownership of objects assoicated with it and therefore free them when context objects where freed. However this isn't the case, all objects allocated by the C# code for the ddlibwaf need to be explicitly freed. This PR does that in a simple way. Revert "Disable ILogger integration by default (open-telemetry#1738)" (DataDog/dd-trace-dotnet#1744) This reverts commit fdbab1aef49b79cc8da9ae3a89d041f2c3ab51ca. Header keys should be lower case (DataDog/dd-trace-dotnet#1743) [Version Bump] 1.28.6 (DataDog/dd-trace-dotnet#1745) * [Version Bump] 1.28.6 * Update Changelog Co-authored-by: andrewlock <andrewlock@users.noreply.github.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Minor tweaks to shared files (DataDog/dd-trace-dotnet#1729) As we are cleaning up the Profiler build to reference `dd-trace-dotnet` instead of `dd-shared-components-dotnet`, we discover the need to minor tweaks. This PR is the collection of such tweaks which I collected into a single PR to minimize churn. The Profiler work is now done (https://github.com/DataDog/dd-continuous-profiler-dotnet/pull/135). Note that while this PR is being reviewed and merged, the Profiler builds cleanly while referencing `dd-trace-dotnet / macrogreg/SharedItemsCatchup`. Once this PR is merged, we must point the profiler to `dd-trace-dotnet / master`. Changes here: * Make packages path the same for all shared items. * Props files must use relative paths to construct shared file references. * Fix upward walking chains in `Directory.Build.props` files. Add minimal test applications that use service bus libraries (DataDog/dd-trace-dotnet#1690) Adds a test application and smoke test for the following service bus libraries. This tests that automatic instrumentation does not encounter issues with the core components of the service bus libraries. - MassTransit - NServiceBus - Rebus [DuckType] - Improve generic methods support. (DataDog/dd-trace-dotnet#1733) * Adds support for Types with generics types inside a generic parameters method. * Changes in the tests. * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Remove unused ISpan/IScope (DataDog/dd-trace-dotnet#1746) Fix Benchmarks build break from removing Datadog.Trace.Abstractions namespace (DataDog/dd-trace-dotnet#1749) Fix compilation directive for NET5.0 (DataDog/dd-trace-dotnet#1731) Exclude liblog from code coverage by filepath (DataDog/dd-trace-dotnet#1753) Adding the "[ExcludeFromCodeCoverage]" attribute apparently causes all sorts of issues Refactor ILogger integration (DataDog/dd-trace-dotnet#1740) * Move ILogger instrumentation into Logging sub-folder We are going to likely have shared components related to ILogger so makes sense to move it now * Call ToString() on SpanId and TraceId Ensures that they are formatted as strings when using custom formatters * Add benchmark for logs injection * Add a sample app demonstrating how to use the ILogger integration The ILogger integration requires auto instrumentation, so uses the Datadog.Monitoring.Distribution to set that up in an easy way * Output a better format string for ILogger Logging Scope Some loggers don't call `GetEnumerator()` on scopes, and instead just call ToString(). Previously we were outputting empty span_id and trace_id values, which looked strange, and might break log parsers etc In addition, according to [this comment](https://github.com/DataDog/dd-trace-dotnet/blob/master/samples/AutomaticTraceIdInjection/SerilogExample/Program.cs#L26), log parsing requires the properties to be a JSON map, so to improve the chance of correct usage, this changes the ToString() value to be a json map. * Update DatadogLoggingScope.ToString() to not be a JSON object Refactor folder locations [latest] (DataDog/dd-trace-dotnet#1748) This PR moves almost all of the code related to development/test/build of the .NET Tracer into a `/tracer` directory to allow for better code partitioning when we add more products into the repo. File changes include: - Directories moved from `/` to `/tracer/` - build - samples - src - test - Individual files moved from `/` to `/tracer/` - Datadog.Trace.proj - Directory.Build.props - GlobalSuppressions.cs - build.cmd - build.ps1 - build.sh - build_in_docker.sh - integrations.json - stylecop.json - Intermediate artifact folders moved from `/src/bin` to `/tracer/src/bin` - artifacts - managed-publish - windows-tracer-home Fix incomplete update to Datadog.Trace.sln for latest sample projects during tracer folder move (DataDog/dd-trace-dotnet#1759) Remove CallTarget Integrations from json file. (DataDog/dd-trace-dotnet#1693) * Initial commit * Fixes x86 PInvoke calls. * fix after merge master * Integrations definition generator. * Changes based on review. * Changes. * Adds Dispose to the NativeCallTargetDefinition instance. * Change WStr("") to EmptyWStr constant * Rollback unnecessary changes. * Rollback unnecessary changes. Fix broken paths after repository move (DataDog/dd-trace-dotnet#1762) * Fix paths in GitHub workflows They were broken during the repository re-arrange * Fix paths in honeypot yml * Remove unused (incorrect) paths from third-party-test-suites They were wrong, but don't appear to have been used, so removed them * Fix path for GitLab build Move tracer snapshots to /tracer/test/snapshots directory (DataDog/dd-trace-dotnet#1766) This should help avoid long path issues with the snapshot files that have extremely descriptive names. Synchronously wait for tasks in HttpClient sample (DataDog/dd-trace-dotnet#1703)

* Catchup to upstream Remove a few benchmarks to make the pipeline faster (https://github.com/DataDog/dd-trace-dotnet/pull/1497) Increase the crank scenario duration to 4 minutes. (https://github.com/DataDog/dd-trace-dotnet/pull/1505) Cleanup project files (redux) (https://github.com/DataDog/dd-trace-dotnet/pull/1499) * delete redundant props file * remove files from solution file * don't build test apps that won't be used * remove conditions for net5.0 * remove checks for empty values * remove <RuntimeIdentifier> and <RuntimeIdentifiers> left over from the 2.x era * move import to the top so we can override default values * delete redundant import * use the recommended import declaration AgentWriter FlushTracesAsync changes (https://github.com/DataDog/dd-trace-dotnet/pull/1501) * Change behavior of the FlushTracesAsync method to enqueue all flush continuations per buffer * AgentWriterEnqueueFlushTasks unit test. CIApp: Update CircleCI spec (https://github.com/DataDog/dd-trace-dotnet/pull/1503) * Update CircleCI spec * fix typo Add Nuke build project and update consolidated pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1476) * Stop git reporting errors in MakeFile when there aren't any There's still one annoying one I can't get rid of: ar: creating archive obj/libre2.a * Add Samples.AspNet472.LoaderOptimizationRegKey to sln so that it's restored with everything else Use "root" packages folder in root instead of own packages subfolder in LoaderOptimization project * Add additional target framework to RazorPages project Causes some issues when skipping dependencies later unfortunately, this was the easiest fix for now * Add Nuke build project Includes build.cmd etc for easily running Includes docker.sh for running inside docker using the same images as we do in CI I haven't added the build project to the solution yet, mostly to avoid rebasing woes, but will add it during cleanup To avoid breaking other pipelines, I added new targets to the proj file. These can be trimmed in the future. * Update the ultimate-pipeline to use Nuke Splits things into extra stages to increase throughput when we're waiting on a particular machine type linux/windows/mac Many more optimizations possible, but may not be worth the effort yet Added new docker-compose targets for now. Can clean these up later There are some steps which haven't been fully converted yet, but this is a good start IMO * git update-index --chmod=+x docker.sh build.sh * Update version number in Nuke script * Skip building Kafka multiple times * Try fix CI test apps - msbuild faffing * Add wait-for-it for dependencies in docker-compose Should add back some resiliency (kafka can sometimes be a bit flakey to start up) * Apply suggestions from code review Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * Ensure we clean the "deps" directory too * Fix macos publish directory * Rename docker.sh -> build_in_docker.sh * Port Arm64 throughput tests * Rename IntegrationTests2 -> NukeIntegrationTests * Restore ReferenceAssemblies reference for dependency libs The .NET Framework 2.0 apps need them, as the SDK doesn't seem to include them by default * Tweak IIS sample runner utility * Skip building Oracle projects in CompileSamples for now (no drive space) * Print drive info to try and see where it's all going * Don't copy CI data around so much Saves 0.5GB of drive space * Add option to print drive space for every target * Revert "Don't copy CI data around so much" This reverts commit fd10d3a528f44234fccaebe0aeeecc86a7083831. * Allow overriding the test filter in integration tests * Revert "Revert "Don't copy CI data around so much"" This reverts commit 82a8afed1ced4ea65d2be5f296679733b5273ef7. * Fix publish logs when multiple job attempts Publish logs for other test steps * Update Version Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> Include Nuke build project in the sln (https://github.com/DataDog/dd-trace-dotnet/pull/1508) * Fix invalid project solution configurations * Add Nuke project to Sln file * Fix typo in SetAllVersions CIApp: Correctness and stability fixes. (https://github.com/DataDog/dd-trace-dotnet/pull/1504) * Fixes an scenario when a NUnit test suite fails on the OneTimeSetUp method gets reported different than the console output. * Fixes NUnit integrations. * Improve render of parameters. * Improvements. * Maps InconclusiveException to a SkipStatus. * Remove unused property. * Increase NUnit test cases with Assert.Pass and Assert.Inconclusive tests. Fix race condition in runner pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1510) Ensure both span buffer are flushed in the FlushTracesAsync() call. (https://github.com/DataDog/dd-trace-dotnet/pull/1514) Avoid the trace copy when flushing spans (https://github.com/DataDog/dd-trace-dotnet/pull/1502) Modify GetReJITParameters callback method to never return E_FAIL HRESULT (https://github.com/DataDog/dd-trace-dotnet/pull/1517) Add net5.0 support to dd-trace (https://github.com/DataDog/dd-trace-dotnet/pull/1516) * Add net5.0 target framework. * Adds a temporal basic nuget description. Crank sample app profiler detector. (https://github.com/DataDog/dd-trace-dotnet/pull/1518) * Adds a profiler detector to the crank sample app. * Exit with an error if the profiler is not correctly attached. Enable calltarget instrumentation by default on all architectures (https://github.com/DataDog/dd-trace-dotnet/pull/1509) Delete project NugetProfilerVersionMismatch (https://github.com/DataDog/dd-trace-dotnet/pull/1520) * remove unused project NugetProfilerVersionMismatch * update nuke pipeline Enable ultimate pipeline for PRs (https://github.com/DataDog/dd-trace-dotnet/pull/1515) * Run ultimate pipeline on all PRs and tags * Schedule ultimate-pipeline to run every night Skip stages that aren't required for schedule * Disable old pipeline triggers * Create a more obvious `benchmarksOnly` variable, and allow "manually" triggering a benchmarking run * Allow skipping benchmarks using skip_benchmarks * Fix environment variable evaluation Convert GitLab package signing build to use Nuke targets (https://github.com/DataDog/dd-trace-dotnet/pull/1512) * Add a package signing target, for use in GitLab * Amend gitlab build script to call Nuke targets * Tweak info task to run first * Remove old gitlab build files * Fix build in Gitlab Use explicit AnyCPU platform, as otherwise dotnet publish looks in the wrong place Gitlab doesn't install MSBuild in a well-known location, so set the path manually * Update GitLab Docker image Incorporates changes from https://github.com/DataDog/datadog-agent-buildimages/pull/135 Adds web build tools and makes a couple of optimisations * Try setting VSToolsPath explicitly * Ensure we sign the dlls that go into the msi and nupkgs * Sign the NuGet bin folder dlls as they get packed Previously we were only signing the "published" dlls, but they aren't used during dotnet pack * Fix default naming of windows-tracer-home.zip Cosmos Db support (https://github.com/DataDog/dd-trace-dotnet/pull/1473) * Cosmos Db support - Call target instrumentation only - Test only run on windows, since the emulator is not supported on other platforms Upgrade spdlog to 1.4 and move to Static lib version (https://github.com/DataDog/dd-trace-dotnet/pull/1507) Fix Nuke pipeline and remove some unused sample projects (https://github.com/DataDog/dd-trace-dotnet/pull/1521) * Fix windows-tracer-home artefact naming * Replace usage of samples-iis in Nuke pipeline Replace with direct publish of aspnet projects instead * Delete unused projects from main sln file * Remove references to non-existant folders * Delete files from unused projects on disk * Split CompileManagedUnitTests in two Should slightly reduce the amount of space required in integration builds, as we only need to build the test helper projects * Switch to creating symlinks between build output Instead of explicit copying to save space Fix CosomsDb tests and disable PR triggers for old pipelines (https://github.com/DataDog/dd-trace-dotnet/pull/1523) * Disable PR triggers for old pipelines Disable scheduled triggers for benchmark/throughput (run by ultimate pipeline) * Cosmosdb buids package versions * Start cosmosdb emulator in test pipeline * Add LinuxUnsupported filter to linux integration tests * Fix Clean target to handle reparse/junction points Add automatic instrumentation for AWS SQS (https://github.com/DataDog/dd-trace-dotnet/pull/1471) - Instrumented Operations - CreateQueue - DeleteQueue - PurgeQueue - ReceiveMessage - SendMessage - SendMessageBatch - Samples.AWS.SQS added to automated integration tests - Verified that the instrumentation adds `aws.region` and `aws.requestId` tags when sending SQS requests to an AWS account Small build improvements (https://github.com/DataDog/dd-trace-dotnet/pull/1524) * xunit/nunit/mstest samples are built using multi api So no point building them standalone * Fix occasional flaky build for xunit sample on arm64 For some (unknown) reason, the arm64 integration test build was occasionally failing with "Type 'AutoGeneratedProgram' already defines a member called 'Main' with the same parameter types" I can reproduce this locally by running ``` ./build_in_docker.sh Clean BuildTracerHome BuildLinuxIntegrationTests ./build_in_docker.sh CompileMultiApiPackageVersionSamples ``` Which makes sense, because the generated .cs file in _other_ obj folders are included when building the samples What doesn't really make sense is why those obj folders sometimes exist when we're doing a clean checkout The solution in this commit resolves the problem anyway * Ignore NuGet package related warnings in CompileMultiApiPackageVersionSamples These are "expected", in that we're using net461 packages on netcoreapp deliberately, and are using "out of bounds" versions of Newtonsoft.Json because we're deliberately using downgrade versions of dependencies. The warnings are all surfaced in the AzDo UI, so this recuces the clutter * Simplify Nuke project yml includes * Set ARFLAGS so that we don't get stderr messages for things that are fine I just repeated the same flags they were trying to use here https://github.com/google/re2/blob/master/Makefile#L23 but aren't using them properly (I think) * Fix AWS SQS tests (not starting SQS container) * Ignore some known warnings when building samples * Don't use obsolete interface * Remove unnecessary global /nowarn:NU1608 * Fix ValueTuple reference * Ignore use of obsolete Mongo API which we have to use * Fix version conflict instead of ignoring it * enable -> restore Fix xUnit 2.2.0 test flakyness (https://github.com/DataDog/dd-trace-dotnet/pull/1526) Custom native IsAlreadyLoaded() implementation for .NET 4.5.x (https://github.com/DataDog/dd-trace-dotnet/pull/1528) * Split IsAlreadyLoaded() implementation depending of the .NET Framework version (<4.6) * change comment. Adds CorLib detection for assembly ref (https://github.com/DataDog/dd-trace-dotnet/pull/1522) * Adds CorLib detection and use that on assembly ref instead of forcing mscorlib reference. * Change AppDomain.Load to Assembly.Load * Changes from review. * fix comment. Downgrade the version of cmake used for MacOs builds (https://github.com/DataDog/dd-trace-dotnet/pull/1529) Version 3.20.x causes the loader dll and pdb to not be embedded correctly. You can confirm this behaviour by running `xcrun size -x -l -m Datadog.Trace.ClrProfiler.Native.dylib` and checking for the `Segment binary` section. In a bad build (with 3.20.x) it will be missing Revert "Enable calltarget instrumentation by default on all architectures (#1509)" (https://github.com/DataDog/dd-trace-dotnet/pull/1532) This reverts commit 1ec618ae4cc21de417c3e7ac7ec59167a22e67b0. Small build improvements (https://github.com/DataDog/dd-trace-dotnet/pull/1531) * Increase dependency warmup timeout to reduce flakiness * Ignore CMake temporary build files (macos) * Don't need the static.a file in the tracer home * Simplify/Unify the build tracer home step for linux + mac Previously we were copying files _into_ the native project's bin folder, and then publishing from there. This change makes things a bit simpler in the pipeline as well as being more consistent wih the output in general. - Only publish the net4 targets in the managed build step on Windows (not required for linux) - Copy the files _to_ the tracer home directly, _from_ the bin output, instead of the other way around * Add build step for generating the tracer home directory in the dd-trace format dd-trace uses a different folder layout for dd-trace. This adds an extra Nuke target for creating a directory with the required structure (required for Linux and Mac) * Add pipeline variable for ddTracerHome * Fix comment * Try fix flakiness in Arm64 Linux Integration builds * Suppress build warnings on macos NUnit integration improvements (https://github.com/DataDog/dd-trace-dotnet/pull/1533) * - Adds support for parameterized fixtures. - Adds support for test declared in a base class. * - Adds support for test.message tag. * fix comment. * Add `test.message` test * - Sets the service name to the repo name in case no value is set. * remove comment. * - Adds the case when the repo doesn't end with .git - Adds the repository name for callsite. * null check. Add an OWIN test application to the integration-tests pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1525) * Treat 501-503 as error codes * Add test application Samples.Owin.WebApi2 Uses same endpoints as SamplesAspNetMvc5 WebApi2 endpoints, for easier testing * Add snapshot tests for Owin app using Verify * Add verify snapshots for owin test * Don't build Samples.Owin.WebApi2 on Linux * Delete duplicate test declaration * Fix link in build project * Switch to using WaitHandle instead of Console.Read, as that was failing in AzDo (as didn't actually wait) Add some extra logging * Remove the invalid HTTP status codes in aspnetcore tests Move the tests cases to TracerSettingsTests instead Fix Enable32Bit flag for IIS container (https://github.com/DataDog/dd-trace-dotnet/pull/1536) For some reason, this _has_ to be uppercase to be picked up. Unlike the other arg. Which must _not_ be uppercase... Fix spans flushing on Testing framework instrumentations (https://github.com/DataDog/dd-trace-dotnet/pull/1535) * Change instrumentation for flushing spans to the Xunit.Sdk.TestAssemblyFinished ctor * Fix MsTest RunCleanup integration. * Fix integration comment Upgrade non windows native builds to C++17 (https://github.com/DataDog/dd-trace-dotnet/pull/1543) * Upgrade macos build to cpp 17 * Fixes linux build Change native format to Microsoft style (https://github.com/DataDog/dd-trace-dotnet/pull/1544) * clang-format with Microsoft style * Improve formatting Add small CI fix and docker optimization (https://github.com/DataDog/dd-trace-dotnet/pull/1551) * Use a build image that is cached on the hosted windows agents, as defined here: https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#cached-docker-images * Fix additional errors with newline character endings Ducktype - Explicit interface method implementation support (https://github.com/DataDog/dd-trace-dotnet/pull/1555) * Adds explicit interface implementation support to DuckTyping * Add support for wildcard. Update ci provider extractor according to specs (https://github.com/DataDog/dd-trace-dotnet/pull/1554) * Update ci provider extractor according to specs * fixes Replace calli call with a normal call if the target method is a generic method. (https://github.com/DataDog/dd-trace-dotnet/pull/1557) Change native format to custom style (https://github.com/DataDog/dd-trace-dotnet/pull/1553) * Change format to custom * changes * changes * Fix for windows Modify CustomSerilogLogProvider to use older LogContext.PushProperties API when LogContext.Push API is not present (https://github.com/DataDog/dd-trace-dotnet/pull/1560) This PR fixes a gap we have today: when the application's Serilog library is old enough that it does not have the LogContext.Push API, we fail to set up our logs injection. The solution involves the following changes: 1. Override the IsLoggerAvailable check to only return true when either the LogContext.Push API is available or the LogContext.PushProperties API is available. Otherwise, this returns false and the built-in LibLog SerilogLogProvider will be used. 2. Update the SerilogEnricher to optionally pass in the enricher object by itself or an array containing the enricher object, based on the Serilog API used for LogContext updates. Enable calltarget by default on .NET 4.6+ runtimes (https://github.com/DataDog/dd-trace-dotnet/pull/1542) * Enable calltarget by default on .NET 4.6+ runtimes Merge hotfix 1.27.1 (https://github.com/DataDog/dd-trace-dotnet/pull/1547) * Bump version to 1.27.1 (https://github.com/DataDog/dd-trace-dotnet/pull/1538) * Custom native IsAlreadyLoaded() implementation for .NET 4.5.x (#1528) (https://github.com/DataDog/dd-trace-dotnet/pull/1539) * Split IsAlreadyLoaded() implementation depending of the .NET Framework version (<4.6) * change comment. (cherry picked from commit 13cdc48e4f70541c07256e574f7d6ae7efeb4ab8) Co-authored-by: Tony Redondo <tony.redondo@datadoghq.com> * copy pipeline triggers from `master` and add `release/*` to ultimate-pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1545) * run pipelines in hotfix branches (https://github.com/DataDog/dd-trace-dotnet/pull/1546) * fix bad merge conflict resolution * update version in build pipeline Co-authored-by: Tony Redondo <tony.redondo@datadoghq.com> Add support for NestedTypes and GenericParameters to EnsureTypeVisibility (https://github.com/DataDog/dd-trace-dotnet/pull/1561) * Add more ducktyping tests for explict interfaces * Add a duck typing test reproducing the specific ILogger issue * Improves EnsuresTypeVisibility with support for NestedTypes and GenericParameters. Co-authored-by: Tony Redondo <tony.redondo@datadoghq.com> Initialize performance counters asynchronously (https://github.com/DataDog/dd-trace-dotnet/pull/1564) Remove callsite benchmarks and set iteration time back to 2 seconds (https://github.com/DataDog/dd-trace-dotnet/pull/1511) Nuke build: overwrite files when copying trace home directory (https://github.com/DataDog/dd-trace-dotnet/pull/1567) Wait 10 more seconds on runtime metrics tests (https://github.com/DataDog/dd-trace-dotnet/pull/1566) hide warnings for EOL .NET Core targets (https://github.com/DataDog/dd-trace-dotnet/pull/1569) Use a UUID for runtime-id instead of container id (https://github.com/DataDog/dd-trace-dotnet/pull/1548) * Use a UUID for runtime-id instead of container id Fix x86 builds in consolidated pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1563) * Rename platform -> buildPlatform Nuke uses the Platform variable to switch between x86/x64 We could also explicitly add `-Platform $(buildPlatform)`; the current approach sets it implicitly * Add missing build configs for WebForms project * Update Nuke integration test runs to use x86 SDK when set to x86 Attempting to load the x86 compiled libs into the x64 bit process causes BadImageFormatExceptions * Add templates for installing the 32 bit .NET SDKs Install the x86 SDKs in Windows integration tests * Add check for using install-dotnet-sdk-32bit wrong Fix race condition in PerformanceCountersListenerTests (https://github.com/DataDog/dd-trace-dotnet/pull/1573) Revert the order in which the log providers are resolved (https://github.com/DataDog/dd-trace-dotnet/pull/1578) Update README (https://github.com/DataDog/dd-trace-dotnet/pull/1576) Reduce dependencies between build tools and helper projects (https://github.com/DataDog/dd-trace-dotnet/pull/1568) * reduce dependencies by using EnvironmentTools.GetSolutionDirectory() instead of EnvironmentHelpers * add TracerVersion to SetAllVersions * remove unused code * remove dependency on Datadog.Core.Tools * add xml-doc comments * make SetAllVersions non-static to remove the static properties and pass path as ctor argument CI tweaks (https://github.com/DataDog/dd-trace-dotnet/pull/1570) * Extract ARM64 jobs from linux The linux hosts have different availability to the linux jobs, so reduces unnecessary blocking if one or the others are not available. As a side effect of splitting these out, it reduces the size of the strategy matrix variables too. * Increase startup timeout on CosmosDB emulator * Don't use Console.Writeline in xUnit tests Adds cruft to otherwise successful test runs * "simplify" the benchmarks/throughput test variables Previously we had two variables to control whether to run the benchmarks, but as there's already a UI for selecting which stages to run, this seems redundant. We are likely going to start running other stages on a scheduled build too, so exposing that directly makes more sense I think * Add a step to the pipeline to upload to the Azure feed Pulls down all our "publish" artefacts and publishes them all as a universal package * Add missing Arm64 dependencies * Fix version extraction * Create a bundled "release" artifact * Fix version number in artifact * Set Platform when publishing This only seems to be necessary locally, but I'm not entirely sure why! * Fix dd-tracer-home Delete old unused pipelines and files (https://github.com/DataDog/dd-trace-dotnet/pull/1513) * Move iis dockerfile to live with others * Remove other pipelines (all handled by ultimate-pipeline) * Remove no-longer-necessary docker-compose targets * Delete unused dockerfiles and scripts * Remove sample targets in docker-compose.yml Somewhat opinionated, but as services are exposed on ports anyway, just as easy to run samples locally instead (Also, there's a nuke target for that) * Start annotating which proj targets we actually use * Remove unused targets from Datadog.Trace.proj * This isn't the YAML you're looking for... * Remove deleted files from solution folder * Remove deleted file from SetAllVersions * Delete unused Samples-iis.sln Fixes writing the environment values in the logs on non windows platforms. (https://github.com/DataDog/dd-trace-dotnet/pull/1581) fix "PrepareRelease msi" command (https://github.com/DataDog/dd-trace-dotnet/pull/1583) Add support for Microsoft.Data.SqlClient 3.* (https://github.com/DataDog/dd-trace-dotnet/pull/1579) * Add support for Microsoft.Data.SqlClient 3.* Fix flaky Kafka test (https://github.com/DataDog/dd-trace-dotnet/pull/1585) In Confluent.Kafka 1.5.3, attempting to consume from a non-existent topic throws an exception. Catching the exception isn't enough, as it causes the consumer to no longer consume, even if the topic does exist later. In the sample app, there was a race condition, whereby if the topic hadn't completely finished being created, the consumer would fail, causing the test to fail. As a workaround, we use a separate consumer for the initial "manual" consumes, to give time for the topic to be ready. To ensure the fix works, we also _explicitly_ consume against a non-existent topic, to ensure we are expecting those errors. > Note, I tried converting this test to snapshot testing, but the actual number of spans is indeterminate, as we can get multiple consume spans for a single message, which makes snapshot testing difficult. Small ASP.NET Snapshot refactoring (https://github.com/DataDog/dd-trace-dotnet/pull/1580) * Update VerifyHelper to reorder tags in snapshots Obviously, being a dictionary, the enumeration order is technically indeterminate, but in practice, this reordering seems to work, and is necessary where the order tags are added to the span at the source may change per invocation * Move snapshots into a Snaphots sub folder Lose the benefits of code folding in VS/Rider, but there's so many, it's too unwieldy within GitHub/filesystem currently * Move ASP.NET Snapshots to sub folder * Update aspnet snapshots to reorder tags and add missing runtime-id * Fix directory char in samples directory Uses correct character regardless of OS This was causing some issues with IisExpress for me locally * Don't pass executable AND applicationPath to StartProcess I personally found this really confusing, especially when it comes to aspnetcore, and has clearly evolved a lot - there's two paths passed to it, sometimes one is executed, sometimes the other etc. Plus double negatives. Instead, I pass the executable, add applicationPath to args if necessary in the callee, and explicitly allow passing the process to profile (optional). I think that makes things a bit clearer, but it could do with more refactoring, and reducing of duplication tbh. * Allow specifying how many web server spans are expected in TestHelper Revert "Delete old unused pipelines and files (#1513)" (https://github.com/DataDog/dd-trace-dotnet/pull/1588) This reverts commit 3d3269ca212ff053607e8b121160c19ba4713b92. Don't create an azure artifact (https://github.com/DataDog/dd-trace-dotnet/pull/1589) It has the potential to cause issues if storage fills up, and was only created to make programmatic consumption easier. We can reevaluate at a later date * Bump version to 1.28.0 Also increase the PrepareRelease timeout Revert "Revert "Delete old unused pipelines and files (#1513)" (#1588)" (https://github.com/DataDog/dd-trace-dotnet/pull/1595) This reverts commit 9bdd9d7f29813f09002bd1126e20259ac130bd28. Fix Linux artifacts (https://github.com/DataDog/dd-trace-dotnet/pull/1587) * Fix Linux artifacts * Fix default paths used in build_in_docker.sh Bitrise spec update. (https://github.com/DataDog/dd-trace-dotnet/pull/1593) Fix native leak in TypeInfo (https://github.com/DataDog/dd-trace-dotnet/pull/1596) * Fix native leak in TypeInfo TypeInfo instance holds 2 pointers but they are never freed Use std::shared_ptr to automate and simplify the memory management of this type. * Use std::make_shared instead of split allocating and object creation Add a custom test framework to monitor execution of unit tests (https://github.com/DataDog/dd-trace-dotnet/pull/1572) * Add a custom test framework to monitor execution of unit tests * Enable xunit diagnostic messages Fix arm64 tar suffix (missing '.') (https://github.com/DataDog/dd-trace-dotnet/pull/1598) Add a public ForceFlushAsync API on the Tracer (https://github.com/DataDog/dd-trace-dotnet/pull/1599) Fixes native memory leaks (https://github.com/DataDog/dd-trace-dotnet/pull/1600) * - Adds a new UniqueBlockingQueue class - Transform the RejitItem struct to use std::unique_ptr<T> - Move all implementations from the headers to the cpp file. * Cleaning and removing more leaks. * Adding more unique_ptr. Bump version to 1.28.1-prerelease (https://github.com/DataDog/dd-trace-dotnet/pull/1603) * Bump version to 1.28.1-prerelease * Update docs/CHANGELOG.md Co-authored-by: Tony Redondo <tony.redondo@datadoghq.com> Co-authored-by: Tony Redondo <tony.redondo@datadoghq.com> Add code coverage to the consolidated pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1527) Update package versions + fix MSTest (https://github.com/DataDog/dd-trace-dotnet/pull/1604) * Update package versions * Fix MSTest for version 2.2.5 Previously, we were receiving only one SimpleSkipParameterizedTest because the skip was executed before iterating the parameters. That changed in 2.2.5, so each parameterised run gets a span Make the benchmark step optional (https://github.com/DataDog/dd-trace-dotnet/pull/1608) Run the npgsql/npgsql tests in our pipelines with the profiler attached (https://github.com/DataDog/dd-trace-dotnet/pull/1431) Adds a new Azure DevOps pipeline to run integration tests from libraries, starting with Npgsql - Runs Npgsql tests at tags v4.1.9 and v5.0.7, on both Ubuntu and Windows, with automatic instrumentation enabled - Downloads .NET Tracer artifacts from the latest master build that produces the dotnet-tool. This requires changes (done in this PR) to automatically tag builds with the appropriate build tag when they build the dotnet-tool - Scheduled to run daily at 3am UTC - Sets up Postgres with several extensions like PostGIS. These setup instructions were copied directly from the source repo Split Windows regression tests and integration tests to save drive space (https://github.com/DataDog/dd-trace-dotnet/pull/1586) * Split the regression tests and the integration tests These test are actually independent (currently) so we can save drive space by splitting them into two jobs, one which builds the regression tests, another which builds the samples. That saves us ~2GB in both jobs, which should stave off drive space issues for a while * Update build/_build/Build.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Merge tools into Nuke (https://github.com/DataDog/dd-trace-dotnet/pull/1605) * Merge GeneratePackageVersions into Nuke * Move UpdateVendors tool to be a Nuke target Embed the template as a resource for simplicity * minor: move Nuke helper files into folder for easier navigation * Move EnvironmentTools.cs to Datadog.Trace.TestHelpers It's only used in test projects anyway * Move TcpPortProvider to TestHelpers, as again, it's only used in tests Some of the sample projects use this type, so rather than reference TestHelpers (and hence transitively Datadog.Trace), added the helper class a link file instead. * Remove remaining references to Datadog.Core.Tools * Move PrepareRelease to Nuke This required quite a lot of work in the GenerateIntegrationDefinitions code to use reflection instead of direct references. Additionally, I purposefully iterate all the built versions of the assembly, and de-dupe the duplicates, to avoid cases where some integrations are not available on some platforms. This required a slightly annoying amount of boilerplate and faffing with AppLoadContexts * Move CIAppTracking project to Nuke * Don't override everything in Nuke's editorconfig * Update build/_build/Build.Utilities.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> * Remove DistributedTracer tool It's for a very specific use-case, and as it's the last remaining tool, I decided to remove it * Apply suggestions from code review Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * Update some target names Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> Add GitHub workflows for automating Release creation (https://github.com/DataDog/dd-trace-dotnet/pull/1611) This creates 4 new workflows: * Auto add PR to `vNext` milestone * Create version bump PR * Auto tag version bump commit * Create draft release from latest tag To make later stages of the release workflow easier, this will automatically run when a pull request is closed, and add it to a `vNext` milestone. Only _merged_ PRs that will be included in the next release should be added to the `vNext` milestone. When we're ready to do a release, the first step is to do a "version bump" commit. This workflow can be _manually_ triggered from the GitHub actions page, where you provide the next version number, and whether or not it's a prerelease. The action first updates the changelog, using the titles of all PRs included in the `vNext` milestone, using the existing format. Next it does the version bump, updates integrations.json, and syncs the MSI contents. After that's complete it renames the `vNext` milestone to the appropriate value, e.g. `1.28.2-prerelease`. Finally, it creates a branch called `version-bump-1.28.2-prerelease` (for example), commits the changed files to it, pushes, and opens a PR. Users can then review the PR, make changes if required, and merge the PR. When the `[Version Bump]` commit above is merged to master, a new workflow is kicked off, that automatically pushes a new tag, `v1.28.2-prerelease` to the repo. This also creates the benchmark branch (as the previous worklfow was doing) ## Create draft release from latest tag The final workflow creates the draft release from the latest tag assets. This assumes that you've already done a version bump PR (it doesn't _require_ that it's the auto-bump, it just needs to have an entry in the changelog, and assets in Azure Devops). The workflow starts by downloading the assets for the `v1.28.2-prerelease` tag build from Azure Devops. We publish a "well known" artifact that it uses to pull everything down, warning/erroring if it can't find an expected build, if the build isn't complete, or if it failed to produce any assets. Once download and extracted, it extracts the release notes from the changelog, and uses these to create a draft release. The release notes include warning/reminders about uploading the nuget packages to nuget, and about attaching the signed MSI artifacts Final publishing is up to a user, there's no automation around that Add analyzer project + ThreadAbortAnlayzer to detect inifinte loops on ThreadAbortException (https://github.com/DataDog/dd-trace-dotnet/pull/1325) * Add Analyzer project with a ThreadAbortException analyzer Unit tests confirm the analyzer detects and can fix the issue The analyzer is designed to catch situation caused by the bug in the runtime https://github.com/dotnet/runtime/issues/9633, triggered by a try catch nested tight in a while loop. * Add ThreadAbortAnalyzer to all the src libraries * Remove RESX file - unnecessary complexity as not localizing * Add missing file headers * Whitespace changes Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * Replace remaining MSTest with XUnit * Remove boilerplate and use helpers from the package instead Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> Fix PInvokes to the native lib on non windows os. (https://github.com/DataDog/dd-trace-dotnet/pull/1612) * Fixes PInvokeMap on non windows platforms. * Add Call from Initialize method. * Addressing PR comments * Changes based in a PR comment. CIApp - Add support to add custom env-vars from dd-trace wrapper (https://github.com/DataDog/dd-trace-dotnet/pull/1594) * Add support to add custom env-vars from dd-trace wrapper. * Update src/Datadog.Trace.Tools.Runner/Utils.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> * Fix compilation errors. * revert solution Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Fix CIApp Feature Tracking (https://github.com/DataDog/dd-trace-dotnet/pull/1616) Fix crank project reference (https://github.com/DataDog/dd-trace-dotnet/pull/1617) * Fix project reference * Remove project reference and use reflection instead. Add fallback for logs injection for NLog versions < 4.1.0 (https://github.com/DataDog/dd-trace-dotnet/pull/1607) Until NLog 4.1.0, there is only one Set API in the MappedDiagnosticsContext class: Set(string, string). Our new CustomNLogLogProvider requires Set(string, object) API, which was introduced in NLog 4.1.0, to use our optimized LogEnricher type. Without this API present, we can't use our CustomNLogLogProvider type so revise the IsLoggerAvailable check to fail when the API is not found. Add fix for log injection with log4net.Ext.Json versions < 2.0.9.1 (https://github.com/DataDog/dd-trace-dotnet/pull/1615) Add fix for log4net.Ext.Json versions < 2.0.9.1 not supporting log4net.Core.IFixingRequired for special-case object rendering. Background The log4net.Ext.Json library is a prominent third-party library that can render log4net objects into JSON. To prepare the final JSON object, the StandardTypesDecorator does a type check against each value in the MDC dictionary and tries to render each object accordingly. If the value is a well-known type, then the idiomatic JSON value is written. If not, the decorator will render the value as a new JSON object and attempt to render each of the original object's members as properties on the new JSON object. Bug In log4net.Ext.Json versions < 2.0.9.1, the interface log4net.Core.IFixingRequired is not included in the set of types specifically recognized as a built-in type, so the custom object Log4NetEnricher+TracerProperty that we insert into the MDC dictionary is rendered as a new JSON object, not a string, which breaks trace-log correlation. Fix Before using our CustomLog4NetLogProvider (which utilizes our Log4NetEnricher for holding the current trace_id and span_id values for logs injection), check if the application is using log4net.Ext.Json version 2.0.9.1 or newer. If not, we'll fallback to the original liblog log provider implementation. Include the project reference to the crank sample to avoid the error trying to locate the managed assembly. (https://github.com/DataDog/dd-trace-dotnet/pull/1618) don't include analyzer when publishing projects (https://github.com/DataDog/dd-trace-dotnet/pull/1619) Fixes the dd-trace exit code always returning 0 (https://github.com/DataDog/dd-trace-dotnet/pull/1621) * Fixes the dd-trace exit code always returning 0 * Removes exit code 1 on --version Removes version from the testFramework tag (https://github.com/DataDog/dd-trace-dotnet/pull/1622) Add ASP.NET Core on IIS Express tests (https://github.com/DataDog/dd-trace-dotnet/pull/1582) * Convert ASP.NET Core tests to snapshot tests * Add snapshots for ASP.NET Core tests * Add tests for middleware-only branches and map branches * Update snapshots for new branches * Add CallTarget and RouteTemplateResourceNames tests * Add snapshots for CallTarget and feature flag for aspnetcore tests * Add additional IIS app types, instead of just classic/integrated To support running asp.net core on iis, both in- and out- of process * Add ASP.NET Core module to applicationHost.config Add dummy config for aspnetcore module. * Add web.config to ASP.NET Core apps * Add ASP.NET Core IIS tests * Add support for x86 tests We have to use the x86 version of dotnet.exe, to run in IIS express. So use the x86 version version on Windows when running x86 tests * Add snapshots for ASP.NET Core IIS tests * Iis tests aren't supported on Linux * Fix CallTarget setting. * Remove incorrect comment * fixup add additional iis * Fix rebasing issues Fix ReJIT and Shutdown log levels. (https://github.com/DataDog/dd-trace-dotnet/pull/1624) Fix ARM64 Integration tests (https://github.com/DataDog/dd-trace-dotnet/pull/1629) * Fix ARM64 Integration tests ARM64 integration tests have not been running due to a typo in the test filter `Exception filtering tests: Incorrect format for TestCaseFilter Error: Missing ')'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.` * Add missing ARM64 AWS SQS config to docker-compose Clear metadata when appdomain unloads (https://github.com/DataDog/dd-trace-dotnet/pull/1630) Exclude more vendored files from code coverage (https://github.com/DataDog/dd-trace-dotnet/pull/1632) Add additional scrubbing for stacktraces in snapshots (https://github.com/DataDog/dd-trace-dotnet/pull/1633) * Stop splitting snapshots by architecture We only needed to do this for the exception messages, but it's annoying, and will only get worse with NGEN support (as .NET Core 2.1 will have different stack traces, for example) * Update Snapshots (delete x86 snapshots, rename x64 snapshots) * Add additional scrubbing to the StackTrace With this change we only keep (and compare) the first "at " in the stack trace. This ensures consistency across platforms and runtimes. * Update snapshots with scrubbed stack traces Logger API refactoring (https://github.com/DataDog/dd-trace-dotnet/pull/1625) * Move logging free functions (Info, Debug, Warn) to static function member of Logger class * Rewrite the logger implementation to allow one logger for the Tracer, for the Profiler, for the Common librairies, and more... Preserve the task cancelled state when using calltarget instrumentation (https://github.com/DataDog/dd-trace-dotnet/pull/1634) Fixes the native logger path issue (https://github.com/DataDog/dd-trace-dotnet/pull/1635) Enable experimental NGEN support (https://github.com/DataDog/dd-trace-dotnet/pull/1628) * Initial basic support. * Finish NGEN experimental support. * Temporal workaround over RequestReJITWithInliners. * remove comment. * Fixes compilation error. * Refactor and ensure always calling EnqueueForRejit either having complete or incomplete data. * Add checks. * update comments. * Fix ReJIT on NGEN. * Apply suggestions from code review Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * Adaptation to the new loader. Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> Add additional logs injection fallback for NLog 1.x (https://github.com/DataDog/dd-trace-dotnet/pull/1614) This work follows from pull request https://github.com/DataDog/dd-trace-dotnet/pull/1607 The previous PR fixes NLog logs injection to work correctly when applications use NLog versions >= 2.0. This PR provides an additional fix for NLog logs injection to work correctly when applications use NLog versions >= 1.0. This is accomplished by adding a new NLog log provider that extends the built-in LibLog NLogLogProvider by also searching for the `NLog.MDC` type for the `Set`/`Remove` APIs which exists in NLog 1.x, in addition to `NLog.MappedDiagnosticsContext` which only exists in NLog 2.x and later. * Make propagator public * ExporterWriter.WriteTrace param to ArraySegment * Cast tracer to IDatadogTracer to accesss Propagator * Use OTel snk file * Add propagator casting for ASP.NET Core Observer * Add exit point after exception on FlushTracesTaskLoopAsync * Re-add internal Propagator property Add better error handling for the Header Tags feature accessing System.Web.HttpResponse.Headers (https://github.com/DataDog/dd-trace-dotnet/pull/1641) Fixes "System.PlatformNotSupportedException: This operation requires IIS integrated pipeline mode" This scenario can occur even when the IIS application is running in the Integrated pipeline mode, which seems to be related to Sitefinity CMS widgets. Without having a reproduction available, this doesn't prevent the bad access from happening. Instead, we have better handling of the scenario: 1. Only attempt to access the System.Web.HttpResponse.Headers field when the application has explicitly configured the DD_TRACE_HEADER_TAGS/TracerSettings.HeaderTags feature, which avoids the issue in some cases 2. Catch the exception and prevents further field accesses when the exception is observed Fix throughput tests (https://github.com/DataDog/dd-trace-dotnet/pull/1650) Throughput tests on Windows have been broken for a week, for no apparent reason. I think something must have changed in the way the `DownloadPipelineArtifact` task runs, as the native windows dll is now being unexpectedly downloaded to a `win-64` sub-folder. The fix here is to move it back to where it should be [Version Bump] 1.28.2 (https://github.com/DataDog/dd-trace-dotnet/pull/1654) * [Version Bump] 1.28.2 Co-authored-by: andrewlock <andrewlock@users.noreply.github.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Add support for a temporary NGEN pipeline (https://github.com/DataDog/dd-trace-dotnet/pull/1642) * Pass through DD_CLR_ENABLE_NGEN to Docker * Pass the DD_CLR_ENABLE_NGEN variable to docker/compose * Don't run some stages for NGEN builds Crank native profiler fix (https://github.com/DataDog/dd-trace-dotnet/pull/1655) * Adds commit hash to the profiler path * Apply the commit_hash sub folder to other crank scenarios. * fix missing variable declaration. * Fail the script in case the crank process return an error ExitCode * Fail the throughput tests if a file doesn't exist (cherry picked from commit 6b5b26eeffc997fb456a2d1eee1bb824e185b268) Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Filter now applied to the samples when compiling (https://github.com/DataDog/dd-trace-dotnet/pull/1653) * Filter now applied to the samples when compiling * Change switch used and make available on CompileSamplesLinux Reduce length of snapshot paths (https://github.com/DataDog/dd-trace-dotnet/pull/1657) * Use the numeric status code in snapshot verifier names Fewer characters, less chance of blowing up MAX_PATH * Rename snapshots * Fix whitespace Update GitHub action release workflows (https://github.com/DataDog/dd-trace-dotnet/pull/1659) * minor: fix job name * Add missing id to versions stage * Automatically check for changed files in version bump commit * Link to the download URL in draft release (instead of a JSON api) Fixes native regex usage on non windows platforms. (https://github.com/DataDog/dd-trace-dotnet/pull/1662) Cache native AssemblyReference instances (https://github.com/DataDog/dd-trace-dotnet/pull/1665) * Cache native assembly reference instances to avoid repeating all regex calls * Adds unordered map mutex * Update src/Datadog.Trace.ClrProfiler.Native/calltarget_tokens.cpp Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> * Fix suggestion compilation error. Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Read performance counter values from environment variables in AAS (https://github.com/DataDog/dd-trace-dotnet/pull/1651) * Read performance counter values from environment variables in AAS Improve error handling for performance counters (https://github.com/DataDog/dd-trace-dotnet/pull/1652) * Log a specific error message when permissions are unsufficient to read performance counters * Disable performance counters when initialization failed Use link shortener for IIS permissions (https://github.com/DataDog/dd-trace-dotnet/pull/1666) Native profiler Initialize callback optimization (https://github.com/DataDog/dd-trace-dotnet/pull/1672) * Native profiler Initialize callback optimization. * Cleanup and fix tests. * Adds total time. * Changes from review. Merge auto-instrumentation code into Datadog.Trace.dll (https://github.com/DataDog/dd-trace-dotnet/pull/1443) * Rename follow-up Fixes crank pipeline on PR merge commits. (https://github.com/DataDog/dd-trace-dotnet/pull/1669) * show folder content * changes. * Copies the managed profiler to the crank runners. Removes Datadog.Trace.ClrProfiler.Managed from the throughput project. * fix pipeline. * Fix app. * list loaded assembly. * Enable debug mode. * Force loader injection in calltarget scenarios. * Fixes assembly name. Disable CallSite scenario from Throughput tests. (https://github.com/DataDog/dd-trace-dotnet/pull/1674) Add instrumentation for Begin/EndGetResponse to WebRequest (https://github.com/DataDog/dd-trace-dotnet/pull/1658) * Add overload to scope factory that allows providing a start time * Add HttpWebRequest integration from BeginGetRequestStream, BeginGetResponse, and EndGetResponse BeginGetRequestStream can be used in both .NET Core and .NET Framework, but BeginGetResponse and EndGetResponse are .NET Framework only, because the .NET Core implementation uses HttpClient under the covers (which we already instrument) * Update WebRequestTests to handle additional scenarios * Fix rename service tests * Fix WebRequest20Tests to reflect additional instrumentation * Add a test for BeginGetResponse when Task.Factory.FromAsync is used * Simplify Task.Factory.FromAsync implementation * Update integrations json after rebase The v0 version of App Sec (https://github.com/DataDog/dd-trace-dotnet/pull/1647) This PR implements a first version of App Sec. Please forgive the message history, it will be squashed away :) The App Sec module introduces two new environment variables to activate App Sec features: `DD_APPSEC_ENABLED` – enables the App Sec feature, any HTTP requests will be examined by the WAF (Web Application Firewall). By default, this feature is disabled. `DD_APPSEC_BLOCKING_ENABLED` – this controls whether the App Sec feature will block suspicious requests. By default, this feature is disabled. The app sec uses the ‘call target’ instrumentation to gather the data about the HTTP request that will be passed to the WAF. It is has not been implemented for ‘call site’ instrumentation and currently there is not plan to support this style of instrumentation. Data is gathered in the instrumentations for ASP.NET and ASP.NET Core and then passed to the ‘instrumentation gateway’ which starts the security flow. Currently the instrumentation gateway simply creates and event with the incoming data. In future version this component will be used to route the data to different security components based on the addresses associated with the data, but this functionality is not necessary for the first version. After the instrumentation gateway the data flow is as follows: ``` InstrumentationGateway -> Security -> PowerWaf -> Native -> Sqreen.dll ``` `Security` – this class coordinates the reaction to security events `PowerWaf` – this class provides .NET idiomatic wrapper to the WAF, as the WAF is implemented in C++ with C style interface `Native` – this class is responsible for finding, loading and providing low level bindings with the WAF `[lib]Sqreen.[dll|so|dylib]` – this are the native binaries containing the WAF implementation. Security events are reported to the agent and backend by `AgentWriter` class, which implements a basic queuing system. Two new samples are provided, that form the basis for the security integration tests. These are `Samples.AspNetMvc5` and `Samples.AspNetCore5` which are used to test ASP.NET and ASP.NET Core respectively. Integration tests based on similar principals to `Datadog.Trace.ClrProfiler.IntegrationTests` have been provided in `Datadog.Trace.Security.IntegrationTests` and are run as part of the integration tests in the `consolidated-pipeline` CI. ARM based Linux systems are not supported yet, but all other environment supported by the .NET libraries are. ASP.NET Core running on .NET Framework is not supported. The instrumentation is a no-op in these scenarios. This is because it though this unlikely to work, since we rely on referencing several `Microsoft.AspNetCore.*` dlls to be able instrument the ASP.NET Core pipeline. Co-authored-by: Anna Yafi <anna.yafi@datadoghq.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Native logger error fix (https://github.com/DataDog/dd-trace-dotnet/pull/1677) Adds NGEN scenario on Throughput test. (https://github.com/DataDog/dd-trace-dotnet/pull/1678) Removes code for Callsite scenario from the throughput tests. (https://github.com/DataDog/dd-trace-dotnet/pull/1679) Small build improvements (https://github.com/DataDog/dd-trace-dotnet/pull/1646) * Allow running an arbitrary dotnet sample app from Nuke i.e. it doesn't have to be part of the solution * Automatically build the "multiPackageProject" list from PackageVersionsGeneratorDefinitions.json There's no point building these twice, so previously we had a list of sample projects to not build. This simplifies things somewhat by loading the list dynamically * Try using a unique docker network per build to avoid issues with concurrent builds on arm64 * Upload artifacts to azure (for use in AAS extension automation) The latest file list can be downloaded from: https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/index.txt and the latest commit sha of the uploads can be downloaded from https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/sha.txt The blobs are stored in a subdirectory * Only push assets to Azure from master * Pull Azure storage location from variable Update CI Visibility specification (https://github.com/DataDog/dd-trace-dotnet/pull/1682) * Update CI Visibility specification * refactor and fixes. replace "minimal" solution with a solution filter (https://github.com/DataDog/dd-trace-dotnet/pull/1631) Add a custom test framework to monitor execution of unit tests in ducktyping library (https://github.com/DataDog/dd-trace-dotnet/pull/1680) Add tests for changes to Datadog.Trace's Public API (https://github.com/DataDog/dd-trace-dotnet/pull/1681) * Add tests for public API and public browsable (intellisense) API Note that we can't use Verify in Datadog.Trace.Tests because it uses v5.0.0 of the Logging abstractions library, which is unfortunately incompatible with v2.x of the aspnetcore libraries. Resorted to a poor man's version instead. * Add public API snapshots based on v1.28.2 (pre assembly merge) * Update snapshots _after_ assembly merge. This adds a _lot_ of surface area * Update snapshots for master * Make Security class internal * Add [Browsable(false)] and [EditorBrowsable(EditorBrowsableState.Never)] to all public APIs in the ClrProfiler library * Exclude non-browsable APIs from "public API" * Try making some things internal Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * More updates * Fix test visibility Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> [Version Bump] 1.28.3-prerelease (https://github.com/DataDog/dd-trace-dotnet/pull/1688) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> Adds Execution time benchmarks (https://github.com/DataDog/dd-trace-dotnet/pull/1687) * Adds initial execution benchmarks * fixes * Adds sleep to the end of the pipeline. Produce NuGet package to deploy automatic instrumentation (https://github.com/DataDog/dd-trace-dotnet/pull/1661) Produces a NuGet package that contains all of the automatic instrumentation assets that will be deployed to the application output directory. The current name is "Datadog.Instrumentation" Details: - TargetFrameworks: net45, net461, netstandard2.0, netcoreapp3.1 (same as Datadog.Trace) - Dependencies: Datadog.Trace - This means customers who try to restore an earlier version of `Datadog.Trace` will face a build-time NU1605 error (https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu1605) - Contents: Publishes the following to application output |-- datadog | |-- linux-arm64 | | |-- Datadog.Trace.ClrProfiler.Native.so | |-- linux-musl-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- linux-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- net45 | | |-- Managed assemblies | |-- net461 | | |-- Managed assemblies | |-- netcoreapp3.1 | | |-- Managed assemblies | |-- netstandard2.0 | | |-- Managed assemblies | |-- osx-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dylib | | |-- libSqreen.dylib | |-- win-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- win-x86 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- createLogPath.sh | |-- integrations.json Additional changes: - Nuke changes: build BuildTracerHome [CreateInstrumentationHome] BuildInstrumentationNuget - CreateInstrumentationHomebuild target: Copies automatic instrumentation binaries from tracer home to the src/Datadog.Instrumentation/home directory, to be used by the Instrumentation, RunnerTool, and StandaloneTool projects - BuildInstrumentationNuget build target: Builds the Datadog.Instrumentation NuGet package with the binaries in the src/Datadog.Instrumentation/home directory - Removes src/Datadog.Trace.Tools.Runner/Datadog.Trace.Tools.Runner.proj - Add a sample application to samples/NugetDeployment directory with a docker-compose file to run the application on each platform supported by automatic instrumentation Appsec/anna/v0 integration/crankscenarios (https://github.com/DataDog/dd-trace-dotnet/pull/1684) * crank scenarios for appsec * Reset sln configs * Include new yaml in sln fix json filename fix name of file for linux profile Scenarios according to acceptance criterias increase timeout limit logging rate restore ultimate pipeline increase timeout change order of tests to check increase timeout for linux Sharing profiles in another file to import Proxy calls to dl (https://github.com/DataDog/dd-trace-dotnet/pull/1686) * Proxy calls to dl Some Linux distros don't have a version of libdl.so without a version prefix. Call from manage code will fail to find the library on these distros. Proxying via a C++ library should cure this, because the C libraries will handle loading in this case. * Add linker command for libdil * Rename exported fuctions to avoid strange issue * Rewrite PInvoke tables for the native methods used by App Sec too * Correct build error on windows Co-authored-by: Robert Pickering <robertfpickering@fastmail.com> Add end-to-end integration tests for logs injection (https://github.com/DataDog/dd-trace-dotnet/pull/1637) New Tests - `LogsInjection.Log4Net`, `LogsInjection.NLog`, `LogsInjection.Serilog` - Behaviors - Logs injection occurs with raw logs and JSON logs - After logs injection setup, a method is invoked another AppDomain to test LogicalCallContext storage - Libraries tested - log4net - .NET Framework: 1.2 - 2.x - .NET Core: 2.0.6+ - NLog - .NET Framework: 1.x - 4.x - .NET Core: 4.5+ - Serilog - .NET Framework: 1.x - 2.x - .NET Core: 2.x - Caveats - **Note**: Windows testing currently does not test multiple package versions. That will need to be enabled at a later point in time - All log4net versions are still vulnerable to serialization issues when crossing AppDomains, so the cross-AppDomain scenario is skipped in the log4net test application Deleted Tests - `Log4Net.SerializationException` regression test (and helper library ApplicationWithLog4Net) - The regression test was added in https://github.com/DataDog/dd-trace-dotnet/pull/551 but is now solved by adding new sample applications that directly cause calls to cross AppDomain boundaries - `NLog10LogsInjection.NullReferenceException` regression test - The regression test was added in https://github.com/DataDog/dd-trace-dotnet/pull/614 but is now solved by the new type FallbackNLogLogProvider Fixes - Disable Serilog logs injection support when the detected Serilog version is earlier than 2.0. This is enforced by adding a new `NoOpSerilogLogProvider` log provider to perform no-ops in this scenario - Add better exception handling in `CustomLog4NetLogProvider.ExtJsonAssemblySupported()` Add Microsoft.Extensions.Logging.ILogger instrumentation (https://github.com/DataDog/dd-trace-dotnet/pull/1663) * Add LogsInjectionTestBase from #1637 * Add ILogger integration When a logger iterates the scopes for a log message we insert ourselves as the first entry. Note that we don't _really_ add a scope (by instrumenting `BeginScope()` or similar) as that would insert our scope in the "scope stack", and make us responsible for removing it at the appropriate time. The approach here is much simpler, and works for all types of instrumentation that are using the standard ILogger infrastructure This essentially works the same way as the "activity" ILogger integration does in the runtime (https://github.com/dotnet/runtime/blob/v5.0.0/src/libraries/Microsoft.Extensions.Logging/src/LoggerFactoryScopeProvider.cs#L36-L52) * Update profiler skipped assembly prefixes We need to instrument Microsoft.Extensions.Logging.Abstractions now, so can't skip the Microsoft.Extensions prefix This adds most of them to the skip list, but there may be a better approach, e.g. an additional "don't skip" list? * Add LogsInjection aspnetcore sample * Add tests for ILogger logs injection * Update ILogger integration to inject dd_service etc even if we don't have a trace_id * Inject dd_span when a span is available Update tests to verify span_id is injected correctly * Update integrations.json after rebase * Convert DatadogLoggingScope indexer to switch expression * Add required [Browsable(false)] and [EditorBrowsable(Never)] attributes to ilogger integration Reduce snapshot file path length (https://github.com/DataDog/dd-trace-dotnet/pull/1696) * Use numeric status code in snapshots + a shorter name numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames * Remove method name from aspnetcore snapshots to reduce file length * Rename snapshots Merging repos: 'dd-shared-components-dotnet' into 'dd-trace-dotnet'. (https://github.com/DataDog/dd-trace-dotnet/pull/1694) This is the first of several self-contained PRs targeted at moving the Shared Code (as in _shared between the Tracer and the Profiler_) from [dd-shared-components-dotnet](https://github.com/DataDog/dd-shared-components-dotnet) into [dd-trace-dotnet](https://github.com/DataDog/dd-trace-dotnet). For this PR, I created a branch `permanent/MoveFrom_dd-shared-components-dotnet` and moved all files from the `dd-shared-components-dotnet` into there, including their complete change history. The branch will remain permanently as a history record, and we will squash-merge this PR into the `master` branch. The files are all inside the `/shared` folder, so that they can be naturally merged into the planned joint directory structure. In addition, this PR also makes the following small changes: * Make whatever tweaks necessary so that _all_ the Shared Code projects build inside `dd-trace-dotnet`. * Set up placeholder folders for the `/tracer` and the `/profiler` products to facilitate the directory refactoring in the subsequent PRs. * Update ReadMe files to describe the changes. Note that this PR keeps the directory structure within the `/shared` folder exactly as it was within the original repo. Subsequent changed will adjust this structure to joint conventions as required. This will require coordination with the profiler build configuration which makes some assumptions about the layout of the relevant folders. This change is self-contained. It will not affect any existing Tracer components, and it will allow the overall repo merge project to proceed in several steps that can be validated independently. This PR will be followed up with additional PR to address the following: * Move Tracer-specific files into the `/tracer` folder. * Ensure that changes contained to shared files that are not (yet) used by the Tracer do not require the Tracer CI validation to pass (this will be updated in the future if and when the Tracer uses more shared components) * Joint build of components used by several products. * Joint release package. * Joint conventions for folder structure. * Etc... Sending more relevant data to backend (https://github.com/DataDog/dd-trace-dotnet/pull/1695) * Adding more data and filtering cookie from the no cookie header waf address Remove usage of non-builtin types in Attribute named arguments (https://github.com/DataDog/dd-trace-dotnet/pull/1601) To workaround an issue discovered in Azure Functions, remove all usages of named arguments in Attributes that are not built-in types. This means removing arguments that use our own custom enums, but keeping arguments that use `string`, etc. While rare, the failing scenario occurs when the corelib attempts to create the `Attribute` object from the metadata but the assembly that gets resolved from the default load context (or one of its handlers) does not resolve to the same assembly as the one we expected, causing type equality to fail. Note: This workaround is only required for attributes that will be instantiated from metadata, like `DuckAttribute`, but it is …

andrewlock added 3 commits August 23, 2021 11:24

Use numeric status code in snapshots + a shorter name

a14ca1e

numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames

Remove method name from aspnetcore snapshots to reduce file length

bfbf9de

Rename snapshots

318d181

andrewlock requested a review from a team as a code owner August 23, 2021 10:30

andrewlock added the area:tests unit tests, integration tests label Aug 23, 2021

kevingosse approved these changes Aug 23, 2021

View reviewed changes

zacharycmontoya approved these changes Aug 23, 2021

View reviewed changes

andrewlock merged commit b4b1a37 into master Aug 23, 2021

andrewlock deleted the andrew/reduce-snapshot-file-path-lengths branch August 23, 2021 15:47

github-actions bot added this to the vNext milestone Aug 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce snapshot file path length #1696

Reduce snapshot file path length #1696

andrewlock commented Aug 23, 2021

macrogreg commented Aug 23, 2021

Reduce snapshot file path length #1696

Reduce snapshot file path length #1696

Conversation

andrewlock commented Aug 23, 2021

macrogreg commented Aug 23, 2021