Refactor folder locations [latest] #1748
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucaspimentel
approved these changes
Sep 7, 2021
zacharycmontoya
force-pushed
the
zach/tracer-folder-organized
branch
from
c6f55f0
to
5c713d6
Compare
Merged
zacharycmontoya
force-pushed
the
zach/tracer-folder-organized
branch
from
5c713d6
to
7d215e4
Compare
andrewlock
reviewed
Sep 8, 2021
100% agree, I'd like to see the entire repo use the same Nuke project but for porting I thought it would be easier to initially move as much as possible into the tracer directory so I took the |
Totally agree. At some point, this something I would like to see happening ;) |
andrewlock
approved these changes
Sep 9, 2021
colin-higgins
approved these changes
Sep 9, 2021
- tracer/build/artifacts/createLogPath.sh - tracer/build/crank/run.sh - tracer/build.sh - tracer/build_in_docker.sh
zacharycmontoya
force-pushed
the
zach/tracer-folder-organized
branch
from
7d215e4
to
864790b
Compare
pjanotti
added a commit
to pjanotti/opentelemetry-dotnet-instrumentation
that referenced
this pull request
Sep 22, 2021
Fixes crank pipeline on PR merge commits. (DataDog/dd-trace-dotnet#1669) * show folder content * changes. * Copies the managed profiler to the crank runners. Removes Datadog.Trace.ClrProfiler.Managed from the throughput project. * fix pipeline. * Fix app. * list loaded assembly. * Enable debug mode. * Force loader injection in calltarget scenarios. * Fixes assembly name. Disable CallSite scenario from Throughput tests. (DataDog/dd-trace-dotnet#1674) Add instrumentation for Begin/EndGetResponse to WebRequest (DataDog/dd-trace-dotnet#1658) * Add overload to scope factory that allows providing a start time * Add HttpWebRequest integration from BeginGetRequestStream, BeginGetResponse, and EndGetResponse BeginGetRequestStream can be used in both .NET Core and .NET Framework, but BeginGetResponse and EndGetResponse are .NET Framework only, because the .NET Core implementation uses HttpClient under the covers (which we already instrument) * Update WebRequestTests to handle additional scenarios * Fix rename service tests * Fix WebRequest20Tests to reflect additional instrumentation * Add a test for BeginGetResponse when Task.Factory.FromAsync is used * Simplify Task.Factory.FromAsync implementation * Update integrations json after rebase The v0 version of App Sec (DataDog/dd-trace-dotnet#1647) This PR implements a first version of App Sec. Please forgive the message history, it will be squashed away :) The App Sec module introduces two new environment variables to activate App Sec features: `DD_APPSEC_ENABLED` – enables the App Sec feature, any HTTP requests will be examined by the WAF (Web Application Firewall). By default, this feature is disabled. `DD_APPSEC_BLOCKING_ENABLED` – this controls whether the App Sec feature will block suspicious requests. By default, this feature is disabled. The app sec uses the ‘call target’ instrumentation to gather the data about the HTTP request that will be passed to the WAF. It is has not been implemented for ‘call site’ instrumentation and currently there is not plan to support this style of instrumentation. Data is gathered in the instrumentations for ASP.NET and ASP.NET Core and then passed to the ‘instrumentation gateway’ which starts the security flow. Currently the instrumentation gateway simply creates and event with the incoming data. In future version this component will be used to route the data to different security components based on the addresses associated with the data, but this functionality is not necessary for the first version. After the instrumentation gateway the data flow is as follows: ``` InstrumentationGateway -> Security -> PowerWaf -> Native -> Sqreen.dll ``` `Security` – this class coordinates the reaction to security events `PowerWaf` – this class provides .NET idiomatic wrapper to the WAF, as the WAF is implemented in C++ with C style interface `Native` – this class is responsible for finding, loading and providing low level bindings with the WAF `[lib]Sqreen.[dll|so|dylib]` – this are the native binaries containing the WAF implementation. Security events are reported to the agent and backend by `AgentWriter` class, which implements a basic queuing system. Two new samples are provided, that form the basis for the security integration tests. These are `Samples.AspNetMvc5` and `Samples.AspNetCore5` which are used to test ASP.NET and ASP.NET Core respectively. Integration tests based on similar principals to `Datadog.Trace.ClrProfiler.IntegrationTests` have been provided in `Datadog.Trace.Security.IntegrationTests` and are run as part of the integration tests in the `consolidated-pipeline` CI. ARM based Linux systems are not supported yet, but all other environment supported by the .NET libraries are. ASP.NET Core running on .NET Framework is not supported. The instrumentation is a no-op in these scenarios. This is because it though this unlikely to work, since we rely on referencing several `Microsoft.AspNetCore.*` dlls to be able instrument the ASP.NET Core pipeline. Co-authored-by: Anna Yafi <anna.yafi@datadoghq.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Native logger error fix (DataDog/dd-trace-dotnet#1677) Adds NGEN scenario on Throughput test. (DataDog/dd-trace-dotnet#1678) Removes code for Callsite scenario from the throughput tests. (DataDog/dd-trace-dotnet#1679) Small build improvements (DataDog/dd-trace-dotnet#1646) * Allow running an arbitrary dotnet sample app from Nuke i.e. it doesn't have to be part of the solution * Automatically build the "multiPackageProject" list from PackageVersionsGeneratorDefinitions.json There's no point building these twice, so previously we had a list of sample projects to not build. This simplifies things somewhat by loading the list dynamically * Try using a unique docker network per build to avoid issues with concurrent builds on arm64 * Upload artifacts to azure (for use in AAS extension automation) The latest file list can be downloaded from: https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/index.txt and the latest commit sha of the uploads can be downloaded from https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/sha.txt The blobs are stored in a subdirectory * Only push assets to Azure from master * Pull Azure storage location from variable Update CI Visibility specification (DataDog/dd-trace-dotnet#1682) * Update CI Visibility specification * refactor and fixes. replace "minimal" solution with a solution filter (DataDog/dd-trace-dotnet#1631) Add a custom test framework to monitor execution of unit tests in ducktyping library (DataDog/dd-trace-dotnet#1680) Add tests for changes to Datadog.Trace's Public API (DataDog/dd-trace-dotnet#1681) * Add tests for public API and public browsable (intellisense) API Note that we can't use Verify in Datadog.Trace.Tests because it uses v5.0.0 of the Logging abstractions library, which is unfortunately incompatible with v2.x of the aspnetcore libraries. Resorted to a poor man's version instead. * Add public API snapshots based on v1.28.2 (pre assembly merge) * Update snapshots _after_ assembly merge. This adds a _lot_ of surface area * Update snapshots for master * Make Security class internal * Add [Browsable(false)] and [EditorBrowsable(EditorBrowsableState.Never)] to all public APIs in the ClrProfiler library * Exclude non-browsable APIs from "public API" * Try making some things internal Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * More updates * Fix test visibility Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> [Version Bump] 1.28.3-prerelease (DataDog/dd-trace-dotnet#1688) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> Adds Execution time benchmarks (DataDog/dd-trace-dotnet#1687) * Adds initial execution benchmarks * fixes * Adds sleep to the end of the pipeline. Produce NuGet package to deploy automatic instrumentation (DataDog/dd-trace-dotnet#1661) Produces a NuGet package that contains all of the automatic instrumentation assets that will be deployed to the application output directory. The current name is "Datadog.Instrumentation" Details: - TargetFrameworks: net45, net461, netstandard2.0, netcoreapp3.1 (same as Datadog.Trace) - Dependencies: Datadog.Trace - This means customers who try to restore an earlier version of `Datadog.Trace` will face a build-time NU1605 error (https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu1605) - Contents: Publishes the following to application output |-- datadog | |-- linux-arm64 | | |-- Datadog.Trace.ClrProfiler.Native.so | |-- linux-musl-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- linux-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- net45 | | |-- Managed assemblies | |-- net461 | | |-- Managed assemblies | |-- netcoreapp3.1 | | |-- Managed assemblies | |-- netstandard2.0 | | |-- Managed assemblies | |-- osx-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dylib | | |-- libSqreen.dylib | |-- win-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- win-x86 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- createLogPath.sh | |-- integrations.json Additional changes: - Nuke changes: build BuildTracerHome [CreateInstrumentationHome] BuildInstrumentationNuget - CreateInstrumentationHomebuild target: Copies automatic instrumentation binaries from tracer home to the src/Datadog.Instrumentation/home directory, to be used by the Instrumentation, RunnerTool, and StandaloneTool projects - BuildInstrumentationNuget build target: Builds the Datadog.Instrumentation NuGet package with the binaries in the src/Datadog.Instrumentation/home directory - Removes src/Datadog.Trace.Tools.Runner/Datadog.Trace.Tools.Runner.proj - Add a sample application to samples/NugetDeployment directory with a docker-compose file to run the application on each platform supported by automatic instrumentation Appsec/anna/v0 integration/crankscenarios (DataDog/dd-trace-dotnet#1684) * crank scenarios for appsec * Reset sln configs * Include new yaml in sln fix json filename fix name of file for linux profile Scenarios according to acceptance criterias increase timeout limit logging rate restore ultimate pipeline increase timeout change order of tests to check increase timeout for linux Sharing profiles in another file to import Proxy calls to dl (DataDog/dd-trace-dotnet#1686) * Proxy calls to dl Some Linux distros don't have a version of libdl.so without a version prefix. Call from manage code will fail to find the library on these distros. Proxying via a C++ library should cure this, because the C libraries will handle loading in this case. * Add linker command for libdil * Rename exported fuctions to avoid strange issue * Rewrite PInvoke tables for the native methods used by App Sec too * Correct build error on windows Co-authored-by: Robert Pickering <robertfpickering@fastmail.com> Add end-to-end integration tests for logs injection (DataDog/dd-trace-dotnet#1637) New Tests - `LogsInjection.Log4Net`, `LogsInjection.NLog`, `LogsInjection.Serilog` - Behaviors - Logs injection occurs with raw logs and JSON logs - After logs injection setup, a method is invoked another AppDomain to test LogicalCallContext storage - Libraries tested - log4net - .NET Framework: 1.2 - 2.x - .NET Core: 2.0.6+ - NLog - .NET Framework: 1.x - 4.x - .NET Core: 4.5+ - Serilog - .NET Framework: 1.x - 2.x - .NET Core: 2.x - Caveats - **Note**: Windows testing currently does not test multiple package versions. That will need to be enabled at a later point in time - All log4net versions are still vulnerable to serialization issues when crossing AppDomains, so the cross-AppDomain scenario is skipped in the log4net test application Deleted Tests - `Log4Net.SerializationException` regression test (and helper library ApplicationWithLog4Net) - The regression test was added in DataDog/dd-trace-dotnet#551 but is now solved by adding new sample applications that directly cause calls to cross AppDomain boundaries - `NLog10LogsInjection.NullReferenceException` regression test - The regression test was added in DataDog/dd-trace-dotnet#614 but is now solved by the new type FallbackNLogLogProvider Fixes - Disable Serilog logs injection support when the detected Serilog version is earlier than 2.0. This is enforced by adding a new `NoOpSerilogLogProvider` log provider to perform no-ops in this scenario - Add better exception handling in `CustomLog4NetLogProvider.ExtJsonAssemblySupported()` Add Microsoft.Extensions.Logging.ILogger instrumentation (DataDog/dd-trace-dotnet#1663) * Add LogsInjectionTestBase from open-telemetry#1637 * Add ILogger integration When a logger iterates the scopes for a log message we insert ourselves as the first entry. Note that we don't _really_ add a scope (by instrumenting `BeginScope()` or similar) as that would insert our scope in the "scope stack", and make us responsible for removing it at the appropriate time. The approach here is much simpler, and works for all types of instrumentation that are using the standard ILogger infrastructure This essentially works the same way as the "activity" ILogger integration does in the runtime (https://github.com/dotnet/runtime/blob/v5.0.0/src/libraries/Microsoft.Extensions.Logging/src/LoggerFactoryScopeProvider.cs#L36-L52) * Update profiler skipped assembly prefixes We need to instrument Microsoft.Extensions.Logging.Abstractions now, so can't skip the Microsoft.Extensions prefix This adds most of them to the skip list, but there may be a better approach, e.g. an additional "don't skip" list? * Add LogsInjection aspnetcore sample * Add tests for ILogger logs injection * Update ILogger integration to inject dd_service etc even if we don't have a trace_id * Inject dd_span when a span is available Update tests to verify span_id is injected correctly * Update integrations.json after rebase * Convert DatadogLoggingScope indexer to switch expression * Add required [Browsable(false)] and [EditorBrowsable(Never)] attributes to ilogger integration Reduce snapshot file path length (DataDog/dd-trace-dotnet#1696) * Use numeric status code in snapshots + a shorter name numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames * Remove method name from aspnetcore snapshots to reduce file length * Rename snapshots Merging repos: 'dd-shared-components-dotnet' into 'dd-trace-dotnet'. (DataDog/dd-trace-dotnet#1694) This is the first of several self-contained PRs targeted at moving the Shared Code (as in _shared between the Tracer and the Profiler_) from [dd-shared-components-dotnet](https://github.com/DataDog/dd-shared-components-dotnet) into [dd-trace-dotnet](https://github.com/DataDog/dd-trace-dotnet). For this PR, I created a branch `permanent/MoveFrom_dd-shared-components-dotnet` and moved all files from the `dd-shared-components-dotnet` into there, including their complete change history. The branch will remain permanently as a history record, and we will squash-merge this PR into the `master` branch. The files are all inside the `/shared` folder, so that they can be naturally merged into the planned joint directory structure. In addition, this PR also makes the following small changes: * Make whatever tweaks necessary so that _all_ the Shared Code projects build inside `dd-trace-dotnet`. * Set up placeholder folders for the `/tracer` and the `/profiler` products to facilitate the directory refactoring in the subsequent PRs. * Update ReadMe files to describe the changes. Note that this PR keeps the directory structure within the `/shared` folder exactly as it was within the original repo. Subsequent changed will adjust this structure to joint conventions as required. This will require coordination with the profiler build configuration which makes some assumptions about the layout of the relevant folders. This change is self-contained. It will not affect any existing Tracer components, and it will allow the overall repo merge project to proceed in several steps that can be validated independently. This PR will be followed up with additional PR to address the following: * Move Tracer-specific files into the `/tracer` folder. * Ensure that changes contained to shared files that are not (yet) used by the Tracer do not require the Tracer CI validation to pass (this will be updated in the future if and when the Tracer uses more shared components) * Joint build of components used by several products. * Joint release package. * Joint conventions for folder structure. * Etc... Sending more relevant data to backend (DataDog/dd-trace-dotnet#1695) * Adding more data and filtering cookie from the no cookie header waf address Remove usage of non-builtin types in Attribute named arguments (DataDog/dd-trace-dotnet#1601) To workaround an issue discovered in Azure Functions, remove all usages of named arguments in Attributes that are not built-in types. This means removing arguments that use our own custom enums, but keeping arguments that use `string`, etc. While rare, the failing scenario occurs when the corelib attempts to create the `Attribute` object from the metadata but the assembly that gets resolved from the default load context (or one of its handlers) does not resolve to the same assembly as the one we expected, causing type equality to fail. Note: This workaround is only required for attributes that will be instantiated from metadata, like `DuckAttribute`, but it is easier to maintain a broader rule that we can always revisit later. Specific changes include: - Replacing all usages of `[Duck(Kind = DuckKind.Field)]` with a new attribute `[DuckField]` - Replacing the one usage of `[InterceptMethod(MethodReplacementAction = MethodReplacementActionType.InsertFirst)]` with `[InsertFirstInterceptMethod]` - Adding a unit test to continuously maintain this coding guideline Don't cache the process instance for runtime metrics (DataDog/dd-trace-dotnet#1698) Add tracer metrics to startup logs (DataDog/dd-trace-dotnet#1689) Use AppDomain.CurrentDomain.BaseDirectory to get current directory for configuration (DataDog/dd-trace-dotnet#1700) Environment.CurrentDirectory does not necessarily point to the directory containing the application. The most obvious example is Windows Services, where Environment.CurrentDirectory points to C:\Windows\System32 Using AppDomain.CurrentDomain.BaseDirectory seems to be a better (and safer) alternative. Tested the following scenarios, and they behave as expected * asp.net sites, (iis express + iis) * asp.net in a virtual directory ((iis express + iis) * Owin self hosted * Owin hosted in a windows service * ASP.NET Core app hosted in iis * `dotnet run` on an asp.net core app (returns the bin directory, whereas Environment.CurrentDirectory returns the app directory/content root) * `dotnet dll`/running .exe directly. Returns app directory instead of the console's current directory (e.g. if entering full path to exe) Prepare the `/shared` folder for consumption by the Profiler. (DataDog/dd-trace-dotnet#1697) This change refactors the file structure within the top-level `/shared` folder into its long-term state. Specifically: * Modify the directory names inside of the top-level shared directories according to team design discussions (lower case). * Simplify references to project-external (shared) `.cs` files. * Clean up relative directory references. * Update T4 auto-generated files. * Add ReadMe information. * Ensure that all shared projects build cleanly. This is the [key ReadMe file describing the shared sources](https://github.com/DataDog/dd-trace-dotnet/tree/macrogreg/master/shared/src#readme). It explains things in detail. It is located under `/shared/src/README.md`. The original PR has a screen shot the folder, drilled in up to the individual libraries: DataDog/dd-trace-dotnet#1697 New version of the WAF (DataDog/dd-trace-dotnet#1692) New version of the WAF The latest version of the C++ binary that analyses the incoming web requests. The configs have moved to a yaml, I have included a test rule, but there will be another later PR that will add yaml parsing and a real default rule set. Add fix for Samples.MultiDomainHost.App.NuGetHttpWithRedirects to send test HTTP traffic to a local HTTP listener instead of contoso.com (DataDog/dd-trace-dotnet#1691) Add PreserveContext attribute for async integrations (DataDog/dd-trace-dotnet#1702) Don't trigger Tracer-specific CI for changes to shared assets not used by the Tracer (DataDog/dd-trace-dotnet#1701) [Version Bump] 1.28.5-prerelease (DataDog/dd-trace-dotnet#1709) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> In the integration_tests_arm64 stage, replace COMPOSE_PROJECT_NAME with the projectName input on the DockerCompose Azure DevOps task (DataDog/dd-trace-dotnet#1715) This should correctly set the project name because the task's projectName input takes precedence over the environment variable with the -p option, and the task will always set a projectName input even if not specified. Don't fail the build if uploading core dumps fails (DataDog/dd-trace-dotnet#1722) By adding `continueOnError: true` to the log upload stage Also removed some "duplicate" uploads for consistency Create new docker-compose services only for arm64 that exposes the ports only via linked services (DataDog/dd-trace-dotnet#1719) This allows multiple stacks to be hosted on one machine, as is the case in the arm64 integration test runs Fix flakiness in Datadog.Trace.ClrProfiler.IntegrationTests.WebRequestTests.SubmitsTraces (DataDog/dd-trace-dotnet#1718) As part of the test, the trace ID and span ID of `spans.First()` is compared against the first distributed tracing headers seen by the in-memory HttpListener. In the most recent test failure, the trace ID comparison failed, and I suspect it's because the in-memory list of spans was not ordered by start time. Ordering the spans should solve this. Enable code coverage for security tests (DataDog/dd-trace-dotnet#1699) * Enable code coverage for security tests * Enable aspnetcore5 tests * Try fixing RabbitMQ coverage Add support for Aerospike (DataDog/dd-trace-dotnet#1717) Update name of NuGet package carrying automatic instrumentation assets (DataDog/dd-trace-dotnet#1720) * Rename Datadog.Instrumentation to Datadog.Monitoring.Distribution, and rename all "instrumentation" references to "distribution" * Update NuGet Deployment test to not expose the Datadog agent to the host, only to the running containers Add FmtLib debug files that were previously forgotten during import of shared files. (DataDog/dd-trace-dotnet#1725) The recent merge of the `dd-shared-components-dotnet` repo included the native [fmt library](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native). During the merge, we included the release version of that library, but we forgot the [debug files](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native/fmt_x64-windows-static/debug/lib). Now, we need to have the Profiler use the shared files from their new location in `dd-trace-dotnet`. However, that is currently not building correctly, because the fmt debug files are missing. This change adds the missing files. There is no other functional change, beyond unblocking the Profiler build. Define 'SharedSrcBaseDir' consistently across projects. (DataDog/dd-trace-dotnet#1724) Several projects in `/shared` and in `dd-continuous-profiler-dotnet` define the `SharedSrcBaseDir` MSBuild property to refer to the folder that has recently been moved to `dd-trace-dotnet\shared\src\managed-src\`. Previously, projects were using different ways to refer to that same folder, which was fragile. This change switches to a consistent reference based on the existing `EnlistmentRoot` MSBuild variable. Add a beta suffix to the Datadog.Monitoring.Distribution NuGet package (DataDog/dd-trace-dotnet#1728) Add a beta01 suffix to the Datadog.Monitoring.Distribution while we ensure it's production-ready Implement shutdown to help fix app sec test flakiness (DataDog/dd-trace-dotnet#1713) Package windows native symbols in a separate archive (DataDog/dd-trace-dotnet#1727) * Package windows native symbols in a separate archive * Only publish symbols in gitlabs Native memory improvements (DataDog/dd-trace-dotnet#1723) * changes * Skip Dynamic and Resource modules from CallTarget. Other improvements. * Fix linux build * Revert RequestRejitForNGenInliners call move. * Move string allocations to contants * Changes based in the review. * Simplify the ModuleLoadFinished callback on CallTarget * use constant * fix test project. * Changes based in the review. Add a previously forgotten comment to a shared editorconfig file. (DataDog/dd-trace-dotnet#1735) Downgrade severity of a shared editoconfig rule to match ongoing Profiler work. (DataDog/dd-trace-dotnet#1736) (The modified `.editorconfig` currently only applies to files used by the Profiler.) Disable AppSec crank till a new runner machine can be created (DataDog/dd-trace-dotnet#1739) Implement 1.0.7 of libddwaf (DataDog/dd-trace-dotnet#1732) * Implement 1.0.7 of libddwaf Our C++ WAF implementation, libddwaf - previously PowerWaf / libSqreen, has been though a big clean up of their APIs and the format of configuration file. This commit synchronizes the API signatures and naming of methods / objects. The details of the new API can be found here: https://github.com/DataDog/libddwaf/blob/master/include/ddwaf.h Disable ILogger integration by default (DataDog/dd-trace-dotnet#1738) * Disable ILogger integration by default In order to better verify the performance impact of the ILogger integration, disable the integration by default, by adding an additional feature flag. * Fix configuration tests broken by disabling ilogger by default Correct memory management for interactions with libddwaf (DataDog/dd-trace-dotnet#1742) I had assumed that libddwaf took ownership of objects assoicated with it and therefore free them when context objects where freed. However this isn't the case, all objects allocated by the C# code for the ddlibwaf need to be explicitly freed. This PR does that in a simple way. Revert "Disable ILogger integration by default (open-telemetry#1738)" (DataDog/dd-trace-dotnet#1744) This reverts commit fdbab1aef49b79cc8da9ae3a89d041f2c3ab51ca. Header keys should be lower case (DataDog/dd-trace-dotnet#1743) [Version Bump] 1.28.6 (DataDog/dd-trace-dotnet#1745) * [Version Bump] 1.28.6 * Update Changelog Co-authored-by: andrewlock <andrewlock@users.noreply.github.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Minor tweaks to shared files (DataDog/dd-trace-dotnet#1729) As we are cleaning up the Profiler build to reference `dd-trace-dotnet` instead of `dd-shared-components-dotnet`, we discover the need to minor tweaks. This PR is the collection of such tweaks which I collected into a single PR to minimize churn. The Profiler work is now done (https://github.com/DataDog/dd-continuous-profiler-dotnet/pull/135). Note that while this PR is being reviewed and merged, the Profiler builds cleanly while referencing `dd-trace-dotnet / macrogreg/SharedItemsCatchup`. Once this PR is merged, we must point the profiler to `dd-trace-dotnet / master`. Changes here: * Make packages path the same for all shared items. * Props files must use relative paths to construct shared file references. * Fix upward walking chains in `Directory.Build.props` files. Add minimal test applications that use service bus libraries (DataDog/dd-trace-dotnet#1690) Adds a test application and smoke test for the following service bus libraries. This tests that automatic instrumentation does not encounter issues with the core components of the service bus libraries. - MassTransit - NServiceBus - Rebus [DuckType] - Improve generic methods support. (DataDog/dd-trace-dotnet#1733) * Adds support for Types with generics types inside a generic parameters method. * Changes in the tests. * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Remove unused ISpan/IScope (DataDog/dd-trace-dotnet#1746) Fix Benchmarks build break from removing Datadog.Trace.Abstractions namespace (DataDog/dd-trace-dotnet#1749) Fix compilation directive for NET5.0 (DataDog/dd-trace-dotnet#1731) Exclude liblog from code coverage by filepath (DataDog/dd-trace-dotnet#1753) Adding the "[ExcludeFromCodeCoverage]" attribute apparently causes all sorts of issues Refactor ILogger integration (DataDog/dd-trace-dotnet#1740) * Move ILogger instrumentation into Logging sub-folder We are going to likely have shared components related to ILogger so makes sense to move it now * Call ToString() on SpanId and TraceId Ensures that they are formatted as strings when using custom formatters * Add benchmark for logs injection * Add a sample app demonstrating how to use the ILogger integration The ILogger integration requires auto instrumentation, so uses the Datadog.Monitoring.Distribution to set that up in an easy way * Output a better format string for ILogger Logging Scope Some loggers don't call `GetEnumerator()` on scopes, and instead just call ToString(). Previously we were outputting empty span_id and trace_id values, which looked strange, and might break log parsers etc In addition, according to [this comment](https://github.com/DataDog/dd-trace-dotnet/blob/master/samples/AutomaticTraceIdInjection/SerilogExample/Program.cs#L26), log parsing requires the properties to be a JSON map, so to improve the chance of correct usage, this changes the ToString() value to be a json map. * Update DatadogLoggingScope.ToString() to not be a JSON object Refactor folder locations [latest] (DataDog/dd-trace-dotnet#1748) This PR moves almost all of the code related to development/test/build of the .NET Tracer into a `/tracer` directory to allow for better code partitioning when we add more products into the repo. File changes include: - Directories moved from `/` to `/tracer/` - build - samples - src - test - Individual files moved from `/` to `/tracer/` - Datadog.Trace.proj - Directory.Build.props - GlobalSuppressions.cs - build.cmd - build.ps1 - build.sh - build_in_docker.sh - integrations.json - stylecop.json - Intermediate artifact folders moved from `/src/bin` to `/tracer/src/bin` - artifacts - managed-publish - windows-tracer-home Fix incomplete update to Datadog.Trace.sln for latest sample projects during tracer folder move (DataDog/dd-trace-dotnet#1759) Remove CallTarget Integrations from json file. (DataDog/dd-trace-dotnet#1693) * Initial commit * Fixes x86 PInvoke calls. * fix after merge master * Integrations definition generator. * Changes based on review. * Changes. * Adds Dispose to the NativeCallTargetDefinition instance. * Change WStr("") to EmptyWStr constant * Rollback unnecessary changes. * Rollback unnecessary changes. Fix broken paths after repository move (DataDog/dd-trace-dotnet#1762) * Fix paths in GitHub workflows They were broken during the repository re-arrange * Fix paths in honeypot yml * Remove unused (incorrect) paths from third-party-test-suites They were wrong, but don't appear to have been used, so removed them * Fix path for GitLab build Move tracer snapshots to /tracer/test/snapshots directory (DataDog/dd-trace-dotnet#1766) This should help avoid long path issues with the snapshot files that have extremely descriptive names. Synchronously wait for tasks in HttpClient sample (DataDog/dd-trace-dotnet#1703)
pjanotti
added a commit
to pjanotti/opentelemetry-dotnet-instrumentation
that referenced
this pull request
Sep 23, 2021
Fixes crank pipeline on PR merge commits. (DataDog/dd-trace-dotnet#1669) * show folder content * changes. * Copies the managed profiler to the crank runners. Removes Datadog.Trace.ClrProfiler.Managed from the throughput project. * fix pipeline. * Fix app. * list loaded assembly. * Enable debug mode. * Force loader injection in calltarget scenarios. * Fixes assembly name. Disable CallSite scenario from Throughput tests. (DataDog/dd-trace-dotnet#1674) Add instrumentation for Begin/EndGetResponse to WebRequest (DataDog/dd-trace-dotnet#1658) * Add overload to scope factory that allows providing a start time * Add HttpWebRequest integration from BeginGetRequestStream, BeginGetResponse, and EndGetResponse BeginGetRequestStream can be used in both .NET Core and .NET Framework, but BeginGetResponse and EndGetResponse are .NET Framework only, because the .NET Core implementation uses HttpClient under the covers (which we already instrument) * Update WebRequestTests to handle additional scenarios * Fix rename service tests * Fix WebRequest20Tests to reflect additional instrumentation * Add a test for BeginGetResponse when Task.Factory.FromAsync is used * Simplify Task.Factory.FromAsync implementation * Update integrations json after rebase The v0 version of App Sec (DataDog/dd-trace-dotnet#1647) This PR implements a first version of App Sec. Please forgive the message history, it will be squashed away :) The App Sec module introduces two new environment variables to activate App Sec features: `DD_APPSEC_ENABLED` – enables the App Sec feature, any HTTP requests will be examined by the WAF (Web Application Firewall). By default, this feature is disabled. `DD_APPSEC_BLOCKING_ENABLED` – this controls whether the App Sec feature will block suspicious requests. By default, this feature is disabled. The app sec uses the ‘call target’ instrumentation to gather the data about the HTTP request that will be passed to the WAF. It is has not been implemented for ‘call site’ instrumentation and currently there is not plan to support this style of instrumentation. Data is gathered in the instrumentations for ASP.NET and ASP.NET Core and then passed to the ‘instrumentation gateway’ which starts the security flow. Currently the instrumentation gateway simply creates and event with the incoming data. In future version this component will be used to route the data to different security components based on the addresses associated with the data, but this functionality is not necessary for the first version. After the instrumentation gateway the data flow is as follows: ``` InstrumentationGateway -> Security -> PowerWaf -> Native -> Sqreen.dll ``` `Security` – this class coordinates the reaction to security events `PowerWaf` – this class provides .NET idiomatic wrapper to the WAF, as the WAF is implemented in C++ with C style interface `Native` – this class is responsible for finding, loading and providing low level bindings with the WAF `[lib]Sqreen.[dll|so|dylib]` – this are the native binaries containing the WAF implementation. Security events are reported to the agent and backend by `AgentWriter` class, which implements a basic queuing system. Two new samples are provided, that form the basis for the security integration tests. These are `Samples.AspNetMvc5` and `Samples.AspNetCore5` which are used to test ASP.NET and ASP.NET Core respectively. Integration tests based on similar principals to `Datadog.Trace.ClrProfiler.IntegrationTests` have been provided in `Datadog.Trace.Security.IntegrationTests` and are run as part of the integration tests in the `consolidated-pipeline` CI. ARM based Linux systems are not supported yet, but all other environment supported by the .NET libraries are. ASP.NET Core running on .NET Framework is not supported. The instrumentation is a no-op in these scenarios. This is because it though this unlikely to work, since we rely on referencing several `Microsoft.AspNetCore.*` dlls to be able instrument the ASP.NET Core pipeline. Co-authored-by: Anna Yafi <anna.yafi@datadoghq.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Native logger error fix (DataDog/dd-trace-dotnet#1677) Adds NGEN scenario on Throughput test. (DataDog/dd-trace-dotnet#1678) Removes code for Callsite scenario from the throughput tests. (DataDog/dd-trace-dotnet#1679) Small build improvements (DataDog/dd-trace-dotnet#1646) * Allow running an arbitrary dotnet sample app from Nuke i.e. it doesn't have to be part of the solution * Automatically build the "multiPackageProject" list from PackageVersionsGeneratorDefinitions.json There's no point building these twice, so previously we had a list of sample projects to not build. This simplifies things somewhat by loading the list dynamically * Try using a unique docker network per build to avoid issues with concurrent builds on arm64 * Upload artifacts to azure (for use in AAS extension automation) The latest file list can be downloaded from: https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/index.txt and the latest commit sha of the uploads can be downloaded from https://apmdotnetci.blob.core.windows.net/apm-dotnet-ci-artifiacts-master/sha.txt The blobs are stored in a subdirectory * Only push assets to Azure from master * Pull Azure storage location from variable Update CI Visibility specification (DataDog/dd-trace-dotnet#1682) * Update CI Visibility specification * refactor and fixes. replace "minimal" solution with a solution filter (DataDog/dd-trace-dotnet#1631) Add a custom test framework to monitor execution of unit tests in ducktyping library (DataDog/dd-trace-dotnet#1680) Add tests for changes to Datadog.Trace's Public API (DataDog/dd-trace-dotnet#1681) * Add tests for public API and public browsable (intellisense) API Note that we can't use Verify in Datadog.Trace.Tests because it uses v5.0.0 of the Logging abstractions library, which is unfortunately incompatible with v2.x of the aspnetcore libraries. Resorted to a poor man's version instead. * Add public API snapshots based on v1.28.2 (pre assembly merge) * Update snapshots _after_ assembly merge. This adds a _lot_ of surface area * Update snapshots for master * Make Security class internal * Add [Browsable(false)] and [EditorBrowsable(EditorBrowsableState.Never)] to all public APIs in the ClrProfiler library * Exclude non-browsable APIs from "public API" * Try making some things internal Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> * More updates * Fix test visibility Co-authored-by: Lucas Pimentel-Ordyna <lucas.pimentel@datadoghq.com> [Version Bump] 1.28.3-prerelease (DataDog/dd-trace-dotnet#1688) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> Adds Execution time benchmarks (DataDog/dd-trace-dotnet#1687) * Adds initial execution benchmarks * fixes * Adds sleep to the end of the pipeline. Produce NuGet package to deploy automatic instrumentation (DataDog/dd-trace-dotnet#1661) Produces a NuGet package that contains all of the automatic instrumentation assets that will be deployed to the application output directory. The current name is "Datadog.Instrumentation" Details: - TargetFrameworks: net45, net461, netstandard2.0, netcoreapp3.1 (same as Datadog.Trace) - Dependencies: Datadog.Trace - This means customers who try to restore an earlier version of `Datadog.Trace` will face a build-time NU1605 error (https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu1605) - Contents: Publishes the following to application output |-- datadog | |-- linux-arm64 | | |-- Datadog.Trace.ClrProfiler.Native.so | |-- linux-musl-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- linux-x64 | | |-- Datadog.Trace.ClrProfiler.Native.so | | |-- libSqreen.so | |-- net45 | | |-- Managed assemblies | |-- net461 | | |-- Managed assemblies | |-- netcoreapp3.1 | | |-- Managed assemblies | |-- netstandard2.0 | | |-- Managed assemblies | |-- osx-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dylib | | |-- libSqreen.dylib | |-- win-x64 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- win-x86 | | |-- Datadog.Trace.ClrProfiler.Native.dll | | |-- Sqreen.dll | |-- createLogPath.sh | |-- integrations.json Additional changes: - Nuke changes: build BuildTracerHome [CreateInstrumentationHome] BuildInstrumentationNuget - CreateInstrumentationHomebuild target: Copies automatic instrumentation binaries from tracer home to the src/Datadog.Instrumentation/home directory, to be used by the Instrumentation, RunnerTool, and StandaloneTool projects - BuildInstrumentationNuget build target: Builds the Datadog.Instrumentation NuGet package with the binaries in the src/Datadog.Instrumentation/home directory - Removes src/Datadog.Trace.Tools.Runner/Datadog.Trace.Tools.Runner.proj - Add a sample application to samples/NugetDeployment directory with a docker-compose file to run the application on each platform supported by automatic instrumentation Appsec/anna/v0 integration/crankscenarios (DataDog/dd-trace-dotnet#1684) * crank scenarios for appsec * Reset sln configs * Include new yaml in sln fix json filename fix name of file for linux profile Scenarios according to acceptance criterias increase timeout limit logging rate restore ultimate pipeline increase timeout change order of tests to check increase timeout for linux Sharing profiles in another file to import Proxy calls to dl (DataDog/dd-trace-dotnet#1686) * Proxy calls to dl Some Linux distros don't have a version of libdl.so without a version prefix. Call from manage code will fail to find the library on these distros. Proxying via a C++ library should cure this, because the C libraries will handle loading in this case. * Add linker command for libdil * Rename exported fuctions to avoid strange issue * Rewrite PInvoke tables for the native methods used by App Sec too * Correct build error on windows Co-authored-by: Robert Pickering <robertfpickering@fastmail.com> Add end-to-end integration tests for logs injection (DataDog/dd-trace-dotnet#1637) New Tests - `LogsInjection.Log4Net`, `LogsInjection.NLog`, `LogsInjection.Serilog` - Behaviors - Logs injection occurs with raw logs and JSON logs - After logs injection setup, a method is invoked another AppDomain to test LogicalCallContext storage - Libraries tested - log4net - .NET Framework: 1.2 - 2.x - .NET Core: 2.0.6+ - NLog - .NET Framework: 1.x - 4.x - .NET Core: 4.5+ - Serilog - .NET Framework: 1.x - 2.x - .NET Core: 2.x - Caveats - **Note**: Windows testing currently does not test multiple package versions. That will need to be enabled at a later point in time - All log4net versions are still vulnerable to serialization issues when crossing AppDomains, so the cross-AppDomain scenario is skipped in the log4net test application Deleted Tests - `Log4Net.SerializationException` regression test (and helper library ApplicationWithLog4Net) - The regression test was added in DataDog/dd-trace-dotnet#551 but is now solved by adding new sample applications that directly cause calls to cross AppDomain boundaries - `NLog10LogsInjection.NullReferenceException` regression test - The regression test was added in DataDog/dd-trace-dotnet#614 but is now solved by the new type FallbackNLogLogProvider Fixes - Disable Serilog logs injection support when the detected Serilog version is earlier than 2.0. This is enforced by adding a new `NoOpSerilogLogProvider` log provider to perform no-ops in this scenario - Add better exception handling in `CustomLog4NetLogProvider.ExtJsonAssemblySupported()` Add Microsoft.Extensions.Logging.ILogger instrumentation (DataDog/dd-trace-dotnet#1663) * Add LogsInjectionTestBase from open-telemetry#1637 * Add ILogger integration When a logger iterates the scopes for a log message we insert ourselves as the first entry. Note that we don't _really_ add a scope (by instrumenting `BeginScope()` or similar) as that would insert our scope in the "scope stack", and make us responsible for removing it at the appropriate time. The approach here is much simpler, and works for all types of instrumentation that are using the standard ILogger infrastructure This essentially works the same way as the "activity" ILogger integration does in the runtime (https://github.com/dotnet/runtime/blob/v5.0.0/src/libraries/Microsoft.Extensions.Logging/src/LoggerFactoryScopeProvider.cs#L36-L52) * Update profiler skipped assembly prefixes We need to instrument Microsoft.Extensions.Logging.Abstractions now, so can't skip the Microsoft.Extensions prefix This adds most of them to the skip list, but there may be a better approach, e.g. an additional "don't skip" list? * Add LogsInjection aspnetcore sample * Add tests for ILogger logs injection * Update ILogger integration to inject dd_service etc even if we don't have a trace_id * Inject dd_span when a span is available Update tests to verify span_id is injected correctly * Update integrations.json after rebase * Convert DatadogLoggingScope indexer to switch expression * Add required [Browsable(false)] and [EditorBrowsable(Never)] attributes to ilogger integration Reduce snapshot file path length (DataDog/dd-trace-dotnet#1696) * Use numeric status code in snapshots + a shorter name numeric values are (mostly) shorter than the long names, and every test is called SubmitTraces, so don't need to include that in the filenames * Remove method name from aspnetcore snapshots to reduce file length * Rename snapshots Merging repos: 'dd-shared-components-dotnet' into 'dd-trace-dotnet'. (DataDog/dd-trace-dotnet#1694) This is the first of several self-contained PRs targeted at moving the Shared Code (as in _shared between the Tracer and the Profiler_) from [dd-shared-components-dotnet](https://github.com/DataDog/dd-shared-components-dotnet) into [dd-trace-dotnet](https://github.com/DataDog/dd-trace-dotnet). For this PR, I created a branch `permanent/MoveFrom_dd-shared-components-dotnet` and moved all files from the `dd-shared-components-dotnet` into there, including their complete change history. The branch will remain permanently as a history record, and we will squash-merge this PR into the `master` branch. The files are all inside the `/shared` folder, so that they can be naturally merged into the planned joint directory structure. In addition, this PR also makes the following small changes: * Make whatever tweaks necessary so that _all_ the Shared Code projects build inside `dd-trace-dotnet`. * Set up placeholder folders for the `/tracer` and the `/profiler` products to facilitate the directory refactoring in the subsequent PRs. * Update ReadMe files to describe the changes. Note that this PR keeps the directory structure within the `/shared` folder exactly as it was within the original repo. Subsequent changed will adjust this structure to joint conventions as required. This will require coordination with the profiler build configuration which makes some assumptions about the layout of the relevant folders. This change is self-contained. It will not affect any existing Tracer components, and it will allow the overall repo merge project to proceed in several steps that can be validated independently. This PR will be followed up with additional PR to address the following: * Move Tracer-specific files into the `/tracer` folder. * Ensure that changes contained to shared files that are not (yet) used by the Tracer do not require the Tracer CI validation to pass (this will be updated in the future if and when the Tracer uses more shared components) * Joint build of components used by several products. * Joint release package. * Joint conventions for folder structure. * Etc... Sending more relevant data to backend (DataDog/dd-trace-dotnet#1695) * Adding more data and filtering cookie from the no cookie header waf address Remove usage of non-builtin types in Attribute named arguments (DataDog/dd-trace-dotnet#1601) To workaround an issue discovered in Azure Functions, remove all usages of named arguments in Attributes that are not built-in types. This means removing arguments that use our own custom enums, but keeping arguments that use `string`, etc. While rare, the failing scenario occurs when the corelib attempts to create the `Attribute` object from the metadata but the assembly that gets resolved from the default load context (or one of its handlers) does not resolve to the same assembly as the one we expected, causing type equality to fail. Note: This workaround is only required for attributes that will be instantiated from metadata, like `DuckAttribute`, but it is easier to maintain a broader rule that we can always revisit later. Specific changes include: - Replacing all usages of `[Duck(Kind = DuckKind.Field)]` with a new attribute `[DuckField]` - Replacing the one usage of `[InterceptMethod(MethodReplacementAction = MethodReplacementActionType.InsertFirst)]` with `[InsertFirstInterceptMethod]` - Adding a unit test to continuously maintain this coding guideline Don't cache the process instance for runtime metrics (DataDog/dd-trace-dotnet#1698) Add tracer metrics to startup logs (DataDog/dd-trace-dotnet#1689) Use AppDomain.CurrentDomain.BaseDirectory to get current directory for configuration (DataDog/dd-trace-dotnet#1700) Environment.CurrentDirectory does not necessarily point to the directory containing the application. The most obvious example is Windows Services, where Environment.CurrentDirectory points to C:\Windows\System32 Using AppDomain.CurrentDomain.BaseDirectory seems to be a better (and safer) alternative. Tested the following scenarios, and they behave as expected * asp.net sites, (iis express + iis) * asp.net in a virtual directory ((iis express + iis) * Owin self hosted * Owin hosted in a windows service * ASP.NET Core app hosted in iis * `dotnet run` on an asp.net core app (returns the bin directory, whereas Environment.CurrentDirectory returns the app directory/content root) * `dotnet dll`/running .exe directly. Returns app directory instead of the console's current directory (e.g. if entering full path to exe) Prepare the `/shared` folder for consumption by the Profiler. (DataDog/dd-trace-dotnet#1697) This change refactors the file structure within the top-level `/shared` folder into its long-term state. Specifically: * Modify the directory names inside of the top-level shared directories according to team design discussions (lower case). * Simplify references to project-external (shared) `.cs` files. * Clean up relative directory references. * Update T4 auto-generated files. * Add ReadMe information. * Ensure that all shared projects build cleanly. This is the [key ReadMe file describing the shared sources](https://github.com/DataDog/dd-trace-dotnet/tree/macrogreg/master/shared/src#readme). It explains things in detail. It is located under `/shared/src/README.md`. The original PR has a screen shot the folder, drilled in up to the individual libraries: DataDog/dd-trace-dotnet#1697 New version of the WAF (DataDog/dd-trace-dotnet#1692) New version of the WAF The latest version of the C++ binary that analyses the incoming web requests. The configs have moved to a yaml, I have included a test rule, but there will be another later PR that will add yaml parsing and a real default rule set. Add fix for Samples.MultiDomainHost.App.NuGetHttpWithRedirects to send test HTTP traffic to a local HTTP listener instead of contoso.com (DataDog/dd-trace-dotnet#1691) Add PreserveContext attribute for async integrations (DataDog/dd-trace-dotnet#1702) Don't trigger Tracer-specific CI for changes to shared assets not used by the Tracer (DataDog/dd-trace-dotnet#1701) [Version Bump] 1.28.5-prerelease (DataDog/dd-trace-dotnet#1709) Co-authored-by: zacharycmontoya <zacharycmontoya@users.noreply.github.com> In the integration_tests_arm64 stage, replace COMPOSE_PROJECT_NAME with the projectName input on the DockerCompose Azure DevOps task (DataDog/dd-trace-dotnet#1715) This should correctly set the project name because the task's projectName input takes precedence over the environment variable with the -p option, and the task will always set a projectName input even if not specified. Don't fail the build if uploading core dumps fails (DataDog/dd-trace-dotnet#1722) By adding `continueOnError: true` to the log upload stage Also removed some "duplicate" uploads for consistency Create new docker-compose services only for arm64 that exposes the ports only via linked services (DataDog/dd-trace-dotnet#1719) This allows multiple stacks to be hosted on one machine, as is the case in the arm64 integration test runs Fix flakiness in Datadog.Trace.ClrProfiler.IntegrationTests.WebRequestTests.SubmitsTraces (DataDog/dd-trace-dotnet#1718) As part of the test, the trace ID and span ID of `spans.First()` is compared against the first distributed tracing headers seen by the in-memory HttpListener. In the most recent test failure, the trace ID comparison failed, and I suspect it's because the in-memory list of spans was not ordered by start time. Ordering the spans should solve this. Enable code coverage for security tests (DataDog/dd-trace-dotnet#1699) * Enable code coverage for security tests * Enable aspnetcore5 tests * Try fixing RabbitMQ coverage Add support for Aerospike (DataDog/dd-trace-dotnet#1717) Update name of NuGet package carrying automatic instrumentation assets (DataDog/dd-trace-dotnet#1720) * Rename Datadog.Instrumentation to Datadog.Monitoring.Distribution, and rename all "instrumentation" references to "distribution" * Update NuGet Deployment test to not expose the Datadog agent to the host, only to the running containers Add FmtLib debug files that were previously forgotten during import of shared files. (DataDog/dd-trace-dotnet#1725) The recent merge of the `dd-shared-components-dotnet` repo included the native [fmt library](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native). During the merge, we included the release version of that library, but we forgot the [debug files](https://github.com/DataDog/dd-shared-components-dotnet/tree/master/src/Shared-Libs/Native/fmt_x64-windows-static/debug/lib). Now, we need to have the Profiler use the shared files from their new location in `dd-trace-dotnet`. However, that is currently not building correctly, because the fmt debug files are missing. This change adds the missing files. There is no other functional change, beyond unblocking the Profiler build. Define 'SharedSrcBaseDir' consistently across projects. (DataDog/dd-trace-dotnet#1724) Several projects in `/shared` and in `dd-continuous-profiler-dotnet` define the `SharedSrcBaseDir` MSBuild property to refer to the folder that has recently been moved to `dd-trace-dotnet\shared\src\managed-src\`. Previously, projects were using different ways to refer to that same folder, which was fragile. This change switches to a consistent reference based on the existing `EnlistmentRoot` MSBuild variable. Add a beta suffix to the Datadog.Monitoring.Distribution NuGet package (DataDog/dd-trace-dotnet#1728) Add a beta01 suffix to the Datadog.Monitoring.Distribution while we ensure it's production-ready Implement shutdown to help fix app sec test flakiness (DataDog/dd-trace-dotnet#1713) Package windows native symbols in a separate archive (DataDog/dd-trace-dotnet#1727) * Package windows native symbols in a separate archive * Only publish symbols in gitlabs Native memory improvements (DataDog/dd-trace-dotnet#1723) * changes * Skip Dynamic and Resource modules from CallTarget. Other improvements. * Fix linux build * Revert RequestRejitForNGenInliners call move. * Move string allocations to contants * Changes based in the review. * Simplify the ModuleLoadFinished callback on CallTarget * use constant * fix test project. * Changes based in the review. Add a previously forgotten comment to a shared editorconfig file. (DataDog/dd-trace-dotnet#1735) Downgrade severity of a shared editoconfig rule to match ongoing Profiler work. (DataDog/dd-trace-dotnet#1736) (The modified `.editorconfig` currently only applies to files used by the Profiler.) Disable AppSec crank till a new runner machine can be created (DataDog/dd-trace-dotnet#1739) Implement 1.0.7 of libddwaf (DataDog/dd-trace-dotnet#1732) * Implement 1.0.7 of libddwaf Our C++ WAF implementation, libddwaf - previously PowerWaf / libSqreen, has been though a big clean up of their APIs and the format of configuration file. This commit synchronizes the API signatures and naming of methods / objects. The details of the new API can be found here: https://github.com/DataDog/libddwaf/blob/master/include/ddwaf.h Disable ILogger integration by default (DataDog/dd-trace-dotnet#1738) * Disable ILogger integration by default In order to better verify the performance impact of the ILogger integration, disable the integration by default, by adding an additional feature flag. * Fix configuration tests broken by disabling ilogger by default Correct memory management for interactions with libddwaf (DataDog/dd-trace-dotnet#1742) I had assumed that libddwaf took ownership of objects assoicated with it and therefore free them when context objects where freed. However this isn't the case, all objects allocated by the C# code for the ddlibwaf need to be explicitly freed. This PR does that in a simple way. Revert "Disable ILogger integration by default (open-telemetry#1738)" (DataDog/dd-trace-dotnet#1744) This reverts commit fdbab1aef49b79cc8da9ae3a89d041f2c3ab51ca. Header keys should be lower case (DataDog/dd-trace-dotnet#1743) [Version Bump] 1.28.6 (DataDog/dd-trace-dotnet#1745) * [Version Bump] 1.28.6 * Update Changelog Co-authored-by: andrewlock <andrewlock@users.noreply.github.com> Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> Minor tweaks to shared files (DataDog/dd-trace-dotnet#1729) As we are cleaning up the Profiler build to reference `dd-trace-dotnet` instead of `dd-shared-components-dotnet`, we discover the need to minor tweaks. This PR is the collection of such tweaks which I collected into a single PR to minimize churn. The Profiler work is now done (https://github.com/DataDog/dd-continuous-profiler-dotnet/pull/135). Note that while this PR is being reviewed and merged, the Profiler builds cleanly while referencing `dd-trace-dotnet / macrogreg/SharedItemsCatchup`. Once this PR is merged, we must point the profiler to `dd-trace-dotnet / master`. Changes here: * Make packages path the same for all shared items. * Props files must use relative paths to construct shared file references. * Fix upward walking chains in `Directory.Build.props` files. Add minimal test applications that use service bus libraries (DataDog/dd-trace-dotnet#1690) Adds a test application and smoke test for the following service bus libraries. This tests that automatic instrumentation does not encounter issues with the core components of the service bus libraries. - MassTransit - NServiceBus - Rebus [DuckType] - Improve generic methods support. (DataDog/dd-trace-dotnet#1733) * Adds support for Types with generics types inside a generic parameters method. * Changes in the tests. * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> * Update src/Datadog.Trace/DuckTyping/DuckType.Methods.cs Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com> Remove unused ISpan/IScope (DataDog/dd-trace-dotnet#1746) Fix Benchmarks build break from removing Datadog.Trace.Abstractions namespace (DataDog/dd-trace-dotnet#1749) Fix compilation directive for NET5.0 (DataDog/dd-trace-dotnet#1731) Exclude liblog from code coverage by filepath (DataDog/dd-trace-dotnet#1753) Adding the "[ExcludeFromCodeCoverage]" attribute apparently causes all sorts of issues Refactor ILogger integration (DataDog/dd-trace-dotnet#1740) * Move ILogger instrumentation into Logging sub-folder We are going to likely have shared components related to ILogger so makes sense to move it now * Call ToString() on SpanId and TraceId Ensures that they are formatted as strings when using custom formatters * Add benchmark for logs injection * Add a sample app demonstrating how to use the ILogger integration The ILogger integration requires auto instrumentation, so uses the Datadog.Monitoring.Distribution to set that up in an easy way * Output a better format string for ILogger Logging Scope Some loggers don't call `GetEnumerator()` on scopes, and instead just call ToString(). Previously we were outputting empty span_id and trace_id values, which looked strange, and might break log parsers etc In addition, according to [this comment](https://github.com/DataDog/dd-trace-dotnet/blob/master/samples/AutomaticTraceIdInjection/SerilogExample/Program.cs#L26), log parsing requires the properties to be a JSON map, so to improve the chance of correct usage, this changes the ToString() value to be a json map. * Update DatadogLoggingScope.ToString() to not be a JSON object Refactor folder locations [latest] (DataDog/dd-trace-dotnet#1748) This PR moves almost all of the code related to development/test/build of the .NET Tracer into a `/tracer` directory to allow for better code partitioning when we add more products into the repo. File changes include: - Directories moved from `/` to `/tracer/` - build - samples - src - test - Individual files moved from `/` to `/tracer/` - Datadog.Trace.proj - Directory.Build.props - GlobalSuppressions.cs - build.cmd - build.ps1 - build.sh - build_in_docker.sh - integrations.json - stylecop.json - Intermediate artifact folders moved from `/src/bin` to `/tracer/src/bin` - artifacts - managed-publish - windows-tracer-home Fix incomplete update to Datadog.Trace.sln for latest sample projects during tracer folder move (DataDog/dd-trace-dotnet#1759) Remove CallTarget Integrations from json file. (DataDog/dd-trace-dotnet#1693) * Initial commit * Fixes x86 PInvoke calls. * fix after merge master * Integrations definition generator. * Changes based on review. * Changes. * Adds Dispose to the NativeCallTargetDefinition instance. * Change WStr("") to EmptyWStr constant * Rollback unnecessary changes. * Rollback unnecessary changes. Fix broken paths after repository move (DataDog/dd-trace-dotnet#1762) * Fix paths in GitHub workflows They were broken during the repository re-arrange * Fix paths in honeypot yml * Remove unused (incorrect) paths from third-party-test-suites They were wrong, but don't appear to have been used, so removed them * Fix path for GitLab build Move tracer snapshots to /tracer/test/snapshots directory (DataDog/dd-trace-dotnet#1766) This should help avoid long path issues with the snapshot files that have extremely descriptive names. Synchronously wait for tasks in HttpClient sample (DataDog/dd-trace-dotnet#1703)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces #1530
Changes
This PR moves almost all of the code related to development/test/build of the .NET Tracer into a
/tracerdirectory to allow for better code partitioning when we add more products into the repo. File changes include:/to/tracer//to/tracer//src/binto/tracer/src/binNote: I recommend reviewing the PR by commit as it's grouped nicely per commit. Notice that in the renaming commits there are a large number of affected files but 0 lines added / removed.
@DataDog/apm-dotnet