Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI Visibility] UDS and NamedPipes support #5634

Merged
merged 25 commits into from
Jun 7, 2024
Merged

[CI Visibility] UDS and NamedPipes support #5634

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c000840
[CI Visibility] UDS and NamedPipes support
tonyredondo May 31, 2024
6ff4327
Fix and add support for the EVP proxy part
tonyredondo May 31, 2024
8a129d8
Reuse AgentTransportStrategy
tonyredondo May 31, 2024
5f70ddf
reuse code and snapshots
tonyredondo May 31, 2024
ebdc23e
NamedPipes tests only on Windows.
tonyredondo Jun 3, 2024
acd2313
Ignore pipes tests on non windows
tonyredondo Jun 3, 2024
cd4eb0e
moar logs
tonyredondo Jun 3, 2024
d7af2aa
Disable parallelism
tonyredondo Jun 3, 2024
d4c1972
Add missing gac installer/uninstaller for windows tests
tonyredondo Jun 4, 2024
35478fc
Fix missing GAC bits for .NET Framework
tonyredondo Jun 5, 2024
469523d
Failure improvements???
tonyredondo Jun 5, 2024
3cb30b3
Merge branch 'master' into tony/civisibility-uds-pipes-support
tonyredondo Jun 5, 2024
8f1600c
moar debug?
tonyredondo Jun 5, 2024
cd856d5
Merge branch 'master' into tony/civisibility-uds-pipes-support
tonyredondo Jun 5, 2024
1c27808
Fix
tonyredondo Jun 5, 2024
16618cf
Fix instrumentation
tonyredondo Jun 5, 2024
34fd92c
stop polluting CI logs with the environment variables
tonyredondo Jun 5, 2024
59b1f4c
fixes
tonyredondo Jun 5, 2024
f0d2eca
Merge branch 'master' into tony/civisibility-uds-pipes-support
tonyredondo Jun 5, 2024
a0f9816
fixes
tonyredondo Jun 5, 2024
9c78586
fixes test helper for dotnet vstest in windows x86
tonyredondo Jun 6, 2024
15956bc
Merge branch 'master' into tony/civisibility-uds-pipes-support
tonyredondo Jun 6, 2024
0feca6b
Removing the DD_TRACE_DEBUG environment variable and increasing the r…
tonyredondo Jun 7, 2024
5dc9c9d
Remove Debug enabled flag
tonyredondo Jun 7, 2024
03fcca7
Merge branch 'master' into tony/civisibility-uds-pipes-support
tonyredondo Jun 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion tracer/src/Datadog.Trace/Ci/Agent/Payloads/EventPlatformPayload.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
// </copyright>

using System;
using Datadog.Trace.Agent;
using Datadog.Trace.Ci.Configuration;
using Datadog.Trace.Telemetry.Metrics;
using Datadog.Trace.Util;
Expand Down Expand Up @@ -142,7 +143,18 @@ private void EnsureUrl()
else
{
// Use Agent EVP Proxy
builder = new UriBuilder(_settings.TracerSettings.ExporterInternal.AgentUriInternal);
switch (_settings.TracerSettings.ExporterInternal.TracesTransport)
{
case TracesTransportType.WindowsNamedPipe:
case TracesTransportType.UnixDomainSocket:
builder = new UriBuilder("http://localhost");
break;
case TracesTransportType.Default:
default:
builder = new UriBuilder(_settings.TracerSettings.ExporterInternal.AgentUriInternal);
break;
}

if (CIVisibility.EventPlatformProxySupport == EventPlatformProxySupport.V4)
{
builder.Path = $"/evp_proxy/v4/{EventPlatformPath}";
Expand Down
104 changes: 71 additions & 33 deletions tracer/src/Datadog.Trace/Ci/CIVisibility.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@
using System.Threading.Tasks;
using Datadog.Trace.Agent;
using Datadog.Trace.Agent.DiscoveryService;
using Datadog.Trace.Agent.StreamFactories;
using Datadog.Trace.Agent.Transports;
using Datadog.Trace.Ci.CiEnvironment;
using Datadog.Trace.Ci.Configuration;
using Datadog.Trace.Configuration;
using Datadog.Trace.HttpOverStreams;
using Datadog.Trace.Logging;
using Datadog.Trace.Pdb;
using Datadog.Trace.PlatformHelpers;
Expand Down Expand Up @@ -425,48 +427,84 @@ internal static IApiRequestFactory GetRequestFactory(ImmutableTracerSettings set
internal static IApiRequestFactory GetRequestFactory(ImmutableTracerSettings tracerSettings, TimeSpan timeout)
{
IApiRequestFactory? factory = null;

var exporterSettings = tracerSettings.ExporterInternal;
var strategy = exporterSettings.TracesTransport;

switch (strategy)
{
case TracesTransportType.WindowsNamedPipe:
Log.Information<string?, int>("Using " + nameof(NamedPipeClientStreamFactory) + " for CI Visibility transport, with pipe name {TracesPipeName} and timeout {TracesPipeTimeoutMs}ms.", exporterSettings.TracesPipeNameInternal, exporterSettings.TracesPipeTimeoutMsInternal);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this identical to the default Agent strategy? Can't you use AgentTransportStrategy if so? What's different here? 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AutomaticDescompression part and the custom timeout I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in 8a129d8 to reuse the strategy if is not tcp

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I suggest an alternative? Just add support for the automatic decompression and custom timeout to the default strategy?

factory = new HttpStreamRequestFactory(
new NamedPipeClientStreamFactory(exporterSettings.TracesPipeNameInternal, exporterSettings.TracesPipeTimeoutMsInternal),
new DatadogHttpClient(new TraceAgentHttpHeaderHelper()),
new Uri("http://localhost"));
break;
case TracesTransportType.UnixDomainSocket:
#if NET5_0_OR_GREATER
Log.Information("Using " + nameof(SocketHandlerRequestFactory) + " for CI Visibility transport, with UDS path {Path}.", exporterSettings.TracesUnixDomainSocketPathInternal);
// use http://localhost as base endpoint
factory = new SocketHandlerRequestFactory(
new UnixDomainSocketStreamFactory(exporterSettings.TracesUnixDomainSocketPathInternal),
AgentHttpHeaderNames.DefaultHeaders,
new Uri("http://localhost"),
timeout: timeout);
#elif NETCOREAPP3_1_OR_GREATER
Log.Information<string?, int>("Using " + nameof(UnixDomainSocketStreamFactory) + " for CI Visibility transport, with Unix Domain Sockets path {TracesUnixDomainSocketPath} and timeout {TracesPipeTimeoutMs}ms.", exporterSettings.TracesUnixDomainSocketPathInternal, exporterSettings.TracesPipeTimeoutMsInternal);
factory = new HttpStreamRequestFactory(
new UnixDomainSocketStreamFactory(exporterSettings.TracesUnixDomainSocketPathInternal),
new DatadogHttpClient(new TraceAgentHttpHeaderHelper()),
new Uri("http://localhost"));
#else
Log.Error("Using Unix Domain Sockets for CI Visibility transport is only supported on .NET Core 3.1 and greater. Falling back to default transport.");
goto case TracesTransportType.Default;
#pragma warning disable CS0162 // Unreachable code detected
#endif
break;
case TracesTransportType.Default:
default:
#if NETCOREAPP
Log.Information("Using {FactoryType} for trace transport.", nameof(HttpClientRequestFactory));
factory = new HttpClientRequestFactory(
tracerSettings.ExporterInternal.AgentUriInternal,
AgentHttpHeaderNames.DefaultHeaders,
handler: new System.Net.Http.HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
},
timeout: timeout);
Log.Information("Using {FactoryType} for trace transport.", nameof(HttpClientRequestFactory));
factory = new HttpClientRequestFactory(
exporterSettings.AgentUriInternal,
AgentHttpHeaderNames.DefaultHeaders,
handler: new System.Net.Http.HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
},
timeout: timeout);
#else
Log.Information("Using {FactoryType} for trace transport.", nameof(ApiWebRequestFactory));
factory = new ApiWebRequestFactory(tracerSettings.ExporterInternal.AgentUriInternal, AgentHttpHeaderNames.DefaultHeaders, timeout: timeout);
Log.Information("Using {FactoryType} for trace transport.", nameof(ApiWebRequestFactory));
factory = new ApiWebRequestFactory(tracerSettings.ExporterInternal.AgentUriInternal, AgentHttpHeaderNames.DefaultHeaders, timeout: timeout);
#endif
var settings = Settings;
if (!string.IsNullOrWhiteSpace(settings.ProxyHttps))
{
var proxyHttpsUriBuilder = new UriBuilder(settings.ProxyHttps);

var settings = Settings;
if (!string.IsNullOrWhiteSpace(settings.ProxyHttps))
{
var proxyHttpsUriBuilder = new UriBuilder(settings.ProxyHttps);
var userName = proxyHttpsUriBuilder.UserName;
var password = proxyHttpsUriBuilder.Password;

var userName = proxyHttpsUriBuilder.UserName;
var password = proxyHttpsUriBuilder.Password;
proxyHttpsUriBuilder.UserName = string.Empty;
proxyHttpsUriBuilder.Password = string.Empty;

proxyHttpsUriBuilder.UserName = string.Empty;
proxyHttpsUriBuilder.Password = string.Empty;
if (proxyHttpsUriBuilder.Scheme == "https")
{
// HTTPS proxy is not supported by .NET BCL
Log.Error("HTTPS proxy is not supported. ({ProxyHttpsUriBuilder})", proxyHttpsUriBuilder);
return factory;
}

if (proxyHttpsUriBuilder.Scheme == "https")
{
// HTTPS proxy is not supported by .NET BCL
Log.Error("HTTPS proxy is not supported. ({ProxyHttpsUriBuilder})", proxyHttpsUriBuilder);
return factory;
}
NetworkCredential? credential = null;
if (!string.IsNullOrWhiteSpace(userName))
{
credential = new NetworkCredential(userName, password);
}

NetworkCredential? credential = null;
if (!string.IsNullOrWhiteSpace(userName))
{
credential = new NetworkCredential(userName, password);
}
Log.Information("Setting proxy to: {ProxyHttps}", proxyHttpsUriBuilder.Uri.ToString());
factory.SetProxy(new WebProxy(proxyHttpsUriBuilder.Uri, true, settings.ProxyNoProxy, credential), credential);
}

Log.Information("Setting proxy to: {ProxyHttps}", proxyHttpsUriBuilder.Uri.ToString());
factory.SetProxy(new WebProxy(proxyHttpsUriBuilder.Uri, true, settings.ProxyNoProxy, credential), credential);
break;
}

return factory;
Expand Down
21 changes: 10 additions & 11 deletions tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,23 +143,22 @@ public IntelligentTestRunnerClient(string? workingDirectory, CIVisibilitySetting
else
{
// Use Agent EVP Proxy
var agentUrl = _settings.TracerSettings.ExporterInternal.AgentUriInternal;
_eventPlatformProxySupport = CIVisibility.EventPlatformProxySupport;
switch (_eventPlatformProxySupport)
{
case EventPlatformProxySupport.V2:
_settingsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v2/{settingsUrlPath}");
_searchCommitsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v2/{searchCommitsUrlPath}");
_packFileUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v2/{packFileUrlPath}");
_skippableTestsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v2/{skippableTestsUrlPath}");
_earlyFlakeDetectionTestsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v2/{efdTestsUrlPath}");
_settingsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v2/{settingsUrlPath}");
_searchCommitsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v2/{searchCommitsUrlPath}");
_packFileUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v2/{packFileUrlPath}");
_skippableTestsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v2/{skippableTestsUrlPath}");
_earlyFlakeDetectionTestsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v2/{efdTestsUrlPath}");
break;
case EventPlatformProxySupport.V4:
_settingsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v4/{settingsUrlPath}");
_searchCommitsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v4/{searchCommitsUrlPath}");
_packFileUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v4/{packFileUrlPath}");
_skippableTestsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v4/{skippableTestsUrlPath}");
_earlyFlakeDetectionTestsUrl = UriHelpers.Combine(agentUrl, $"evp_proxy/v4/{efdTestsUrlPath}");
_settingsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v4/{settingsUrlPath}");
_searchCommitsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v4/{searchCommitsUrlPath}");
_packFileUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v4/{packFileUrlPath}");
_skippableTestsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v4/{skippableTestsUrlPath}");
_earlyFlakeDetectionTestsUrl = _apiRequestFactory.GetEndpoint($"evp_proxy/v4/{efdTestsUrlPath}");
break;
default:
throw new NotSupportedException("Event platform proxy not supported by the agent.");
Expand Down
Loading
Loading