Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash tracking #5451

Merged
merged 102 commits into from
May 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
78f509d
Adding a ReportCrash API
kevingosse Mar 7, 2024
fb28703
Report crash
kevingosse Mar 8, 2024
a93ffe7
update code
kevingosse Mar 20, 2024
a1964d9
Remove header
kevingosse Mar 20, 2024
39c9416
Update libdatadog
kevingosse Mar 20, 2024
13d880d
Add a test createdump command
kevingosse Mar 20, 2024
1e37f7b
Add cout
kevingosse Mar 20, 2024
795e14e
Change path
kevingosse Mar 20, 2024
999c099
Fix endpoint
kevingosse Mar 20, 2024
4258900
Add messages
kevingosse Mar 20, 2024
9cd5b1e
ddog_crashinfo_drop
kevingosse Mar 20, 2024
d803bc9
Fix double-free
kevingosse Mar 20, 2024
49a1f57
One more message
kevingosse Mar 20, 2024
c67844a
Add another stacktrace
kevingosse Mar 20, 2024
6ae9554
Add tags and metadata
kevingosse Mar 20, 2024
1fff89a
Making progress
kevingosse Mar 25, 2024
a7375c8
Add imports
kevingosse Mar 25, 2024
1c42d9c
Fix type
kevingosse Mar 25, 2024
043aa37
Delete file
kevingosse Mar 25, 2024
72c9e0b
Missing include
kevingosse Mar 25, 2024
2bfaed7
Fix variable name
kevingosse Mar 25, 2024
11230bd
Remove using
kevingosse Mar 25, 2024
a37f627
Factory
kevingosse Mar 25, 2024
d2f0914
Remove the tags
kevingosse Mar 25, 2024
6a2d962
Add ctor/dtor
kevingosse Mar 25, 2024
3790797
Add required dependencies
kevingosse Mar 25, 2024
95de977
Update createdump command
kevingosse Mar 25, 2024
9c0b715
Fix cs
kevingosse Mar 25, 2024
3e219e2
Fix encoding
kevingosse Mar 25, 2024
3f6a932
grmbl
kevingosse Mar 25, 2024
070b527
Resolve managed
kevingosse Mar 25, 2024
05a1d8c
Let ClrMD handle ptrace
kevingosse Mar 25, 2024
02cae1d
Add logs
kevingosse Mar 27, 2024
d785b0a
logs
kevingosse Mar 27, 2024
6ded882
Fix
kevingosse Mar 27, 2024
6c21dba
Actually push the frames
kevingosse Mar 27, 2024
cc568a5
Retrieve module
kevingosse Mar 29, 2024
e898272
Implement FindModule
kevingosse Mar 29, 2024
83abcde
Fix
kevingosse Mar 29, 2024
33b9fb2
Fix type
kevingosse Mar 29, 2024
c93f2f3
Resolve info
kevingosse Mar 29, 2024
4009b22
Fixity fix
kevingosse Mar 29, 2024
69d4549
Try a full test
kevingosse Mar 29, 2024
bd0adff
Syntax
kevingosse Mar 29, 2024
7a2d7b1
try/catch
kevingosse Mar 29, 2024
64a44fb
Moar logs
kevingosse Mar 29, 2024
5c4c835
hex
kevingosse Mar 29, 2024
8d09439
Clear env
kevingosse Mar 29, 2024
7bc73d2
Remove diagnostics, add headers
kevingosse Mar 29, 2024
31225d3
Remove some debug messages
kevingosse Mar 29, 2024
64a12c1
Cleanup
kevingosse Apr 4, 2024
093fceb
Checkpoint
kevingosse Apr 5, 2024
002241f
Add demangling
kevingosse Apr 5, 2024
7b8c5af
Checkpoint
kevingosse Apr 10, 2024
bdf520f
Extract SetMetadata
kevingosse Apr 10, 2024
d6bc523
Checkpoint
kevingosse Apr 10, 2024
d67035d
CHeckpoint
kevingosse Apr 12, 2024
4e7eed2
Implement passthrough
kevingosse Apr 12, 2024
936294d
Implement implicit passthrough
kevingosse Apr 12, 2024
060154c
Handle COMPlus
kevingosse Apr 12, 2024
ce3effc
Checkpoint
kevingosse Apr 16, 2024
72778c5
Add manual parsing of the arguments
kevingosse Apr 17, 2024
8c16674
better filtering, group error messages
kevingosse Apr 18, 2024
8f854fd
Add plenty of stuff
kevingosse Apr 18, 2024
00dd3c7
Fix compilation on Windows
kevingosse Apr 18, 2024
e5f2d90
Fix compilation
kevingosse Apr 19, 2024
13a6bef
Add integration tests
kevingosse Apr 19, 2024
df27aeb
Fix tests
kevingosse Apr 19, 2024
062fb70
Update to passthrough logic
kevingosse Apr 22, 2024
6aacbb7
Start adding artifact tests
kevingosse Apr 22, 2024
ade12c9
Adding more tests
kevingosse Apr 23, 2024
adf50f6
Force minidump type
kevingosse Apr 23, 2024
ed52b2d
Update tests
kevingosse Apr 24, 2024
b27546f
Filter async continuations
kevingosse Apr 25, 2024
0129d99
Update tests
kevingosse Apr 29, 2024
75bfd13
override telemetry
kevingosse Apr 29, 2024
1f7d127
Add telemetry tests
kevingosse Apr 29, 2024
f01d1a9
Add debug info for CI
kevingosse Apr 29, 2024
5549a4c
override telemetry, take 2
kevingosse Apr 30, 2024
192d8e4
Fix tests
kevingosse Apr 30, 2024
eb8a245
Show the full exception
kevingosse Apr 30, 2024
d8e6010
Ignore tests on ARM
kevingosse Apr 30, 2024
02874f1
Test the stacktrace in the report
kevingosse Apr 30, 2024
caab6c7
Remove catchsegv
kevingosse May 2, 2024
837fea5
Implement obfuscation
kevingosse May 2, 2024
14aa6ae
Add missing headers
kevingosse May 2, 2024
099e972
Update libdatadog
kevingosse May 2, 2024
1164203
Try reenabling tests on docker
kevingosse May 2, 2024
f975f5b
Try custom version of clrmd
kevingosse May 2, 2024
d2777cf
Update ClrMD to a cleaner version
kevingosse May 3, 2024
ee073bd
Clean the mess
kevingosse May 3, 2024
184af38
Address PR feedback
kevingosse May 3, 2024
e8abd38
Fix obfuscation
kevingosse May 3, 2024
bf3852a
Add unit test for commandline
kevingosse May 3, 2024
91d5999
Add test case
kevingosse May 3, 2024
ac5ceaa
Collect stackpointer
kevingosse May 6, 2024
20626e6
Change the way managed callstacks are resolved
kevingosse May 6, 2024
6152fe0
Fix null ref
kevingosse May 6, 2024
4c07518
Address PR feedback
kevingosse May 6, 2024
5bb8348
Update libdatadog to v9 + small fixes
kevingosse May 14, 2024
53c8639
Update libdatadog in vcxproj
kevingosse May 14, 2024
3d0f86b
Update libdatadog in tests
kevingosse May 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions build/cmake/FindLibdatadog.cmake
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
include(FetchContent)

set(LIBDATADOG_VERSION "v8.0.0" CACHE STRING "libdatadog version")
set(LIBDATADOG_VERSION "v9.0.0" CACHE STRING "libdatadog version")

if (CMAKE_SYSTEM_PROCESSOR STREQUAL aarch64 OR CMAKE_SYSTEM_PROCESSOR STREQUAL arm64)
if (DEFINED ENV{IsAlpine} AND "$ENV{IsAlpine}" MATCHES "true")
set(SHA256_LIBDATADOG "09640a86f6beb5069c25ff7772f8b1a336d68f33dd94469711be9b98388fee0b" CACHE STRING "libdatadog sha256")
set(SHA256_LIBDATADOG "c2c6c32f612e8c1682131e72bd50492d809bac973f516e56e163e797435eef75" CACHE STRING "libdatadog sha256")
set(FILE_TO_DOWNLOAD libdatadog-aarch64-alpine-linux-musl.tar.gz)
else()
set(SHA256_LIBDATADOG "ff3c3f4650ac45d5700917026a84953b1d52c07652b834a12ada2f80c4cda47e" CACHE STRING "libdatadog sha256")
set(SHA256_LIBDATADOG "97c4fc46f92580b8929e8fcc3f51b47226836e29bce0b57ac8d3387a27a81ce1" CACHE STRING "libdatadog sha256")
set(FILE_TO_DOWNLOAD libdatadog-aarch64-unknown-linux-gnu.tar.gz)
endif()
else()
if (DEFINED ENV{IsAlpine} AND "$ENV{IsAlpine}" MATCHES "true")
set(SHA256_LIBDATADOG "17360251a8585450d8499ae5e323161a1486e487937b257571adc4c06a2a3688" CACHE STRING "libdatadog sha256")
set(SHA256_LIBDATADOG "68e67c5e87616f830289bc85626d2062277bef54694cc6dbb445105c66fe8885" CACHE STRING "libdatadog sha256")
set(FILE_TO_DOWNLOAD libdatadog-${CMAKE_SYSTEM_PROCESSOR}-alpine-linux-musl.tar.gz)
else()
set(SHA256_LIBDATADOG "1856e9b209f3a414536930eea1a8d4c854539ccfbf395ba1740f0d8f71bc2938" CACHE STRING "libdatadog sha256")
set(SHA256_LIBDATADOG "cd89cbe480db0b828a43afd161ddd83e57319dbe3d412fa4a2d096daae244595" CACHE STRING "libdatadog sha256")
set(FILE_TO_DOWNLOAD libdatadog-${CMAKE_SYSTEM_PROCESSOR}-unknown-linux-gnu.tar.gz)
endif()
endif()
Expand Down
6 changes: 6 additions & 0 deletions build/cmake/FindLibunwind.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ ExternalProject_Add(libunwind
BUILD_ALWAYS false
BUILD_BYPRODUCTS ${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-${CMAKE_SYSTEM_PROCESSOR}.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-ptrace.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-dwarf-common.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-dwarf-generic.a
)


Expand All @@ -26,6 +29,9 @@ target_include_directories(libunwind-lib INTERFACE
target_link_libraries(libunwind-lib INTERFACE
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-${CMAKE_SYSTEM_PROCESSOR}.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-ptrace.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-dwarf-common.a
${LIBUNWIND_BINARY_DIR}/src/.libs/libunwind-dwarf-generic.a
)

add_dependencies(libunwind-lib libunwind)
5 changes: 5 additions & 0 deletions profiler/src/Demos/Samples.ExceptionGenerator/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ public enum Scenario
GenericExceptions = 4,
TimeItExceptions = 5,
MeasureExceptions = 6,
Unhandled = 7
}

public class Program
Expand Down Expand Up @@ -90,6 +91,10 @@ public static void Main(string[] args)
Console.WriteLine(" ########### Measuring exceptions...");
break;

case Scenario.Unhandled:
Console.WriteLine(" ########### Crashing...");
throw new InvalidOperationException("Task failed successfully.");

default:
Console.WriteLine($" ########### Unknown scenario: {scenario}.");
break;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
#include <signal.h>
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

/* dl_iterate_phdr wrapper
The .NET profiler on Linux uses a classic signal-based approach to collect thread callstack.
Expand Down Expand Up @@ -57,6 +59,49 @@ unsigned long long dd_inside_wrapped_functions()
return functions_entered_counter;
}

__attribute__((constructor))
void initLibrary(void) {
// If crashtracking is enabled, check the value of DOTNET_DbgEnableMiniDump
// If set, set DD_TRACE_CRASH_HANDLER_PASSTHROUGH to indicate dd-dotnet that it should call createdump
// If not set, set it to 1 so that .NET calls createdump in case of crash
// (and we will redirect the call to dd-dotnet)
char* crashHandler = getenv("DD_TRACE_CRASH_HANDLER");

if (crashHandler != NULL && crashHandler[0] != '\0')
{
char* enableMiniDump = getenv("DOTNET_DbgEnableMiniDump");

if (enableMiniDump == NULL)
{
enableMiniDump = getenv("COMPlus_DbgEnableMiniDump");
}

if (enableMiniDump != NULL && enableMiniDump[0] == '1')
{
// If DOTNET_DbgEnableMiniDump is set, the crash handler should call createdump when done
char* passthrough = getenv("DD_TRACE_CRASH_HANDLER_PASSTHROUGH");

if (passthrough == NULL || passthrough[0] == '\0')
{
// If passthrough is already set, don't override it
// This handles the case when, for example, the user calls dotnet run
// - dotnet run sets DOTNET_DbgEnableMiniDump=1
// - dotnet then launches the target app
// - the target app thinks DOTNET_DbgEnableMiniDump has been set by the user and enables passthrough
setenv("DD_TRACE_CRASH_HANDLER_PASSTHROUGH", "1", 1);
}
}
else
{
// If DOTNET_DbgEnableMiniDump is not set, we set it so that the crash handler is called,
// but we instruct it to not call createdump afterwards
setenv("COMPlus_DbgEnableMiniDump", "1", 1);
setenv("DOTNET_DbgEnableMiniDump", "1", 1);
setenv("DD_TRACE_CRASH_HANDLER_PASSTHROUGH", "0", 1);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we look at this env var DD_TRACE_CRASH_HANDLER_PASSTHROUGH ?

Copy link
Collaborator Author

@kevingosse kevingosse May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The value is then used by dd-dotnet (in CreatedumpCommand.cs) to know whether it should call createdump or not.

}
}
}

/* Function pointers to hold the value of the glibc functions */
static int (*__real_dl_iterate_phdr)(int (*callback)(struct dl_phdr_info* info, size_t size, void* data), void* data) = NULL;

Expand Down Expand Up @@ -121,6 +166,90 @@ int dladdr(const void* addr_arg, Dl_info* info)
return result;
}

/* Function pointers to hold the value of the glibc functions */
static int (*__real_execve)(const char* pathname, char* const argv[], char* const envp[]) = NULL;

static char* ddTracePath = NULL;

int execve(const char* pathname, char* const argv[], char* const envp[])
{
if (__real_execve == NULL)
{
__real_execve = dlsym(RTLD_NEXT, "execve");

ddTracePath = getenv("DD_TRACE_CRASH_HANDLER");

if (ddTracePath != NULL && ddTracePath[0] == '\0')
{
ddTracePath = NULL;
}
}

#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wtautological-compare"
if (ddTracePath != NULL && pathname != NULL)
{
size_t length = strlen(pathname);

if (length >= 11 && strcmp(pathname + length - 11, "/createdump") == 0)
{
// Execute the alternative crash handler, and prepend "createdump" to the arguments

// Count the number of arguments (the list ends with a null pointer)
int argc = 0;
while (argv[argc++] != NULL);

// We add two arguments: the path to dd-dotnet, and "createdump"
char** newArgv = malloc((argc + 2) * sizeof(char*));

// By convention, argv[0] contains the name of the executable
// Insert createdump as the first actual argument
newArgv[0] = ddTracePath;
newArgv[1] = "createdump";

// Copy the remaining arguments
memcpy(newArgv + 2, argv, sizeof(char*) * argc);

size_t envp_count;
for (envp_count = 0; envp[envp_count]; ++envp_count);
char** new_envp = malloc((envp_count + 1) * sizeof(char*)); // +1 for NULL terminator

int index = 0;

for (size_t i = 0; i < envp_count; ++i) {
gleocadie marked this conversation as resolved.
Show resolved Hide resolved
if (strncmp(envp[i], "LD_PRELOAD=", strlen("LD_PRELOAD=")) == 0) {
continue;
}

if (strncmp(envp[i], "CORECLR_ENABLE_PROFILING=", strlen("CORECLR_ENABLE_PROFILING=")) == 0) {
continue;
}

if (strncmp(envp[i], "DOTNET_DbgEnableMiniDump=", strlen("DOTNET_DbgEnableMiniDump=")) == 0) {
continue;
}

if (strncmp(envp[i], "COMPlus_DbgEnableMiniDump=", strlen("COMPlus_DbgEnableMiniDump=")) == 0) {
continue;
}

new_envp[index++] = envp[i];
}
new_envp[index] = NULL; // NULL terminate the array

int result = __real_execve(ddTracePath, newArgv, new_envp);

free(newArgv);
free(new_envp);

return result;
}
}
#pragma clang diagnostic pop

return __real_execve(pathname, argv, envp);
}

#ifdef DD_ALPINE

/* Function pointers to hold the value of the glibc functions */
Expand Down
Loading
Loading