Crashdump hanging

[rminderhoud]

Hello, I just integrated sentry-native into our CMake project and everything seems to be working well with sentry but it looks like the call to crashpad_handler.exe seems to be hanging.

OS: Windows 10 (64-bit)
Compiler: MSVC 14.35.32215
Build: sentry-native 0.6.5 (static, crashpad, winhttp)

I've confirmed that sentry_capture_event works if the program executes normally. However once I introduce a crash, e.g. throw or *((int*)0) = 0; it hands on invoking crash_handler.exe. I've configured sentry database location and can see the dumps being created. I also configured the path to the crash_handler.exe rather than relying on the default:

// In WinMain.cpp
    sentry_options_t* options = sentry_options_new();
    sentry_options_set_dsn(options, dsn);
    sentry_options_set_database_path(options, sentry_database_path.string().c_str());
    sentry_options_set_handler_path(options, handler_path.string().c_str());
    sentry_options_set_release(options, GIT_COMMIT_HASH);

#ifdef DEBUG
    sentry_options_set_debug(options, 1);
#endif

    sentry_init(options);

Here are the debug logs from my last test:

[sentry] INFO using database path "C:\Users\Ralph\AppData\Local\Rednim Games\ROSE Online\data\sentry"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG background worker thread started
[sentry] DEBUG starting crashpad backend with handler "C:\dev\rose\rose-online\bin\crashpad_handler.exe"
[sentry] DEBUG using minidump URL "https://<REMOVED>.ingest.sentry.io:443/api/<REMOVED>/minidump/?sentry_client=sentry.native/0.6.5&sentry_key=<REMOVED>"
[sentry] INFO started crashpad client handler
[sentry] DEBUG processing and pruning old runs
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG executing task on worker thread
[sentry] DEBUG sending request using winhttp to "https://<REMOVED>.ingest.sentry.io:443/api/<REMOVED>/envelope/":
x-sentry-auth:Sentry sentry_key=<REMOVED>, sentry_version=7, sentry_client=sentry.native/0.6.5
content-type:application/x-sentry-envelope
content-length:337

[sentry] DEBUG merging scope into event
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] INFO flushing session and queue before crashpad handler
[sentry] DEBUG sending envelope
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG serializing envelope into buffer
[sentry] DEBUG dumped 2 in-flight envelopes to disk
[sentry] INFO handing control over to crashpad

Any advice on how I can debug this? Watching the process manager I don't see crashpad_handler.exe called

[supervacuus]

Hi @rminderhoud! To clarify:

Do you mean your process is stuck after the last log line (handing control over...) you posted?

What do you mean when you say

Watching the process manager I don't see crashpad_handler.exe called

? Is the crashpad_handler.exe still running? Has it ever run?

Just for a little context of why I ask these questions:

• The crashpad_handler.exe should be running in parallel with your application starting with this logline: [sentry] INFO started crashpad client handler; You can verify this with the debugger by adding a breakpoint after sentry_init(). If the process isn't running, that would be very unexpected and require further investigation. The Windows event log could provide more context. If you compile the Native SDK with the Debug configuration, crashpad may provide more logging output.
• if the process is stuck after the last logline, it typically means that the crashpad-client (the library part of crashpad loaded in your process) is waiting for the crashpad_handler.exe to respond (after it is finished with the minidump) so it can shut down the application. Of course, if no crashpad_handler.exe is running, it will never receive a response.
• This wait has a timeout of 60 seconds. Can you verify that the application shuts down after ~60 seconds? You can also try to attach the VS debugger to your process while it is stuck to verify where it is stuck (please build with Debug or RelWithDebInfo in this case).

Minor warning: Please don't keep the debugger running during the crash because the debugger will receive the crash instead of the handler in the Native SDK. You can run the debugger before the crash (to verify the start of the external process) or attach it after the crash (if the process is stuck long enough).

[rminderhoud]

@supervacuus Thank you very much for the quick and thorough response. The context was very helpful. I can confirm that crashpad_handler.exe is correctly running once sentry is initiated. The reason I didn't see crashpad_handler.exe running was because I was checking after the process was handed over to the handler. It appears the crash handler is "crashing". I have everything compiled in debug but I see that the handler crashes. I am digging in to try to get more information but this is the current call stack:

 	crashpad_handler.exe!logging::LogMessage::~LogMessage() Line 376	C++
>	crashpad_handler.exe!crashpad::MinidumpXSaveAMD64CetU::InitializeFromSnapshot(const crashpad::CPUContextX86_64 * context_snapshot) Line 363	C++
 	crashpad_handler.exe!crashpad::MinidumpContextAMD64Writer::InitializeFromSnapshot(const crashpad::CPUContextX86_64 * context_snapshot) Line 268	C++
 	crashpad_handler.exe!crashpad::MinidumpContextWriter::CreateFromSnapshot(const crashpad::CPUContext * context_snapshot) Line 74	C++
 	crashpad_handler.exe!crashpad::MinidumpExceptionWriter::InitializeFromSnapshot(const crashpad::ExceptionSnapshot * exception_snapshot, const std::map<unsigned __int64,unsigned int,std::less<unsigned __int64>,std::allocator<std::pair<unsigned __int64 const ,unsigned int>>> & thread_id_map) Line 50	C++
 	crashpad_handler.exe!crashpad::MinidumpFileWriter::InitializeFromSnapshot(const crashpad::ProcessSnapshot * process_snapshot) Line 120	C++
 	crashpad_handler.exe!crashpad::CrashReportExceptionHandler::ExceptionHandlerServerException(void * process, unsigned __int64 exception_information_address, unsigned __int64 debug_critical_section_address) Line 105	C++
 	crashpad_handler.exe!crashpad::ExceptionHandlerServer::OnCrashDumpEvent(void * ctx, unsigned char __formal) Line 544	C++

Looking at the stack at that crash looks like a LOG_FATAL is triggered with the following message: [30196:24984:20230918,092700.935:FATAL minidump_context_writer.cc:362] Check failed: context_snapshot->xstate.cet_u.cetmsr == 1ull (0 vs. 1).

[rminderhoud]

Okay looks like this seems to simply be a problem with debug builds, I re-tested with release and it worked. Thanks for the help!

[supervacuus]

Thanks for the update, @rminderhoud! It makes sense that this only happens in the debug build because the crashpad CHECKs are only enabled in debug builds. I still want to keep this on my radar, so thanks for the report.