RUST_BACKTRACE=full can't find source files if the running application changes its working directory

[yaahc]

This version will print source lines

    panic!();
    std::env::set_current_dir(git_dir)?;

this version will hit the subsequent line in color-backtrace/src/lib.rs and skip printing all the src lines.

    std::env::set_current_dir(git_dir)?;
    panic!();

214             Err(ref e) if e.kind() == ErrorKind::NotFound => return Ok(()),

The filenames in my binary all use relative paths, I noticed that in your screenshot it seems to use absolute paths.

I imagine the fix is as simple as generating the rust binaries with absolute paths in the debug info, but I have no idea how to force it to compile that way. Either that or store the crate root somewhere in the binary and use that during filename resolution rather than expecting it to work relative to the current directory.

[athre0z]

This appears to be a platform thing. Windows and macOS use absolute pathes while Linux has relative ones. I guess we could do some simple source path discovery using std::env::current_exe as the origin, traversing up through the enclosing folders.

[yaahc]

Is current_exe the path that the binary is in? I worry that this will break for the binary after its been cargo install-ed into someplace like ~/.cargo/bin

[athre0z]

Yes, that would indeed break things.

I did a bit of debugging through backtrace-rs and it appears as if it is the libbacktrace symbolizer specifically that returns those relative pathes. Sweeping through the binary, a string with the absolute build directory appears to exist. Also, the optional (feature gated) gimli-rs pure Rust backtrace-rs symbolizer backend for DWARF parsing returns absolute pathes on Linux as well. Unfortunately, enabling it increases the amount of transitive dependencies from 15 to 99, which increases build time of a simple hello world with color_backtrace enabled from 2 to 30 seconds on my machine, so I really really don't want to make that to be the default.

[yaahc]

This might be too hacky but I was thinking you could try using

https://doc.rust-lang.org/std/macro.option_env.html

to read $PWD in at compile time and then, if you fail to look up the file normally and PWD is_some, try looking for files relative to PWD. This will at least let you avoid pulling in new dependencies.

[athre0z]

Mhh, this would require some pretty complex path discovery logic since you can cargo build in any subdirectory of a cargo project, existence of workspace Cargo.toml complicates things even further, cargo install --path would be broken completely. Essentially similar issues as with currect_exe.

I think it would be best to just fix libbacktrace. There is some code in there to create abs paths, however it is only used for symbols without line number information. Moving it out of the if & changing the callback invocations to receive the abs_filename (just like when there are no line numbers) fixes our issues. I think it's rather inconsistent to return different kind of paths depending on whether there are line number or not, but perhaps there is some logic behind this.

https://github.com/ianlancetaylor/libbacktrace/blob/master/dwarf.c#L2874

I'll create an issue in libbacktrace in order to figure this out -- if we're lucky, we can just move the abs path code out of the if.

[yaahc]

Hey, I just want to see where we are at with this issue. I see the comments in the libbacktrace issue you filed and the backtrace-rs issue you recently commented on, particularly about libbacktrace no longer being maintained.

I'd love to get this issue and #1 resolved, and I'm interested in contributing PRs to help fix both issues so I can finally start using this wonderful crate. With libbacktrace no longer maintained what do you imagine is the next step to getting the path issue resolved?

[athre0z]

Hi Jane,

thank you for your interest in helping out! Right now, we're in a worse place than before, with libbacktrace now being the default on macOS as well. On the positive side, due to either improvements in rustc or gimli, gimli now "only" takes 19s to build (vs. the previous 30s) on my 6c/12t machine. Pretty meh, but still better than essentially killing source printing of this crate on Linux in a lot of situations, I would think.

For now, with the option of just fixing libbacktrace becoming very unlikely, I propose we change backtrace-rs to use coresymbolication on macOS again and gimli on Linux. Oddly, this also builds gimli on Mac despite only adding the feature in a conditional cfg(target_os="linux") section of the Cargo.toml. Appears to be a bug / weird feature in Cargo (the amount of issues we have with such a simple lib .. we're really unlucky here, lol). Either way, I wouldn't consider this a deal-breaker either. I pushed this to a branch, please let me know what you think about this, perhaps you have an idea why it would also build gimli on platforms != Linux.

If you want to help out, it would likely be best to talk to the gimli / backtrace-rs people on how to push gimli further to allow it to be the default in backtrace-rs one day.

Regarding issue #1, I think this is mostly resolved, but let's talk about this in the other issue. I'll post there in a moment.

[yaahc]

As far as I know @alexcrichton is the definition of "gimli/backtrace-rs" people, so I'm gonna ping him from this issue so it ends up in his notification queue and when hes around and has time we can talk to him about it.

[alexcrichton]

For now, with the option of just fixing libbacktrace becoming very unlikely

I think this is reality as well unfortunately :(

I propose we change backtrace-rs to use coresymbolication on macOS again and gimli on Linux

I would think the same! I'd actually recommend using gimli on macOS as well if you can, and gimli should also work on MinGW now, entirely obsoleting libbacktrace in theory

Oddly, this also builds gimli on Mac

Yes that's a Cargo issue, I found the same thing when compiling backtrace itself :).

FWIW I'd like to work more on the compile time of the gimli feature of backtrace, it's not currently in a state where I'd be comfortable turning it on by default. Build time is indeed a big concern, but I think there's still a few features missing as well.

If you want to help out, it would likely be best to talk to the gimli / backtrace-rs people on how to push gimli further to allow it to be the default in backtrace-rs one day.

I've mostly just been recently idly working on this issue - rust-lang/backtrace-rs#189 - but if others are willing to help out that'd be great! At this point I think the major issues are:

• Reducing the build time from what it is today (or at least trying hard to do so)
• Verifying gimli works for more binaries in more situations (e.g. just using it)
• And... that may be it!

I'm curious to push this forward as well!

[yaahc]

Well, sign me up to help on rust-lang/backtrace-rs#189, I've never really spent time trying to optimize build times so that sounds like a fun project.

[athre0z]

Thanks for the response @alexcrichton! I now also switched to gimli on macOS. I had previously skimmed through the gimli issue in backtrace-rs and read something about it not working on macOS yet, so I didn't even try that before. Seems to be working fine!

[timotheecour]

ianlancetaylor/libbacktrace#72 seems relevant and has a reproducible bug

For now, with the option of just fixing libbacktrace becoming very unlikely,

why? https://github.com/ianlancetaylor/libbacktrace still seems maintained, from looking at recent commits

[athre0z]

When originally posting this, the library hadn't received any commits on 1+ year (as can be seen by the hole in the activity graph here). Looks like it being more actively maintained again now, which is good to hear!