Bear never finishes

[FredeEB]

Describe the bug
I invoked bear with bear -- make <make-options> and the process is stuck in wait on

To Reproduce
Run the above command on a very large complex codebase.

Expected behavior
Bear would finish and generate a compile_commands.json file

Environment:

• OS name: Linux
• OS version: Ubuntu 22.04
• OS architecture: x86_64
• Bear version: 3.0.18
• Bear install method: apt

Additional context
I need a bit of help debugging this. I've installed the version above and run it like so

bear -- make -j30 all

The issue is that when it completes compiling, it seems like bear is just hanging. ps -aux | grep bear | wc yields 208 lines all related to this build, however, all processes I've tried to attach to with GDB are stuck in wait on all threads, which seems off to me.

Bear has at this point managed to generate a compile_commands.events.json file that's about 1.5GB, which I also understand could take a considerable amount of time to handle, although I haven't found any threads actually working towards that.

The unfortunate part, is that this isn't a open-source project, so I can't share the codebase, however, I can try to find a codebase that shows the same behaviour.

Is this something anyone has seen before?

[rizsotto]

Hey @FredeEB , thanks for the report.

Version < 3.0.9 had similar issue. That time the compile_commands.events.json were slow to write (it was a SQLight file back then), and caused the intercept to not been able to write fast enough, and hold back the build. (Bear executes two commands one after another. First it executes intercept which executes the build and produce the event file. Then executes citnames, which reads the event file and produce the compilation database.) High parallelism (like make -j30) can be the cause if intercept can't write the file fast enough, or the gRPC server is the bottleneck. My approach was to execute the intercept command (instead of bear), and monitor the CPU usage of the process... The other problem can be the citnames which recognize the process execution events, and create the compilation database. This is a single threaded code, and reading through 1.5GB code might be time consuming, and might look like the process is hanging.

[FredeEB]

@rizsotto - thanks for getting back on this so promptly

For reference, I let this run overnight, and when I came in the next morning and ctrl-c'd the process because it wasn't done, it had been running for ~19 hours, where a build of the code base normally takes 15-20 minutes. Do you know of anyone who's tried running bear with llvm or chromium or similar large projects? It would be interesting to see if it's this specific code base or a generic bottleneck that bear reaches at some point. I'll try and run this for llvm, and see it I can make it pass. It might be easier to debug when having something to compare it to.

PS. I worked around the issue by using bear V2.4.4, however, I'd much prefer using the version available in apt, so I don't rely on a locally built version.

[rizsotto]

@FredeEB , the 19-hours-run sounds really bad. Sorry for that!

Yes, Bear is used against large projects. I did use the linux kernel as benchmark in the past.

• What is the build system that your project is using?
• Is the project doing cross compilations? (Or just a native target?)
• Is the project using any special tools in the build? (eg.: scripts to generate code)
• Are any of the tools involved is statically linked?
• Does the same thing happens if you run the build with less parallelism?
• Can you run intercept and citnames instead of bear? (To see which part is failing.)
• Can you run these commands with verbose logging? (Just to pass -vvvv before the --)

[FredeEB]

@rizsotto - Thanks for the pointers.

• The project I'm working on includes the Linux kernel, along with a lot of C++ on top, It is built using handwritten makefiles
• It's always crosscompiling using GCC to whatever platform, altough mostly the arm-linux-g++ variant.
• That's a maybe. We do use a lot of templates and macros if that counts, but scripts that generate code, would be in the style of protoc for protocol-buffers
• I haven't tried, however, if I cannot run with this level of parallism, using bear becomes close to pointless. Building the project with -j30 takes 15-20 minutes, so decreasing that would really start to hurt performance, especially if bear ends up adding a significant overhead
• Can you elaborate on how you would invoke each of those?
• I'll give that a shot tomorrow

By the way, I tried generating a CDB for llvm using bear, by having cmake generate a make based build system and run that under bear, and it worked flawlessly.

Thanks for taking time out of your calendar for this. It's greatly appreciated

[rizsotto]

Hey @FredeEB , thanks for coming back.

I am not suggesting to use your build with no parallelism as a workflow. (I understand that developer productivity is an important thing.) It would be a one-shot, to debug Bear if this might be the cause of the problem.

Run intercept instead of bear is easy. It has the same command line parameters. intercept -vvvv -- make all. This will produce the event file, that citnames can read. (Both program has --help flag to get more information about the usage, and their man page can be also helpful.)

[rizsotto]

Hey @FredeEB , I've just released a new version of Bear.

There is a change in this new release, which tries to fix hanging processes. Is there any chance that you can try it out? (Since I was not able to reproduce it.)

[FredeEB]

Thanks for the heads-up @rizsotto.

I haven't been able to debug the issue further (work got in the way), but I'll definitely give it a shot

Thanks

[FredeEB]

Hello again!
Sorry for the long turnaround time. I just built bear from the latest master, however, I still get very slow execution of the intercept stage of bear when compared to running just make.
I've not dug into the code yet, but is it possible that some commands are slow when run under intercept? The buildsystem in question uses a lot of different unix commands?
I'll probably have a look at the code to see if I can do some profiling tonight, and find out where it's spending all the time.

[rizsotto]

Hey @FredeEB , that's a good question.

There is a big chance that build runs slower with Bear. This section of the wiki explains how the tool is working. You can see that for every process executions, there is an extra process running. (Although this is a small executable, but it runs for every command your build executes.)

In the past slowness was caused by a performance bottle neck (writing down the events were too slow), but this is fixed now. I presume this is a different problem. I guess there are processes which does not work with a wrapper process around them. (Probably using some IPC, which is not mock-ed properly in the wrapper.)

[PixelyIon]

I run into a similar issue while building wine with bear -- make -j (on a 20 logical core CPU) after running autoconf/configure. After checking with pstree, it seemed like there were no compiler processes running only bear's wrapper which were seemingly stuck in gRPC calls.