Add support for common pool and vthread pool metrics

[pditommaso]

This PR adds a a Micrometer binder for tracking ForkJoin commons pool and virtual threads default pool metrics

[pditommaso]

I'm testing this branch in stage, I can see ForkJoin pools in prometheus metrics

» curl -s https://wave.stage-seqera.io/prometheus  | grep executor_
# HELP executor_active An estimate of the number of threads that are currently stealing or executing tasks
# TYPE executor_active gauge
executor_active{name="ForkJoin.commonPool"} 0.0
executor_active{name="ForkJoin.virtualPool"} 1.0
# HELP executor_active_threads The approximate number of threads that are actively executing tasks
# TYPE executor_active_threads gauge
executor_active_threads{name="io"} 0.0
executor_active_threads{name="scheduled"} 0.0
executor_active_threads{name="stream-executor"} 0.0
# HELP executor_completed_tasks_total The approximate total number of tasks that have completed execution
# TYPE executor_completed_tasks_total counter
executor_completed_tasks_total{name="io"} 0.0
executor_completed_tasks_total{name="scheduled"} 171.0
executor_completed_tasks_total{name="stream-executor"} 0.0
# HELP executor_idle_seconds  
# TYPE executor_idle_seconds summary
executor_idle_seconds_count{name="ForkJoin.commonPool"} 0
executor_idle_seconds_sum{name="ForkJoin.commonPool"} 0.0
executor_idle_seconds_count{name="ForkJoin.virtualPool"} 0
executor_idle_seconds_sum{name="ForkJoin.virtualPool"} 0.0
# HELP executor_idle_seconds_max  
# TYPE executor_idle_seconds_max gauge
executor_idle_seconds_max{name="ForkJoin.commonPool"} 0.0
executor_idle_seconds_max{name="ForkJoin.virtualPool"} 0.0
# HELP executor_pool_core_threads The core number of threads for the pool
# TYPE executor_pool_core_threads gauge
executor_pool_core_threads{name="io"} 0.0
executor_pool_core_threads{name="scheduled"} 16.0
executor_pool_core_threads{name="stream-executor"} 32.0
# HELP executor_pool_max_threads The maximum allowed number of threads in the pool
# TYPE executor_pool_max_threads gauge
executor_pool_max_threads{name="io"} 2.147483647E9
executor_pool_max_threads{name="scheduled"} 2.147483647E9
executor_pool_max_threads{name="stream-executor"} 32.0
# HELP executor_pool_size_threads The current number of threads in the pool
# TYPE executor_pool_size_threads gauge
executor_pool_size_threads{name="io"} 0.0
executor_pool_size_threads{name="scheduled"} 16.0
executor_pool_size_threads{name="stream-executor"} 0.0
# HELP executor_queue_remaining_tasks The number of additional elements that this queue can ideally accept without blocking
# TYPE executor_queue_remaining_tasks gauge
executor_queue_remaining_tasks{name="io"} 0.0
executor_queue_remaining_tasks{name="scheduled"} 2.147483647E9
executor_queue_remaining_tasks{name="stream-executor"} 2.147483647E9
# HELP executor_queued An estimate of the total number of tasks currently held in queues by worker threads
# TYPE executor_queued gauge
executor_queued{name="ForkJoin.commonPool"} 0.0
executor_queued{name="ForkJoin.virtualPool"} 0.0
# HELP executor_queued_tasks The approximate number of tasks that are queued for execution
# TYPE executor_queued_tasks gauge
executor_queued_tasks{name="io"} 0.0
executor_queued_tasks{name="scheduled"} 4.0
executor_queued_tasks{name="stream-executor"} 0.0
# HELP executor_running An estimate of the number of worker threads that are not blocked waiting to join tasks or for other managed synchronization threads
# TYPE executor_running gauge
executor_running{name="ForkJoin.commonPool"} 0.0
executor_running{name="ForkJoin.virtualPool"} 1.0
# HELP executor_seconds  
# TYPE executor_seconds summary
executor_seconds_count{name="ForkJoin.commonPool"} 0
executor_seconds_sum{name="ForkJoin.commonPool"} 0.0
executor_seconds_count{name="ForkJoin.virtualPool"} 0
executor_seconds_sum{name="ForkJoin.virtualPool"} 0.0
executor_seconds_count{name="io"} 0
executor_seconds_sum{name="io"} 0.0
executor_seconds_count{name="scheduled"} 171
executor_seconds_sum{name="scheduled"} 0.203916342
executor_seconds_count{name="stream-executor"} 0
executor_seconds_sum{name="stream-executor"} 0.0
# HELP executor_seconds_max  
# TYPE executor_seconds_max gauge
executor_seconds_max{name="ForkJoin.commonPool"} 0.0
executor_seconds_max{name="ForkJoin.virtualPool"} 0.0
executor_seconds_max{name="io"} 0.0
executor_seconds_max{name="scheduled"} 0.001152989
executor_seconds_max{name="stream-executor"} 0.0
# HELP executor_steals_total Estimate of the total number of tasks stolen from one thread's work queue by another. The reported value underestimates the actual total number of steals when the pool is not quiescent
# TYPE executor_steals_total counter
executor_steals_total{name="ForkJoin.commonPool"} 802.0
executor_steals_total{name="ForkJoin.virtualPool"} 1095.0

But still aren't reported in Grafana. @gavinelder @seqeralabs/admins any clue what may be wrong?

[bebosudo]

I'm seeing them in mimir, see here.

[munishchouhan]

Yes, I have tested this version in dev too
I can see two executers:

1. ForkJoin.commonPool
2. ForkJoin.virtualPool

[pditommaso]

I'd like to see this in stage with more real usage. Wave backend is already deployed.

[munishchouhan]

I'd like to see this in stage with more real usage. Wave backend is already deployed.

ok I will create a panel for it

[munishchouhan]

executor_active matrics only returns max 2 even when I did stress test on stage.
And forkjoin pool doesnot come when using executor_active_threads.
I will dig further to see how we can expose other matrics for forkjoin pool

[munishchouhan]

After doing tests in local I found that those active thread count seems correct, because prometheus does specify that executor_active returns number if active threads:

and I also printed the count of active threads while running 10 parallel transfers and builds but get this:

=== Thread Pool Details at Tue Dec 03 14:06:05 CET 2024 ===
ForkJoin Common Pool Threads: 0
Virtual Thread count: 0

=== Thread Pool Details at Tue Dec 03 14:06:35 CET 2024 ===
ForkJoin Common Pool Threads: 0
Virtual Thread count: 0

=== Thread Pool Details at Tue Dec 03 14:07:05 CET 2024 ===
ForkJoin Common Pool Threads: 0
Virtual Thread count: 0

Used this code:

@Singleton
class ThreadInspector {

    @Scheduled(fixedRate = "30s")
    void printThreadCounts() {
        println "\n=== Thread Pool Details at ${new Date()} ==="
        printForkJoinThreadCount()
        printVirtualThreadCarrierCount()
    }

    private void printForkJoinThreadCount() {
        ForkJoinPool commonPool = ForkJoinPool.commonPool()
        println "ForkJoin Common Pool Threads: ${commonPool.activeThreadCount}"
    }

    private void printVirtualThreadCarrierCount() {
        try {
            // Create a virtual thread executor
            Class<?> VirtualThread = Class.forName("java.lang.VirtualThread");

            // Use reflection to get the internal ForkJoinPool
            Field poolField = VirtualThread.getDeclaredField("DEFAULT_SCHEDULER");
            poolField.setAccessible(true);
            ForkJoinPool virtualThreadPool = (ForkJoinPool) poolField.get(null);

            println "Virtual Thread count: ${virtualThreadPool.activeThreadCount}"
        }
        catch (Exception e) {
            log.error "Unable to registry carrier threads pool metrics", e
        }
    }

}

[munishchouhan]

So I will move forward with the panel in Grafana for this