Can't see mesmer metrics in Prometheus

[vasigorc]

I am trying to get some actor related metrics (Actor Core) while running our application but I am failing to do so and was wondering if the community could help me understand what am I doing wrong.

Here are a few bits of information that could be important:

• mesmer dependency version: 0.8.0.RC1

    "io.scalac" %% "mesmer-akka-extension" % "0.8.0.RC1"
      exclude (org = "com.fasterxml.jackson.module", name = "jackson-module-scala_2.13"),

• mesmer extension jar version: 0.8.0.RC1
• opentelemetry api version: 1.14.0
• opentelemetry javaagent version: 1.13.0
• opentelemetry protocol: http

otel.exporter.otlp.protocol=http/protobuf
otel.javaagent.extensions=src/pack/lib/opentelemetry/mesmer-otel-extension.jar

• this block in my application.conf:

  io.scalac.mesmer.actor.rules {
    "/user" = group
    "/user/*" = instance
    "/system" = group
  }

• and also this configuration in the same document

akka.actor.typed.extensions=["io.scalac.mesmer.extension.AkkaMonitoring"]

• opentelemetry collector exporter: prometheus

I don't know how important this is but I get these error messages from mesmer which I ignored because we don't use Akka Clustering:

[main] [] [] [] [] ERROR i.s.mesmer.extension.AkkaMonitoring - Ref akka.actor.LocalActorRefProvider@2474df51 is not instance of akka.cluster.ClusterActorRefProvider|
[info] 2022-08-24T19:07:28.756+0000 [][main] [] [] [] [] DEBUG i.s.mesmer.extension.AkkaMonitoring - Start akka stream service|
[info] 2022-08-24T19:07:28.770+0000 [][main] [] [] [] [] DEBUG i.s.mesmer.extension.AkkaMonitoring - Starting up module akkastream|
[info] 2022-08-24T19:07:28.770+0000 [][main] [] [] [] [] DEBUG i.s.mesmer.extension.AkkaMonitoring - Start stream monitor|
[info] 2022-08-24T19:07:28.792+0000 [][main] [] [] [] [] DEBUG i.s.mesmer.extension.AkkaMonitoring - Start akka cluster service|
[info] 2022-08-24T19:07:28.797+0000 [][main] [] [] [] [] ERROR i.s.mesmer.extension.AkkaMonitoring - ActorSystem is not properly configured to start cluster monitor of type ClusterEventsMonitor$|
[info] 2022-08-24T19:07:28.798+0000 [][main] [] [] [] [] ERROR i.s.mesmer.extension.AkkaMonitoring - ActorSystem is not properly configured to start cluster monitor of type ClusterRegionsMonitorActor$|
[info] 2022-08-24T19:07:28.802+0000 [][main] [] [] [] [] ERROR i.s.mesmer.extension.AkkaMonitoring - ActorSystem is not properly configured to start cluster monitor of type ClusterSelfNodeEventsActor$|

After I run a few of our automation tests against our Akka instrumented app I can see normal metrics collected and exported via Opentelemetry (the view below is from Prometheus):

but I can't find any of the Akka Core metrics that I hoped Mesmer would collect.

I am not sure what am I doing wrong 🤷‍♂️

[vasigorc]

CC @lgajowy

[lgajowy]

I will try to look into this. Thanks for reporting.

[vasigorc]

Thanks!
Small update: I've changed the actor.rules to the below:

  io.scalac.mesmer.actor.rules {
    "/my-app/user" = group
    "/my-app/user/*" = instance
    "/my-app/system" = group
  }

because when we instantiate ActorSystem we do it like this ActorSystem(my-app) but this doesn't change anything. By the way I think that it would be helpful to include these rules in the documentation if they are indispensable for metrics reporting.

Not sure how helpful it would be, but I've also added a breakpoint to the matches methods of classes implementing PathMatcher but it never breaks there.