Explanation on supported metrics

[vasigorc]

Hello team 👋

I've a few questions about the supported metrics page. You mention that there are three supported measurement types: gauge, counter and histogram. This aligns well with the interfaces from opentelemetry-api's io.opentelemetry.api.metrics package. However in the subsequent section you mention a recorder metric, is this a typo? Did you mean histogram instead?

My second question is around metrics' attributes. Typically Opentelemetry measurements come in two flavours with and without attributes (example). Do you use attributes? If so what are the attributes' keys, e.g. actorName or dispatcherName etc. Being able to view these values, filter and group by them is essential for us and probably not only for us.

[lgajowy]

However in the subsequent section you mention a recorder metric, is this a typo? Did you mean histogram instead?

There are errors - thanks for catching this. Recorder is a type of metric that was there in OpenTelemetry but was removed (in 1.7.0 I think). For Akka Persistence it is indeed a histogram. The section should look like this:

## Akka Persistence

- Persisted events - histogram
- Event persistence time - histogram
- Recovery total - counter
- Recovery time - histogram
- Snapshots - counter

When it comes to akka metrics:

## Akka core

- Running actors - gauge
- Mailbox size - gauge
- Stashed messaged - counter
- Mailbox time - gauge # not recorder
- Processed messages - counter
- Processing time - gauge # not recorder 
- Sent messages - counter

Please note however, not sure if it's correct for akka actor - we currently have gauges for mailbox time sum/min/max/count. At least for sum and count it would be enough to have a histogram. @vasigorc do you see it the same way from the user perspective?

My second question is around metrics' attributes. Typically Opentelemetry measurements come in two flavours with and without attributes (example). Do you use attributes? If so what are the attributes' keys, e.g. actorName or dispatcherName etc. Being able to view these values, filter and group by them is essential for us and probably not only for us.

Yes, we use attributes but I must admit it could be more extensive. For akka Actor we currently use the following:

actor_path="/system", node="akka://accounts@127.0.0.1:2551"}

However, when it comes to path, it's very limited in it's current state. It's either "/system" or "/user". We discussed it internally and just created an issue to push this further: #383. Of course, your input is more than welcome.

What other attributes besides actorName and dispatcherName would be useful?

[vasigorc]

Thank you for a quick follow-up @lgajowy !

Please note however, not sure if it's correct for akka actor - we currently have gauges for mailbox time sum/min/max/count.

Do gauges allow for averages as well? Histograms allow aggregation over time periods which could be useful.

However, when it comes to path, it's very limited in it's current state. It's either "/system" or "/user".

I haven't checked the implementation. With ActorStack's wrappedReceive you can get access to the full path of the receiver and the sender. Similar to this example (they didnd't use ActorStack but it is the same principle). Of course this approach requires doing more work on the application side 🤷 ...overriding the default mailbox could work, but is a bit invasive, e.g.:

akka {
  actor {
    default-mailbox {
      mailbox-type = "path...MesmerMonitorMailboxType"
    }
  }
}

What other attributes besides actorName and dispatcherName would be useful?

Below is a list of what we could think of (we could add more later on):

• receiving_actor_name
• sending_actor_name
• receiving_actor_path
• sending_actor_path
• dispatcher_name
• actor_system
• message_name (could be a simple _.toString and is probably only of interest for Processing time)

[worekleszczy]

Hi @vasigorc 👋

Do gauges allow for averages as well? Histograms allow aggregation over time periods which could be useful.

Not directly - for this we use a pattern similar to prometheus summary - you can compute the average using sum and count metrics.

As for tracking other actors - we have a support for a configuration option where you can define what actor you want to tract. For now there are 3 different types supported: group, instance and disabled. With group you will combine all of the metrics of this actor and its children and grandchildren etc. With instance mesmer will only report metrics for this actor not taking into account its offspring. disabled remove particular actor from being reported.

As actor systems might grow indefinitely we want users to define in which actors they are interested in - this is a safeguard not to flood monitoring backend with extraneous information too granular to be useful.

Here is an example of the config:

io.scalac.mesmer.actor.rules {
  "/user" = group
  "/user/*" = instance
  "/system" = group
  "/system/sharding/accounts" = group
  "/system/sharding/accounts/*" = instance
}

This will:

• combine all of the metrics for /user and /system subtree
• report by instance all children of /user actor
• combine all of the metrics for /system/sharding/accounts
• report by instance all children of /system/sharding/accounts/ actor

Currently we support only one kind of wildcard which is an * at the end of the actorPath - asterisk anywhere else will be ignored. However it's pretty trivial to add a new one - check out PathMatcher

[vasigorc]

Hey @worekleszczy, thanks for jumping in!

Here is an example of the config:

This looks like HOCON syntax, does this go into a .conf file? I didn't find it in the current documentation.

However it's pretty trivial to add a new one - check out PathMatcher

It is but only from your end (and wisely so) 😄 since the trait is sealed and case classes are final (so no sub-classing). Anyway these cases suffice for us.

What do you think about adding some of the attributes that we mentioned? If these sound feasible/appropriate could you roughly predict when might this become available?

[worekleszczy]

This looks like HOCON syntax, does this go into a .conf file? I didn't find it in the current documentation.

This indeed it's HOCON - if you paste that part into your application.conf it will work automatically.

It is but only from your end (and wisely so) 😄 since the trait is sealed and case classes are final (so no sub-classing).

You're right, but we're open for pull requests 😄

What do you think about adding some of the attributes that we mentioned? If these sound feasible/appropriate could you roughly predict when might this become available?

We've been thinking about it for some time and is here is the summary of it:

• receiving_actor_name - it's already there as a last segment of actorPath - do you a concrete use case for this to have it duplicated in a separate attribute?
• sending_actor_path - it's already available as actorPath attribute for sent_message metric, but you should understand it's limitations, as it's not supported in akka typed (@lgajowy if this is not in the docs, we should really put it there 🙃 ). All messages in akka are wrapped in final case class Envelope private (message: Any, sender: ActorRef) and this is maintained in typed one, but sender is always set to null. This is due to the design change, that if you want to be able to reply to an actor, you should pass the ref manually with the sent message.
• receiving_actor_path - this is already present as an actorPath
• sending_actor_name - similar to receiving_actor_name
• dispatcher_name - this one seems a good candidate for an additional attributes 👍
• actor_system - same as dispatcher_name 👍
• message_name - hmm, this is a tricky one - I don't think this would be reasonable to add this - this would have too high cardinality and from my experience this would not be really useful but would add additional stress on the metrics backend. Do you maybe have a concrete use case in mind for this? We were thinking instead if something like message_type as actors typically have a protocol defined, so cardinality for this would be pretty manageable.

[vasigorc]

Thank you for the follow-up

it's already there as a last segment of actorPath

We could play with this to address our needs for a few of the suggested attributes.

For message_name a use case is when a few actors share the same blocking dispatcher but one of them for some message may take a super long processing time - seeing the average processing time per message could help quickly identify such bottleneck.

[vasigorc]

@worekleszczy I apologize for the long pause. Is it possible to document the configuration from your previous answer as well as the supported attributes in your guide?

[lgajowy]

Hi. We're now working on filling the mesmer.io page with the proper documentation. I will create an issue to document the config as well. Thanks for the suggestion :)

[lgajowy]

#487