QixPerformance-Log

[chevcheli0s]

Is there a way to use QixPerformance-Log?
It will be useful to see a table in grafana with high resource consumption by users/applications.

[mountaindude]

Yes in principle it's very doable.
The latest versions of Butler SOS already has a mechanism for getting arbitrary log data from QSEoW to Butler SOS, so we could just add the QIX logs to that.

In fact - Butler SOS can these days be made to gather data on ANY log events generated by QSEoW.

Data granularity could be a challenge though.
This data would be most useful if we get it frequently, let's say every 20-30 sec. But that would (probably) generate lots of log files in Sense.
On the other hand, if a sampling interval of for example 5 minutes is used, that would not guarantee to capture an event causing the engine to lock up for 4 minutes and 30 sec.

Once the data is in InfluxDB you can use a retention policy to only keep the last days worth of data, so no problems with data volumes there.
Then again... I guess you could create (and schedule a daily run of) a Powershell script that prunes the Sense logs older than x days... effectively achieving a log retention policy in Sense too.

[mountaindude]

Forgot to mention in previous comment.

There will only be data in the QixPerformance_Engine los if telemetry is enabled.
And enabling telemetry comes with a general performance penalty (hard to say exactly how big it is, a few percent has been mentioned).
I think... at least I am not getting anything in that file, and I have telemetry switched off.

So: Bringing this data into Butler SOS comes with a performance penalty on the Sense server itself. May or may not be an issue.

Not sure what happened earlier, but after a restart of the engine service, with "QIX performance log level" set to Info in the QMC there are now log rows in that log file.
Lot's of rows in fact - in a large Sense cluster there will be lots of data to deal with.

No need to turn on telemetry after all.
Unless those extra metrics are also needed.

[mountaindude]

Time to revisit this topic.

I have done some preliminary work around this and decided to at least explore the idea a bit more.
No guarantees at this point that it will end up in the main Butler SOS tool.

Concept

• A new log appender is added in QS, capturing INFO level messages from QixPerformance.Engine.Engine. The messages are sent using UDP to Butler SOS.
• There will be lots of messages sent. Opening an app can result in 25 messages, same when interacting with an object in an app. A large/busy QS environment could easily cause thousands of messages per minute.

Configuration

Due to data volumes involved it is not realistic to store all info about all apps.
It must be possible to limit which apps, and possibly also which app objects should be monitored.

Possible solution:

• Specify in YAML config file which apps/app objects/message types should be monitored. Zero or more items allowed in the config file.
• Add custom property (on apps) that enable/disable monitoring for that app. This would make it easy for an admin to enable monitoring of an app that is misbehaving.
• By adding filters in the appender file on the QS server(s) it may be possible to limit the number of UDP messages sent to Butler SOS. At this point unclear if QS supports this though, the documentation QS Log4Net implementation is rudimentary, to say the least.

Consequences

• Lots of UDP messages could mean the Butler SOS server must be scaled up. More disk/cpu needed for InfluxDB.
• Butler SOS has been stress tested in the past, but not really on this level. There may be bugs that surface when the load on Butler SOS increase by a lot. Good/scalable design and thorough testing needed.

Config file changes

The config file could get new entries like these:

Butler-SOS:
  logEvents:
    ...
    appPerformanceMonitor:
      enable: true                     # Should app performance data be extracted from log events?
      customProperty:                  # App-specific custom property control of what performance data to extract from log events 
        customPropertyName: PerformanceMonitor
        valueAllObjects: AllObjects  # Should all objects in the app be monitored?
        valueSomeObjects: SomeObjects # Should only some objects in the app be monitored?
        customPropertyAppOjects: PerformanceMonitor_AppObjects # Custom property specifying what objects should be monitored in the app. Used if customPropertyName is set to valueSomeObjects.
        valueAllTypes: AllTypes      # Should all object types be monitored?
        valueSomeTypes: SomeTypes    # Should only some object types be monitored?
        customPropertyObjectTypes: PerformanceMonitor_ObjectTypes # Custom property specifying what object types should be monitored. Used if customPropertyName is set to valueSomeTypes.
      monitorObject:                   # What objects should be monitored? Entire apps or just specific objects within some specific apps?
        allApps: false                 # Should all apps be monitored?
        app:                           # App specific monitoring.
          - appId: 2c5e4b7e-4b7b-4b7b-4b7b-4b7b4b7b4b7b
            allObjects: false          # Should all objects in the app be monitored? Leave appObjects array empty if this is true.
            appObject:                 # What app objects (charts, tables, ...) should be monitored for this app?
              - objectId: YJEpPT       # Object ID of the object to monitor
                method:                # What methods should be monitored for this object?
                  - GenericObject::GetLayout
                  - Doc::GetScriptEx
            allTypes: false            # Should all object types be monitored?
            objectType:                # What object types should be monitored? 
              - sheet
              - barchart
              - linechart
              - map
            allMethods: false               # Should all methods be monitored?
            method:                    # What methods should be monitored? Additive to the app objects above.
              - GenericObject::GetLayout
              - GenericObject::GetHyperCubeContinuousData
        allTypes: false                # Should all object types be monitored?
        objectType:                    # What object types should be monitored for all apps?
          - sheet
          - barchart
          - linechart
          - map
        allMethods: false              # Should all methods be monitored?
        method:                        # What methods should be monitored for all apps?
          - Global::OpenApp
          - Doc::GetAppLayout
          - Doc::CreateSessionObject

[mountaindude]

Note: Final config file syntax WILL differ from above.
https://butler-sos.ptarmiganlabs.com will as always contain the final, authoritative documentation.