Can we have more regularly updated Prom metrics?

[odscjames]

(This issues looks in detail at one stage only, but I think it applies to all stages).

As a sys admin doing operations
I want to see more regularly updated metrics
so that I can monitor the system with accurancy.

I'm currently running an operation to resolerize all documents so ~9000. This operation takes hours. I want to see detailed metrics of how far through the operation is so I can monitor performance, and see the effects of any changes I make (such as giving the system more resources half way through).

However, the current stats this software gives to Solr are only updated at the start of the main loop for solrize, not during. https://github.com/IATI/refresher/blob/develop/src/library/solrize.py#L432

This means that what Promethous sees is:

I'd like the stats to be updated during the loop, so I see the graph running down over time.

(Ideally I'd like the stats to be updated totally independently of the loop, so that if there is a problem with the main loop the stats still continue to be updated - tho in this case that doesn't matter, as if the loops break "documents to process" won't go down! But maybe it would in the case of other stats)

Acceptance criteria

1. Functional Criteria (Product features or functions that developers must implement to enable users to accomplish their tasks)
   1.1. Stats should update during operations

2. Non-functional Criteria (Include availability, maintainability, performance, reliability, scalability, security, and usability criteria)
   2.1. The updating of stats shouldn't break normal work loop
   2.2. Ideally a broken work loop wouldn't stop stats updating (tho ideally a broken work loop would raise an alarm somehow somewhere too!)

[odscjames]

Thought later on: It would be good to get the metrics from the database directly, as then we get a record as new tasks come in as well as tasks are processed.