Improve warning for insufficient file resources for PQ max_bytes

[donoghuc]

Release notes

[Release Notes] Improve warning message for insufficient disk space when configuring persistent queues for pipelines.

What does this PR do?

This commit refactors the PersistedQueueConfigValidator class to provide a
more detailed, accurate and actionable warning when pipeline's PQ configs are at
risk of running out of disk space. See
#14839 for design considerations. The
highlights of the changes include accurately determining the free resources on a
filesystem disk and then providing a breakdown of the usage for each of the
paths configured for a queue.

Why is it important/What is the impact to the user?

The user will now have a much more detailed and accurate warning they can meaninfully take action with. Here is an example output (as generated using test mocks in the unit tests)

Persistent queues require more disk space than is available on multiple filesystems:

Filesystem 'disk1':
- Total space required: 600gb
- Currently free space: 100gb
- Current PQ usage: 80gb
- Additional space needed: 520gb

Individual queue requirements:
  /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/studtmp-62dcd546c57212e54f73ac3c9d0ed599ce41b9e534c3eb57f22d326bb4e8/main:
    Current size: 30gb
    Maximum size: 300gb
  /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/studtmp-62dcd546c57212e54f73ac3c9d0ed599ce41b9e534c3eb57f22d326bb4e8/secondary:
    Current size: 50gb
    Maximum size: 300gb

Filesystem 'disk2':
- Total space required: 300gb
- Currently free space: 50gb
- Current PQ usage: 25gb
- Additional space needed: 275gb

Individual queue requirements:
  /var/folders/cw/q_xjr4md1wj_w_c1xwfrnxdw0000gn/T/studtmp-c8a3ca76784464d59b207b16b96b7557e6e28943c31f3bf16f6133b3a775/third:
    Current size: 25gb
    Maximum size: 300gb

Please either:
1. Free up disk space
2. Reduce queue.max_bytes in your pipeline configurations
3. Move PQ storage to a filesystem with more available space
Note: Logstash may fail to start if this is not resolved.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

The unit tests should cover the basics, though i've extensively stubbed the filesystem I will be looking at standing this up in a real build.

Related issues

• Closes Warning Message when insufficient space to hold multiple persistent queues on a file system could be clearer  #14839

Use cases

Screenshots

Logs

[donoghuc]

TODO:
- Linting (didnt want to distract from the actual changes yet)

• Figure out other process stuff... Labels, PR templates etc
• Build logstash and give it a real configuration instead of mocking everything in rspec tests

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9b408b4

History

• 💛 Build #1827 was flaky 557a159

[robbavey]

Review left in wrong PR 🤦