Investigate why certain runners are offline, including cases where runners are actually online and consuming jobs

[tt-rkim]

No description provided.

[tt-rkim]

An example was vm-77. Don't have the links but not a big deal.

Looks like if a GitHub runner's host is completely out of space, the runner's various internal services start to die. I noticed stack traces in the logs for the runner service. Probably not able to write service logs.

This prevents the runner from calling home to be able to listen to jobs.

Fix was to clear.

@ttmchiou @TT-billteng I put a check into the reset scripts for VMs to look for >=99% disk usage, and will do a rm -rf .cache and docker system prune and delete some random docker logs.

[TT-billteng]

An example was vm-77. Don't have the links but not a big deal.

Looks like if a GitHub runner's host is completely out of space, the runner's various internal services start to die. I noticed stack traces in the logs for the runner service. Probably not able to write service logs.

This prevents the runner from calling home to be able to listen to jobs.

Fix was to clear.

@ttmchiou @TT-billteng I put a check into the reset scripts for VMs to look for >=99% disk usage, and will do a rm -rf .cache and docker system prune and delete some random docker logs.

Should the check kick in earlier?

[tt-rkim]

How earlier?

It's right after checking if the cards are in use in reset script