Worker thread crashes (all API requests timeout) after disk has been full

[ErikBjare]

My system drive got full the other day, and today I noticed aw-server-rust wasn't responding to API requests (but was happily serving the web UI).

After a while I realized what may have caused it, and indeed found in the logs:

thread '<unnamed>' panicked at 'Failed to commit datastore transaction! database or disk is full', aw-datastore/src/worker.rs:182:29

So, when the disk gets full, the DB worker thread panics, and future DB requests will just fail to respond (since the worker is dead).

[johan-bjareholt]

I've seen this a few times before as well, have forgotten to document it.

What do you think the correct behaviour should be?
The options i see are:

• Shut down aw-server-rust completely so the user gets notified by aw-qt
• Drop all transactions until the disk is no longer full, but print warnings in the log.

[ErikBjare]

I think the server should respond with HTTP code 503, with a suitable error in the log. That way, watchers can queue events in memory, as appropriate.

…

On Mon, Nov 15, 2021 at 10:54 PM Johan Bjäreholt ***@***.***> wrote: I've seen this a few times before as well, have forgotten to document it. What do you think the correct behaviour should be? The options i see are: - Shut down aw-server-rust completely so the user gets notified by aw-qt - Drop all transactions until the disk is no longer full, but print warnings in the log. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKXDOTMBMVHWTQZ6E6K6XDUMF6RNANCNFSM5ICMDH7A> .