Database slowdown over time

[gh0st42]

We used serval to distribute sensor data, taking snapshots every 1-5 minutes. This included about 5 journals (temperature, pressure, etc) per sensor node (2 in total) and a 8MP image taken every now and then. After a few days serval became almost unusable.
Listing all rhizome entries takes quite long (rest and cmdline) and inserting new content is almost impossible (all done via rest but one insert takes over a minute).
One node had around 5000 entries and the other around 2800. Restarting serval-dna or calling rhizome clean does not really make it better. We still had plenty diskspace and inodes left on the devices...

I'm not sure whether the problem is the growing journals or the large number of images. Since you guys have more long term experience and heavily use meshms (which are just fancy journals) the bug is probably more related to the total number of files.

[lakeman]

Turning on debug.profiling and repeating each of the restful requests might help to pin down where the time is going.

[lakeman]

We should probably create subfolders in the rhizome blob directory to limit the size of any single folder.