Beta 18 temp space size

[loppefaaret]

Seems like we got some new temp sizes with this new beta 18.
If people could report here, that would be great.
At minimum, report your executing command, if from GUI, all the settings used.
Assuming it still peaks right around start of phase 3, depending on -2 location.
Thank you

[loppefaaret]

There might be some issue of files not being removed after processing the different tables and phases, and that is the cause of growing temp space

[dkinsk]

Hi, Im getting a
Caught plotting error: Not enough memory for sort in memory. Need to sort 1.625000Gib
Using chia beta 18 gui
Using my local ssd for the temp files and writing finals to a 6tb storage drive
It seems to be stuck on this

[loppefaaret]

that is a misconfigured plotter - you are asking too much of it, without giving it enough RAM to do the job...
are you plotting k33 by any chance ?
anyways, you need more RAM allocated for it to complete...
to use as little RAM as possible, you go with k=32, then giving it something like RAM max usage= 2048
and finally, i really recommend using Number of buckets= 128

[dkinsk]

thank you for the help,
my plotter settings are plot size k32, max ram usage 3072, threads 2, buckets auto, stripe size 65536 = basically the default settings

[loppefaaret]

then add into the Number of buckets= 128 its a known bug that should be fixed soon

[dkinsk]

ok so changed the ram usage to 2048 and buckets to 128 but same problem

Im thinking of going back to beta 17 ?

[dorkmo]

I got similar during phase 3

[dorkmo]

buckets auto selected 32

[loppefaaret]

yup, up the bucket count, unless you got a really slow seeking old mechanical HDD, always go 128 buckets
halving the bucket, doubles the RAM requirement

[loppefaaret]

after more people have gotten back with reports and their debuggin tools, we can conclude that temp space is back up to 500GB~, and the RAM requirements have grown, and processing time has increased 20-25% - developers are calling this regression unacceptable, and are going to pull it back down nearer previous levels come beta 19...

A few quotes from Keybase chat:

beta17:
First computation pass time: 1661.261 seconds. CPU (100.000%) Sat Dec 5 07:42:34 2020
Bucket 0 uniform sort. Ram: 3.777GiB, u_sort min: 3.250GiB, qs min: 1.021GiB.
Bucket 1 uniform sort. Ram: 3.777GiB, u_sort min: 3.250GiB, qs min: 1.020GiB.
Bucket 2 uniform sort. Ram: 3.777GiB, u_sort min: 3.250GiB, qs min: 1.021GiB.
Bucket 3 uniform sort. Ram: 3.777GiB, u_sort min: 3.250GiB, qs min: 1.021GiB.
beta18:
First computation pass time: 2419.457 seconds. CPU (80.930%) Sun Dec 6 01:22:32 2020
Bucket 0 QS. Ram: 2.970GiB, u_sort min: 3.250GiB, qs min: 1.021GiB. force_qs: 0
Bucket 1 QS. Ram: 2.970GiB, u_sort min: 3.250GiB, qs min: 1.021GiB. force_qs: 0
Bucket 2 QS. Ram: 2.970GiB, u_sort min: 3.250GiB, qs min: 1.021GiB. force_qs: 0
Bucket 3 QS. Ram: 2.970GiB, u_sort min: 3.250GiB, qs min: 1.021GiB. force_qs: 0

as I explained earlier I did already assign more RAM to it. With beta17 and -b 4608 on the same hardware worked fine, all used the uniform sort. I upped the RAM to -b 6144. I noticed a 5000 seconds increase with beta18. Most of the time is 'lost' in phase3. Looks like it now sorts the buckets twice (2 entries in the output) and the buckets are sorted randomly. Looks like some sort of parallelism going wrong.

b17 vs b18 with my mem + file logging, and here's the results:
-k 32 -b 5000 -u 128 -r 1 -s 65536 -t NVMe -2 NVMe -d NVMe
Plotting Time
| ver | P1 | P2 | P3 | P4 | Total |
|---|---|---|---|---|---|
| b17 | 10315 | 3661 | 6876 | 735 | 21589 |
| b18 | 10931 | 7466 | 10617 | 736 | 29752 |
Max Memory Usage
| ver | P1 | P2 | P3 | P4 |
|---|---|---|---|---|
| b17 | 5277536 | 5277536 | 5277536 | 5277536 |
| b18 | 6449024 | 6449024 | 6449024 | 6449024 |
Max Disk Space
| ver | P1 | P2 | P3 | P4 |
|---|---|---|---|---|---|
| b17 | 313427332 | 375332576 | 375332576 | 375332576 |
| b18 | 313474048 | 533873360 | 561327708 | 561327708 |
Max File Count
| ver | P1 | P2 | P3 | P4 |
|---|---|---|---|---|
| b17 | 265 | 265 | 265 | 265 |
| b18 | 265 | 777 | 802 | 802 |

So basicly, go back to beta 17 is probably the best choice until we can get our hands on beta 19:
Beta 17 release