Speeding up runtime

[walkrr]

I was worried my computer wasn't going to finish fetching all the data in time, so I wrote a bash script to run the program simultaneously on each directory using threading. Needs a compiled release version and ctr-common-1.pem in the directory you run it. Hope someone finds this helpful.

#!/bin/bash
commands=(
    "./saveShop --directory 1095226 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095227 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095228 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095229 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095230 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095231 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095232 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095233 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095234 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095235 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095236 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095237 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095238 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095239 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095240 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095241 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095242 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095243 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095244 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095245 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095246 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095247 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095248 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095249 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095250 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095251 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095252 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095253 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095254 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095255 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095256 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095257 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095258 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095259 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095260 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095261 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095262 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095263 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095264 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095265 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095266 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095267 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095268 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095269 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095270 --regions US fetch-all --cert ctr-common-1.pem "
    "./saveShop --directory 1095271 --regions US fetch-all --cert ctr-common-1.pem "
)

# Define a function to run each command in a separate thread
run_command() {
    cmd=$1
    $cmd &
}

# Loop through the commands and run each in a separate thread
for cmd in "${commands[@]}"
do
    run_command "$cmd"
done

# Wait for all threads to finish
wait

[neobrain]

Good idea, but this is dangerous. You'll miss titles that aren't part of any directories, and media caching won't work reliably (you might actually corrupt it by simultaneously writing to http_log from multiple processes).

If you end up using this script, I'd advise starting a normal saveShop invocation in a different directory separately. If that runs through fine before the shutdown, it'll be the more reliable archive.

Multi-threading saveShop itself would've been cool, but there's only so much I can do :)

[walkrr]

Okay that makes sense thank you for the heads up! I will definitely try that. Thanks for all your work on this