How to compact the database? (struct.error: 'H' format requires 0 <= number <= 65535)

[jimbojw]

Could you supply simple instructions on how a user of lukechilds/electrumx can execute the compact_history command?

I've been using the lukechilds/electrumx docker image successfully for many months. However recently my ElectrumX server raises exceptions as reported in kyuupichan/electrumx#185

INFO:BlockProcessor:our height: 657,506 daemon: 658,515 UTXOs 266MB hist 386MB
INFO:BlockProcessor:processed 16 blocks size 20.12 MB in 8.2s
INFO:BlockProcessor:processed 8 blocks size 10.87 MB in 4.5s
INFO:BlockProcessor:processed 15 blocks size 18.34 MB in 7.3s
INFO:BlockProcessor:processed 17 blocks size 20.65 MB in 9.2s
INFO:BlockProcessor:our height: 657,563 daemon: 658,515 UTXOs 306MB hist 449MB
INFO:DB:flushed filesystem data in 0.03s
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:BlockProcessor:flushing to DB for a clean shutdown...
INFO:DB:flushed filesystem data in 0.00s
INFO:SessionManager:closing down server for rpc://0.0.0.0:8000
ERROR:root:task crashed: <Task finished coro=<Controller.serve() done, defined at /electrumx/electrumx/server/controller.py:81> exception=error("'H' format requires 0 <= number <= 65535")>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.3-py3.7.egg/aiorpcx/util.py", line 309, in check_task
    task.result()
  File "/electrumx/electrumx/server/controller.py", line 134, in serve
    await group.spawn(wait_for_catchup())
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.3-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.3-py3.7.egg/aiorpcx/curio.py", line 211, in join
    raise task.exception()
  File "/electrumx/electrumx/server/block_processor.py", line 669, in fetch_and_process_blocks
    await self.flush(True)
  File "/electrumx/electrumx/server/block_processor.py", line 342, in flush
    await self.run_in_thread_with_lock(flush)
  File "/electrumx/electrumx/server/block_processor.py", line 198, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/electrumx/electrumx/server/block_processor.py", line 197, in run_in_thread_locked
    return await run_in_thread(func, *args)
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.3-py3.7.egg/aiorpcx/curio.py", line 68, in run_in_thread
    return await get_event_loop().run_in_executor(None, func, *args)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/electrumx/electrumx/server/block_processor.py", line 341, in flush
    self.estimate_txs_remaining)
  File "/electrumx/electrumx/server/db.py", line 210, in flush_dbs
    self.flush_history()
  File "/electrumx/electrumx/server/db.py", line 279, in flush_history
    self.history.flush()
  File "/electrumx/electrumx/server/history.py", line 133, in flush
    flush_id = pack_be_uint16(self.flush_count)
struct.error: 'H' format requires 0 <= number <= 65535

I understand from the aforementioned bug that I'm supposed to run something like this:

$ docker-compose exec electrumx electrumx_compact_history

However, when I try to run docker-compose, it complains thus:

ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

I am a novice at docker, this is the only container that I run.

[Mave95]

You can get a shell inside the container with the following command:
docker exec -it electrumx /bin/sh
There you can run electrumx_compact_history
You might need to replace "electrumx" in the docker exec with the name of your docker container or it's id.

[jimbojw]

Thanks @Mave95, I think I'm making progress.

How can I stop the electrumx_server command without terminating the container?

Here's what I've tried so far. With the container running, I ran docker container ls which produced these results:

$ docker container ls
CONTAINER ID        IMAGE                  COMMAND             CREATED             STATUS              PORTS                                                      NAMES
fda9a23863c2        lukechilds/electrumx   "init"              3 minutes ago       Up 3 minutes        8000/tcp, 50001/tcp, 50004/tcp, 0.0.0.0:50002->50002/tcp   busy_bhabha

The CONTAINER ID works for opening up a shell.

$ docker exec -it fda9a23863c2 /bin/sh
/data # 

Once there, I can see that there's a python3 process electrumx_server running with PID 1:

/data # ps
PID   USER     TIME  COMMAND
    1 root      4:33 python3 /electrumx/electrumx_server
   53 root      0:00 /bin/sh
   58 root      0:00 ps

I tried to run the /electrumx/electrumx_compact_history command, but it failed due to DB lock:

/data # /electrumx/electrumx_compact_history
INFO:root:Starting history compaction...
INFO:electrumx.server.db.DB:switching current directory to /data
INFO:electrumx.server.db.DB:using leveldb for DB backend
Traceback (most recent call last):
  File "/electrumx/electrumx_compact_history", line 73, in main
    loop.run_until_complete(compact_history())
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/electrumx/electrumx_compact_history", line 51, in compact_history
    await db.open_for_compacting()
  File "/electrumx/electrumx/server/db.py", line 147, in open_for_compacting
    await self._open_dbs(True, True)
  File "/electrumx/electrumx/server/db.py", line 123, in _open_dbs
    self.utxo_db = self.db_class('utxo', for_sync)
  File "/electrumx/electrumx/server/storage.py", line 31, in __init__
    self.open(name, create=self.is_new)
  File "/electrumx/electrumx/server/storage.py", line 83, in open
    max_open_files=mof)
  File "plyvel/_plyvel.pyx", line 247, in plyvel._plyvel.DB.__init__
  File "plyvel/_plyvel.pyx", line 88, in plyvel._plyvel.raise_for_status
plyvel._plyvel.IOError: b'IO error: lock utxo/LOCK: Resource temporarily unavailable'
CRITICAL:root:History compaction terminated abnormally

I tried to shut down the server according to the documentation for terminating ElectrumX by using the electrumx_rpc command but that failed:

/data # electrumx_rpc stop
cannot connect - is ElectrumX catching up, not running, or is 8000 the wrong RPC port?

So instead, I tried sending the INT signal using the kill command:

/data # kill -s INT 1

This worked in shutting down the ElectrumX server, but it also shut down the container and booted me right out of the shell.

$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

I'm not sure what to do next. How can I stop the electrumx_server command without terminating the container?

[Mave95]

You have to modify the entrypoint of your container to start it without running electrumx_server. Here's a tutorial that explains how: https://thorsten-hans.com/how-to-run-commands-in-stopped-docker-containers
Note that you might have to add your container configuration (volumes, env vars, ...) to the docker run command.

[sancoder]

@jimbojw take a look here
kyuupichan/electrumx#185 (comment)

[jimbojw]

@sancoder Yes, thanks, I believe that's eventually what I did, to my recollection.

If we're gonna live with this compact_history stuff it'd be great to have such a pause for service merged into the electrumx_server script.

I 100% agree with this comment of yours. Having to muck about with containers is not a good UX. If we have to run periodic maintenance like compacting the database, it should at least be easy to perform. This doesn't have to be a manual process.

[dorveille]

Here's a one-liner I constructed to perform history compact operations. It should work on any system that has docker installed and that has r/w access to the electrum data set.

docker run --rm --env COIN=BitcoinSegwit --env DB_DIRECTORY=/data -v "${PWD}/electrumx:/data" --entrypoint=/usr/local/bin/electrumx_compact_history lukechilds/electrumx:v1.16.0

Regarding manual maintenance tasks, this specific issue (struct.error: 'H' format) occurs often enough for me, that I believe electrumx itself should gracefully detect and recover from these sort of issues.