Container keeps crashing with error: "struct.error: 'H' format requires 0 <= number <= 65535"

[maltokyo]

I cant work out what to do here, I keep getting this, every time the server starts..
Please advise if you have encountered this.

root@ssd:~# docker logs -f generated_electrumx_1
INFO:electrumx:ElectrumX server starting
INFO:electrumx:logging level: DEBUG
INFO:Controller:Python version: 3.7.9 (default, Dec 18 2020, 05:56:48)  [GCC 9.3.0]
INFO:Controller:software version: ElectrumX 1.16.0
INFO:Controller:aiorpcX version: 0.18.5
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 200 blocks
INFO:Daemon:daemon #1 at bitcoind:43782/ (current)
INFO:DB:switching current directory to /data
INFO:DB:using leveldb for DB backend
ERROR:Daemon:connection problem - check your daemon is running.  Retrying occasionally...
INFO:Daemon:running normally
INFO:DB:opened UTXO DB (for sync: True)
INFO:DB:UTXO DB version: 8
INFO:DB:coin: BitcoinSegwit
INFO:DB:network: mainnet
INFO:DB:height: 696,707
INFO:DB:tip: 0000000000000000000042feb97d359861fda2bcb306e3d17220d8b2425ae44d
INFO:DB:tx count: 664,111,409
INFO:DB:flushing DB cache at 2,000 MB
INFO:History:history DB version: 1
INFO:History:flush count: 65,535
INFO:SessionManager:RPC server listening on 0.0.0.0:8000
INFO:Prefetcher:catching up to daemon height 697,011 (304 blocks behind)
INFO:BlockProcessor:our height: 696,717 daemon: 697,012 UTXOs 16MB hist 18MB
INFO:BlockProcessor:processed 10 blocks size 14.20 MB in 149.3s
INFO:BlockProcessor:our height: 696,724 daemon: 697,014 UTXOs 28MB hist 31MB
INFO:BlockProcessor:processed 7 blocks size 10.50 MB in 31.7s
INFO:BlockProcessor:our height: 696,738 daemon: 697,014 UTXOs 47MB hist 56MB
INFO:BlockProcessor:processed 14 blocks size 21.33 MB in 58.3s
INFO:BlockProcessor:our height: 696,751 daemon: 697,014 UTXOs 63MB hist 78MB
INFO:BlockProcessor:processed 13 blocks size 18.93 MB in 27.3s
INFO:BlockProcessor:processed 7 blocks size 10.56 MB in 14.9s
INFO:BlockProcessor:our height: 696,772 daemon: 697,014 UTXOs 86MB hist 108MB
INFO:BlockProcessor:processed 14 blocks size 18.68 MB in 29.3s
INFO:BlockProcessor:our height: 696,814 daemon: 697,014 UTXOs 104MB hist 136MB
INFO:BlockProcessor:processed 42 blocks size 28.92 MB in 32.8s
INFO:BlockProcessor:our height: 696,848 daemon: 697,014 UTXOs 120MB hist 158MB
INFO:BlockProcessor:processed 34 blocks size 23.53 MB in 34.3s
INFO:BlockProcessor:processed 25 blocks size 18.77 MB in 18.1s
INFO:BlockProcessor:our height: 696,887 daemon: 697,014 UTXOs 146MB hist 192MB
INFO:BlockProcessor:processed 14 blocks size 13.97 MB in 13.9s
INFO:BlockProcessor:processed 10 blocks size 13.38 MB in 15.0s
INFO:BlockProcessor:our height: 696,913 daemon: 697,014 UTXOs 163MB hist 218MB
INFO:BlockProcessor:processed 16 blocks size 14.90 MB in 12.5s
INFO:BlockProcessor:processed 26 blocks size 19.40 MB in 18.3s
INFO:BlockProcessor:our height: 696,974 daemon: 697,014 UTXOs 190MB hist 257MB
INFO:BlockProcessor:processed 35 blocks size 23.44 MB in 22.7s
INFO:BlockProcessor:our height: 697,009 daemon: 697,014 UTXOs 202MB hist 275MB
INFO:BlockProcessor:processed 35 blocks size 23.32 MB in 30.5s
INFO:BlockProcessor:processed 5 blocks size 4.92 MB in 10.1s
INFO:BlockProcessor:caught up to height 697014
INFO:DB:flushed filesystem data in 0.04s
INFO:Prefetcher:cancelled; prefetcher stopping 
INFO:SessionManager:closing down server for rpc://0.0.0.0:8000
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
  File "/electrumx/electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/electrumx/electrumx/lib/server_base.py", line 129, in run
    await server_task
  File "/electrumx/electrumx/lib/server_base.py", line 102, in serve
    await self.serve(shutdown_event)
  File "/electrumx/electrumx/server/controller.py", line 134, in serve
    await group.spawn(wait_for_catchup())
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.5-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.5-py3.7.egg/aiorpcx/curio.py", line 211, in join
    raise task.exception()
  File "/electrumx/electrumx/server/block_processor.py", line 702, in fetch_and_process_blocks
    await group.spawn(self._process_prefetched_blocks())
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.5-py3.7.egg/aiorpcx/curio.py", line 242, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.5-py3.7.egg/aiorpcx/curio.py", line 211, in join
    raise task.exception()
  File "/electrumx/electrumx/server/block_processor.py", line 654, in _process_prefetched_blocks
    await self._first_caught_up()
  File "/electrumx/electrumx/server/block_processor.py", line 670, in _first_caught_up
    await self.flush(True)
  File "/electrumx/electrumx/server/block_processor.py", line 358, in flush
    await self.run_in_thread_with_lock(flush)
  File "/electrumx/electrumx/server/block_processor.py", line 212, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/electrumx/electrumx/server/block_processor.py", line 211, in run_in_thread_locked
    return await run_in_thread(func, *args)
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.5-py3.7.egg/aiorpcx/curio.py", line 68, in run_in_thread
    return await get_event_loop().run_in_executor(None, func, *args)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/electrumx/electrumx/server/block_processor.py", line 357, in flush
    self.estimate_txs_remaining)
  File "/electrumx/electrumx/server/db.py", line 244, in flush_dbs
    self.flush_history()
  File "/electrumx/electrumx/server/db.py", line 313, in flush_history
    self.history.flush()
  File "/electrumx/electrumx/server/history.py", line 160, in flush
    flush_id = pack_be_uint16(self.flush_count)
struct.error: 'H' format requires 0 <= number <= 65535

[maltokyo]

This looks like the same problem. But without any long term solution..
#60

[maltokyo]

OK, how I solved this (super hacky)

1. Stop the container
2. Clone this git repo onto your server/computer
3. Change the init file to run the compaction (change last line to this: exec /electrumx/electrumx_compact_history )
4. Build a new image from the clone from using (for example docker build -t lukechilds/electrumx_temp )
5. Then run this new image, and it will do everything needed. Shut down when finished (check logs to make sure it is finished)
6. Revert back to the normal image.

[4oo4]

@maltokyo Thanks, I found another way to do that is just do a one-off docker run -it (interactive shell) on the same image. I ran into issues but this was because I forgot to mount my /data volume.

docker run -v /path/to/electrum/data:/data -it electrumx /bin/sh

Then inside the container:

cd /electrumx

# You can avoid this if you pass these to your `docker run -it ....` command
export DAEMON_URL="http://rpcuser:rpcpass@x.x.x.x:8332
export COIN=BitcoinSegWit
export NET=mainnet
export PEER_ANNOUNCE=""
export PEER_DISCOVERY=""
export SERVICES="ssl://172.20.0.2:50002"
export REPORT_SERVICES=""
export INITIAL_CONCURRENT=100
export COST_SOFT_LIMIT=100000
export COST_HARD_LIMIT=1000000
export REQUEST_SLEEP=30000 

./electrumx_compact_history

[dorveille]

Here's a one-liner to perform history compact operations. It should work on any system that has docker installed and that has r/w access to the electrum data set.

docker run --rm --env COIN=BitcoinSegwit --env DB_DIRECTORY=/data -v "${PWD}/electrumx:/data" --entrypoint=/usr/local/bin/electrumx_compact_history lukechilds/electrumx:v1.16.0

Regarding long term solution, I don't think this has much to do with lukechilds' container, it might be helpful to comment on the issue on the electrumx repo, and maybe even the issue on the upstream repo.

[tuwid]

If someone is running this on kubernetes with helm charts Id recommend the following with a compacting.enabled set to True in the value files of

    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "service.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      initContainers:
      - name: volume-mount-hack
        image: busybox
        command: ["sh", "-c", "chown -R 999:999 /var/lib/electrumx"]
        volumeMounts:
        - name: electrumx-data
          mountPath: /var/lib/electrumx
      {{- if .Values.compacting.enabled }}
      - name: compacting-hack
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
        command: ["/usr/local/bin/electrumx_compact_history"]
        volumeMounts:
        - name: electrumx-data
          mountPath: /var/lib/electrumx
        envFrom:
          - secretRef:
              name: {{ include "service.fullname" . }}-secrets
          - configMapRef: 
              name: {{ include "service.fullname" . }}-configmap
      {{- end }}          
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          volumeMounts:
          - name: ssl-key
            mountPath: "/etc/electrumx.key"
            subPath: "electrumx.key"
            readOnly: true
          - name: ssl-cert
            mountPath: "/etc/electrumx.cert"     
            subPath: "electrumx.cert"       
            readOnly: true
          - name: electrumx-data
            mountPath: /var/lib/electrumx         
          envFrom:
            - secretRef:
                name: {{ include "service.fullname" . }}-secrets
            - configMapRef: 
                name: {{ include "service.fullname" . }}-configmap
          ports:
            - name: rpc
              containerPort: {{ .Values.config.service.rpcPort }}
            - name: tcp
              containerPort: {{ .Values.config.service.tcpPort }}