Bitcoin core bugging out since what I assume was a bad shutdown. Mostly fixed (I think) but stalling trying to resync txindex. Here's the history:
- Core wouldn't connect to peers or load the latest blocks - I figured something must have bugged out if shut down incorrectly
- I ran -reindex-chainstate which successfully rechecked all block headers. Once this was complete, peers started connecting again and new blocks were being downloaded and synced. Side note: this took ages, a week maybe. I'm running the data directory on an external SSD (Samsung T7) and bumped up the speed to dbcache=4000 (50% of my computer's 8GB RAM)
- Wallet couldn't see latest transactions - I tried getrawtransaction and it stated txindex wasn't synced. I checked gettxindex which showed "synced": false. Checked the debug log - txindex script seemed to have stalled, wasn't making any progress
- Shut down core - shutdown didn't complete properly, after 30 mins or so I did an 'end task'
- Restarting core went ok, and txindex starts syncing. New peers connecting ok, new blocks downloading ok
- txindex seems to stall after 10-30mins each time, debug log doesn't show any progress but new peers connecting and new blocks downloading ok
- Restarting core seems to restart the synchronising txindex process ok - core still doesn't shut down correctly and needs a force close, and when restarting it runs the synchronise txindex script for a while then stalls again
Any ideas? Any more detailed diagnostics I can run to find out where and why it's stalling? Thanks in advance.
Here's a dump from the debug log:
2023-11-05T20:41:19Z Bitcoin Core version v22.0.0 (release build)
2023-11-05T20:41:19Z Qt 5.12.11 (static), plugin=windows (static)
2023-11-05T20:41:19Z Static plugins:
2023-11-05T20:41:19Z QWindowsIntegrationPlugin, version 330752
2023-11-05T20:41:19Z QWindowsVistaStylePlugin, version 330752
2023-11-05T20:41:19Z Style: windowsvista / QWindowsVistaStyle
2023-11-05T20:41:19Z System: Windows 10 (10.0), x86_64-little_endian-llp64
2023-11-05T20:41:19Z Screen: \\.\DISPLAY1 1920x1080, pixel ratio=1.0
2023-11-05T20:41:20Z Assuming ancestors of block 00000000000000000008a89e854d57e5667df88f1cdef6fde2fbca1de5b639ad have valid signatures.
2023-11-05T20:41:20Z Setting nMinimumChainWork=00000000000000000000000000000000000000001fa4663bbbe19f82de910280
2023-11-05T20:41:20Z Using the 'shani(1way,2way)' SHA256 implementation
2023-11-05T20:41:20Z Using RdSeed as additional entropy source
2023-11-05T20:41:20Z Using RdRand as an additional entropy source
2023-11-05T20:41:20Z GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely…"
2023-11-05T20:41:20Z Default data directory C:\Users\jonny\AppData\Roaming\Bitcoin
2023-11-05T20:41:20Z Using data directory D:\Bitcoin
2023-11-05T20:41:20Z Config file: D:\Bitcoin\bitcoin.conf
2023-11-05T20:41:20Z Config file arg: dbcache="1000"
2023-11-05T20:41:20Z Config file arg: server="1"
2023-11-05T20:41:20Z Config file arg: txindex="1"
2023-11-05T20:41:20Z Using at most 125 automatic connections (2048 file descriptors available)
2023-11-05T20:41:20Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2023-11-05T20:41:20Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2023-11-05T20:41:20Z Script verification uses 7 additional threads
2023-11-05T20:41:20Z scheduler thread start
2023-11-05T20:41:20Z HTTP: creating work queue of depth 16
2023-11-05T20:41:20Z Using random cookie authentication.
2023-11-05T20:41:20Z Generated RPC authentication cookie D:\Bitcoin\.cookie
2023-11-05T20:41:20Z HTTP: starting 4 worker threads
2023-11-05T20:41:20Z Using wallet directory D:\Bitcoin\wallets
2023-11-05T20:41:20Z init message: Verifying wallet(s)…
2023-11-05T20:41:20Z init message: Loading banlist…
2023-11-05T20:41:20Z SetNetworkActive: true
2023-11-05T20:41:20Z Using /16 prefix for IP bucketing
2023-11-05T20:41:20Z Cache configuration:
2023-11-05T20:41:20Z * Using 2.0 MiB for block index database
2023-11-05T20:41:20Z * Using 124.8 MiB for transaction index database
2023-11-05T20:41:20Z * Using 8.0 MiB for chain state database
2023-11-05T20:41:20Z * Using 865.2 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2023-11-05T20:41:20Z init message: Loading block index…
2023-11-05T20:41:20Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2023-11-05T20:41:20Z Opening LevelDB in D:\Bitcoin\blocks\index
2023-11-05T20:41:20Z Opened LevelDB successfully
2023-11-05T20:41:20Z Using obfuscation key for D:\Bitcoin\blocks\index: 0000000000000000
2023-11-05T20:41:23Z LoadBlockIndexDB: last block file = 3922
2023-11-05T20:41:23Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=33, size=53851357, heights=815456...815488, time=2023-11-05...2023-11-05)
2023-11-05T20:41:23Z Checking all blk files are present...
2023-11-05T20:41:27Z Opening LevelDB in D:\Bitcoin\chainstate
2023-11-05T20:41:28Z Opened LevelDB successfully
2023-11-05T20:41:28Z Using obfuscation key for D:\Bitcoin\chainstate: dc6979cd4bd813b3
2023-11-05T20:41:29Z Loaded best chain: hashBestChain=0000000000000000000270c62ee0ca93f9c0992a724470e214ceaf6bab585ad4 height=815488 date=2023-11-05T20:37:29Z progress=0.999999
2023-11-05T20:41:29Z init message: Verifying blocks…
2023-11-05T20:41:29Z Verifying last 6 blocks at level 3
2023-11-05T20:41:29Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2023-11-05T20:41:51Z No coin database inconsistencies in last 6 blocks (21212 transactions)
2023-11-05T20:41:51Z block index 30774ms
2023-11-05T20:41:51Z Opening LevelDB in D:\Bitcoin\indexes\txindex
2023-11-05T20:42:17Z Opened LevelDB successfully
2023-11-05T20:42:17Z Using obfuscation key for D:\Bitcoin\indexes\txindex: 0000000000000000
2023-11-05T20:42:17Z block tree size = 815492
2023-11-05T20:42:17Z nBestHeight = 815488
2023-11-05T20:42:17Z txindex thread start
2023-11-05T20:42:17Z loadblk thread start
2023-11-05T20:42:17Z Syncing txindex with block chain from height 616291
2023-11-05T20:42:17Z AddLocal([2a04:4a43:947f:f514:fdcc:b1cc:2ce3:4c78]:8333,1)
2023-11-05T20:42:17Z Discover: Sam - 2a04:4a43:947f:f514:fdcc:b1cc:2ce3:4c78
2023-11-05T20:42:17Z AddLocal([2a04:4a43:947f:f514:392b:e2c3:3462:4c74]:8333,1)
2023-11-05T20:42:17Z Discover: Sam - 2a04:4a43:947f:f514:392b:e2c3:3462:4c74
2023-11-05T20:42:17Z torcontrol thread start
2023-11-05T20:42:17Z Bound to 127.0.0.1:8334
2023-11-05T20:42:17Z Leaving InitialBlockDownload (latching to false)
2023-11-05T20:42:17Z Bound to [::]:8333
2023-11-05T20:42:17Z Bound to 0.0.0.0:8333
2023-11-05T20:42:17Z init message: Loading P2P addresses…
2023-11-05T20:42:17Z Loaded 58287 addresses from peers.dat 110ms
2023-11-05T20:42:17Z Loaded 2 addresses from "anchors.dat"
2023-11-05T20:42:17Z 2 block-relay-only anchors will be tried for connections.
2023-11-05T20:42:17Z init message: Starting network threads…
2023-11-05T20:42:17Z net thread start
2023-11-05T20:42:17Z dnsseed thread start
2023-11-05T20:42:17Z Waiting 300 seconds before querying DNS seeds.
2023-11-05T20:42:17Z addcon thread start
2023-11-05T20:42:17Z opencon thread start
2023-11-05T20:42:17Z msghand thread start
2023-11-05T20:42:17Z init message: Done loading
2023-11-05T20:42:17Z GUI: Platform customization: "windows"
2023-11-05T20:42:20Z New outbound peer connected: version: 70016, blocks=815488, peer=0 (block-relay-only)
2023-11-05T20:42:21Z New outbound peer connected: version: 70016, blocks=815488, peer=1 (block-relay-only)
2023-11-05T20:42:30Z New outbound peer connected: version: 70016, blocks=815488, peer=2 (outbound-full-relay)
2023-11-05T20:42:31Z New outbound peer connected: version: 70015, blocks=815488, peer=4 (outbound-full-relay)
2023-11-05T20:42:39Z New outbound peer connected: version: 70016, blocks=815488, peer=5 (outbound-full-relay)
2023-11-05T20:42:39Z P2P peers available. Skipped DNS seeding.
2023-11-05T20:42:39Z dnsseed thread exit
2023-11-05T20:42:42Z New outbound peer connected: version: 70016, blocks=815488, peer=7 (outbound-full-relay)
2023-11-05T20:42:49Z Syncing txindex with block chain from height 616346
2023-11-05T20:43:00Z New outbound peer connected: version: 70016, blocks=815488, peer=10 (outbound-full-relay)
2023-11-05T20:43:06Z New outbound peer connected: version: 70015, blocks=815488, peer=11 (outbound-full-relay)
2023-11-05T20:43:13Z New outbound peer connected: version: 70016, blocks=815488, peer=13 (outbound-full-relay)
2023-11-05T20:43:16Z New outbound peer connected: version: 70016, blocks=815488, peer=14 (outbound-full-relay)
2023-11-05T20:43:20Z Syncing txindex with block chain from height 616398
2023-11-05T20:43:51Z Syncing txindex with block chain from height 616433
2023-11-05T20:43:58Z New outbound peer connected: version: 70016, blocks=815488, peer=21 (outbound-full-relay)
2023-11-05T20:44:22Z Syncing txindex with block chain from height 616463
2023-11-05T20:44:44Z New outbound peer connected: version: 70016, blocks=815488, peer=22 (outbound-full-relay)
2023-11-05T20:49:39Z UpdateTip: new best=00000000000000000001a9debe40ab106fc7a9328d9d0ae6e39ccfc0302976fb height=815489 version=0x2c1a0000 log2_work=94.521130 tx=914569581 date='2023-11-05T20:49:05Z' progress=1.000000 cache=41.6MiB(289338txo)
2023-11-05T20:49:56Z UpdateTip: new best=00000000000000000001304514a6a3080875bd23ef495d98276fb5c362ee35d8 height=815490 version=0x2217e000 log2_work=94.521144 tx=914573253 date='2023-11-05T20:49:21Z' progress=1.000000 cache=43.3MiB(301523txo)
2023-11-05T20:53:58Z Imported mempool transactions from disk: 52234 succeeded, 415 failed, 0 expired, 20 already there, 0 waiting for initial broadcast
2023-11-05T20:53:58Z loadblk thread exit
2023-11-05T20:58:18Z New outbound peer connected: version: 70016, blocks=815490, peer=25 (outbound-full-relay)
2023-11-05T20:58:25Z New outbound peer connected: version: 70016, blocks=815490, peer=26 (outbound-full-relay)
2023-11-05T20:58:25Z New outbound peer connected: version: 70016, blocks=815488, peer=27 (outbound-full-relay)
2023-11-05T20:58:26Z New outbound peer connected: version: 70016, blocks=815490, peer=28 (outbound-full-relay)
2023-11-05T20:58:32Z New outbound peer connected: version: 70016, blocks=815490, peer=29 (outbound-full-relay)
2023-11-05T20:58:33Z New outbound peer connected: version: 70016, blocks=815490, peer=30 (outbound-full-relay)
2023-11-05T20:59:34Z New outbound peer connected: version: 70016, blocks=815490, peer=31 (outbound-full-relay)
2023-11-05T20:59:41Z New outbound peer connected: version: 70016, blocks=815489, peer=32 (outbound-full-relay)
2023-11-05T21:07:12Z UpdateTip: new best=00000000000000000002ed61af02abb053e53e287ff98f1a9f5898fc9b3edf57 height=815491 version=0x3fff0000 log2_work=94.521157 tx=914576806 date='2023-11-05T21:06:41Z' progress=1.000000 cache=55.9MiB(403631txo)
2023-11-05T21:11:26Z UpdateTip: new best=00000000000000000001754ba92729f0085950aec003537733de92854bfdbc84 height=815492 version=0x24424000 log2_work=94.521171 tx=914578785 date='2023-11-05T21:10:54Z' progress=1.000000 cache=57.3MiB(414237txo)
2023-11-05T21:14:47Z UpdateTip: new best=00000000000000000000bf127ef8c2af5e9bbac6866bf993f2ee5b04530c49ce height=815493 version=0x2e990000 log2_work=94.521185 tx=914582250 date='2023-11-05T21:14:05Z' progress=1.000000 cache=58.6MiB(424487txo)
2023-11-05T21:17:17Z UpdateTip: new best=000000000000000000042045e42d958931fceef136506301ab7801f2ffdbb724 height=815494 version=0x2309c000 log2_work=94.521198 tx=914585717 date='2023-11-05T21:16:50Z' progress=1.000000 cache=60.0MiB(435051txo)
And here's the debug log when I try and shutdown bitcoin core:
2023-11-05T21:30:58Z tor: Thread interrupt
2023-11-05T21:30:58Z addcon thread exit
2023-11-05T21:30:58Z opencon thread exit
2023-11-05T21:30:58Z torcontrol thread exit
2023-11-05T21:30:58Z Shutdown: In progress...
2023-11-05T21:30:58Z net thread exit
2023-11-05T21:30:58Z msghand thread exit
2023-11-05T21:30:59Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2023-11-05T21:30:59Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.01s)
2023-11-05T21:30:59Z scheduler thread exit
2023-11-05T21:31:23Z Writing 0 unbroadcast transactions to disk.
2023-11-05T21:31:23Z Dumped mempool: 0.031262s to copy, 23.6728s to dump
2023-11-05T21:31:23Z FlushStateToDisk: write coins cache to disk (440587 coins, 63659kB) started
2023-11-05T21:31:23Z FlushStateToDisk: write coins cache to disk (440587 coins, 63659kB) completed (0.36s)
then nothing happens so I have to manually end task. Don't know if this is related to the synchronising txindex script stalling
Edit: maybe it's not stalling, but just unbelievably slow. Left running overnight, started at: 2023-11-05T21:46:32Z Syncing txindex with block chain from height 616487 and now on: 2023-11-06T07:33:02Z Syncing txindex with block chain from height 638839 so thats c.2500tx/hr so will take another 3 days to complete. Is that about right? Note I did drop dbcache to =1000 in case it was using too much memory
Thanks
