CKB v114.0 Full node sync is stopped.

[silySuper]

Bug Report

Full node sync is stopped.

Current Behavior

[logs 2.zip](https://github.com/nervosnetwork/ckb/files/15315402/logs.2.zip)

Environment

• CKB version: v114.0
• Chain: testnet

testnet/data has been replaced by https://download.magickbase.com/backup_20240513.tar.gz

[eval-exec]

(The link you provided, https://download.magickbase.com/backup_20240513.tar.gz is returning a 404 error.)

When you say, "Full node is stopped," do you specifically mean that the CKB has been synchronizing for a long time but the block height hasn't increased? How long has it been syncing?

Could you provide the output of:

curl -X POST 127.0.0.1:8114 -H 'Content-Type: application/json' -d '{ "id": 42, "jsonrpc": "2.0", "method": "sync_state", "params": [ ] }'

and

curl -X POST 127.0.0.1:8114 -H 'Content-Type: application/json' -d '{ "id": 42, "jsonrpc": "2.0", "method": "get_peers", "params": [ ] }'

[silySuper]

It has been sync from yesterday afternoon to now.

[eval-exec]

The get_peers RPC returns an empty result, indicating that the CKB node isn't maintaining network connections with other peers, hence it's unable to synchronize blocks.

1. Have you made any changes to the default ckb.toml file? If so, how did you modify it? Did you edit the configuration related to the white-list in ckb.toml?
2. Can you share the complete log file (./data/logs/run.log)?

[silySuper]

run.log

I do not change ckb.toml.This part of ckb.toml.

### Whitelist-only mode
# whitelist_only = false
### Whitelist peers connecting from the given IP addresses
# whitelist_peers = []
### Enable `SO_REUSEPORT` feature to reuse port on Linux, not supported on other OS yet
# reuse_port_on_linux = true

max_peers = 125
max_outbound_peers = 8
# 2 minutes
ping_interval_secs = 120
# 20 minutes
ping_timeout_secs = 1200
connect_outbound_interval_secs = 15
# If set to true, try to register upnp
upnp = false
# If set to true, network service will add discovered local address to peer store, it's helpful for private net development
discovery_local_address = false
# If set to true, random cleanup when there are too many inbound nodes
# Ensure that itself can continue to serve as a bootnode node
bootnode_mode = false

[eval-exec]

I found there are some ERROR in run.log:

2024-05-14 20:18:58.451 +00:00 log flusher ERROR sled::flusher  failed to fsync from periodic flush thread: Input/output error (os error 5)

1. What happened in 2024-05-14 20:18:58.451?
2. Could you provide data/network/peer_store/addr_manager.db and data/network/ban_list.db files?
3. Could you provide:

curl -X POST 127.0.0.1:8114 -H 'Content-Type: application/json' -d '{ "id": 42, "jsonrpc": "2.0", "method": "get_banned_addresses", "params": [ ] }'

and

curl -X POST 127.0.0.1:8114 -H 'Content-Type: application/json' -d '{ "id": 42, "jsonrpc": "2.0", "method": "get_tip_header", "params": [ ] }'

[silySuper]

1.At 2024-05-14 20:18:58.451 ,computer is sleeping(v114.0 is in my hard disk).
2.
addr_manager.db.zip
ban_list.db.zip

[silySuper]

now it shows error in ckb server.

[eval-exec]

Is your hard drive malfunctioning?
Could you change [loggger].filter to "debug" in ckb.toml, then restart ckb node, then provide the log file?

[silySuper]

My hard drive does not throw error before，I will find a tool to check whether it is malfunctioning
logs.zip

[eval-exec]

I suspect there might be an issue with the [network] configuration in your config file.

What's the configuration for support_protocols in your ckb.toml file?

Could you share the complete configuration from your ckb.toml file?

[eval-exec]

I found your ckb process is buzy on serving an RPC:
I guess it's Indexer's get_cells or get_transactions RPC.
I observed that the "id" field in the RPC request is not consistent. Is the ckb process operating as a public node?

❯ cat logs/run.log | grep -i rpc | head
2024-05-15 07:21:02.362 +00:00 main INFO ckb_rpc::server  Listen HTTP RPCServer on address: 127.0.0.1:8114
2024-05-15 07:21:02.944 +00:00 GlobalRt-7 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0xc9ed7d","id":8362}.
2024-05-15 07:21:02.953 +00:00 GlobalRt-6 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x10639e0895502b5688a6be8cf69460d76541bfa4821629d86d62ba0aae3f9606","id":8716}.
2024-05-15 07:21:02.964 +00:00 GlobalRt-6 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"alerts":[],"chain":"ckb_testnet","difficulty":"0x1c794aac","epoch":"0x70800e40021fd","is_initial_block_download":true,"median_time":"0x18f70071de4"},"id":2884}.
2024-05-15 07:21:02.971 +00:00 GlobalRt-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x10639e0895502b5688a6be8cf69460d76541bfa4821629d86d62ba0aae3f9606","id":1528}.
2024-05-15 07:21:02.972 +00:00 GlobalRt-6 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"alerts":[],"chain":"ckb_testnet","difficulty":"0x1c794aac","epoch":"0x70800e40021fd","is_initial_block_download":true,"median_time":"0x18f70071de4"},"id":2529}.
2024-05-15 07:21:02.975 +00:00 GlobalRt-6 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x10639e0895502b5688a6be8cf69460d76541bfa4821629d86d62ba0aae3f9606","id":3712}.
2024-05-15 07:21:03.926 +00:00 GlobalRt-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0xc9ed7d","id":804}.
2024-05-15 07:21:04.931 +00:00 GlobalRt-4 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0xc9ed7d","id":8224}.
2024-05-15 07:21:05.190 +00:00 GlobalRt-6 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"compact_target":"0x1d08fda0","dao":"0xa58aea41d2aa324d7444ff8c4625280007bc9da5479673060085b8d95c40d408","epoch":"0x70800e40021fd","extra_hash":"0x167c593d80e706b9c2e52c9a6aeebf39fdd08574c55b6deb5df128a1484677cb","hash":"0xf47a17392103d8c423089c6fb42ea3bd15cd44b5a8268c7ae18a72d517906ce2","nonce":"0xc108f9e52219af493fa13978b8cb7429","number":"0xc9ed7d","parent_hash":"0xcdc8fe751ae2731595b674b07f116ab240eefde45b4a911e702b79f58cffc441","proposals_hash":"0xf6d454599fdf28e3c73fb8e7d0d93e3f6128ad650a21b0f745cb9fba8bb0742f","timestamp":"0x18f70094a66","transactions_root":"0x9ff138b4bd1cbb83065343651537ac465209abb26455f0dd603978c6447ea6f4","version":"0x0"},"id":3662}.
/tmp/t
❯ cat logs/run.log | grep -i rpc | tail
2024-05-15 07:22:08.718 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":3306}.
2024-05-15 07:22:08.719 +00:00 GlobalRt-8 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":7302}.
2024-05-15 07:22:08.719 +00:00 GlobalRt-8 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":9305}.
2024-05-15 07:22:08.719 +00:00 GlobalRt-8 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":8167}.
2024-05-15 07:22:08.720 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":3193}.
2024-05-15 07:22:08.721 +00:00 GlobalRt-8 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":4394}.
2024-05-15 07:22:08.721 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":3780}.
2024-05-15 07:22:08.721 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":285}.
2024-05-15 07:22:08.722 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":3175}.
2024-05-15 07:22:08.723 +00:00 GlobalRt-1 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"last_cursor":"0x","objects":[]},"id":8217}.
/tmp/t
❯ cat logs/run.log | grep -i rpc | wc -l
116692

How's the CPU and IO load on the machine where the ckb process is running?

Which client is making a large number of RPC requests to the ckb node? Could you try turning off the client to see if it makes a difference?

[silySuper]

I does not find any other rpc requests to the ckb node.