You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
New channel doesn't become active without a disconnect #5366
Opened a channel to a working, online LND node.
There were no errors with bitcoind or lightningd.
Still after 16 confirmations the state was CHANNELD_AWAITING_LOCKIN
A simple forced disconnect solved it: lightning-cli disconnect NODE_ID true
Logs:
INFO NODE_ID-chan#82: Peer transient failure in CHANNELD_AWAITING_LOCKIN: disconnect command
INFO NODE_ID-chan#82: State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_NORMAL
The channel should switch to CHANNELD_NORMAL automatically.
2022-07-01T10:35:55.331Z INFO [peer-pubkey]-channeld-chan#16792: Peer connection lost
2022-07-01T10:35:55.332Z INFO [peer-pubkey]-chan#16792: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
This is what the LND peer's log say:
2022-07-01 12:29:37.247 [INF] PEER: unable to read message from 038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5@[144.76.24.71:9735](http://144.76.24.71:9735/): read tcp [192.168.255.10:43282](http://192.168.255.10:43282/)->[144.76.24.71:9735](http://144.76.24.71:9735/): use of closed network connection
2022-07-01 12:29:37.353 [INF] SRVR: Established connection to: 038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5@[144.76.24.71:9735](http://144.76.24.71:9735/)
2022-07-01 12:29:37.353 [INF] SRVR: Finalizing connection to 038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5@[144.76.24.71:9735](http://144.76.24.71:9735/), inbound=false
2022-07-01 12:29:37.399 [INF] PEER: NodeKey(038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5) loading ChannelPoint([txid:outnum])
2022-07-01 12:29:37.399 [INF] PEER: Negotiated chan series queries with 038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5
2022-07-01 12:29:37.400 [INF] DISC: Creating new GossipSyncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5
2022-07-01 12:29:37.448 [INF] DISC: Removing GossipSyncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5
2022-07-01 12:29:57.244 [WRN] RPCS: Gossip syncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5 not found
2022-07-01 12:30:18.886 [WRN] RPCS: Gossip syncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5 not found
2022-07-01 12:30:39.219 [WRN] RPCS: Gossip syncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5 not found
2022-07-01 12:31:00.375 [WRN] RPCS: Gossip syncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5 not found
2022-07-01 12:31:21.257 [WRN] RPCS: Gossip syncer for peer=038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5 not found
UPDATE - according to my node the channel was opened and in CHANNELD_NORMAL, but for the remote side it was still pending. So my report might be an unrelated LND issue. The channel became active for the counterparty after restarting LND.
Any debug logs from the CLN node? Specifically we're looking for peer_out FUNDING_LOCKED and/or peer_in FUNDING_LOCKED which signals the transition from awaiting lockin to nomal, and only once both sides have sent and received it we can switch state.
#5349
If you're trying to get debug logs from one peer in particular without the firehose of all peers this PR allows you to specify just one peer pubkey for log-level=debug:.
The problem here is that selectively enabling logs for one peer would require that we know ahead of time that the issue might occur, which we don't, so I don't see how per-peer log level selection helps here, especially since this might also happen while the operator is sleeping, e.g., for a slow confirmation or an incoming funding request, so how could they enable logging on that? I think the firehose method is the one that'll ultimately get us an example.
I have set the logs to io and opened a channel on testnet to my LND v0.15.0. Could not reproduce the problem.
Got "state": "CHANNELD_NORMAL", after looking in 9 confirmations.
I have set the logs to io and opened a channel on testnet to my LND v0.15.0. Could not reproduce the problem. Got "state": "CHANNELD_NORMAL", after looking in 9 confirmations.
It's totally random for me too. Sometimes it happens.
I thought this should already been fixed in 0.12 but I have a node initiated channel to me and it’s sitting in "CHANNELD_AWAITING_LOCKIN:Sent reestablish, waiting for theirs” for a long time , but as soon as force disconnected / reconnected, it immediately locked in.
Maybe this is issue is still lurking around and could become the root cause of other issues like stale connection leading to force close.
I have the same issue but the disconnect/connect trick didn't change the state CHANNELD_AWAITING_LOCKIN.
I also tried to restart CLN without success. I'm using Core Lightning v22.11.1.
EDIT: It worked after the third time trying to disconnect from peer.
Notice that the issue here manifests when we are waiting on the channel_ready message (formerly funding_locked) from the peer. If that peer does not send the message on reconnect we won't progress to the CHANNELD_NORMAL state. So it's not just our local behavior that causes this, but also the peer's behavior.
I opened a channel today with v24.05, no idea what node the other side is running (my guess is LND), which remained stuck in CHANNELD_AWAITING_LOCKIN after confirmation. The disconnect and reconnect trick above fixed it.
Every time a new block came in the log showed:
2024-06-26T16:38:26.293Z DEBUG ...-chan#13945: Funding tx ... depth 6 of 3
2024-06-26T16:38:26.293Z DEBUG ...-chan#13945: Sending towire_channeld_funding_depth with channel state CHANNELD_AWAITING_LOCKIN
At the three confirmation mark I don't see anything interesting (within grep -n3 distance):
DEBUG lightningd: Got depth change 2->3 for ...
DEBUG ...-chan#13945: Funding tx ... depth 3 of 3
DEBUG ...-chan#13945: Sending towire_channeld_funding_depth with channel state CHANNELD_AWAITING_LOCKIN
DEBUG lightningd: Feerate estimate for 2 blocks set to 6138 (was 6137)
Activity
zerofeerouting commentedon Jul 1, 2022
It seems I am having the same issue:
This is what the LND peer's log say:
UPDATE - according to my node the channel was opened and in
CHANNELD_NORMAL, but for the remote side it was still pending. So my report might be an unrelated LND issue. The channel became active for the counterparty after restarting LND.TrezorHannes commentedon Jul 1, 2022
Logging this here for completion.
The disconnect from the CLN Node didn't have any effects. LND gossip syncer still hung up on a closed network connection
channelinfo didn't show any issue like outdated timestamps on last_update (see my other CLN peer channel issue logged here with LND, so not sure what was causing my gossip to get stuck
I had to restart LND to get the gossipSyncer into a healthy state
cdecker commentedon Jul 1, 2022
Any debug logs from the CLN node? Specifically we're looking for
peer_out FUNDING_LOCKEDand/orpeer_in FUNDING_LOCKEDwhich signals the transition from awaiting lockin to nomal, and only once both sides have sent and received it we can switch state.zerofeerouting commentedon Jul 1, 2022
Unfortunately my loglevel is INFO at the moment. Maybe @openoms can be of help.
wtogami commentedon Jul 2, 2022
#5349
If you're trying to get debug logs from one peer in particular without the firehose of all peers this PR allows you to specify just one peer pubkey for
log-level=debug:.cdecker commentedon Jul 3, 2022
The problem here is that selectively enabling logs for one peer would require that we know ahead of time that the issue might occur, which we don't, so I don't see how per-peer log level selection helps here, especially since this might also happen while the operator is sleeping, e.g., for a slow confirmation or an incoming funding request, so how could they enable logging on that? I think the firehose method is the one that'll ultimately get us an example.
openoms commentedon Jul 4, 2022
I have set the logs to io and opened a channel on testnet to my LND v0.15.0. Could not reproduce the problem.
Got
"state": "CHANNELD_NORMAL",after looking in 9 confirmations.devastgh commentedon Jul 4, 2022
It's totally random for me too. Sometimes it happens.
btweenthebars commentedon Sep 2, 2022
I thought this should already been fixed in 0.12 but I have a node initiated channel to me and it’s sitting in "CHANNELD_AWAITING_LOCKIN:Sent reestablish, waiting for theirs” for a long time , but as soon as force disconnected / reconnected, it immediately locked in.
Maybe this is issue is still lurking around and could become the root cause of other issues like stale connection leading to force close.
JssDWt commentedon Dec 12, 2022
Having this issue regularly on regtest here: #5808
olwe1 commentedon Jan 11, 2023
I have the same issue but the disconnect/connect trick didn't change the state
CHANNELD_AWAITING_LOCKIN.I also tried to restart CLN without success. I'm using Core Lightning v22.11.1.
EDIT: It worked after the third time trying to disconnect from peer.
cdecker commentedon Jan 13, 2023
Notice that the issue here manifests when we are waiting on the
channel_readymessage (formerlyfunding_locked) from the peer. If that peer does not send the message on reconnect we won't progress to theCHANNELD_NORMALstate. So it's not just our local behavior that causes this, but also the peer's behavior.Sjors commentedon Jun 26, 2024
I opened a channel today with v24.05, no idea what node the other side is running (my guess is LND), which remained stuck in
CHANNELD_AWAITING_LOCKINafter confirmation. The disconnect and reconnect trick above fixed it.Every time a new block came in the log showed:
At the three confirmation mark I don't see anything interesting (within
grep -n3distance):