This is the opinion editor of Shinobi, a self-taught educator and tech-focused Bitcoin podcast host in the Bitcoin space.
This is the second time in about a month that btcd/LND has exploited a bug that caused them to deviate from Bitcoin Core in consensus. Once again, Burak was the developer who triggered the vulnerability – this time clearly on purpose – and once again, it was a problem with the code for parsing Bitcoin transactions above the consensus layer. As I discussed in my article about the previous bug triggered by Burak, before Taproot there were limits on how large the script and witness data could be in a transaction. With the activation of Taproot, these restrictions have been removed, placing restrictions only on the block size limit itself to limit these parts of individual transactions. The problem with the last bug was that while the consensus code in btcd was properly upgraded to reflect this change, the code handling the peer-to-peer transfer was not properly upgraded, including parsing data before sending or receiving. So before the code was actually accepted to be approved for consensus, the processing blocks and transactions failed the data, never passed it to the consensus validation logic, and the block in question was never validated.
A very similar thing happened this time. Another limitation in the peer-to-peer section of the codebase was the incorrect application of the limit on witness data, limiting it to a maximum of 1/8 of the block size as opposed to the full block size. Burak developed a transaction with witness data that exceeded the strict limit by only one weight unit and once again suspended btcd and LND nodes at block height. This operation was a non-standard operation, meaning that while perfectly valid by consensus rules, it is not valid according to the default mempool policy, and therefore nodes will not pass it over the network. It is entirely possible to mine it into a block, but the only way to do that is to submit it directly to a miner, which Burak did with the help of F2Pool.
This really needs to be heavily vetted to ensure that any piece of code whose purpose is to analyze and validate Bitcoin data is compatible with what Bitcoin Core will do. It doesn’t matter if this code is a consensus engine for a node application or just a piece of code that relays operations for a Lightning node. This was the second mistake it’s been right on the word since last month in the codebase. It wasn’t even detected by anyone at Lightning Labs. AJ Towns reported this on October 11th, two days after the original bug was triggered by Burak’s 998 to 999 multisig operation. It was publicly posted on Github for 10 hours before being removed. A fix was made with the intention of quietly fixing the problem in the next release of LND, but it was not released.
Now, this is pretty much standard procedure for a serious vulnerability, especially with a project like Bitcoin Core where such a vulnerability could actually cause serious damage to the underlying network/protocol. But in this particular case, it presented a serious risk to LND users’ funds, and given that a previous bug with the same risks was literally right next to it, the chances of it being found and exploited were very high, as demonstrated by Burak. This begs the question of where the quiet patch approach should go when it comes to vulnerabilities like this that could leave users open to money theft (since their node can’t detect old channel states and penalize them properly).
As I covered in my last bug article, if a malicious actor found the bugs before a well-intentioned developer, they would tactically open new channels to vulnerable nodes, redirect the entire content of those channels to themselves, and then exploit the bug. From there, they would be able to keep those funds under their control and also close the channel back to the original state, literally doubling their money. Burak’s active exploitation of this issue, ironically, actually protected LND users from such an attack.
Once deployed, users were open to such attacks from peers who already had open channels, but they could no longer be specifically targeted with new channels. Their nodes were frozen and would never recognize or process payments through channels that someone tried to open after a block that stopped their node. So while it didn’t completely eliminate the risk of users being exploited, it did limit that risk for people who already had a channel. Burak’s action softens it. Personally, I think that such actions in response to a mistake make sense; it limited the damage, alerted users to the risk, and prompted a quick fix.
LND wasn’t the only thing affected either. Liquid the closing process was also brokenupdates are required from the federation functionaries to fix it. Older versions of Rust Bitcoin was also affected, causing the block to affect some block explorers and electricity samples (backend server implementation for Electrum Wallet). Now, with the exception of Liquid’s stake, which eventually exposes the funds after the time block expires to emergency recovery keys held by Blockstream – and in a realistic heist movie plot where Blockstream steals these funds, everyone knows exactly who to follow – these others issues never put anyone’s funds at risk. Also, Rust Bitcoin actually fixed this particular bug in newer versions, apparently not contacting the maintainers of other codebases to highlight the potential for such problems. It was only the active use of the bug on the network that widely exposed the problem as being present in multiple codebases.
When it comes to vulnerabilities like this in Bitcoin’s Layer 2 software, this poses some big problems. First, the rigor of checking these codebases for security bugs and prioritizing this over the integration of new features. I think it’s pretty telling that security isn’t always a priority given that this second bug wasn’t even found by the maintainers of the codebase it was in, despite being next to the original bug discovered last month. Wasn’t that code base internally audited after one major bug that put users’ funds at risk? Did it take an outsider to scout the project? This does not demonstrate the priority of protecting users’ funds over creating new features to attract more users. Second, the fact that this issue has already been fixed in Rust Bitcoin shows that there is a lack of communication between the maintainers of different codebases regarding bugs like this. This is quite understandable, because finding a bug in one of completely different codebases does not immediately lead a person to think: “I need to contact other teams writing similar software in completely different programming languages to warn them about the potential for this kind of bug.” You don’t find a bug in Windows and immediately consider reporting the bug to the Linux kernel maintainers. As a distributed consensus protocol on a global network, Bitcoin is a very different animal; maybe Bitcoin developers should start thinking along these lines when it comes to vulnerabilities in Bitcoin software. Especially when it comes to analyzing and interpreting data related to consensus.
Finally, when it comes to protocols like Lightning, which depend on observing the blockchain at all times to be able to react to legacy channel states, perhaps to maintain security, independent analysis and verification of data should be absolutely minimized. not completely deleted and not committed to Bitcoin Core or data derived directly from it. Core Lightning is built this way by connecting to the Bitcoin Core instance and depending entirely on it for validating blocks and transactions. If LND worked the same way, none of these bugs in btcd would have affected LND users in a way that put their funds at risk.
Regardless of how things are handled—either outsourcing validation entirely, or simply minimizing internal validation and being more careful about it—this incident shows that something needs to change in the approach to how Layer 2 software handles interactions with consensus-related data. Still, everyone is very lucky that it wasn’t exploited by a malicious actor, but by a developer proving a point. However, Bitcoin cannot rely on luck or hoping there are no malicious actors.
Developers and users should focus on improving processes to prevent incidents like this from happening again, and not play the blame game like a hot potato.
This is a guest post by Shinobi. The views expressed are entirely their own and do not necessarily reflect the views of BTC Inc or Bitcoin Magazine.