wallet+waddrmgr: sync and store up to MaxReorgDepth blocks #618
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @halseth |
|
for reference: #555 |
halseth
suggested changes
May 15, 2019
cfromknecht
reviewed
May 22, 2019
|
Pushed a significant rewrite that addresses some comments @cfromknecht and I discussed offline where it was possible for us to miss scanning relevant blocks if the birthday was before the reorg safe height. I also included some nice refactors along the way. PTAL @cfromknecht @halseth! |
cfromknecht
reviewed
May 27, 2019
There was a problem hiding this comment.
@wpaulino latest diff looking solid, dig the recovery refactor as well 👌
cfromknecht
approved these changes
Jun 6, 2019
halseth
reviewed
Jun 8, 2019
halseth
suggested changes
Jun 11, 2019
In this commit, we modify the wallet's block hash index to only store up to MaxReorgDepth blocks. This allows us to reduce consumed storage, as we'd be mostly storing duplicate data. We choose to store up to MaxReorgDepth to ensure we can recover from a potential long reorg.
In this commit, we add a migration that will be used by existing wallets to ensure they can adhere to the new requirement of storing up to MaxReorgDepth entries within the block hash index.
halseth
reviewed
Jun 13, 2019
wallet/wallet.go
Outdated
|
|
||
| // Clear the batch of all processed blocks to reuse the same | ||
| // memory for future batches. | ||
| blocks = blocks[:0] |
There was a problem hiding this comment.
are blocks actually used anywhere?
There was a problem hiding this comment.
They're used in the database transaction above.
| "%v", err) | ||
| } | ||
|
|
||
| c.bestBlock.Hash = *bestHash |
There was a problem hiding this comment.
It's meant to be there since BlockStamp can be called at any point, which acquires the lock.
halseth
suggested changes
Jun 13, 2019
| if err != nil { | ||
| return err | ||
| } | ||
| blocks = append(blocks, &waddrmgr.BlockStamp{ |
There was a problem hiding this comment.
should blocks before our birthday be added to the slice? If yes, add comment what this achieves.
There was a problem hiding this comment.
There's already a comment at the top of the loop explaining the rationale?
Since the recovery process itself acts as rescan, we'll also update our wallet's synced state along the way to reflect the blocks we process and prevent rescanning them later on.
cfromknecht
reviewed
Jun 14, 2019
| "%v", err) | ||
| } | ||
|
|
||
| c.bestBlock.Hash = *bestHash |
This ensures the wallet can properly do an initial sync, a recovery, or detect if it's on a stale branch before attempting to process new blocks at tip. Since the rescan will be triggered synchronously as well, we'll need to catch the wallet's quit chan when handling rescan batches in order to allow for clean shutdowns.
IsCurrent allows us to determine if the chain backend considers itself "current" with the chain.
This serves as groundwork for only storing up to MaxReorgDepth blocks upon initial sync. To do so, we want to make sure the chain backend considers itself current so that we can only fetch the latest MaxReorgDepth blocks from it.
We do this as the wallet will no longer store blocks all the way from genesis to the tip of the chain. Instead, in order to find a reasonable birthday block, we resort to performing a binary search for a block timestamp that's within +/-2 hours of the birthday timestamp.
This commit serves as another building point to allow the wallet to not store blocks all the way from genesis to the tip of chain. We modify the wallet's recovery logic to now start from either its birthday block, or the current reorg safe height if it's before the birthday, to ensure the wallet properly only stores MaxReorgDepth blocks. We also refactor things a bit in hopes of making the logic a bit more readable.
Currently, wallet rescans start from its known tip of the chain. Since we no longer store blocks all the way from genesis to the tip of the chain, performing a rescan would cause us to scan blocks all the way from genesis, which we want to avoid. To prevent this, we set the wallet's tip to be the current reorg safe height. This ensures that we're unable to scan any blocks before it, and that we maintain MaxReorgDepth blocks stored.
We use the recently introduced locateBirthdayBlock function within birthdaySanityCheck as it serves as a more optimized alternative that achieves the same purpose.
If a block arrives while the wallet is rescanning, users would be shown a log message that is confusing and not very helpful.
One could argue that the behavior before this commit was incorrect, as the ChainClient interface expects a call to NotifyBlocks before notifying blocks at tip, so we decide to fix this. Since we now wait for the chain backend to be considered "current" before proceeding to sync the wallet with it, any blocks that were processed while waiting would result in being notified and scanned twice, once by processing it at tip, and another while rescanning the wallet, which is not desirable.
Roasbeef
approved these changes
Jun 14, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR, we modify the wallet's block hash index to only store up to MaxReorgDepth blocks. This allows us to reduce consumed storage, as we'd be mostly storing duplicate data. We choose to store up to MaxReorgDepth to ensure we can recover from a potential long reorg.
We also add a migration that will be used by existing wallets to ensure they can adhere to the new requirement of storing up to MaxReorgDepth entries within the block hash index.
As a result of only storing up to MaxReorgDepth, we can also speed up the initial sync process for the wallet as we can just fetch the latest MaxReorgDepth blocks of the chain.
Depends on #617 to address some
bitcoindissues.Partly addresses #513, except we do not batch the requests to the chain backend.
Fixes #616.
NOTE: Since this is changing the initial sync logic, we should make sure to test manually and also through the set of tests within
lndto ensure there are no regressions.