No transaction history when address has a lot (100+) of transactions #9203
Labels
Comments
|
@rachelhamlin, included this in V1 scope since highly likely the issue to be seen by users |
|
Yes, let's try to fix. 👍 |
|
as a quick simple solution, show only new transactions in history |
|
@yenda do you have any thoughts on how we could improve this one? Step 1 - show newest transactions first/only (until when is 'new'?) |
|
@rachelhamlin I would say
|
|
Yesterday I checked how it works on Although we definitely can fallback to etherscan, as far as understand we switched to a new implementation because we planned to not rely on centralised services and so we would need to remove ethers dependency later anyway. Before switching to etherscan I would try to have ~4 separate Also it would be great to have some feedback from Let me know if I missed some discussion on this regard and Etherscan has already been chosen as a best solution for this case. |
|
@hesterbruikman @errorists @rachelhamlin i also think we shouldn't retrieve any history by default and have an option for |
|
@flexsurfer I don't disagree but do you imagine that would be a manual fetch, so at first it's all empty right. you press a button to load last 10 items in history, as you start scrolling down you fetch backwards in batches of 10. what I'm saying is, I don't fancy placing buttons there like we did for chat to fetch missing gaps in history, those suck |
|
Just tried to play with that The way to fix it is to rewrite scanning on |
|
@rasom correct observations, I think though that @rachelhamlin @andremedeiros have already been discussing with etherscan about using their API. |
|
@rasom yes that is the one I'm talking about, it takes a block argument so you can use it with a dichotomic search to find a timespan than contains for instance 10 transactions. I'm not 100% sure about token transactions though, outgoing would be included because that's a contract call and still a transaction, but I think you can only be aware of incoming token transactions to your address through events.So that could be a partial solution to improve the on-chain lookups, but at this point I think we are going for etherescan it's much simpler, and the current way can stay as a fallback, and also nothing stops us from verifying the correctness of the transaction data provided by etherescan. Later we can explore a third method with something like the graph. |
|
Sounds great Roman. So user should see transactions from last N (50?) blocks, hopefully within a matter of seconds. As user scrolls, more transactions appear. |
yep, that actually sounds very confusing. Let's just say that user should see N last transaction in a matter of seconds :) |
rasom
added a commit
to status-im/status-go
that referenced
this issue
Dec 20, 2019
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 2, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 2, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 3, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 3, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 10, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 15, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 16, 2020
|
@rasom how is this going? I think we'll start dogfooding a release candidate with the team & ambassadors before this fix is complete, but we will want it in the final build for users. |
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 17, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 21, 2020
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 21, 2020
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 21, 2020
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
rasom
added a commit
that referenced
this issue
Jan 21, 2020
- all messages are not shown right away, in order to paginate history a user has to press "load more" button - added link to etherscan before transfers list - there is a new "fetch more" button at the end of the list - rest of changes can be found here status-im/status-go#1775
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 22, 2020
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
rasom
added a commit
that referenced
this issue
Jan 22, 2020
- all messages are not shown right away, in order to paginate history a user has to press "load more" button - added link to etherscan before transfers list - there is a new "fetch more" button at the end of the list - rest of changes can be found here status-im/status-go#1775
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 22, 2020
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
rasom
added a commit
that referenced
this issue
Jan 23, 2020
- all messages are not shown right away, in order to paginate history a user has to press "load more" button - added link to etherscan before transfers list - there is a new "fetch more" button at the end of the list - rest of changes can be found here status-im/status-go#1775
rasom
added a commit
to status-im/status-go
that referenced
this issue
Jan 23, 2020
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
each address of a new empty multiaccount this scan required
1. two `eth_getBalance` requests to find out that there is no any
balance change between zero and the last block, for eth transfers
2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
also accounts were not deduplicated. So even for an empty multiacc we
scanned chain twice for each chat and main wallet addresses, in result
app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
there were any eth transfers, and that caused empty history in case
if user already used all available eth (so that both zero and latest
blocks show 0 eth for an address). There might have been transactions
but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
scan, so it could require indefinite number of requests; the scanning
algorithm was written so that we persisted the whole history of
transactions or tried to scan form the beginning again in case of
failure, giving up only after 10 minutes of failures. In result
addresses with sufficient number of transactions would never be fully
scanned and during these 10 minutes app could use gigabytes of
internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
requests. These requests return significantly bigger responses than
`eth_getBalance`/`eth_transactionsCount` and it is likely that
execution of thousands of them in parallel caused failures for
accounts with hundreds of transactions. Even for an account with 12k
we could successfully determine blocks with transaction in a few
minutes using `eth_getBalance` requests, but `eth_getBlock...`
couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
caused in average 3-4 times more such requests than is needed.
*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
then check erc20 in the range where these eth transactions were found.
For an empty address in multiacc this means:
1. two `eth_getBalance` transactions to determine that there was no
balance change between zero and the last block; two
`eth_transactionsCount` requests to determine there are no outgoing
transactions for this address; total 4 requests for eth transfers
2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
account, so a new multiacc requires ~25 (we also request latest block
number and probably execute a few other calls) request to determine
that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
1. determine the range which contains 20-25 outgoing eth/erc20
transactions. This usually requires up to 10 `eth_transactionCount`
requests
2. then we scan chain for eth transfers using `eth_getBalance` and
`eth_transactionCount` (for double checking zero balances)
3. we make sure that we do not scan db for more than 30 blocks with
transfers. That's important for accounts with mostly incoming
transactions, because the range found on the first step might
contain any number of incoming transfers, but only 20-25 outgoing
transactions
4. when we found ~30 blocks in a given range, we update initial
range `from` block using the oldest found block
5. and now we scan db for erc20transfers using `eth_getLogs`
`oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
6. when all blocks which contain incoming/outgoing transfers for a
given address are found, we save these blocks to db and mark that
transfers from these blocks are still to be fetched
7. Then we select latest ~30 (the number can be adjusted) blocks from
these which were found and fetch transfers, this requires 3-4
requests per transfer.
8. we persist scanned range so that we know were to start next time
9. we dispatch an event which tells client that transactions are found
10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
20 transfers, if not we scan chain again and return transfers after
small fixes
rasom
added a commit
that referenced
this issue
Jan 23, 2020
- all messages are not shown right away, in order to paginate history a user has to press "load more" button - added link to etherscan before transfers list - there is a new "fetch more" button at the end of the list - rest of changes can be found here status-im/status-go#1775
|
yes |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Expected behavior updated 1/11/19. Issue is ready for pick-up. [RH]
Problem
For the recovered accounts having a lot of transactions it very long to fetch and display transaction history in History tab. Another note is that we apparently fetch them from the earliest to latest transaction which makes much less sense than if we reverse this order (so currently I see TXs I had 2 years ago and not those I sent/received today).
So with the account having ~650 transaction during the last 1.5 year on Mainnet it took ~10 minutes for last transactions (ETH and ERC20) I had couple of days ago to be displayed.
And for the account with 11000+ transactions (in Ropsten network):
It might take forever for me to see tx sent/receive today for this
11k-tx account(seed phrase of this acc provided in steps below)Expected behavior
[TBD as it needs input from dev, design but for now...]
Disregard the number of transactions address has I see last 10 transactions in less than 1 second performed with this address.
Step 1 / Fetch latest transactions before earliest. Rely on Etherscan to get latest 20.
Step 2 / Fetch if on wifi or user scrolls.
Step 3 / Take over for Etherscan by Infura if fetching slows.
Actual behavior
It takes 20+ minutes for list of 600 transactions to appear on Wallet -> Account -> History tab
Notes
It seems that issue appeared after we moved Wallet to
status-go.Acceptance Criteria
Reproduction
tree weekend ceiling awkward universe pyramid glimpse raven pair lounge grant griefWallet->Status account->HistorytabAdditional Information
0.14.0 (2910101102)The text was updated successfully, but these errors were encountered: