replace mapNextTx with slimmer setSpends #7997

kazcw · 2016-05-03T21:21:11Z

mapNextTx entries comprise about 15% of application memory usage once the mempool
warms up; most of this space is taken by the key type, which contains a uint256.
The same uint256 is accessible from the map's value type.

setSpends represents the information as a set of CSpendRefs. A CSpendRef is,
logically, a reference to a COutPoint; the COutPoint may be either "free" (as in
the case of setSpends.find(COutPoint(...))), or in an element of the vin of a
CTransaction (as in all the values inserted into setSpend). If the COutPoint is
associated with a spending CTransaction, the CSpendRef is also capable of
providing access to the corresponding transaction.

[This object bimodality is necessary because lookup on a set requires
constructing an object of the set's key type, an inconvenience corrected in
C++14.]

Saves about 10% of total memory usage (an entry in mapNextTx is 48 bytes; an
entry in setSpends in 16 bytes). Since the mempool is DynamicUsage-regulated,
this will translate to a larger mempool in the same amount of space.

(Supersedes #7991, which addressed the same issue less efficiently)

sdaftuar · 2016-05-05T17:37:18Z

Code review ACK, will test.

dcousens · 2016-05-06T02:51:34Z

[This object bimodality is necessary because lookup on a set requires
constructing an object of the set's key type, an inconvenience corrected in
C++14.]

I thought we were only at C++11? Or are you suggesting this is something else that would be corrected by moving to C++14?

kazcw · 2016-05-06T03:36:30Z

I'm just blaming C++<=11 for my tagged union. When we eventually switch to C++14 there's a much cleaner solution that doesn't require putting two things that should be different types in the same class.

dcousens · 2016-05-06T04:05:54Z

src/txmempool.h

noexcept, then a throw

Did I misunderstand the meaning of this qualifier?

noexcept guarantees std::terminate occurs if an exception escapes the function; it's a hard assert that's allowed inside a constexpr.

TIL, thanks @kazcw

sdaftuar · 2016-05-06T18:41:17Z

ACK. Nice improvement!

dcousens · 2016-05-07T02:25:58Z

utACK 2642766

sipa · 2016-05-09T19:33:41Z

Concept ACK

kazcw · 2016-05-09T22:32:03Z

I just realized that no users of mapNextTx need to know which CTxIn of a transaction spends a particular prevout. This makes a map-based approach possible with the same memory usage as this set implementation. I'm going to replace this with the map implementation because it's definitely cleaner that way (doesn't require the CSpendRef trick, and a map better reflects the function of the data structure).

TheBlueMatt · 2016-05-15T19:29:28Z

src/indirectmap.h

Feel like this might lead to errors at some point? I'd kinda prefer if you only do this for insert().

I guess it's the correct thing to do if you want to mimic the std::map interface as good as possible, but operator[] automatically constructing elements is annoying...

TheBlueMatt · 2016-05-15T19:50:05Z

ACK 507437a0f067dd696c1fe9698f4ad60a3801ce3f +/- removing [] and forcing insertion through insert(). It'd also be nice to get a test for indirectmap, or at least the following, if you're feeling lazy:

diff --git a/src/txmempool.cpp b/src/txmempool.cpp
index 2f867c8..42c3da4 100644
--- a/src/txmempool.cpp
+++ b/src/txmempool.cpp
@@ -673,6 +673,7 @@ void CTxMemPool::check(const CCoinsViewCache *pcoins) const
             // Check whether its inputs are marked in mapNextTx.
             auto it3 = mapNextTx.find(txin.prevout);
             assert(it3 != mapNextTx.end());
+            assert(it3->first == &txin.prevout);
             assert(it3->second == &tx);
             i++;
         }

kazcw · 2016-05-16T21:38:21Z

The find-and-maybe-insert operator does seem especially squirrelly in this case. Added a squashme with insert(...) instead; added the assertion.

sipa · 2016-05-16T22:34:09Z

utACK f379416127f0832ba4aa73c1af874df79b58124c

TheBlueMatt · 2016-05-17T05:05:26Z

Seems you forgot to add the assertion?

kazcw · 2016-05-17T05:22:07Z

Oops, I hadn't saved the changes to the file yet

TheBlueMatt · 2016-05-17T17:26:36Z

re-ACK 64949c7f9c7d47df170fa3814bb5aa89235c8e74

sipa · 2016-06-02T15:25:33Z

Can you squash?

Saves about 10% of application memory usage once the mempool warms up. Since the mempool is DynamicUsage-regulated, this will translate to a larger mempool in the same amount of space. Map value type: eliminate the vin index; no users of the map need to know which input of the transaction is spending the prevout. Map key type: replace the COutPoint with a pointer to a COutPoint. A COutPoint is 36 bytes, but each COutPoint is accessible from the same map entry's value. A trivial DereferencingComparator functor allows indirect map keys, but the resulting syntax is misleading: `map.find(&outpoint)`. Implement an indirectmap that acts as a wrapper to a map that uses a DereferencingComparator, supporting a syntax that accurately reflect the container's semantics: inserts and iterators use pointers since they store pointers and need them to remain constant and dereferenceable, but lookup functions take const references.

kazcw · 2016-06-02T19:34:27Z

Squashed all my commits from 64949c7 -> 9805f4a

sipa · 2016-06-02T23:18:44Z

utACK 9805f4a, but well-tested by the CTxMemPool::check function.

9805f4a mapNextTx: use pointer as key, simplify value (Kaz Wesley)

Ref: PR bitcoin#7997 in Bitcoin core

9b9c616 Fix missing zapwallettxes mode in wallet_hd.py functional test (furszy) d6d0ad9 [logs] fix zapwallettxes startup logs (John Newbery) 006c503 [wallet] fix zapwallettxes interaction with persistent mempool (John Newbery) c6d45c6 Adapting and connecting mempool_persist.py functional test to the test runner. (furszy) 4f26a4e Control mempool persistence using a command line parameter. (John Newbery) 5d949de [Qt] Do proper shutdown (Jonas Schnelli) e60da98 Allow shutdown during LoadMempool, dump only when necessary (Jonas Schnelli) c0a0e81 Moving TxMempoolInfo tx member to CTransactionRef (furszy) f0c2255 Add mempool.dat to doc/files.md (furszy) 8e52226 Add DumpMempool and LoadMempool (Pieter Wuille) 44c635d Add AcceptToMemoryPoolWithTime function (Pieter Wuille) 6bbc6a9 Add feedelta to TxMempoolInfo (Pieter Wuille) 9979f3d [mempool] move removed and conflicts transaction ref list to vector. (furszy) 4f672c2 Make removed and conflicted arguments optional to remove (Pieter Wuille) e51c4b8 Bypass removeRecursive in removeForReorg (Pieter Wuille) 54cf7c0 Get rid of CTxMempool::lookup() entirely (furszy) 35bc2a9 Finished the switch CTransaction storage in mempool to CTransactionRef and introduced the timeLastMempoolReq coming from btc#8080. (furszy) d10583b An adapted version of btc@b5599147533103efea896a1fc4ff51f2d3ad5808 (furszy) cb4fc6c An adapted version of btc@ed7068302c7490e8061cb3a558a0f83a465beeea (furszy) 9645775 Split up and optimize transaction and block inv queues (furszy) 68bc68f Don't do mempool lookups for "mempool" command without a filter (furszy) 7624823 mapNextTx: use pointer as key, simplify value (furszy) 191c62e Return mempool queries in dependency order (Pieter Wuille) 23c9f3e Eliminate TX trickle bypass, sort TX invs for privacy and priority. (furszy) 6ebfd17 tiny test fix for mempool_tests (Alex Morcos) 8c0016e Check all ancestor state in CTxMemPool::check() (furszy) 91c6096 Add ancestor feerate index to mempool (Suhas Daftuar) 64e84e2 Add ancestor tracking to mempool This implements caching of ancestor state to each mempool entry, similar to descendant tracking, but also including caching sigops-with-ancestors (as that metric will be helpful to future code that implements better transaction selection in CreatenewBlock). (furszy) 8325bb4 Fix mempool limiting for PrioritiseTransaction Redo the feerate index to be based on mining score, rather than fee. (furszy) 1fa40ac Remove work limit in UpdateForDescendants() The work limit served to prevent the descendant walking algorithm from doing too much work by marking the parent transaction as dirty. However to implement ancestor tracking, it's not possible to similarly mark those descendant transactions as dirty without having to calculate them to begin with. This commit removes the work limit altogether. With appropriate chain limits (-limitdescendantcount) the concern about doing too much work inside this function should be mitigated. (furszy) ba32375 Rename CTxMemPool::remove -> removeRecursive remove is no longer called non-recursively, so simplify the logic and eliminate an unnecessary parameter (furszy) c30fa16 CTxMemPool::removeForBlock now uses RemoveStaged (furszy) Pull request description: Ending up 2020 with a large PR :). Included a good number of performance and privacy improvements over the mempool and inv processing/sending areas + added mempool cache dump/load. Almost finishing with #1726 work, getting closer to 9725, and getting closer to be able to decouple the message processing thread <-> wallet and the wallet <-> GUI locks dependencies (which will be another long story.. but well, step by step). The final goal is clear, a much faster syncing process using pivx-qt (a good speed up for pivxd as well), smoother visual navigation when big wallets are syncing, less thread synchronization issues, among all of the improvements that will be coming with the backports adaptations. Adapted the following PRs to our sources: dashpay#7062 —> only eb30666. bitcoin#7174 —> complete. bitcoin#7562 —> only c5d746a. bitcoin#7594 —> complete. bitcoin#7840 —> complete. bitcoin#7997 —> complete. bitcoin#8080 —> complete bitcoin#8126 —> except e9b4780 (we don't have the mapRelay) and c2a4724 (we don't have the relay expiration vector). bitcoin#8448 —> complete bitcoin#8515 —> complete bitcoin#9408 —> complete bitcoin#9966 —> complete bitcoin#10330 —> complete ACKs for top commit: random-zebra: ACK 9b9c616 Fuzzbawls: ACK 9b9c616 Tree-SHA512: 186bd09bbb19b55ec0d46dc27291663a0df2d60d8238c661a8c9b8cbf3677b6f8a92fa56bd7346a3cb5293712eeccc5d6542ee3c441d35a4f61e7d12e2ce489a

maflcko added Refactoring Resource usage Mempool labels May 4, 2016

dcousens reviewed May 6, 2016
View reviewed changes

kazcw force-pushed the setSpends branch from 2642766 to 507437a Compare May 9, 2016 23:05

TheBlueMatt reviewed May 15, 2016
View reviewed changes

kazcw force-pushed the setSpends branch from bee55a0 to 64949c7 Compare May 17, 2016 05:27

kazcw force-pushed the setSpends branch from 64949c7 to 9805f4a Compare June 2, 2016 19:32

sipa merged commit 9805f4a into bitcoin:master Jun 2, 2016

sipa added a commit that referenced this pull request Jun 2, 2016

Merge #7997: replace mapNextTx with slimmer setSpends

a82f033

9805f4a mapNextTx: use pointer as key, simplify value (Kaz Wesley)

gmaxwell mentioned this pull request Jun 4, 2016

std::shared_ptr based CTransaction storage in mempool #8126

Merged

codablock pushed a commit to codablock/dash that referenced this pull request Dec 22, 2017

Merge bitcoin#7997: replace mapNextTx with slimmer setSpends

fb88e9b

9805f4a mapNextTx: use pointer as key, simplify value (Kaz Wesley)

andvgal pushed a commit to energicryptocurrency/gen2-energi that referenced this pull request Jan 6, 2019

Merge bitcoin#7997: replace mapNextTx with slimmer setSpends

7896756

9805f4a mapNextTx: use pointer as key, simplify value (Kaz Wesley)

wqking added a commit to wqking-misc/Phore that referenced this pull request Dec 13, 2019

Replaced mapNextTx with smaller map to reduce memory usage.

3a7c813

Ref: PR bitcoin#7997 in Bitcoin core

furszy mentioned this pull request Jul 3, 2020

Up to CTransactionRef point back ports to do list. PIVX-Project/PIVX#1726

Closed

21 tasks

furszy mentioned this pull request Jan 13, 2021

[NET] Inv, mempool and validation improvements PIVX-Project/PIVX#2118

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

replace mapNextTx with slimmer setSpends #7997

replace mapNextTx with slimmer setSpends #7997

Uh oh!

Conversation

kazcw commented May 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdaftuar commented May 5, 2016

Uh oh!

dcousens commented May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kazcw commented May 6, 2016

Uh oh!

dcousens May 6, 2016

Choose a reason for hiding this comment

Uh oh!

kazcw May 6, 2016

Choose a reason for hiding this comment

Uh oh!

dcousens May 6, 2016

Choose a reason for hiding this comment

Uh oh!

sdaftuar commented May 6, 2016

Uh oh!

dcousens commented May 7, 2016

Uh oh!

sipa commented May 9, 2016

Uh oh!

kazcw commented May 9, 2016

Uh oh!

TheBlueMatt May 15, 2016

Choose a reason for hiding this comment

Uh oh!

sipa May 16, 2016

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt commented May 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kazcw commented May 16, 2016

Uh oh!

sipa commented May 16, 2016

Uh oh!

TheBlueMatt commented May 17, 2016

Uh oh!

kazcw commented May 17, 2016

Uh oh!

TheBlueMatt commented May 17, 2016

Uh oh!

sipa commented Jun 2, 2016

Uh oh!

kazcw commented Jun 2, 2016

Uh oh!

sipa commented Jun 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kazcw commented May 3, 2016 •

edited

Loading

dcousens commented May 6, 2016 •

edited

Loading

TheBlueMatt commented May 15, 2016 •

edited

Loading