{"push_id":959214251,"size":10,"distinct_size":8,"ref":"refs/heads/blockchain_database2","head":"e08d4573cd078b7350406bd97eb87f855278f7a7","before":"f3cccd322fb382b470722a2ea75a2960b2e443ee","commits":[{"sha":"0e83746168c6ce5a689b3e40857392b57e5e879a","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"database: Major redesign of database package.\n\nThis commit contains a complete redesign and rewrite of the database\npackage that approaches things in a vastly different manner than the\nprevious version.  This is the first part of several stages that will be\nneeded to ultimately make use of this new package.\n\nSome of the reason for this were discussed in #255, however a quick\nsummary is as follows:\n\n- The previous database could only contain blocks on the main chain and\n  reorgs required deleting the blocks from the database.  This made it\n  impossible to store orphans and could make external RPC calls for\n  information about blocks during the middle of a reorg fail.\n- The previous database interface forced a high level of bitcoin-specific\n  intelligence such as spend tracking into each backend driver.\n- The aforementioned point led to making it difficult to implement new\n  backend drivers due to the need to repeat a lot of non-trivial logic\n  which is better handled at a higher layer, such as the blockchain\n  package.\n- The old database stored all blocks in leveldb.  This made it extremely\n  inefficient to do things such as lookup headers and individual\n  transactions since the entire block had to be loaded from leveldb (which\n  entails it doing data copies) to get access.\n\nIn order to address all of these concerns, and others not mentioned, the\ndatabase interface has been redesigned as follows:\n\n- Two main categories of functionality are provided: block storage and\n  metadata storage\n- All block storage and metadata storage are done via read-only and\n  read-write MVCC transactions with both manual and managed modes\n  - Support for multiple concurrent readers and a single writer\n  - Readers use a snapshot and therefore are not blocked by the writer\n- Some key properties of the block storage and retrieval API:\n  - It is generic and does NOT contain additional bitcoin logic such spend\n    tracking and block linking\n  - Provides access to the raw serialized bytes so deserialization is not\n    forced for callers that don't need it\n  - Support for fetching headers via independent functions which allows\n    implementations to provide significant optimizations\n  - Ability to efficiently retrieve arbitrary regions of blocks\n    (transactions, scripts, etc)\n- A rich metadata storage API is provided:\n  - Key/value with arbitrary data\n  - Support for buckets and nested buckets\n  - Bucket iteration through a couple of different mechanisms\n  - Cursors for efficient and direct key seeking\n- Supports registration of backend database implementations\n- Comprehensive test coverage\n- Provides strong documentation with example usage\n\nThis commit also contains an implementation of the previously discussed\ninterface named ffldb (flat file plus leveldb metadata backend).  Here\nis a quick overview:\n\n- Highly optimized for read performance with consistent write performance\n  regardless of database size\n- All blocks are stored in flat files on the file system\n- Bulk block region fetching is optimized to perform linear reads which\n  improves performance on spindle disks\n- Anti-corruption mechanisms:\n  - Flat files contain full block checksums to quickly an easily detect\n    database corruption without needing to do expensive merkle root\n    calculations\n  - Metadata checksums\n  - Open reconciliation\n- Extensive test coverage:\n  - Comprehensive blackbox interface testing\n  - Whitebox testing which uses intimate knowledge to exercise uncommon\n    failure paths such as deleting files out from under the database\n  - Corruption tests (replacing random data in the files)\n\nIn addition, this commit also contains a new tool under the new database\ndirectory named dbtool which provides a few basic commands for testing the\ndatabase.  It is designed around commands, so it could be useful to expand\non in the future.\n\nFinally, this commit addresses the following issues:\n\n- Adds support for and therefore closes #255\n- Fixes #199\n- Fixes #201\n- Implements and closes #256\n- Obsoletes and closes #257\n- Closes #247 once the required chain and btcd modifications are in place\n  to make use of this new code","distinct":false,"url":"https://api.github.com/repos/davecgh/btcd/commits/0e83746168c6ce5a689b3e40857392b57e5e879a"},{"sha":"68da827da45b85709e951139956fb150fd46cffa","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"Implement database cache layer.\n\nThis commit adds a database cache layer to the ffldb database backend so\nthat callers can commit multiple transactions ithout having to incur the\noverhead of a disk sync on every new block.","distinct":false,"url":"https://api.github.com/repos/davecgh/btcd/commits/68da827da45b85709e951139956fb150fd46cffa"},{"sha":"f8ee51a70260a1ad6a121872494fdd0869c2fcdb","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"WIP: blockchain: Rework to use new db interface.\n\nThis commit is the first stage of several that are planned to convert\nthe blockchain package into a concurrent safe package that will\nultimately allow support for multi-peer download and concurrent chain\nprocessing.  The goal is to update btcd proper after each step so it can\ntake advantage of the enhancements as they are developed.\n\nIn addition to the aforementioned benefit, this staged approach has been\nchosen since it is absolutely critical to maintain consensus.\nSeparating the changes into several stages makes it easier for reviewers\nto logically follow what is happening and therefore helps prevent\nconsensus bugs.  Naturally there are significant automated tests to help\nprevent consensus issues as well.\n\nThe main focus of this stage is to convert the blockchain package to use\nthe new database interface and implement the chain-related functionality\nwhich it no longer handles.  It also aims to improve efficiency in\nvarious areas by making use of the new database and chain capabilities.\n\nThe following is an overview of the chain changes:\n\n- Update to use the new database interface\n- Add chain-related functionality that the old database used to handle\n  - Main chain structure and state\n  - Transaction spend tracking\n  - Transaction indexing\n- Make best chain state lways set when the chain instance is created\n  - Remove now unnecessary logic for dealing with unset best state\n- Replace the old TxStore type with a new UtxoViewpoint which paves the\n  way towards ultimately supporting a separate pruned utxo set without\n  reliance on a transaction index\n- Make all exported functions concurrent safe\n  - Currently using a single chain state lock as it provides a straight\n    forward and easy to review path forward however this can be improved\n    with more fine grained locking\n- Optimize various cases where full blocks were being loaded when only\n  the header is needed to help reduce the I/O load\n- Add the ability for callers to get a snapshot of the current best\n  chain stats in a concurrent safe fashion\n  - Does not block callers while new blocks are being processed\n- Make error messages that reference transaction outputs consistently\n  use <transaction hash>:<output index>\n- Update tests and examples to reflect the changes\n\nThe following is an overview of the btcd changes:\n\n- Update to use the new database and chain interfaces\n- Temporarily remove all code related to the address index\n- Convert all code that uses transaction stores to use the new utxo\n  store\n- Rework several calls that required the block manager for safe\n  concurrency to use the chain package directly now that it is\n  concurrent safe\n- Change all calls to obtain the best hash to use the new best state\n  snapshot capability from the chain package\n- Remove workaround for limits on fetching height ranges since the new\n  database interface no longer imposes them\n- Correct the gettxout RPC handler to return the best chain hash as\n  opposed the hash the txout was found in\n- Optimize various RPC handlers:\n  - Change several of the RPC handlers to use the new chain snapshot\n    capability to avoid needlessly loading data\n  - Update several handlers to use new functionality to avoid accessing\n    the block manager so they are able to return the data without\n    blocking when the server is busy processing blocks\n  - Update non-verbose getblock to avoid deserialization and\n    serialization overhead\n  - Update getblockheader to request the block height directly from\n    chain and only load the header\n  - Update getdifficulty to use the new cached data from chain\n  - Update getmininginfo to use the new cached data from chain\n  - Update non-verbose getrawtransaction to avoid deserialization and\n    serialization overhead\n  - Update gettxout to use the new utxo store versus loading\n    full transactions using the transaction index\n\nThe following is an overview of the utility changes:\n- Update addblock to use the new database and chain interfaces\n- Update findcheckpoint to use the new database and chain interfaces\n- Remove the dropafter utility which is no longer supported\n\nNOTE: In order to get testing started on this code this commit is being\npushed via a pull request before everything in done although it should\nbe stable.  Before it can be merged to master, the following additional\nthings are needed:\n\n- Implement a proper pruned utxo set and reorg journal\n- Reimplement the address index using the new database interface\n  including implementing the functionality that used to live in the old\n  database, but will now be needed in btcd proper\n- Increase test coverage\n\nblockchain: Remove b.bestNode nil checks.\n\nSince b.bestNode can no longer be nil, remove the remaining checks.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/f8ee51a70260a1ad6a121872494fdd0869c2fcdb"},{"sha":"f002e7761f83d8473067230e67fe9ba39d94dffb","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Introduce AssertError.\n\nThe blockchain contains several internal consistency checks which are\ncurrently returned as generic errors.  This introduces a new exported\nerror type named AssertError which can be used by callers to easily\ndifferentiate them via a type assertion.\n\nThis will ultimately allow code that deals with the blockchain package\nto detect this class error and force a server shutdown should one occur.\nIn practice, these conditions should never happen.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/f002e7761f83d8473067230e67fe9ba39d94dffb"},{"sha":"bfd60e1e662f93efaaa9a4ef339cfab3a0135224","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"Implement optimized utxo set and view.\n\nThis removes the dependency on the transaction index for looking up\ninput transaction scripts and instead implements a utxo set that, as it\nname implies, provides access specifically to the unspent transaction\noutputs.  The serialized utxo entries use a domain specific compression\nalgorithm that understands the standard transasction scripts in order to\nsignificantly compress them.\n\nTo go along with this, the chain code has been reworked to make the view\na rolling view as opposed to solely being a static storage mechanism.\nThis change is basically required for efficiently updating the utxo set\nwithout having to calculate spent and added utxos twice since block\nconnection is checked before it is actually updated.  This also paves\nthe way for a utxo cache which will provide even more signficant\nperformance improvements.\n\nIn addition to providing significantly faster access to the inputs\nneeded to validate transactions, this has several other benefits such as\npaving the way for block pruning and various proofs.\n\nFinally, full tests for all compression and serialization code have been\nadded and the old utxostore data set used in the tests has been updated\nto use the new utxo serialization format.  The reduced the data from\n1.4M to 36K.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/bfd60e1e662f93efaaa9a4ef339cfab3a0135224"},{"sha":"a5fc4f6b04ea28ec5255272d025a081736949ba7","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Correct unordered utxo serialization.\n\nThis corrects the utxo entry serialization to account for out of order\npacked indexes in the outputs and adds a test to trigger the original\nfailing case.\n\nIn the normal case, the packed outputs are always in order, however when\nresurrecting transactions they are packed into the outputs backwards, so\nthe serialization code needs to ensure they are properly ordered in\nascending order prior to serializing them.\n\nThis solution here isn't particuarly elegant, but the internal structure\ndata struct used to represent the entries need to be changed in order to\nhandle this better, so for now, just sort them before serializing.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/a5fc4f6b04ea28ec5255272d025a081736949ba7"},{"sha":"30fc8a716637542c9197f156ada5aafc918e2821","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Address review items.\n\n- Make fields in initial chain state print consistent\n- Rename UtxoEntry.Amount to UtxoEntry.AmountByIndex\n- Ensure any replace utxos have the compressed flag set to false\n- Correct comment on FetchUtxoView","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/30fc8a716637542c9197f156ada5aafc918e2821"},{"sha":"db679ff303c505a0db170ff466125c1a10fff5d0","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Rework UtxoEntry to only use map.\n\nThis gets rid of the separate \"packed\" outputs array in favor of sparse\nmap to utxoOutput pointers for a few reasons:\n\n- The upcoming cache uses the UtxoEntry struct, so it needs to be able\n  to efficiently remove random utxos as opposed to the entire entry\n- Holding a reference to a specific utxo, such as when it's cached for\n  write to the database, doesn't require the entire backing array to be\n  held into due to an interior reference.\n- Now that there is a spend journal which needs to populate the entries\n  in reverse order, having a packed index that was originally based on a\n  txindex where everything could be done in a set order no longer applies","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/db679ff303c505a0db170ff466125c1a10fff5d0"},{"sha":"8c58efe648e21e6cca40f2578291c97a3be25951","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Add best hash to UtxoViewpoint.\n\nThis adds tracking for the current best hash a UtxoViewpoint represents\nalong with an assert in checkConnectBlock to enforce its correctness.\n\nIn addition to helping ensure correctness of the view's state, it is\nuseful for the upcoming cache as well since it will have its own best\nhash as well.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/8c58efe648e21e6cca40f2578291c97a3be25951"},{"sha":"e08d4573cd078b7350406bd97eb87f855278f7a7","author":{"email":"40f085f3e2c05dc127b0a330228985d077d6c8f7@conformal.com","name":"Dave Collins"},"message":"blockchain: Cleanup and comment BlockChain struct.\n\nThis adds some comments and logically groups the various fields of the\nBlockChain struct.","distinct":true,"url":"https://api.github.com/repos/davecgh/btcd/commits/e08d4573cd078b7350406bd97eb87f855278f7a7"}]}