tiamak/salvium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)

Browse Source 
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
This commit is contained in:
NoodleDoodleNoodleDoodleNoodleDoodleNoo
2015-07-10 13:09:32 -07:00
parent 1f83444d3d
commit e5d2680094
33 changed files with 4052 additions and 2373 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/cryptonote_core/cryptonote_core.cpp
+106 -7
View File
@@ -46,7 +46,7 @@ using namespace epee;
#include "cryptonote_core/checkpoints_create.h"
#include "blockchain_db/blockchain_db.h"
#include "blockchain_db/lmdb/db_lmdb.h"
#ifndef STATICLIB
#if defined(BERKELEY_DB)
#include "blockchain_db/berkeleydb/db_bdb.h"
#endif
@@ -212,18 +212,27 @@ namespace cryptonote
#if BLOCKCHAIN_DB == DB_LMDB
std::string db_type = command_line::get_arg(vm, daemon_args::arg_db_type);
std::string db_sync_mode = command_line::get_arg(vm, daemon_args::arg_db_sync_mode);
bool fast_sync = command_line::get_arg(vm, daemon_args::arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, daemon_args::arg_prep_blocks_threads);
BlockchainDB* db = nullptr;
uint64_t BDB_FAST_MODE = 0;
uint64_t BDB_FASTEST_MODE = 0;
uint64_t BDB_SAFE_MODE = 0;
if (db_type == "lmdb")
{
db = new BlockchainLMDB();
}
else if (db_type == "berkeley")
{
#ifndef STATICLIB
#if defined(BERKELEY_DB)
db = new BlockchainBDB();
BDB_FAST_MODE = DB_TXN_WRITE_NOSYNC;
BDB_FASTEST_MODE = DB_TXN_NOSYNC;
BDB_SAFE_MODE = DB_TXN_SYNC;
#else
LOG_ERROR("BlockchainBDB not supported on STATIC builds");
LOG_ERROR("BerkeleyDB support disabled.");
return false;
#endif
}
@@ -240,9 +249,71 @@ namespace cryptonote
LOG_PRINT_L0("Loading blockchain from folder " << folder.string() << " ...");
const std::string filename = folder.string();
// temporarily default to fastest:async:1000
blockchain_db_sync_mode sync_mode = db_async;
uint64_t blocks_per_sync = 1000;
try
{
db->open(filename);
uint64_t db_flags = 0;
bool islmdb = db_type == "lmdb";
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
for(const auto &option : options)
LOG_PRINT_L0("option: " << option);
// temporarily default to fastest:async:1000
uint64_t DEFAULT_FLAGS = islmdb ? MDB_WRITEMAP | MDB_MAPASYNC | MDB_NORDAHEAD | MDB_NOMETASYNC | MDB_NOSYNC :
BDB_FASTEST_MODE;
if(options.size() == 0)
{
// temporarily default to fastest:async:1000
db_flags = DEFAULT_FLAGS;
}
bool safemode = false;
if(options.size() >= 1)
{
if(options[0] == "safe")
{
safemode = true;
db_flags = islmdb ? MDB_NORDAHEAD : BDB_SAFE_MODE;
sync_mode = db_nosync;
}
else if(options[0] == "fast")
db_flags = islmdb ? MDB_NOMETASYNC | MDB_NOSYNC | MDB_NORDAHEAD : BDB_FAST_MODE;
else if(options[0] == "fastest")
db_flags = islmdb ? MDB_WRITEMAP | MDB_MAPASYNC | MDB_NORDAHEAD | MDB_NOMETASYNC | MDB_NOSYNC : BDB_FASTEST_MODE;
else
db_flags = DEFAULT_FLAGS;
}
if(options.size() >= 2 && !safemode)
{
if(options[1] == "sync")
sync_mode = db_sync;
else if(options[1] == "async")
sync_mode = db_async;
}
if(options.size() >= 3 && !safemode)
{
blocks_per_sync = atoll(options[2].c_str());
if(blocks_per_sync > 5000)
blocks_per_sync = 5000;
if(blocks_per_sync == 0)
blocks_per_sync = 1;
}
bool auto_remove_logs = command_line::get_arg(vm, daemon_args::arg_db_auto_remove_logs) != 0;
db->set_auto_remove_logs(auto_remove_logs);
db->open(filename, db_flags);
if(!db->m_open)
return false;
}
catch (const DB_ERROR& e)
{
@@ -250,7 +321,13 @@ namespace cryptonote
return false;
}
m_blockchain_storage.set_user_options(blocks_threads,
blocks_per_sync, sync_mode, fast_sync);
r = m_blockchain_storage.init(db, m_testnet);
bool show_time_stats = command_line::get_arg(vm, daemon_args::arg_show_time_stats) != 0;
m_blockchain_storage.set_show_time_stats(show_time_stats);
#else
r = m_blockchain_storage.init(m_config_folder, m_testnet);
#endif
@@ -587,6 +664,25 @@ namespace cryptonote
{
return m_blockchain_storage.add_new_block(b, bvc);
}
//-----------------------------------------------------------------------------------------------
bool core::prepare_handle_incoming_blocks(const std::list<block_complete_entry> &blocks)
{
#if BLOCKCHAIN_DB == DB_LMDB
m_blockchain_storage.prepare_handle_incoming_blocks(blocks);
#endif
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync)
{
#if BLOCKCHAIN_DB == DB_LMDB
m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
#endif
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_block(const blobdata& block_blob, block_verification_context& bvc, bool update_miner_blocktemplate)
{
@@ -678,7 +774,8 @@ namespace cryptonote
return m_blockchain_storage.get_block_id_by_height(height);
}
//-----------------------------------------------------------------------------------------------
bool core::get_block_by_hash(const crypto::hash &h, block &blk) {
bool core::get_block_by_hash(const crypto::hash &h, block &blk)
{
return m_blockchain_storage.get_block_by_hash(h, blk);
}
//-----------------------------------------------------------------------------------------------
@@ -723,11 +820,13 @@ namespace cryptonote
return true;
}
//-----------------------------------------------------------------------------------------------
void core::set_target_blockchain_height(uint64_t target_blockchain_height) {
void core::set_target_blockchain_height(uint64_t target_blockchain_height)
{
m_target_blockchain_height = target_blockchain_height;
}
//-----------------------------------------------------------------------------------------------
uint64_t core::get_target_blockchain_height() const {
uint64_t core::get_target_blockchain_height() const
{
return m_target_blockchain_height;
}
//-----------------------------------------------------------------------------------------------