V27.1

What's Changed

Networking overhaul

Significant portions of the networking structure have been rewritten to improve reliability, maintainability, and performance.

• Merge server_socket in to tcp_listener by @clemahieu in #4354
• Preliminary network class cleanup by @pwojcikdev in #4490
• Reorganize tcp_listener by @pwojcikdev in #4515
• Networking cleanup continued by @pwojcikdev in #4495
• Fix tcp_server constructor by @pwojcikdev in #4499
• Cleanup strand binding in socket class by @pwojcikdev in #4498
• Keep io context running when stopping test system by @pwojcikdev in #4525
• Remove channel comparison functions by @pwojcikdev in #4500
• Rework handshake visitor by @pwojcikdev in #4504
• Store io_context as shared pointer by @pwojcikdev in #4506
• Keep weak pointer to node in socket objects by @pwojcikdev in #4542
• Use strand for socket operations by @pwojcikdev in #4576
• Use a dedicated thread for tcp listener by @pwojcikdev in #4523
• Peer cache by @pwojcikdev in #4574
• Use coroutines in tcp listener by @pwojcikdev in #4581
• Stop using boost::asio::ip::udp::endpoint by @dsiganos in #4585
• Run UPNP port mapping on a dedicated thread by @pwojcikdev in #4591
• Remove temporary channels by @pwojcikdev in #4595
• Replace tcp_message_manager by @pwojcikdev in #4609
• Internal node io context by @pwojcikdev in #4618
• Avoid creating temporary inproc channels by @pwojcikdev in #4625
• Transport classes cleanup by @pwojcikdev in #4651
• Differentiate between node IO runners and general IO thread roles by @pwojcikdev in #4643
• Remove tls support by @pwojcikdev in #4646
• Fix stopping of bootstrap workers by @pwojcikdev in #4667
• Incremental backoff for local_block_broadcaster by @pwojcikdev in #4662
• Legacy bootstrap throttling by @pwojcikdev in #4669

Fair queuing

• Fair queuing for block processor by @pwojcikdev in #4476
• Fair queuing for vote processor by @pwojcikdev in #4536
• Fair queuing for bootstrap server by @pwojcikdev in #4584
• Wait until fair queue source drains before removal by @pwojcikdev in #4597
• Fair queuing for request aggregator by @pwojcikdev in #4598

Logging and stats overhaul

• Logging overhaul by @pwojcikdev in #4375
• Avoid printing warning about cerr and colors in default cli mode by @dsiganos in #4400
• Tracing by @pwojcikdev in #4412
• Fix typo in ipc session_id debug log by @gr0vity-dev in #4448
• Allow easy printing ranges of objects by @pwojcikdev in #4488
• Log improvements by @pwojcikdev in #4580
• Stats overhaul by @pwojcikdev in #4583
• Stats logging by @pwojcikdev in #4596
• Enum utils by @pwojcikdev in #4601

Election/block handling improvements

• Add a bucket that lands in the range typically used by faucets / tipping services. by @gr0vity-dev #4661
• Remove atomic enum election::state_m and instead require locking election::mutex. by @clemahieu in #4318
• Remove block_processor::flush function by @clemahieu in #4404
• Remove gap cache by @pwojcikdev in #4422
• Move block observer logic into active_transactions by @pwojcikdev in #4421
• Adjust election block rebroadcasting by @pwojcikdev in #4423
• Blockprocessor context by @pwojcikdev in #4424
• Cleanup unused broadcast_confirm_req variants by @pwojcikdev in #4440
• Local block broadcaster by @pwojcikdev in #4454
• Ledger block abstraction by @clemahieu in #4458
• Improve block field functions by accessing sideband information internally by @clemahieu in #4465
• Remove ledger::block_destination and use block::destination by @clemahieu in #4478
• Remove usages of frontiers table from ledger_processor block checks by @clemahieu in #4460
• Simplify the nano::ledger::successor function to make its implementation more obvious. by @clemahieu in #4484
• Election vote result enum by @pwojcikdev in #4497
• Election age statistics by @RickiNano in #4537
• Limit batch size for block processor by @pwojcikdev in #4655

Rep handling overhaul

• Rep crawler overhaul by @pwojcikdev in #4449
• Dedicated rep_tiers class by @pwojcikdev in #4471
• Multithreaded request aggregator by @RickiNano in #4469
• Changed description of request_aggregator_threads by @RickiNano in #4477
• rep_weights part 1 - Add data store for representative weights by @simpago in #4482
• rep_weights part 2 - Store representative weights in database by @simpago in #4483
• rep_weights part 3 - Add mininum weight to representative cache by @simpago in #4485
• Convert rep_weights mutex to a shared_mutex since the majority of time it's being read by @clemahieu in #4628
• Track rep query response time by @pwojcikdev in #4658
• Use memory weight instead of exact weight when computing wallet reps. by @clemahieu in #4555

Vote handling improvements

• Enable large votes by default where support was added in V26 #4350 by @clemahieu #4683
• Store full votes in vote cache by @pwojcikdev in #4518
• Vote generator write queue by @clemahieu in #4556
• Multithreaded vote processor by @pwojcikdev in #4600
• Separate vote_router from active_transactions by @clemahieu in #4607
• Add 'rebroadcasted' flag to confirm_ack message by @pwojcikdev in #4621
• More vote cache optimizations by @pwojcikdev in #4629
• Address performance regression in vote_cache::top by @clemahieu in #4627
• Use read/write transaction variant in vote generator by @pwojcikdev in #4639
• Offload vote cache lookup to a separate thread by @pwojcikdev in #4631
• Optimize vote_cache::collect_container_info by @pwojcikdev in #4644
• Reduce vote generator max batch size by @pwojcikdev in #4650
  CI/GitHub improvements
• Update clang-format by @pwojcikdev in #4414
• Analyze core dumps on test failure by @pwojcikdev in #4430
• Sanitizer workflow for macOS by @pwojcikdev in #4431
• Separate leak sanitizer from ASAN run by @pwojcikdev in #4435
• Reintroduce RC builds for releases branches by @gr0vity-dev in #4381
• Ignore chacha.hpp ubsan errors by @pwojcikdev in #4538
• Enable reporting TSAN errors by @pwojcikdev in #4539
• disable rpm builds by @gr0vity-dev in #4554
• Add NANO_TRACING argument to Dockerfile by @gr0vity-dev in #4603
• During release process, create pull request to increment major version on instead of a direct commit by @gr0vity-dev in #4380
• Add Continuous Test Results as GitHub Badge by @gr0vity-dev in #4635
• Add flamegraph generation in CI build by @clemahieu in #4638
• Introduce nano-nightly dockerhub registry for live network builds by @gr0vity-dev in #4513

3rd party library updates

• Update minimum OSX deployment target to 13.3 which is required by the OSX 14.4 SDK. by @clemahieu in #4464
• Update googletest to v1.14 by @clemahieu in #4503
• Update rocksdb to 9.2.2 by @RickiNano #4652

Test improvements

• very minor assert improvement by @dsiganos in #4360
• Fix conflicts.add_two unit test by @dsiganos in #4361
• Improve assert statements of failing test to see why it is failing by @dsiganos in #4363
• Move rpc_context and test_response classes into their own files. by @dsiganos in #4370
• Use assert_eq and assert_timely_eq by @RickiNano in #4372
• Assert conversions part 2 by @RickiNano in #4376
• Fix for node.fork_publish_inactive by @dsiganos in #4382
• Fixing a race condition in active_transactions.fork_replacement_tally by @dsiganos in #4385
• Fix rpc receivable unconfirmed and rpc.accounts_receivable_confirmed by @dsiganos in #4387
• Some helpful comments about unit test socket_timeout.connect by @dsiganos in #4389
• Fix active_transactions.confirm_frontier by @dsiganos in #4386
• Simplify system::get_available_port() by @dsiganos in #4388
• Remove rule from code inspector script that enforces nano locks by @dsiganos in #4395
• Fix nano timed locks by @dsiganos in #4396
• Fix bootstrap unit tests by @dsiganos in #4383
• Fix failing TEST (active_transactions, fifo) by @gr0vity-dev in #4394
• Fix unit test node.search_receivable_confirmed by @dsiganos in #4408
• Fix active transactions vote replays by @dsiganos in #4401
• Clean up node.fork_multi_flip test. by @clemahieu in #4411
• Remove github cache saving and restoring from unit tests by @dsiganos in #4399
• Update macOS runner by @pwojcikdev in #4415
• Separate node.expire test by @pwojcikdev in #4417
• Re-enable unit test active_transactions.confirm_new by @dsiganos in #4438
• Minor unused test code bugfix (nano::test::account_height) by @dsiganos in #4466
• Add work_pool to nano::test::context::ledger_context by @clemahieu in #4494
• Fix pruning tests that did not confirm blocks before pruning. by @clemahieu in #4532
• Fix and re-enable unit test rpc.send_epoch_2 by @dsiganos in #4551
• Fix and re-enable unit test rpc.wallet_history by @dsiganos in #4552
• Fix race condition in test election_scheduler.no_vacancy by @dsiganos in #4570
• Rename to core_test_main to entry for consistency by @pwojcikdev in #4572
• Test daemon interrupt handling by @pwojcikdev in #4563
• Fix a TSAN issue in active_transactions.confirm_new by @dsiganos in #4582
• Fix unit test bootstrap_processor.lazy_pruning_missing_block by @dsiganos in #4575
• Improve unit test telemetry.basic by @dsiganos in #4589
• A system test to exercise the rpc stop command by @dsiganos in #4573
• Fix rpc.block_confirm_confirmed by @clemahieu in #4608
• Directly confirm blocks within unit tests rather than starting/forcing elections by @clemahieu in #4605
• Fix rare race in request_aggregator.one_update by @gr0vity-dev in #4622
• Fix node.block_confirm test by @pwojcikdev in #4666

Miscellaneous improvements

• Remove ledger_walker by @pwojcikdev in #4357
• Remove unused includes by @RickiNano in #4349
• Bound depth of hinted scheduler block traversal by @pwojcikdev in #4359
• Search for receivables as a wallet action, rather than a background node task. by @clemahieu in #4365
• Build fix for thread_runner.cpp with NANO_ASIO_HANDLER_TRACKING enabled by @clemahieu in #4373
• Fix bug/warning about temporary object lifetime by @dsiganos in #4374
• Fix for queuing wallets action which doesn't go through node.wallets by @clemahieu in #4377
• Applying [[noreturn]] attribute to asset_internal by @clemahieu in #4392
• Convenience function for block_or_pruned_exists by @RickiNano in #4393
• Remove cleanup functions related to unchecked blocks. by @clemahieu in #4384
• Improve enum conversions by @pwojcikdev in #4397
• Use make_final_vote function by @RickiNano in #4407
• Set maximum unchecked blocks in config by @RickiNano in #4342
• Use the make_vote function in unit tests by @RickiNano in #4410
• Remove unnecessary unlock/lock of a mutex by @dsiganos in #4416
• Allow overriding of default application path directory by @pwojcikdev in #4418
• Store pointer to boost::asio::signal_set and cancel when ipc_server::stop is called. by @clemahieu in #4367
• Initiating Beta Reset to Address Large Ledger Size by @gr0vity-dev in #4426
• Update the beta genesis hash to the new one, fixes the systests by @dsiganos in #4427
• Link nano_lib with boost::property_tree which is at least referenced by nano/lib/blocks.hpp by @clemahieu in #4432
• Use erase_if pattern by @pwojcikdev in #4429
• Change collect_container_info to be a member of nano::block_processor by @clemahieu in #4434
• Remove overload of ledger::amount that was only used in one test and didn't have a meaningful function. by @clemahieu in #4436
• Include thread header in threading.hpp by @clemahieu in #4439
• Clean up ledger::process return value. by @clemahieu in #4442
• Remove node::rep_block function that was only used in tests. by @clemahieu in #4444
• Documentation and code clarity improvements to work_pool unit tests by @dsiganos in #4445
• Convert ledger::process to take a shared_ptr argument instead of a block reference. by @clemahieu in #4443
• Introduce opencl_work_func_t to make code more readable by @dsiganos in #4446
• Merge v26.1 commits into develop by @dsiganos in #4453
• Use start/stop pattern in vote_processor by @pwojcikdev in #4455
• Remove unused ledger::could_fit by @clemahieu in #4457
• Do not return node_id private key in RPC call by @dsiganos in #4459
• Ledger query simplification by @clemahieu in #4461
• Blocks.hpp recompile time reduction by @clemahieu in #4468
• fix some typos by @cuinix in #4467
• Use start/stop pattern in block_processor by @pwojcikdev in #4462
• Reduce recompile time of ledger.hpp. by @clemahieu in #4475
• Remove pragma from pending_info.cpp by @clemahieu in #4480
• Remove unused function ledger::root_exists by @clemahieu in #4481
• Convert store::pending::get to return an optional rather than using an out-argument and returning an error code. by @clemahieu in #4479
• Fix build error with new optional pending_info by @simpago in #4487
• Add functions to allow pending_key objects to be in containers. by @clemahieu in #4489
• Add operator* to store::iterator. by @clemahieu in #4493
• Adding abstraction of iteration over receivable entries by @clemahieu in #4496
• Replace some additional direct usages of store.pending with calls to ledger::pending_info by @clemahieu in #4501
• Remove final vote canary code as final votes are now always enabled. by @clemahieu in #4502
• Fix active_transactions missed notifications by @clemahieu in #4505
• Block cemented callback simplify by @clemahieu in #4508
• Header reduction for election, vote_generator, vote_spacing, and active_transactions by @clemahieu in #4509
• Return per hash vote result by @pwojcikdev in #4510
• Config option for disabling hinted scheduler by @pwojcikdev in #4512
• Rename block::hash(blake2b_state&) overload to block::generate_hash(blake2b_state&) by @clemahieu in #4514
• Remove debug_assert for a tautology which was causing a warning. uint8_t can't be larger than 256 by @clemahieu in #4517
• Fix missing vote_code enumeration for websockets. by @clemahieu in #4516
• Apply the count parameter in json_handler::accounts_receivable() by @simpago in #4520
• Improve write_database_queue::wait description and add [[nodiscard]] by @clemahieu in #4522
• Reimplement and simplify confirmation_height_processor by @clemahieu in #4519
• Remove frontiers table from ledger by @clemahieu in #4425
• Add missing lib/logger.hpp include. by @clemahieu in #4524
• Fixing shared->unique assignment and no return in control flow. by @clemahieu in #4529
• Encapsulate ledger cache counts by @clemahieu in #4528
• Use ledger::confirm by @clemahieu in #4531
• Removing unconfirmed_frontiers command line option. by @clemahieu in #4530
• RPC endpoint for election statistics by @RickiNano in #4533
• Adding const to ledger::weight* functions. by @clemahieu in #4534
• Move write database queue by @clemahieu in #4535
• Move inactive_node and node_wrapper into their own files by @dsiganos in #4541
• Remove conditional compiling. by @clemahieu in #4544
• map::iterator is a dependent typename and needs the typename keyword. by @clemahieu in #4545
• Fix intermittent failure of request_aggregator.split by @clemahieu in #4546
• Fixes TSAN issues. by @clemahieu in #4547
• Bugfix add_outer_node(), it should be added to disconnected set of nodes by @dsiganos in #4548
• Max election age precision by @RickiNano in #4549
• Documentation for pending key and info classes and some other improvements whilst investigating by @dsiganos in #4553
• Move node::receive_confirmed to wallets::receive_confirmed by @clemahieu in #4557
• Consistent log capitalization by @pwojcikdev in #4561
• Clean up and remove recursive transaction usage by @clemahieu in #4559
• Constexpr check NANO_ASIO_HANDLER_TRACKING by @pwojcikdev in #4562
• Signal manager thread role by @pwojcikdev in #4565
• Fix race conditions in accessing election confirmation counters by @dsiganos in #4564
• Generate indented toml config files for improved legibility by @RickiNano in #4558
• Creating nano::secure::transaction type by @clemahieu in #4543
• Cleaner daemon stopping by @pwojcikdev in #4560
• Start/stop pattern for unchecked_map by @pwojcikdev in #4577
• Feature: disable_activate_successors flag by @dsiganos in #4586
• Async task helper by @pwojcikdev in #4590
• Introduce ledger confirmed_set/any_set classes to separate confirmed-only and confirmed/unconfirmed operations on the ledger by @clemahieu in #4486
• Node configuration auto updater by @RickiNano in #4579
• Style fix, pass transaction as first argument. by @clemahieu in #4592
• Minor fixes by @dsiganos in #4594
• Remove block_process_timeout by @pwojcikdev in #4599
• Build fixes. by @clemahieu in #4602
• Add comment/warning to node::block_confirmed_or_being_confirmed functions by @clemahieu in #4540
• Introduce active_transactions_config by @gr0vity-dev in #4604
• Move recently_cemented_cache and recently_confirmed_cache to their own files. by @clemahieu in #4606
• Fix race condition between election creation and vote_cache triggering by @clemahieu in #4610
• Fixed three compiler warnings (C26498) by @RickiNano in #4617
• Configuration and environment variable improvements by @pwojcikdev in #4613
• Remove active_elections::trim by @clemahieu in #4611
• Scheduler test cleanup by @clemahieu in #4619
• Simplify system.add_node checks by @pwojcikdev in #4614
• Add cli support for disable_max_peers... node flags by @gr0vity-dev in #4620
• Clean up priority scheduler bucket look ups by @clemahieu in #4632
• Move confirmation processing on to worker threads rather than io threads. by @clemahieu in #4634
• Fix build errors converting and comparing to int. by @clemahieu in #4633
• Optimize election scheduler activate function by @pwojcikdev in #4636
• Use deque rather than vector to support large initialization sequences by @clemahieu in #4641
• Add environment variable NANO_MEMORY_INTENSIVE by @clemahieu in #4640
• Optimize active_elections::notify_observers by @pwojcikdev in #4637
• Optimize cemented callbacks by @pwojcikdev in #4642
• Fix rocksdb migration by @RickiNano in #4645
• Bundle ledger transaction and write guard into a single object by @pwojcikdev in #4653
• Increase beta network connection limits by @pwojcikdev in #4657
• Avoid bootstrap notifications from live traffic by @pwojcikdev in #4463
• Limit number of rocksdb background threads by @pwojcikdev in #4660
• Do not wait for cancellation signal by @pwojcikdev in #4664

Full Changelog: V26.1...V27.1