Overhaul of the Logging Infrastructure #1596

Dimi1010 · 2024-09-30T14:48:20Z

Overhaul of the Logging Infrastructure

Changes

Breaking changes: LogLevel has been updated to an enum class and moved to the pcpp namespace.
- Logger retains aliases to the new enum.
- Impact: May cause issues for users performing arithmetic operations on LogLevel, but this is not expected to be a major concern.
Logging Improvements:
- Fixed incorrect or missing LogModule values in the LoggingModule enum.
- Introduces helper classes to organize logging data:
  - LogSource: Encapsulates file, function, line number and module information.
  - LogContext: Encapsulates the context of a single log message.
API Enhancements:
- Formalized internal methods of Logger class into public methods with documentation.
  - Replaced internalCreateLogStream with createLogContext.
  - Replaced internalPrintLogMessage with emit.
- Added helper log* functions to Logger class to allow logging, through runtime methods.
- Standardized all PCPP_LOG_* macros to pass through PCPP_LOG.
Performance Optimizations:
- Introduced an optional object pooling mechanism for LogContext to reduce allocation overhead.
- Added a generic ObjectPool<T> class to facilitate pooling for reusable objects.
- The optimization is enabled by default and can be disabled by calling useContextPooling(false) on the Logger instance.
Compile-Time Logging Exclusions:
- Added support for excluding log messages defined through PCPP_LOG macros at compile time via PCPP_ACTIVE_LOG_LEVEL preprocessor variable.
Code Quality Enhancements:
- Moved ostream& operator<< overloads for IPAddress and MacAddress inside pcpp to satisfy ADL (Argument-Dependent Lookup).

LogContext Pooling Optimization

When enabled, the pooling optimization introduces the following behavior:

Allocation on createLogContext:
- When createLogContext is called, the pool is queried for an available LogContext.
- If no available LogContext is found, a new one is created immediately.
Releasing on emit:
- When the emit method with a unique_ptr<LogContext> parameter is called, the LogContext is returned to the pool after the operation completes.
- If the pool has not reached its maximum retention size, the LogContext is stored for reuse.
- If the pool exceeds its maximum retention size, the LogContext is deleted.

Optimization Benchmarks

Performance benchmarks highlight the impact of enabling the pooling optimization:

Windows 10 - MSVC x64 (Release Build)

Warmup Messages: 5,000
Benchmarked Messages: 300,000
Results:
- Pooling Enabled: 104,387,800 ns
- Pooling Disabled: 179,670,900 ns

WSL (Ubuntu 22.04) - GCC x64 (Release Build)

Warmup Messages: 5,000
Benchmarked Messages: 300,000
Results:
- Pooling Enabled: 64,135,976 ns
- Pooling Disabled: 125,265,749 ns

Notes

The pooling optimization offers significant performance gains by reducing the cost of frequent allocations and deallocations.
Benchmarks suggest up to ~42% improvement on Windows and ~48% improvement on Linux.
While the changes to LogLevel and other enums may introduce minor compatibility issues, they align with modern C++ best practices.

…ace to satisfy ADL requirements.

- LogLevel is now a top level enum class. - Added a new log level Off to disable output from a specific module. - Logger::LogLevel is a deprecated alias to LogLevel. - Logger::Info, Debug, Error are deprecated aliases to LogLevel::... - Removed public "internal*" functions from Logger.Logger - Added LogSource struct to encapsulate source information. - Added shouldLog method to check if a log should be emitted for a given level and module. - Removed nonfunctional artifacts "m_LogStream" and "Logger::operator<<" - Added templated "log" functions that are friends to Logger. - Reworked PCPP_LOG macros to no longer utilize the now removed internal functions. - Added PCPP_LOG_INFO macro level. - Changed PCPP_LOG_ERROR to now check if the log should be emitted. - Fixed NetworkUtils log module name overlapping with NetworkUtils class. - Fixed missing enum value for PacketLogModuleSll2Layer.

codecov · 2024-09-30T15:28:49Z

Codecov Report

Attention: Patch coverage is 83.33333% with 27 lines in your changes missing coverage. Please review.

Project coverage is 83.12%. Comparing base (3929558) to head (bc82caf).

Files with missing lines	Patch %	Lines
Common++/src/Logger.cpp	69.44%	10 Missing and 1 partial ⚠️
Common++/header/ObjectPool.h	75.60%	7 Missing and 3 partials ⚠️
Common++/header/Logger.h	88.09%	5 Missing ⚠️
Pcap++/src/PcapLiveDeviceList.cpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1596      +/-   ##
==========================================
- Coverage   83.15%   83.12%   -0.03%     
==========================================
  Files         277      278       +1     
  Lines       48196    48307     +111     
  Branches     9991    10085      +94     
==========================================
+ Hits        40078    40157      +79     
+ Misses       7225     7040     -185     
- Partials      893     1110     +217

Flag	Coverage Δ
fedora40	`75.15% <81.25%> (-0.02%)`	⬇️
macos-13	`80.63% <71.42%> (-0.03%)`	⬇️
macos-14	`80.63% <71.42%> (-0.02%)`	⬇️
macos-15	`80.60% <71.21%> (-0.03%)`	⬇️
mingw32	`70.83% <75.53%> (-0.09%)`	⬇️
mingw64	`70.79% <75.53%> (-0.09%)`	⬇️
npcap	`85.08% <56.45%> (-0.23%)`	⬇️
rhel94	`74.98% <82.29%> (-0.04%)`	⬇️
ubuntu2004	`58.57% <76.34%> (-0.03%)`	⬇️
ubuntu2004-zstd	`58.70% <76.34%> (-0.02%)`	⬇️
ubuntu2204	`74.89% <82.29%> (-0.05%)`	⬇️
ubuntu2204-icpx	`61.21% <62.50%> (-0.19%)`	⬇️
ubuntu2404	`75.16% <82.10%> (-0.07%)`	⬇️
unittest	`83.12% <83.33%> (-0.03%)`	⬇️
windows-2019	`85.12% <56.45%> (-0.23%)`	⬇️
windows-2022	`85.15% <56.45%> (-0.22%)`	⬇️
winpcap	`85.11% <56.45%> (-0.21%)`	⬇️
xdp	`50.41% <20.83%> (-0.09%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

seladb · 2024-10-14T02:28:51Z

Common++/header/Logger.h

+			std::ostringstream sstream;                                                                                \
+			sstream << message;                                                                                        \


Can you please check if this change affects the PcapPlusPlus binary sizes?

I made a change in this PR a couple of years ago to significantly reduce the binary sizes: #748

This is a breakdown on the install directory for the x64 release w/npcap.

Folder1 is the new size. Folder2 is the old size.

File Folder1Size Folder2Size AbsoluteDelta RelativeDelta ---- ----------- ----------- ------------- ------------- bin\Arping.exe 714240 639488 74752 10.47% bin\ArpSpoofing.exe 703488 631296 72192 10.26% bin\benchmark.exe 599552 562688 36864 6.15% bin\DisplayLiveDevices.exe 30208 30208 0 0.00% bin\DNSResolver.exe 748032 664064 83968 11.23% bin\DnsSpoofing.exe 747520 671232 76288 10.21% bin\HttpAnalyzer.exe 832512 743936 88576 10.64% bin\IcmpFileTransfer-catcher.exe 730624 658432 72192 9.88% bin\IcmpFileTransfer-pitcher.exe 742400 666112 76288 10.28% bin\IPDefragUtil.exe 711680 655360 56320 7.91% bin\IPFragUtil.exe 693248 637952 55296 7.98% bin\PcapPrinter.exe 658432 612352 46080 7.00% bin\PcapSearch.exe 661504 618496 43008 6.50% bin\PcapSplitter.exe 720384 667648 52736 7.32% bin\SSLAnalyzer.exe 822272 733184 89088 10.83% bin\TcpReassembly.exe 799232 710144 89088 11.15% bin\TLSFingerprinting.exe 796672 712704 83968 10.54% lib\Common++.lib 5045102 4887832 157270 3.12% lib\Packet++.lib 19849740 18454316 1395424 7.03% lib\Pcap++.lib 3620658 2896688 723970 20.00%

Full breakdown: npcap-x64-release.txt

Did you test it with MSVC or MinGW? Also, did you have a chance to test on Linux also?

I don't remember in which platform the impact was the highest, but I think it was Linux 🤔

Msvc, I dont think I have mingw setup. I should be able to test the sizes on Unix via WSL.

@Dimi1010 I see you're still working on it. Please let me know when the changes are done and when you want me to review the PR again.
It'd be good to make sure binary sizes don't increase. I'd test at least on Windows and Linux

@seladb Ok, ended up keeping the optimization.

As bc82caf: Release builds ended up being:

Windows

Static Libraries

Common++.lib - 4.7MB -> 5MB

Packet++.lib - 17.6MB -> 19.7MB

Pcap++.lib - 2.8MB -> 3.3MB

Examples

Same binary size or <=10KB increase.

Linux (WSL)

Static Libraries

libCommon++.a - 408KB -> 444KB

libPakcet++.a - 3.1MB -> 3.3MB

libPcap++.a - 676KB -> 748KB

Examples

On average ~0.1MB increase on each binary.

This reverts commit be95ab8.

- Added optional compile time elimination of log calls below set level.

- Renamed printLogMessage to emit and changed visibility to public. - Added new class LogContext to encapsulate a single emittable log message. - Added new methods createLogContext which is practically a rework of internalCreateLogStream but returns a LogContext. - Added optional use of object pooling optimization for reusing log contexts. (Enabled via preprocessor flag PCPP_LOG_USE_OBJECT_POOL)

… minimum log level set to too high severity..

…ntext pooling. - Removed preprocessor variable PCPP_LOG_USE_OBJECT_POOL. - Disabled context pooling for the unit tests as it interferes with the memory leak checker.

… time minimum log level set to too high severity..

- Added documentation to new methods and classes. - Addressed explicit constructor warnings. - Moved disable of context pooling for unit tests to the main.cpp files of the respective tests.

Dimi1010 added 9 commits September 20, 2024 21:40

Pulled ostream operators for IPAddress and MacAddress into the namesp…

0293115

…ace to satisfy ADL requirements.

Merge remote-tracking branch 'upstream/dev' into refactor/log-templates

dfb98c5

Cleanup and fixes.

9062c48

Added the new Off log level to the string conversion.

1e856f0

Fixed wrong variable name.

02618fc

Added documentation to log source.

32ff8f9

Lint.

12492d2

Fixed docstring for LogSource.

f21f85d

Dimi1010 force-pushed the refactor/log-templates branch from bd53c74 to f21f85d Compare September 30, 2024 15:09

Fixed extra /

0faf2e8

Dimi1010 added 2 commits October 7, 2024 02:03

Merge remote-tracking branch 'upstream/dev' into refactor/log-templates

ae72e81

Fixed explicit warning.

43d7b1c

Dimi1010 marked this pull request as ready for review October 7, 2024 09:50

Dimi1010 requested a review from seladb as a code owner October 7, 2024 09:50

Dimi1010 requested review from clementperon, tigercosmos and egecetin October 7, 2024 09:50

Dimi1010 added breaking change Pull request contains a breaking change to the public API. refactoring labels Oct 7, 2024

seladb reviewed Oct 14, 2024

View reviewed changes

Dimi1010 added 8 commits October 14, 2024 16:38

Merge remote-tracking branch 'upstream/dev' into refactor/log-templates

87042f5

Moved log functions inside logger.

be95ab8

Revert "Moved log functions inside logger."

4b21a99

This reverts commit be95ab8.

Merge remote-tracking branch 'upstream/dev' into refactor/log-templates

00034e9

Moved the log functions to the Logger class.

b2055c0

- Added optional compile time elimination of log calls below set level.

Merge branch 'dev' into refactor/log-templates

cfc3d56

Fixed typo in macro names.

ac2a383

Changed value param to const-ref.

3776653

Dimi1010 added 3 commits November 14, 2024 18:50

Added "venv" and "./out" to ignored directories by codespell.

76079a7

Fixed warnings about unreferenced local variables if the compile time…

c71dd27

… minimum log level set to too high severity..

Dimi1010 marked this pull request as draft November 14, 2024 20:25

Dimi1010 added 7 commits November 14, 2024 22:37

Removed useless variable.

803c98a

Fixed friend class definition.

9d44f76

Fixed variable assignment.

af63a9b

Added method useContextPooling to control if the logger should use co…

02e717a

…ntext pooling. - Removed preprocessor variable PCPP_LOG_USE_OBJECT_POOL. - Disabled context pooling for the unit tests as it interferes with the memory leak checker.

Fixed more warnings about unreferenced local variables if the compile…

3410231

… time minimum log level set to too high severity..

Addressed warnings and documentation.

379c074

- Added documentation to new methods and classes. - Addressed explicit constructor warnings. - Moved disable of context pooling for unit tests to the main.cpp files of the respective tests.

Fixed include.

bc82caf

Dimi1010 changed the title ~~Logger overhaul.~~ Overhaul of the Logging Infrastructure Nov 16, 2024

Dimi1010 marked this pull request as ready for review November 16, 2024 18:38

Dimi1010 requested a review from seladb November 16, 2024 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul of the Logging Infrastructure #1596

Overhaul of the Logging Infrastructure #1596

Dimi1010 commented Sep 30, 2024 •

edited

Loading

codecov bot commented Sep 30, 2024 •

edited

Loading

seladb Oct 14, 2024

Dimi1010 Oct 14, 2024 •

edited

Loading

seladb Oct 15, 2024

Dimi1010 Oct 15, 2024 •

edited

Loading

seladb Oct 17, 2024

Dimi1010 Nov 16, 2024

Overhaul of the Logging Infrastructure #1596

Are you sure you want to change the base?

Overhaul of the Logging Infrastructure #1596

Conversation

Dimi1010 commented Sep 30, 2024 • edited Loading

Overhaul of the Logging Infrastructure

Changes

LogContext Pooling Optimization

Optimization Benchmarks

Windows 10 - MSVC x64 (Release Build)

WSL (Ubuntu 22.04) - GCC x64 (Release Build)

Notes

codecov bot commented Sep 30, 2024 • edited Loading

Codecov Report

seladb Oct 14, 2024

Choose a reason for hiding this comment

Dimi1010 Oct 14, 2024 • edited Loading

Choose a reason for hiding this comment

seladb Oct 15, 2024

Choose a reason for hiding this comment

Dimi1010 Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

seladb Oct 17, 2024

Choose a reason for hiding this comment

Dimi1010 Nov 16, 2024

Choose a reason for hiding this comment

Dimi1010 commented Sep 30, 2024 •

edited

Loading

codecov bot commented Sep 30, 2024 •

edited

Loading

Dimi1010 Oct 14, 2024 •

edited

Loading

Dimi1010 Oct 15, 2024 •

edited

Loading