feat: additional data extraction

[weliasz]

Description

Implemented extraction of additional information from metric name suffix based on regular expression defined per OID famil

Type of change

• New feature

How Has This Been Tested?

Placed regexes for TCP-MIB and UDP-MIB and checked in Splunk if were extracted properly

Checklist

• My commit message is conventional
• I have run pre-commit on all files before creating the PR
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have checked my code and corrected any misspellings

[codecov-commenter]

Codecov Report

Merging #177 (f7451d6) into develop (42cb2c3) will increase coverage by 0.95%.
The diff coverage is 95.23%.

@@             Coverage Diff             @@
##           develop     #177      +/-   ##
===========================================
+ Coverage    49.57%   50.52%   +0.95%     
===========================================
  Files           22       22              
  Lines         1410     1425      +15     
===========================================
+ Hits           699      720      +21     
+ Misses         711      705       -6     

Impacted Files	Coverage Δ
...lunk_connect_for_snmp_poller/manager/hec_sender.py	72.58% <95.23%> (+9.27%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42cb2c3...f7451d6. Read the comment docs.

[github-actions]

license-eye has totally checked 265 files.

Valid	Invalid	Ignored	Fixed
52	1	212	0

Click to see the invalid file list

• tests/test_additional_data_extraction.py

[github-actions]

Expected release notes (by @omrozowicz-splunk)

features:
additional data extraction (0e0263f)
don't put empty values and remove trailing index numbers from m… (#173) (e6a2f5b)
linear backoff used while waiting for MIB (#172) (b3c6f0e)
just as for events time for metrics is assigned on SC4SNMP (#165) (1874618)
add check for missing varBinds (#166) (58587cd)
issue a warning when assigned profile is missing (#162) (96b7f8d)
force inventory refresh (#149) (8026248)
add additionalvarbinds to enricher (#126) (98aeaf3)
enrich data with linked tables (#101) (75bdb13)
extract enricher data from config (#97) (123864f)
extract IF-MIB data from an SNMPWALK (#96) (7699591)
support polling the same IP address using multiple profiles (#92) (a6320ca)

fixes:
delete duplicated license (f7451d6)
build fix (15f258e)
build fix (40765c5)
build fix (11a037b)
build fix (c298f58)
build fix (9aee0a4)
build fix (0e756e1)
build fix (ac0c3c9)
build fix (49d8d7d)
missing profiles check (#174) (b26f5c9)
add check for if walk (#171) (461f245)
warning if profile does not exist (#170) (d8e14dc)
fix issue with onetime walk not running (#168) (490fff6)
fix issue with KeyError metric, double _ids in mongo and problem with new hosts added (#167) (588cb2e)
profiles are now loaded from file every time (#164) (823ff11)
new execution of enricher (#163) (b646e19)
onetime walk retry (#158) (d69a9c7)
removal of unsafe eval call (#157) (843c26a)
bring back previous changes (#161) (0e87f17)
implement new way for enricher to work (8e24841)
onetime walk with enricher (#154) (11a42ca)
matching on both dysDescr and sysObjectId (#156) (15158e8)
default event formatting (#155) (ed96f44)
fetch frequency (#152) (4602304)
allow * in profile (#151) (880dbce)
changed order of validation (#147) (8dd65e4)
add some restrictions for getting profiles from mib server (#145) (3c690a5)
add missing "break" (#144) (9b728d6)
error with no snmp received before timeout (#143) (b50480f)
'get_snmp_data() missing 1 required positional argument: 'additional_metric_fields'' and change error logger to debug (494ecbb)
change github.repository_owner to github.actor (#129) (89a3fb5)
add toJson() method to onetime_task (#127) (04b16f7)
update splunk indexes with meta_index (#116) (6d3f1bc)
changed indexes (#114) (9644663)
change way of checking if profile is an OID to use regex (#112) (277a050)
mongo real time data check (#107) (4526b0c)
change the way of creating host name which is an id in the database (#104) (bbf2733)
forced commit for creating new release (f3081c1)

others (will not be included in Semantic-Release notes):
59a17d2 Update tests/test_additional_data_extraction.py
pre-commit autoupdate (#175) (42cb2c3)
0f0e621 [ADDON-43861] correct sourcetype for walk
92e091e Fix/default freq dynamic (#160)
456f6db [ADDON-43460] retry and fail if mib unavailable
048ba1d [ADDON-43435] don't remove host if other record uses it
861a40e [ADDON-43362] special builder for error message
bf2293d Feature/dynamic profile (#138)
adding pull request template (7314352)
better exception handling for mib request (a41fab9)
1.5.0-develop.9 (51d7678)
5a8a418 added 2 missing variables
0c2faf1 encapsulated hec sending request + test + correct pycov coverage report
803ff1a added sending error messages to hec isntead of ignoring them
remove local actionlint (3cc058e)
8ea42f0 added missing error messages
c23399b fixing logging level
1.5.0-develop.8 (8e7cc3d)
change image path (7167c4c)
edf4fc9 adding frequency to all metric data and profile if it is set in conf
4dd2c28 Feat/async rework (#122)
disable shellcheck SC1090 (734d523)
add actionlint as pre-commit step (52ac6f8)
allow dependabot[bot] and remove custom allow policy (437745f)
pre-commit autoupdate [ci skip] (e0544a4)
7ef1a09 refactor
remove black from dev dependency (1b07dd8)
add pre-commit step and remove lint step (9ef6611)
update severity threshold for snyk to "high" (#113) (7540a5b)
pre-commit autoupdate [ci skip] (6ff7b00)
pre-commit and debug statements [ci skip] (bb5e0ff)
set threshold for jscdp linter (#105) (c01fb4d)
1.5.0-develop.3 (7306e5d)
fix of duplicated code linter problem (c855088)
fix shellcheck (2fa415a)
apply mypy and flake8 (ec32122)
apply pre-commit isort (9c906f9)
exclude json files from copyright check (f3d6b2d)
update README to include pre-commit (cd6a9eb)
apply pre-commit black (c53b8b6)
pre-commit (9a24536)
33acbbf feat: Replace multiple SNMP walks with one SNMP Bulk (#100)
ebe3ebf Feature/redo walk (#95)
d5879c5 Stubs for lint
9cff184 Stubs for lint
adddbea Stubs for lint
5dc1549 Stubs for lint
2c5a5c3 Feat/multiple profiles2 (#93)

• I hereby acknowledge these release notes are 🥙 AWESOME 🥙

[ryanfaircloth]

🎉 This PR is included in version 1.5.0-develop.36 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀