STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication #2408

zuo · 2023-09-13T17:29:16Z

This is a bunch of updates, fixes and enhancements related to integration with STOMP-based services, especially with the n6's one.

The most important change is related to the upcoming switch of authentication to the n6 Stream API: from client certificate-based to login-and-passcode-based. Other changes focus on compatibility with newer versions of the stomp.py library as well as on minor fixes/cleanups/improvements (mostly consistency-related).

The key changes concern the STOMP collector bot and the STOMP output bot:

Each of those two bots obtained three new config parameters -- necessary to accommodate to the incoming changes regarding the n6 Stream API's authentication mechanism, i.e., switching from the legacy client-certificate-based one to a new STOMP-login-and-passcode-based one. The new config parameters are:
- auth_by_ssl_client_certificate: if true (the default), the legacy certificate-based authentication will be attempted (so ssl_client_certificate and ssl_client_certificate_key need to be specified); if false, the new login-based authentication will be attempted (so the config parameters described below need to be specified);
- ~~stomp_login~~ username: a STOMP login (in the case of n6 it is just the user's login, the same which you use to log in to n6 Portal);
- ~~stomp_passcode~~ password: a STOMP passcode (in the case of n6 it is the user's API key which can be obtained via n6 Portal).
From now on, newer versions of the stomp.py library are supported, including the latest (8.1.0).

See also: the commit messages.

sebix · 2023-09-13T18:04:01Z

docs/user/bots.rst

 * `ssl_ca_certificate`: path to CA file
-* `ssl_client_certificate`: path to client cert file
-* `ssl_client_certificate_key`: path to client cert key file
+auth_by_ssl_client_certificate`: Boolean, default: true (note: set to false for new *n6* auth)


Suggested change

auth_by_ssl_client_certificate`: Boolean, default: true (note: set to false for new *n6* auth)

* `auth_by_ssl_client_certificate`: Boolean, default: true (note: set to false for new *n6* auth)

Note: now I added also CHANGELOG.md.

Now: a few corrections in CHANGELOG.md.

Now (among others): more corrections/additions in CHANGELOG.md.

sebix · 2023-09-13T18:04:39Z

docs/user/bots.rst

+* `stomp_login`: STOMP login (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false
+* `stomp_passcode`: STOMP passcode (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false


what about calling them username and password?

I used these names because login and passcode are STOMP-(protocol)-specific terms/keywords.

But if you think username and password will be better here, I will change them; it's not a problem. :-)

So, what is your final decision, @sebix?

@kamil-certat @gethvi what do you think?

username/password would be consistent with the parameters of other bots and independent of the underlying protocol.
login/passcode is in line with the specific protocol

I tend towards consistency and username/password.

OK, done. :-)

I would use usernam/password or login/passcode (stomp prefix doesn't look like necessary when we are speaking about Stomp bot :))

zuo · 2023-09-16T07:29:52Z

@sebix Hm, the only failing test does not seem to be related to this PR:

=========================== short test summary info ============================
FAILED intelmq/tests/bots/experts/reverse_dns/test_expert.py::TestReverseDnsExpertBot::test_invalid_ptr2 - AssertionError: {'__type': 'Event', 'source.ip': '5.157.80.[47 chars].nl'} != {'source.ip': '5.157.80.221', '__type': 'Event'}
+ {'__type': 'Event', 'source.ip': '5.157.80.221'}
- {'__type': 'Event',
-  'source.ip': '5.157.80.221',
-  'source.reverse_dns': 'aliancys.peopleinc.nl'}
====== 1 failed, 1374 passed, 236 skipped, 3 xfailed in 175.49s (0:02:55) ======

Maybe re-run could help?...

sebix · 2023-09-16T07:38:16Z

Maybe re-run could help?...

Yes, it did

kamil-certat

Looks very well, just a small suggestion. Do you have any date when the transition from certs to credentials occurs?

kamil-certat · 2023-09-19T13:57:09Z

intelmq/bots/collectors/stomp/collector.py


        # check if certificates exist
-        for f in [self.ssl_ca_cert, self.ssl_cl_cert, self.ssl_cl_cert_key]:
+        for f in ssl_params.values():


As the ssl_cs_certtificate is still important, maybe let's add the check if the file exists in the check method as well? It is used during verification if IntelMQ is properly configured, see e.g.:

intelmq/intelmq/bots/outputs/file/output.py

Lines 92 to 104 in 85c215d

@staticmethod

def check(parameters):

if 'file' not in parameters:

return [["error", "Parameter 'file' not given."]]

dirname = os.path.dirname(parameters['file'])

if not os.path.exists(dirname) and '{ev' not in dirname:

path = Path(dirname)

try:

path.mkdir(mode=0o755, parents=True, exist_ok=True)

except OSError:

return [["error", "Directory (%r) of parameter 'file' does not exist and could not be created." % dirname]]

else:

return [["info", "Directory (%r) of parameter 'file' did not exist, but has now been created." % dirname]]

OK, I will add such checks, together with some refactoring (to reduce code duplication...).

Great, thanks!

Now I just added (among others) those checks.

kamil-certat · 2023-09-19T14:00:10Z

docs/user/bots.rst

+* `stomp_login`: STOMP login (e.g., *n6* user login), used only if `auth_by_ssl_client_certificate` is false
+* `stomp_passcode`: STOMP passcode (e.g., *n6* user API key), used only if `auth_by_ssl_client_certificate` is false


I would use usernam/password or login/passcode (stomp prefix doesn't look like necessary when we are speaking about Stomp bot :))

zuo · 2023-09-20T19:22:00Z

Do you have any date when the transition from certs to credentials occurs?

Soon -- in a few days -- CERT.pl will contact all interested parties (i.e., organizations that connect to the n6 Stream API), in particular (obviously) CERT.at, conveying necessary information on the upcoming auth mechanism change.

As far as I know, it is planned to turn off the old mechanism (client-certificate-based auth) around mid-November.

kamil-certat · 2023-09-21T09:11:55Z

Thanks for the info!

zuo · 2023-09-21T20:00:42Z

The latest version is a refactored one + with additional changes. In particular, now both bots implement the check() hook, other checks have been added/improved, some auto-reconnect-related fixes have been made... For more details, see the commit messages and changelog entries.

zuo · 2023-09-22T23:53:52Z

A few tweaks more... Now, it should be ready. :-) (I hope...)

zuo · 2023-09-25T12:54:04Z

@kamil-certat @sebix Sorry for the fuss with the recent forced pushes... Now the stuff is ready.

The updates, fixes and improvements regard the *STOMP collector* and *STOMP output* bots. Important changes are described below... From now on, newer versions of the `stomp.py` package are supported -- including the latest (8.1.0). Now both STOMP bots coerce the `port` configuration parameter to int -- so that a string representing an integer number is also acceptable (even if not recommended) as a value of that parameter. In the *STOMP output* bot, a bug has been fixed: `AttributeError` caused by attempts to get unset attributes (`ssl_ca_cert` and companions...). The *STOMP collector*'s reconnection mechanism has been fixed: from now on, no reconnection attempts are made after `shutdown()`. Apart from that, reconnection is not attempted at all for versions of `stomp.py` older than 4.1.21 (as it did not work properly anyway). Also regarding the *STOMP collector* bot, the following (undocumented and unused) attributes of `StompCollectorBot` instances are no longer set in `init()`: `ssl_ca_cert`, `ssl_cl_cert`, `ssl_cl_cert_key`. Various checks have been improved/enhanced. Now, for example, both STOMP bot classes implement the `check()` static/class method -- whose role is to check ("statically", without the need to run the bot) configuration parameters; in particular, it checks whether necessary certificate files are accessible. When it comes to runtime (on-initialization) checks, one notable improvement is that now also the *STOMP output* bot will raise a `MissingDependencyError` if the `stomp.py` version is older than 4.1.8 (an analogous check has already been implemented by *STOMP collector*). The code of those bot classes have also been significantly refactored -- in particular, several common operations have been factored out and placed in a new mix-in class: `intelmq.lib.mixins.StompMixin`; its definition resides in a new module: `intelmq.lib.mixins.stomp`.

Each of the *STOMP collector* and *STOMP output* bots obtained the following new configuration parameters: * `auth_by_ssl_client_certificate` (a Boolean flag; it is `True` by default -- to keep backward compatibility); * `username` and `password` -- to be used as STOMP authentication credentials (login and passcode), but *only* if the aforementioned parameter `auth_by_ssl_client_certificate` is `False`. If `auth_by_ssl_client_certificate` is `False`, then the (supported also previously...) `ssl_client_certificate` and `ssl_client_certificate_key` parameters are ignored (i.e., not only left unused, but also there are *no checks* whether the files they refer to actually exist).

The changes include also those regarding *feeds* (values of certain properties of the CERT.PL's "N6 Stomp Stream" feed entry have been updated/improved) and the *changelog*.

zuo · 2023-09-27T09:52:49Z

Now just rebased on the latest develop, no real changes made.

zuo mentioned this pull request Sep 13, 2023

Stomp collector failed #2342

Closed

sebix reviewed Sep 13, 2023

View reviewed changes

zuo force-pushed the jk/stomp-and-n6-related-updates-fixes branch 3 times, most recently from 25ca767 to d4fee3e Compare September 14, 2023 20:08

sebix added the component: bots label Sep 14, 2023

sebix added this to the 3.2.2 milestone Sep 14, 2023

kamil-certat approved these changes Sep 19, 2023

View reviewed changes

zuo force-pushed the jk/stomp-and-n6-related-updates-fixes branch from d4fee3e to a247811 Compare September 21, 2023 19:55

zuo requested review from kamil-certat and sebix September 21, 2023 20:01

zuo force-pushed the jk/stomp-and-n6-related-updates-fixes branch from a247811 to bfc6d4f Compare September 22, 2023 23:52

zuo changed the title ~~STOMP-and-n6-related updates and fixes, especially adding login-based authentication~~ STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication Sep 23, 2023

zuo force-pushed the jk/stomp-and-n6-related-updates-fixes branch 2 times, most recently from e752d58 to 93ab34b Compare September 25, 2023 12:52

sebix approved these changes Sep 26, 2023

View reviewed changes

zuo added 3 commits September 27, 2023 11:52

doc: add/update/improve stuff related to STOMP bots and *n6*

7c59d49

The changes include also those regarding *feeds* (values of certain properties of the CERT.PL's "N6 Stomp Stream" feed entry have been updated/improved) and the *changelog*.

zuo force-pushed the jk/stomp-and-n6-related-updates-fixes branch from eb8e0b4 to 7c59d49 Compare September 27, 2023 09:52

kamil-certat approved these changes Sep 27, 2023

View reviewed changes

sebix merged commit b0eaac6 into certtools:develop Sep 27, 2023
24 checks passed

zuo mentioned this pull request Oct 10, 2023

Stomp Collector failed #2364

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication #2408

STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication #2408

zuo commented Sep 13, 2023 •

edited

Loading

sebix Sep 13, 2023

zuo Sep 13, 2023

zuo Sep 13, 2023

zuo Sep 14, 2023

zuo Sep 21, 2023

sebix Sep 13, 2023

zuo Sep 13, 2023 •

edited

Loading

zuo Sep 14, 2023

sebix Sep 14, 2023

zuo Sep 14, 2023

kamil-certat Sep 19, 2023

zuo commented Sep 16, 2023

sebix commented Sep 16, 2023

kamil-certat left a comment

kamil-certat Sep 19, 2023

zuo Sep 20, 2023

kamil-certat Sep 21, 2023

zuo Sep 21, 2023

kamil-certat Sep 19, 2023

zuo commented Sep 20, 2023 •

edited

Loading

kamil-certat commented Sep 21, 2023

zuo commented Sep 21, 2023

zuo commented Sep 22, 2023

zuo commented Sep 25, 2023

zuo commented Sep 27, 2023

	auth_by_ssl_client_certificate`: Boolean, default: true (note: set to false for new n6 auth)
	* `auth_by_ssl_client_certificate`: Boolean, default: true (note: set to false for new n6 auth)

		* `stomp_login`: STOMP login (e.g., n6 user login), used only if `auth_by_ssl_client_certificate` is false
		* `stomp_passcode`: STOMP passcode (e.g., n6 user API key), used only if `auth_by_ssl_client_certificate` is false

	@staticmethod
	def check(parameters):
	if 'file' not in parameters:
	return [["error", "Parameter 'file' not given."]]
	dirname = os.path.dirname(parameters['file'])
	if not os.path.exists(dirname) and '{ev' not in dirname:
	path = Path(dirname)
	try:
	path.mkdir(mode=0o755, parents=True, exist_ok=True)
	except OSError:
	return [["error", "Directory (%r) of parameter 'file' does not exist and could not be created." % dirname]]
	else:
	return [["info", "Directory (%r) of parameter 'file' did not exist, but has now been created." % dirname]]

STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication #2408

STOMP-and-n6-related updates, fixes and enhancements, especially adding login-based authentication #2408

Conversation

zuo commented Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuo Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuo commented Sep 16, 2023

sebix commented Sep 16, 2023

kamil-certat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuo commented Sep 20, 2023 • edited Loading

kamil-certat commented Sep 21, 2023

zuo commented Sep 21, 2023

zuo commented Sep 22, 2023

zuo commented Sep 25, 2023

zuo commented Sep 27, 2023

zuo commented Sep 13, 2023 •

edited

Loading

zuo Sep 13, 2023 •

edited

Loading

zuo commented Sep 20, 2023 •

edited

Loading