Scripted update to add missing example attributes #400

dabdine · 2022-01-26T21:45:44Z

Description

Adds missing example attributes to <example> elements in recog fingerprint XML files, greatly reducing recog_verify warnings.

This is only round 1. Subsequent rounds will target some hand modifications to target fingerprints missing examples altogether. Additionally, there is one warning that is not fixed with this patch due to it being an edge case: The fingerprint regex conditionally matches a version, but the only example does not supply one. That will be fixed by hand as well in a subsequent PR.

The script that was used to make these changes is here:
https://gist.github.com/dabdine/6c641407e805ab33530ec67459a94c3a

Motivation and Context

Clean up warnings so that more strict enforcement of quality can eventually be turned on.

How Has This Been Tested?

Ran recog_verify

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

I have updated the documentation accordingly (or changes are not required).
I have added tests to cover my changes (or new tests are not required).
All new and existing tests passed.

dabdine · 2022-01-27T09:37:52Z

Fails for golang, works for C/JRuby & Java. The failing cases are because of captured junk text in a service version. I'll tweak the regex to see if I can get the behavior to work consistently across go/ruby/Java.

dabdine · 2022-01-28T08:36:50Z

xml/operating_system.xml

@@ -28,7 +28,7 @@

  <fingerprint pattern="^(?i:(?:Microsoft )?Windows 10 Mobile(?:\s([a-z]+))?(?: Edition)?)$">
    <description>Windows 10 Mobile</description>
-    <example os.product="Windows 10 Mobile">Windows 10 Mobile Edition</example>
+    <example os.product="Windows 10 Mobile" os.edition="Edition">Windows 10 Mobile Edition</example>


This looks buggy--should be "Mobile," but it's a current shipping defect. This patch only makes it obvious since it interpolates the field value from the match groups derived from the example string.

dabdine · 2022-01-28T08:40:15Z

xml/pop_banners.xml

-    <example>Qpopper (version 4.0.3) at domain starting.  &lt;xxx@domain&gt;</example>
+    <example service.version="4.0.16" host.domain="foo.example.com">Qpopper (version 4.0.16) at foo.example.com</example>
+    <example service.version="2.53" host.domain="domain starting.  &lt;xxx@domain&gt;">QPOP (version 2.53) at domain starting.  &lt;xxx@domain&gt;</example>
+    <example service.version="4.0.3" host.domain="domain starting.  &lt;xxx@domain&gt;">Qpopper (version 4.0.3) at domain starting.  &lt;xxx@domain&gt;</example>


Bugged regex

dabdine · 2022-01-28T08:43:15Z

xml/pop_banners.xml

@@ -112,7 +112,7 @@

  <fingerprint pattern="^Microsoft Exchange 2000 POP3 server version (\d+\.\d+\.\d+\.\d+) (.+) ready.$">
    <description>Microsoft Exchange Server 2000</description>
-    <example>Microsoft Exchange 2000 POP3 server version 6.0.6603.0 (host) ready.</example>
+    <example service.version="6.0.6603.0" host.name="(host)">Microsoft Exchange 2000 POP3 server version 6.0.6603.0 (host) ready.</example>


Bugged, no parens should exist in host.name

dabdine · 2022-01-28T15:02:24Z

Regarding the comments about bugs above, I'm simply noting them for some hand mods later. I don't intend to fix those as part of this (automated) patch

mkienow-r7 · 2022-01-28T15:29:30Z

Regarding the comments about bugs above, I'm simply noting them for some hand mods later. I don't intend to fix those as part of this (automated) patch

Do you intend to make those fixes or is that an exercise left to the reader? :)

Is there a reason for not making those corrections as part of this PR or a PR against your PR branch?

dabdine · 2022-01-28T15:45:58Z

@mkienow-r7 i'm open to suggestions! I was going to fix some of these today assuming this patch lands. I think it'd be better to isolate the hand-crafted fixes from this patch though and do it post-merge

tsellers-r7 · 2022-01-28T16:04:31Z

xml/ftp_banners.xml

-    <example hw.product="Q7406">AXIS Q7406 Video Encoder Blade 5.01 (Aug 01 2008) ready.</example>
-    <example hw.product="241Q">AXIS 241Q Video Server 4.47.2 (Dec 11 2008) ready.</example>
-    <example hw.version="5.07.2">AXIS P7701 Video Decoder 5.07.2 (Apr 20 2010) ready.</example>
+    <example hw.product="Q7406" hw.version="Blade">AXIS Q7406 Video Encoder Blade 5.01 (Aug 01 2008) ready.</example>


Marking this for follow up: Incorrect capture by the existing regex.

tsellers-r7 · 2022-01-28T16:07:26Z

xml/hp_pjl_id.xml

+    <example os.product="2200">HP LaserJet 2200</example>
+    <example os.product="4050">LASERJET 4050</example>
+    <example os.product="4 PLUS">LASERJET 4 PLUS</example>
+    <example os.product="Professional P1606dn">HP LaserJet Professional P1606dn</example>


Marking this for follow up PR: This should be hw.product not os.product

tsellers-r7 · 2022-01-28T16:08:03Z

xml/hp_pjl_id.xml

+    <example os.product="designjet 110plus">hp designjet 110plus</example>
+    <example os.product="DESIGNJET 1050C">DESIGNJET 1050C</example>
+    <example os.product="DESIGNJET 1055CM">DESIGNJET 1055CM</example>
+    <example os.product="DESIGNJET 700">DESIGNJET 700</example>


Marking this for follow up PR: This should be hw.product not os.product

Similar elsewhere in this file.

tsellers-r7 · 2022-01-28T16:21:19Z

xml/snmp_sysdescr.xml

-    <example>3COM VCX Connect 200</example>
-    <example>3COM VCX Server</example>
-    <example>3COM VCX V7005</example>
+    <example os.product="VCX Connect 100">3COM VCX Connect 100</example>


Marking this for follow up PR: Should be hw.product
Similar below.

tsellers-r7 · 2022-01-28T16:24:18Z

xml/snmp_sysdescr.xml

-    <example>CX400 - Flare 2.19.0.400.5.040</example>
-    <example>CX500 - Flare 2.07.0.500.5.020</example>
-    <example>CX700 - Flare 2.24.0.700.5.022</example>
+    <example os.product="CX3-10c" os.version="3.26.0.10.5.032">CX3-10c - Flare 3.26.0.10.5.032</example>


Marking this for follow up PR: Should be hw.product

Ugh, sooooooo much of this.

mkienow-r7 · 2022-01-28T16:25:47Z

@dabdine It seems I had things a little mixed up from the verify workflow failure the other day. I agree, and I'm fine with collaborative post-merge corrections.

mkienow-r7 · 2022-01-28T20:06:24Z

It's really nice to see the warning count drop dramatically. Thank you for the contribution @dabdine!

Scripted update to add missing example attributes

5c280f4

Fix regular expression to work with go, ruby and java regex

2680933

dabdine commented Jan 28, 2022

View reviewed changes

tsellers-r7 reviewed Jan 28, 2022

View reviewed changes

mkienow-r7 merged commit f32f3a4 into rapid7:master Jan 28, 2022

This was referenced Jan 28, 2022

Fix additional warnings due to untested attributes #404

Merged

Fix examples in architecture.xml, ftp_banners.xml, and ssh_banners.xml #406

Merged

mkienow-r7 added a commit to mkienow-r7/recog that referenced this pull request Feb 23, 2022

Fix regex capture issues pointed out in rapid7#400

94d35e3

mkienow-r7 mentioned this pull request Feb 23, 2022

Fix regex capture issues pointed out in #400 #425

Merged

3 tasks

mkienow-r7 added a commit to mkienow-r7/recog that referenced this pull request Feb 25, 2022

Fix regex capture issues pointed out in rapid7#400

11ba27a

mkienow-r7 added a commit that referenced this pull request Feb 26, 2022

Fix regex capture issues mentioned in #400 (#425)

762acd7

mkienow-r7 mentioned this pull request Nov 9, 2022

Correct fingerprint hardware parameters #510

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scripted update to add missing example attributes #400

Scripted update to add missing example attributes #400

dabdine commented Jan 26, 2022 •

edited

Loading

dabdine commented Jan 27, 2022 •

edited

Loading

dabdine Jan 28, 2022 •

edited

Loading

dabdine Jan 28, 2022

dabdine Jan 28, 2022

dabdine commented Jan 28, 2022

mkienow-r7 commented Jan 28, 2022

dabdine commented Jan 28, 2022

tsellers-r7 Jan 28, 2022

tsellers-r7 Jan 28, 2022

tsellers-r7 Jan 28, 2022 •

edited

Loading

tsellers-r7 Jan 28, 2022 •

edited

Loading

tsellers-r7 Jan 28, 2022

mkienow-r7 commented Jan 28, 2022 •

edited

Loading

mkienow-r7 commented Jan 28, 2022

Scripted update to add missing example attributes #400

Scripted update to add missing example attributes #400

Conversation

dabdine commented Jan 26, 2022 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

dabdine commented Jan 27, 2022 • edited Loading

dabdine Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

dabdine Jan 28, 2022

Choose a reason for hiding this comment

dabdine Jan 28, 2022

Choose a reason for hiding this comment

dabdine commented Jan 28, 2022

mkienow-r7 commented Jan 28, 2022

dabdine commented Jan 28, 2022

tsellers-r7 Jan 28, 2022

Choose a reason for hiding this comment

tsellers-r7 Jan 28, 2022

Choose a reason for hiding this comment

tsellers-r7 Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

tsellers-r7 Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

tsellers-r7 Jan 28, 2022

Choose a reason for hiding this comment

mkienow-r7 commented Jan 28, 2022 • edited Loading

mkienow-r7 commented Jan 28, 2022

dabdine commented Jan 26, 2022 •

edited

Loading

dabdine commented Jan 27, 2022 •

edited

Loading

dabdine Jan 28, 2022 •

edited

Loading

tsellers-r7 Jan 28, 2022 •

edited

Loading

tsellers-r7 Jan 28, 2022 •

edited

Loading

mkienow-r7 commented Jan 28, 2022 •

edited

Loading