adding RN106 and RN107 #4643

dantavori · 2024-11-04T13:06:58Z

Related Issues

fixes: https://jira-dc.paloaltonetworks.com/browse/CIAC-12004

Description

NGINXApiModule -> should raise RN106 for ExportIndicators pack and RN107 for TAXII Server 2 integration, but not EDL which has a RN
SplunkPy - should raise RN107 (RN exists but not entry) due to Splunk Generic playbook change
MacOS - should raise RN106 due to modeling rule schema change
CortexXDR - should raise RN106 due to integration description change
Mattermost - should raise RN106 due to integration script change
Tripwire - only TPB was changed so shouldn't raise a validation error
Note that in case of RN106, RN107 won't be raised
ServiceNow_v2 - has a RN entry so no validation error
ignored files which shouldn't raise validation errors: Packs/EWS/CONTRIBUTORS.json, Packs/Gem/Author_image.png, Packs/Whois/.pack-ignore, Packs/Slack/Integrations/SlackV3/SlackV3_test.py

Packs/SplunkPy/Playbooks/playbook-Splunk_Generic.yml: [RN107] - No release note entry was found for the playbook "Splunk Generic" in the SplunkPy pack. Please rerun the update-release-notes command without -u to generate an updated template. If you are trying to exclude an item from the release notes, please refer to the documentation found here - https://xsoar.pan.dev/docs/integrations/changelog#excluding-items
Packs/TAXIIServer/Integrations/TAXII2Server/TAXII2Server.yml: [RN107] - No release note entry was found for the integration "TAXII2 Server" in the TAXIIServer pack. Please rerun the update-release-notes command without -u to generate an updated template. If you are trying to exclude an item from the release notes, please refer to the documentation found here - https://xsoar.pan.dev/docs/integrations/changelog#excluding-items
Packs/MacOS: [RN106] - Release notes were not found. Please run `demisto-sdk update-release-notes -i Packs/MacOS -u (major|minor|revision|documentation)` to generate release notes according to the new standard. You can refer to the documentation found here: https://xsoar.pan.dev/docs/integrations/changelog for more information.
Packs/CortexXDR: [RN106] - Release notes were not found. Please run `demisto-sdk update-release-notes -i Packs/CortexXDR -u (major|minor|revision|documentation)` to generate release notes according to the new standard. You can refer to the documentation found here: https://xsoar.pan.dev/docs/integrations/changelog for more information.
Packs/Mattermost: [RN106] - Release notes were not found. Please run `demisto-sdk update-release-notes -i Packs/Mattermost -u (major|minor|revision|documentation)` to generate release notes according to the new standard. You can refer to the documentation found here: https://xsoar.pan.dev/docs/integrations/changelog for more information.
Packs/ExportIndicators: [RN106] - Release notes were not found. Please run `demisto-sdk update-release-notes -i Packs/ApiModules -u (major|minor|revision|documentation)` to generate release notes according to the new standard. You can refer to the documentation found here: https://xsoar.pan.dev/docs/integrations/changelog for more information.

new validate job:
https://gitlab.xdr.pan.local/xdr/cortex-content/content/-/jobs/10466680
old validate job: https://gitlab.xdr.pan.local/xdr/cortex-content/content/-/jobs/10429679

for some reason, old validate raised RN106 for SplunkPy (instead of RN107).

github-actions · 2024-11-04T13:24:21Z

Changelog(s) in markdown:

Moved RN106 to the new validate format. The validation ensures there are no missing release notes. #4643

github-actions · 2024-11-12T17:41:13Z

Changelog(s) in markdown:

Moved RN106 and RN107 to the new validate format. These validations ensure there are no missing release notes. #4643

github-actions · 2024-11-13T13:44:57Z

Changelog(s) in markdown:

Moved RN106 and RN107 to the new validate format. These validations ensure there are no missing release notes. #4643
Fixed an issue where the git statuses of content objects' related files were not calculated correctly. #4643

dantavori · 2024-11-13T13:26:16Z

demisto_sdk/commands/common/git_util.py

+    def _check_file_status(
+        self,
+        file_path: str,
+        remote: str,
+        branch: str,
+        feature_branch_or_hash: Optional[str] = None,
+    ) -> str:
        """Get the git status of a given file path
        Args:
            file_path (str): the file path to check
            remote (str): the used git remote
            branch (str): the used git branch
+            feature_branch_or_hash (str | None): compare against a specific branch or commit
        Returns:
            str: the git status of the file (M, A, R, D).
        """
-        current_branch_or_hash = self.get_current_git_branch_or_hash()
+        if not feature_branch_or_hash:
+            feature_branch_or_hash = self.get_current_git_branch_or_hash()

        if remote:
            diff_line = self.repo.git.diff(
                "--name-status",
-                f"{remote}/{branch}...{current_branch_or_hash}",
+                f"{remote}/{branch}...{feature_branch_or_hash}",
                "--",
                file_path,
            )

        # if remote does not exist we are checking against the commit sha1
        else:
            diff_line = self.repo.git.diff(
-                "--name-status", f"{branch}...{current_branch_or_hash}", "--", file_path
+                "--name-status", f"{branch}...{feature_branch_or_hash}", "--", file_path


to be able to get related files' git statuses per specific commits (for our use-case: current branch / master), I allowed passing a specific commit

dantavori · 2024-11-13T13:27:21Z

demisto_sdk/commands/common/tools.py

+def get_pack_latest_rn_version(pack_path: str, git_sha: Optional[str] = None) -> str:
    """
    Extract all the Release notes from the pack and return the highest version of release note in the Pack.

    Return:
        (str): The lastest version of RN.
    """
-    list_of_files = glob.glob(pack_path + "/ReleaseNotes/*")
+    if git_sha:
+        git_util = GitUtil.from_content_path()
+        list_of_files = git_util.list_files_in_dir(f"{pack_path}/ReleaseNotes", git_sha)
+    else:
+        list_of_files = glob.glob(f"{pack_path}/ReleaseNotes/*")


this change makes the Pack's latest_rn_version field correct per commit

dantavori · 2024-11-13T13:29:15Z

demisto_sdk/commands/content_graph/objects/pack.py

+            git_sha=self.git_sha,
+            prev_ver=self.old_base_content_object.git_sha
+            if self.old_base_content_object
+            else None,


now we can compare these versions to get the actual git status. we should do it for other related files.

Can you open a ticket for that?

dantavori · 2024-11-13T13:52:51Z

TestSuite/pack.py

I had two different tests with a pack named ApiModule and the test failed on FileExistsError

YuvHayun

Amazing job!
Approved with minor comments.

YuvHayun · 2024-11-14T11:19:03Z

demisto_sdk/commands/validate/initializer.py

+        git_util = GitUtil.from_content_path()
+        current_git_sha = git_util.get_current_commit_hash()


Not sure I fully understand all the change here, but are we sure it won't affect the way we load changed files?

renaming git_sha arg to prev_ver just to avoid confusion between prev and current versions

passed current_git_sha to the object's git_sha. from what I saw, the git_sha attribute is used only in:
a) related_files.py (which is what I fixed)
b) integration and script parsers to get the code - if git_sha not passed, it uses the current working dir (equivalent to current git sha), so allegedly nothing was changed.

passed prev_ver to old object's git_sha.

With 2+3, we can calculate the git_status of related files correctly for each object (see changes in related_files.py and git_utils.py)

demisto_sdk/commands/validate/tools.py

demisto_sdk/commands/validate/tests/RN_validators_test.py

dantavori added 2 commits November 4, 2024 15:06

adding RN106

d53b848

skip new packs, added changelog

9e84337

dantavori and others added 17 commits November 4, 2024 15:27

fix cond

47fb1cc

fixes

67c76e0

improved git status

af6c08f

added tests, wip

36d2fe7

Merge branch 'master' into convert_RN106

2993b34

test fix wip

bb2db2c

test fixes

9bd3b6b

run only rn106

7ac78e1

test log

8e0da4a

logs

831de66

fixes wip

e88a3c6

revert git status changes

0856552

warning for debugging

7069cc8

fixes

34f7e0b

fixes

87110d1

fixes

8f3216e

rn107

e4e54ab

dantavori changed the title ~~adding RN106~~ adding RN106 and RN107 Nov 12, 2024

dantavori added 4 commits November 12, 2024 20:07

missing filetype

790b8a8

fixes and tests

19d9e52

move common code to tools and fix test

f4285b0

fix mocker

29cbec9

dantavori marked this pull request as ready for review November 13, 2024 15:21

dantavori requested a review from RosenbergYehuda as a code owner November 13, 2024 15:21

dantavori commented Nov 13, 2024

View reviewed changes

dantavori requested a review from YuvHayun November 13, 2024 15:21

fix api module

1bf49b5

YuvHayun approved these changes Nov 14, 2024

View reviewed changes

dantavori and others added 2 commits November 19, 2024 10:59

cr notes

f801084

Merge branch 'master' into convert_RN106

beb2b8b

dantavori merged commit febe16b into master Nov 19, 2024
25 checks passed

dantavori deleted the convert_RN106 branch November 19, 2024 09:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding RN106 and RN107 #4643

adding RN106 and RN107 #4643

dantavori commented Nov 4, 2024 •

edited

Loading

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 12, 2024

github-actions bot commented Nov 13, 2024

dantavori Nov 13, 2024

dantavori Nov 13, 2024

dantavori Nov 13, 2024

YuvHayun Nov 14, 2024

dantavori Nov 13, 2024

YuvHayun left a comment

YuvHayun Nov 14, 2024

dantavori Nov 14, 2024

		git_util = GitUtil.from_content_path()
		current_git_sha = git_util.get_current_commit_hash()

adding RN106 and RN107 #4643

adding RN106 and RN107 #4643

Conversation

dantavori commented Nov 4, 2024 • edited Loading

Related Issues

Description

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 12, 2024

github-actions bot commented Nov 13, 2024

dantavori Nov 13, 2024

Choose a reason for hiding this comment

dantavori Nov 13, 2024

Choose a reason for hiding this comment

dantavori Nov 13, 2024

Choose a reason for hiding this comment

YuvHayun Nov 14, 2024

Choose a reason for hiding this comment

dantavori Nov 13, 2024

Choose a reason for hiding this comment

YuvHayun left a comment

Choose a reason for hiding this comment

YuvHayun Nov 14, 2024

Choose a reason for hiding this comment

dantavori Nov 14, 2024

Choose a reason for hiding this comment

dantavori commented Nov 4, 2024 •

edited

Loading