Create a periodic prow job to implement KEP-3203

[nehaLohia27]

Related to kubernetes/enhancements#3203

Create a periodic prow job to query GitHub API for new CVEs announced (every 5 minutes).
If new CVE announced, push the new content tok/sig-security/sig-security-tooling/feeds/official-cve-feed.json

This prow job will call the shell/golang/python script to query the github apis and populate the json with the below data.

The sample json format will be as :-

[
        {
            "cve_id": "CVE-2021-25741",
            "cve_url": "https://www.cvedetails.com/cve-details.php?t=1&cve_id=CVE-2021-25741",
            "number": "104980",
            "summary": "Symlink Exchange Can Allow Host Filesystem Access",
            "issue_url": "https://github.com/kubernetes/kubernetes/issues/104980",
            "google_group_url": "https://groups.google.com/g/kubernetes-announce/search?q=CVE-2021-25741"
        },
        {
            "cve_id": "CVE-2020-8565",
            "cve_url": "https://www.cvedetails.com/cve-details.php?t=1&cve_id=CVE-2020-8565",
            "number": "95623",
            "summary": "Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel >= 9",
            "issue_url": "https://github.com/kubernetes/kubernetes/issues/95623",
            "google_group_url": "https://groups.google.com/g/kubernetes-announce/search?q=CVE-2020-8565"
        }
]

[nehaLohia27]

/sig security

[nehaLohia27]

/sig testing

[nehaLohia27]

/assign

[PushkarJ]

With the new flow, we may need to create a prow job that pushes to gcs-bucket instead of Github repo. We can use the same prow job to also call the build-hook for k/website. As a pre-cursor, I think it might be worth working with @sftim to trigger the build hook with the relevant secrets as POC on your laptop or some other playground machine to ensure it works as expected.

I will work on requesting a gcs-bucket in parallel in the meantime

/retitle Create a periodic prow job to implement KEP-3203

[nehaLohia27]

@PushkarJ Thanks. The above flow sounds good. @sftim I would like to play with the build hook part . Will need some help on this. Let me go through the docs first and then we can connect for any queries.

[PushkarJ]

Bucket request has been made here kubernetes/k8s.io#3494

[nehaLohia27]

@PushkarJ @sftim I have finished the python script to create the json blob using github apis.. Should I create one draft PR in this repo with the python script ? Also I think I will start working on creating the prow job to write to gcs bucket. @PushkarJ I see you have posted references of how to write to gcs bucket using prow in this issue #33. I can work with @rajaskakodkar to complete this.. Also when we will get the gcs bucket access any idea.?

[nehaLohia27]

Also the location of prow job can be the same as synk scan .? We can create new yaml in the same location or update existing yaml also https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-k8s-infra/trusted/sig-security-trusted.yaml

[PushkarJ]

@nehaLohia27 yes draft PR is always welcome! Looks like two things need to be implemented next in prow job:

1. Push to GCS bucket the JSON blob generated from Github API
2. Trigger k/website build and add json blob dynamically blob dynamically like here: kubernetes/website@main...PushkarJ:generate-content

For GCS bucket, we may need to wait for it to get provisioned for a few days, but maybe its worth doing PoC for step 2 and testing the external build for now ?

@sftim for option 2, is this a good example for how to do it: https://github.com/kubernetes/website/blob/main/.github/workflows/netlify-periodic-build.yml ? If yes, what's the best way to get the TOKEN variable in the file?

[nehaLohia27]

Sure . Will work on doing the POC for external build for step2.

Also @PushkarJ I think we cannot create the draft PR in this repo because we are not going to consume the script from this repo..May be I will work on prow job yaml and create the draft PR in test-infra. Should we create one separate prow job yaml under this location or something else ?
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-k8s-infra/trusted/sig-security-trusted.yaml

For now posting the script below for any comments. Let me know any feedback on this.

import json
import requests

url = 'https://api.github.com/search/issues?q=is:issue+CVSS:3+in:body+label:committee/security-response+state:closed+repo:kubernetes/kubernetes'
headers = {'Authorization': 'token xxxxxxx', 'Accept': 'application/vnd.github.v3+json'}
res = requests.get(url, headers=headers)
cve_arr = res.json()

cve_list = []

for item in cve_arr['items']:
    cve = {"issue_url": None, "number": None, "cve_id": None, "summary": None, "cve_url": None, "google_group_url": None}
    cve['issue_url'] = item['html_url']
    cve['number'] = item['number']
    title=item['title'].split(":")
    if len(title) == 1:
        cve_id=None
        cve['cve_id'] = None
        cve['cve_url'] = None
        cve['summary'] = item['title'].split(":")[0]
        cve['google_group_url'] = None
    else:
        cve_id=item['title'].split(":")[0]
        cve['cve_id'] = item['title'].split(":")[0]
        cve['summary'] = item['title'].split(":")[1]
        cve['cve_url'] = f"https://www.cvedetails.com/cve-details.php?t=1&cve_id={cve_id}"
        cve['google_group_url'] = f"https://groups.google.com/g/kubernetes-announce/search?q={cve_id}"
    cve_list.append(cve)
    
cves = json.dumps(cve_list, sort_keys=True, indent=4)
print(cves)

# write the final cves to official_cve_feed.json
with open("official_cve_feed.json", "w") as cvejson:
    cvejson.write(cves)

Also the link to official_cve_feed.json from my github gists is updated. This is just a temporary URL.
https://gist.githubusercontent.com/nehaLohia27/ffc2c57f0e32ab338d9f2a02b2fc9e7c/raw/2dfe97c65bc8e4a145867d8175ad5b144e00c4cd/issues

@sftim Can you also take a look at the script above?

[PushkarJ]

May be I will work on prow job yaml and create the draft PR in test-infra. Should we create one separate prow job yaml under this location or something else ?

Yes that makes perfect sense to do as next step!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nehaLohia27]

/remove-lifecycle stale

[PushkarJ]

Infra k/test-infra PR(s):
kubernetes/test-infra#26896
kubernetes/test-infra#26988
kubernetes/test-infra#26990
Security k/sig-security PR(s):
#55
#57

Fixed by above PRs
/close

[k8s-ci-robot]

@PushkarJ: Closing this issue.

In response to this:

Infra k/test-infra PR(s):
kubernetes/test-infra#26896
kubernetes/test-infra#26988
kubernetes/test-infra#26990
Security k/sig-security PR(s):
#55
#57

Fixed by above PRs
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.