Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate NVD importer to aboutcode pipeline #1587

Merged
merged 3 commits into from
Sep 27, 2024
Merged

Conversation

keshav-space
Copy link
Member

@keshav-space keshav-space self-assigned this Sep 11, 2024
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@TG1999 TG1999 added 1-next and removed 0-next labels Sep 27, 2024
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
Signed-off-by: Keshav Priyadarshi <git@keshav.space>
@keshav-space
Copy link
Member Author

Import log from a clean run.

python3 manage.py import nvd_importer
INFO 2024-09-27 19:47:58.696 Pipeline [NVDImporterPipeline] starting
INFO 2024-09-27 19:47:58.696 Step [collect_and_store_advisories] starting
INFO 2024-09-27 19:48:00.292 Collecting 264,063 advisories
INFO 2024-09-27 19:48:00.293 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.json.gz`
INFO 2024-09-27 19:48:18.226 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.json.gz`
INFO 2024-09-27 19:48:23.767 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz`
INFO 2024-09-27 19:48:31.574 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz`
INFO 2024-09-27 19:48:45.962 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz`
INFO 2024-09-27 19:49:07.016 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.json.gz`
INFO 2024-09-27 19:49:24.757 Progress: 10% (26407/264063) ETA: 760 seconds (12.7 minutes)
INFO 2024-09-27 19:49:28.764 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2008.json.gz`
INFO 2024-09-27 19:49:49.508 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.json.gz`
INFO 2024-09-27 19:50:08.944 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.json.gz`
INFO 2024-09-27 19:50:26.638 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.json.gz`
INFO 2024-09-27 19:50:45.825 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.json.gz`
INFO 2024-09-27 19:50:59.168 Progress: 20% (52813/264063) ETA: 716 seconds (11.9 minutes)
INFO 2024-09-27 19:51:05.487 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz`
INFO 2024-09-27 19:51:26.084 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.json.gz`
INFO 2024-09-27 19:51:50.088 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.json.gz`
INFO 2024-09-27 19:52:10.179 Progress: 30% (79219/264063) ETA: 583 seconds (9.7 minutes)
INFO 2024-09-27 19:52:11.737 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2016.json.gz`
INFO 2024-09-27 19:52:38.310 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.json.gz`
INFO 2024-09-27 19:53:14.678 Progress: 40% (105626/264063) ETA: 472 seconds (7.9 minutes)
INFO 2024-09-27 19:53:17.808 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2018.json.gz`
INFO 2024-09-27 19:53:59.035 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz`
INFO 2024-09-27 19:54:26.547 Progress: 50% (132032/264063) ETA: 386 seconds (6.4 minutes)
INFO 2024-09-27 19:54:40.599 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.json.gz`
INFO 2024-09-27 19:55:26.447 Progress: 60% (158438/264063) ETA: 297 seconds (5.0 minutes)
INFO 2024-09-27 19:55:32.114 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2021.json.gz`
INFO 2024-09-27 19:56:21.949 Progress: 70% (184845/264063) ETA: 215 seconds (3.6 minutes)
INFO 2024-09-27 19:56:22.700 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2022.json.gz`
INFO 2024-09-27 19:57:22.294 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2023.json.gz`
INFO 2024-09-27 19:57:48.767 Progress: 80% (211251/264063) ETA: 147 seconds (2.5 minutes)
INFO 2024-09-27 19:58:24.242 Progress: 90% (237657/264063) ETA: 69 seconds (1.1 minutes)
INFO 2024-09-27 19:58:26.356 Fetching `https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2024.json.gz`
INFO 2024-09-27 19:59:18.636 Successfully collected 261,761 advisories
INFO 2024-09-27 19:59:18.636 Step [collect_and_store_advisories] completed in 680 seconds (11.3 minutes)

Copy link
Contributor

@TG1999 TG1999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@keshav-space keshav-space merged commit 021b568 into main Sep 27, 2024
9 checks passed
@keshav-space keshav-space deleted the nvd-importer-pipeline branch September 27, 2024 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

2 participants