Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: wrong insertion location in changelog #13

Merged
merged 16 commits into from
Sep 10, 2024
Merged

fix: wrong insertion location in changelog #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
861dcc5
fix: updating old ci.cd folder
wphyojpl Sep 9, 2024
0468c16
fix: updating github actions
wphyojpl Sep 9, 2024
fafc1b2
fix: feature release update
wphyojpl Sep 9, 2024
e939f32
Merge branch 'develop' of https://github.com/unity-sds/mdps-ds-lib in…
wphyojpl Sep 10, 2024
6220c00
fix: valid pypi version
wphyojpl Sep 10, 2024
68ebf6d
fix: install twine
wphyojpl Sep 10, 2024
16b73b8
fix: add twine
wphyojpl Sep 10, 2024
fec2e49
fix: wrong string format
wphyojpl Sep 10, 2024
6455002
Merge branch 'develop' into sample-feature
wphyojpl Sep 10, 2024
0842519
fix: typo in version
wphyojpl Sep 10, 2024
fae34d9
Merge branch 'sample-feature' of https://github.com/unity-sds/mdps-ds…
wphyojpl Sep 10, 2024
2d41738
Merge branch 'develop' into sample-feature
wphyojpl Sep 10, 2024
f413ac3
Merge branch 'develop' of https://github.com/unity-sds/mdps-ds-lib in…
wphyojpl Sep 10, 2024
2522330
feat: inject at correct place in CHANGELOG
wphyojpl Sep 10, 2024
a3c82ee
Merge branch 'sample-feature' of https://github.com/unity-sds/mdps-ds…
wphyojpl Sep 10, 2024
1be0f51
chore: add example file
wphyojpl Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ci/update_setup_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ def update_change_log(self):
]
with open(change_log_path, 'r') as change_log_file:
change_logs = change_log_file.read().splitlines()
pattern = r"## \[\d+\.\d+\.\d+\] - \d{4}-\d{2}-\d{2}"
pattern = r"## \[\d+\.\d+\.\d+.*\] - \d{4}-\d{2}-\d{2}"

inserting_line = 0
for i, each_line in enumerate(change_logs):
Expand Down
303 changes: 303 additions & 0 deletions examples/stage-in-out-via-jupyter.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,303 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "279d1d9b-35cf-411e-803c-c287b1e7dbb1",
"metadata": {},
"source": [
"## Performing Stage-In programatically via Python NoteBook.\n",
"### Steps\n",
"- Download and install uds library from https://pypi.org/project/mdps-ds-lib/\n",
"- Set Log Level\n",
"- Create environment variables\n",
"- Call stage-in class to see them downloaded. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "313d285a-1700-470e-9c63-b242119eb58a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting mdps-ds-lib\n",
" Using cached mdps_ds_lib-0.1.0-py3-none-any.whl.metadata (1.3 kB)\n",
"Requirement already satisfied: boto3<2.0.0,>=1.26.51 in /opt/conda/lib/python3.11/site-packages (from mdps-ds-lib) (1.34.34)\n",
"Collecting elasticsearch==7.13.4 (from mdps-ds-lib)\n",
" Using cached elasticsearch-7.13.4-py2.py3-none-any.whl.metadata (7.7 kB)\n",
"Collecting fastjsonschema<3.0.0,>=2.19.1 (from mdps-ds-lib)\n",
" Using cached fastjsonschema-2.20.0-py3-none-any.whl.metadata (2.1 kB)\n",
"Collecting jsonschema<5.0.0,>=4.23.0 (from mdps-ds-lib)\n",
" Using cached jsonschema-4.23.0-py3-none-any.whl.metadata (7.9 kB)\n",
"Requirement already satisfied: pystac==1.9.0 in /opt/conda/lib/python3.11/site-packages (from mdps-ds-lib) (1.9.0)\n",
"Requirement already satisfied: requests==2.31.0 in /opt/conda/lib/python3.11/site-packages (from mdps-ds-lib) (2.31.0)\n",
"Collecting requests-aws4auth==1.2.3 (from mdps-ds-lib)\n",
" Using cached requests_aws4auth-1.2.3-py2.py3-none-any.whl.metadata (18 kB)\n",
"Requirement already satisfied: tenacity==8.2.3 in /opt/conda/lib/python3.11/site-packages (from mdps-ds-lib) (8.2.3)\n",
"Collecting xmltodict==0.13.0 (from mdps-ds-lib)\n",
" Using cached xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB)\n",
"Requirement already satisfied: urllib3<2,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from elasticsearch==7.13.4->mdps-ds-lib) (1.26.18)\n",
"Requirement already satisfied: certifi in /opt/conda/lib/python3.11/site-packages (from elasticsearch==7.13.4->mdps-ds-lib) (2024.2.2)\n",
"Requirement already satisfied: python-dateutil>=2.7.0 in /opt/conda/lib/python3.11/site-packages (from pystac==1.9.0->mdps-ds-lib) (2.8.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests==2.31.0->mdps-ds-lib) (3.3.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests==2.31.0->mdps-ds-lib) (3.4)\n",
"Requirement already satisfied: six in /opt/conda/lib/python3.11/site-packages (from requests-aws4auth==1.2.3->mdps-ds-lib) (1.16.0)\n",
"Requirement already satisfied: botocore<1.35.0,>=1.34.34 in /opt/conda/lib/python3.11/site-packages (from boto3<2.0.0,>=1.26.51->mdps-ds-lib) (1.34.34)\n",
"Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.11/site-packages (from boto3<2.0.0,>=1.26.51->mdps-ds-lib) (1.0.1)\n",
"Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /opt/conda/lib/python3.11/site-packages (from boto3<2.0.0,>=1.26.51->mdps-ds-lib) (0.10.0)\n",
"Requirement already satisfied: attrs>=22.2.0 in /opt/conda/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.23.0->mdps-ds-lib) (23.1.0)\n",
"Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.23.0->mdps-ds-lib) (2023.7.1)\n",
"Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.23.0->mdps-ds-lib) (0.30.2)\n",
"Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.23.0->mdps-ds-lib) (0.10.6)\n",
"Using cached mdps_ds_lib-0.1.0-py3-none-any.whl (73 kB)\n",
"Using cached elasticsearch-7.13.4-py2.py3-none-any.whl (356 kB)\n",
"Using cached requests_aws4auth-1.2.3-py2.py3-none-any.whl (24 kB)\n",
"Using cached xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)\n",
"Using cached fastjsonschema-2.20.0-py3-none-any.whl (23 kB)\n",
"Using cached jsonschema-4.23.0-py3-none-any.whl (88 kB)\n",
"Installing collected packages: fastjsonschema, xmltodict, elasticsearch, requests-aws4auth, jsonschema, mdps-ds-lib\n",
" Attempting uninstall: fastjsonschema\n",
" Found existing installation: fastjsonschema 2.18.1\n",
" Uninstalling fastjsonschema-2.18.1:\n",
" Successfully uninstalled fastjsonschema-2.18.1\n",
" Attempting uninstall: jsonschema\n",
" Found existing installation: jsonschema 4.19.1\n",
" Uninstalling jsonschema-4.19.1:\n",
" Successfully uninstalled jsonschema-4.19.1\n",
"Successfully installed elasticsearch-7.13.4 fastjsonschema-2.20.0 jsonschema-4.23.0 mdps-ds-lib-0.1.0 requests-aws4auth-1.2.3 xmltodict-0.13.0\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install mdps-ds-lib"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "bc93f96b-6fd7-4fbe-9dfb-409f8ecd710e",
"metadata": {},
"outputs": [],
"source": [
"import logging\n",
"logging.basicConfig(level=30, format=\"%(asctime)s [%(levelname)s] [%(name)s::%(lineno)d] %(message)s\")\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cf77b209-db34-4af2-ac4a-663a33c211e2",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"os.environ['AWS_ACCESS_KEY_ID'] = 'ddd'\n",
"os.environ['AWS_SECRET_ACCESS_KEY'] = 'ddd'\n",
"os.environ['AWS_SESSION_TOKEN'] = 'ddd'\n",
"\n",
"\n",
"os.environ['EDL_USERNAME'] = '/unity/uds/user/abcd/edl_username' # Parameter Store Key for EarthData Login Username\n",
"os.environ['EDL_PASSWORD'] = '/unity/uds/user/abcd/edl_pass' # Parameter Store Key for EarthData Login Password\n",
"os.environ['EDL_PASSWORD_TYPE'] = 'PARAM_STORE' # Can hardcode it to PARAM_STORE if that is used.\n",
"os.environ['EDL_BASE_URL'] = 'urs.earthdata.nasa.gov' # Earthdata Login URL to get Earthdata token to download files\n",
"os.environ['GRANULES_DOWNLOAD_TYPE'] = 'DAAC' # Download type to choose download class. DAAC, HTTP, and so on.\n",
"os.environ['DOWNLOADING_KEYS'] = 'data,metadata' # Which asset keys to download\n",
"\n",
"os.environ['STAC_JSON'] = 'https://raw.githubusercontent.com/GodwinShen/emit-ghg/main/test/catalog.json' # URL to direct which granules + assets to download\n",
"# os.environ['STAC_JSON'] = os.path.join(os.getcwd(), 'stage_in.json') # Alternatively, you can store the file locally, and point it as a path\n",
"os.environ['DOWNLOAD_DIR'] = os.path.join(os.getcwd(), 'downloaded_files') # Base directory where files will be downloaded\n",
"os.environ['OUTPUT_FILE'] = os.path.join(os.getcwd(), 'stage_in_result.json') # file path where the result is written locally for review"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "5017392a-bc34-46f8-b557-b4ae5fed38d8",
"metadata": {},
"outputs": [],
"source": [
"from mdps_ds_lib.lib.utils.file_utils import FileUtils\n",
"FileUtils.mk_dir_p(os.environ['DOWNLOAD_DIR']) # Creating a base directory if not created. They can be created manually w/o calling this. \n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "4a43641f-60a9-49fe-a47c-fd3b58acbea5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"done\n"
]
}
],
"source": [
"from mdps_ds_lib.stage_in_out.download_granules_factory import DownloadGranulesFactory\n",
"from mdps_ds_lib.stage_in_out.stage_in_out_utils import StageInOutUtils\n",
"\n",
"# Hardcoded method call. All params are set via environment previously\n",
"result_str = DownloadGranulesFactory().get_class(os.getenv('GRANULES_DOWNLOAD_TYPE', 'MISSING_GRANULES_DOWNLOAD_TYPE')).download()\n",
"StageInOutUtils.write_output_to_file(result_str)\n",
"print('done')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "7555fd58-139b-4cc9-bbed-19018b93bec9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['/home/jovyan/downloaded_files/G2721220118-LPCLOUD.xml', '/home/jovyan/downloaded_files/EMIT_L1B_RAD_001_20230620T084426_2317106_011.nc', '/home/jovyan/downloaded_files/EMIT_L2A_RFL_001_20230620T084426_2317106_011.nc', '/home/jovyan/downloaded_files/downloaded_feature_collection.json', '/home/jovyan/downloaded_files/G2721699381-LPCLOUD.xml']\n"
]
}
],
"source": [
"from glob import glob\n",
"print(list(glob(os.path.join(os.environ['DOWNLOAD_DIR'], '*')))) # Checking if files are downloaded. "
]
},
{
"cell_type": "markdown",
"id": "10b9e21e-939e-4d04-9559-253c7e4d2ea6",
"metadata": {},
"source": [
"## Performing Stage-out programatically as auxiliary files\n",
"#### Steps\n",
"- Create environment variables\n",
"- Call stage-in class to see them uploaded."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "ff40e26a-2dc4-4ba3-8d12-f771737ed03b",
"metadata": {},
"outputs": [],
"source": [
"os.environ['GRANULES_UPLOAD_TYPE'] = 'UPLOAD_AUXILIARY_FILE_AS_GRANULE' # Setting uploading as auxiliary\n",
"os.environ['STAGING_BUCKET'] = 'uds-test-cumulus-internal' # S3 bucket where they will reside\n",
"os.environ['VERIFY_SSL'] = 'FALSE' # Optional param.\n",
"os.environ['RESULT_PATH_PREFIX'] = 'stage_out' # Optional. prefix to store the result in S3 which will trigger auto catalog.\n",
"os.environ['PARALLEL_COUNT'] = '1' # How many threads are used to upload it\n",
"\n",
"os.environ['OUTPUT_DIRECTORY'] = os.path.join(os.getcwd(), 'output_result_dir') # the success / failure results to be stored locally for rewview\n",
"os.environ['BASE_DIRECTORY'] = os.path.join(os.getcwd(), 'downloaded_files') # Base folder to upload\n",
"os.environ['OUTPUT_FILE'] = os.path.join(os.getcwd(), 'stage_out_result.json') # file path where the result overview is written locally for review"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "41e92f64-e540-4fd9-8d0e-57e5176e25d2",
"metadata": {},
"outputs": [],
"source": [
"tenant = 'UDS_DEMO'\n",
"tenant_venue = 'TEST' \n",
"collection_name = 'UDS_UNIT_COLLECTION'\n",
"collection_version = '24.09.10.11.00'.replace('.', '') \n",
"temp_collection_id = f'URN:NASA:UNITY:{tenant}:{tenant_venue}:{collection_name}___{collection_version}'\n",
"os.environ['COLLECTION_ID'] = temp_collection_id # Setting Collection ID\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "e650999a-73f6-4361-825f-3b001c08109d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"done\n"
]
}
],
"source": [
"from mdps_ds_lib.stage_in_out.upoad_granules_factory import UploadGranulesFactory\n",
"from mdps_ds_lib.stage_in_out.stage_in_out_utils import StageInOutUtils\n",
"\n",
"FileUtils.mk_dir_p(os.environ['OUTPUT_DIRECTORY'])\n",
"\n",
"# Hardcoded method call. All params are set via environment previously\n",
"upload_result_str = UploadGranulesFactory().get_class(os.getenv('GRANULES_UPLOAD_TYPE', UploadGranulesFactory.UPLOAD_S3_BY_STAC_CATALOG)).upload()\n",
"StageInOutUtils.write_output_to_file(upload_result_str)\n",
"print('done')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "be1a00d7-c56b-401a-94f2-dfcc576bfdb0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:EMIT_L1B_RAD_001_20230620T084426_2317106_011/EMIT_L1B_RAD_001_20230620T084426_2317106_011.nc', 1852557979)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:EMIT_L1B_RAD_001_20230620T084426_2317106_011/EMIT_L1B_RAD_001_20230620T084426_2317106_011.nc.stac.json', 1724)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:EMIT_L2A_RFL_001_20230620T084426_2317106_011/EMIT_L2A_RFL_001_20230620T084426_2317106_011.nc', 1851092294)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:EMIT_L2A_RFL_001_20230620T084426_2317106_011/EMIT_L2A_RFL_001_20230620T084426_2317106_011.nc.stac.json', 1724)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:G2721220118-LPCLOUD/G2721220118-LPCLOUD.xml', 6119)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:G2721220118-LPCLOUD/G2721220118-LPCLOUD.xml.stac.json', 1567)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:G2721699381-LPCLOUD/G2721699381-LPCLOUD.xml', 7513)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:G2721699381-LPCLOUD/G2721699381-LPCLOUD.xml.stac.json', 1567)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:downloaded_feature_collection/downloaded_feature_collection.json', 6079)\n",
"('URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100/URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:URN:NASA:UNITY:UDS_DEMO:TEST:UDS_UNIT_COLLECTION___2409101100:downloaded_feature_collection/downloaded_feature_collection.json.stac.json', 1632)\n"
]
}
],
"source": [
"from mdps_ds_lib.lib.aws.aws_s3 import AwsS3\n",
"\n",
"s3 = AwsS3()\n",
"# Check S3 if they are uploaded.\n",
"s3_keys = [str(k) for k in s3.get_child_s3_files(os.environ['STAGING_BUCKET'], os.environ['COLLECTION_ID'])]\n",
"print('\\n'.join(s3_keys))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "faa41166-7f32-4cf9-b819-bb6dce784e88",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading