Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Landsat 1-5 (MSS and TM) and 7 (ETM) Item creation #13

Merged
merged 47 commits into from
Mar 22, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
3509796
adds STAC Item creation for MSS, TM, and ETM data
pjhartzell Mar 3, 2022
22f599c
edit command documentation
pjhartzell Mar 3, 2022
1e5b45b
add documentation
pjhartzell Mar 4, 2022
f0181fb
make oli-tirs fragment directory name compatable (oli_tirs)
pjhartzell Mar 4, 2022
02912bc
add landsat-c2-l1 collection
pjhartzell Mar 4, 2022
6933d76
change asset keys to common names, add create_collection command
pjhartzell Mar 5, 2022
1939507
finish collections, update readme and tests
pjhartzell Mar 5, 2022
fafd5ae
make docker cibuild work
pjhartzell Mar 6, 2022
54090ce
revert legacy_l8 to make existing tests untouched.
pjhartzell Mar 6, 2022
15f4f5f
add some tests, update temporal extents, fix default values
pjhartzell Mar 7, 2022
2482cfb
fix typing error
pjhartzell Mar 7, 2022
a95e9be
sundry review edits
pjhartzell Mar 8, 2022
99a3e47
make asset hrefs absolute when creating assets
pjhartzell Mar 8, 2022
50e126d
remove relative asset paths from item command, import formatting
pjhartzell Mar 8, 2022
22b5edb
remove "Science Product" and "Data Product" from Ttem descriptions
pjhartzell Mar 8, 2022
cbd051b
replace alternate link with via links to usgs stac server
pjhartzell Mar 8, 2022
50f6cd2
move legacy landsat 8 logic into mtl_metadata class
pjhartzell Mar 9, 2022
e5cbbbe
add title to doi Link
pjhartzell Mar 9, 2022
2b89a98
add doi links to collection summaries field
pjhartzell Mar 9, 2022
57ddb43
change how scientific doi links are included in collections
pjhartzell Mar 9, 2022
4fba799
remove sr_ and st_ prefixes from asset keys
pjhartzell Mar 9, 2022
d62b68a
remove "unitless" units from raster extension
pjhartzell Mar 9, 2022
033ac29
fix: finish removing "unitless" unit from oli_tirs
pjhartzell Mar 9, 2022
4507d34
remove eo extension center_wavelength and full_width_half_max in coll…
pjhartzell Mar 9, 2022
43a3c1d
fix a slew of errors and add spatial_resolution to raster extension
pjhartzell Mar 9, 2022
cac4992
update gsd
pjhartzell Mar 9, 2022
998f9e6
fix: remove 'name' field from MSS raster extension
pjhartzell Mar 10, 2022
761c601
fix: update fragments with traceable info.
pjhartzell Mar 10, 2022
8b0cf8d
change eo:bands layout, other edits to meet requested format
pjhartzell Mar 11, 2022
45b338b
correct swir16 asset description in TM and ETM fragments
pjhartzell Mar 13, 2022
1ad6f86
sundry fragment edits
pjhartzell Mar 14, 2022
a1eaf81
condense MSS eo:bands summary, move band designation to description
pjhartzell Mar 15, 2022
56bdfbb
attempt to better standardize and explain eo and asset information
pjhartzell Mar 15, 2022
cea2979
move back to repeated bands in eo:bands summary
pjhartzell Mar 15, 2022
17a7381
fix doi colon error
pjhartzell Mar 17, 2022
540bd2b
update c2-l1 examples
pjhartzell Mar 17, 2022
b6e9c0e
add new stactools antimeridian check
pjhartzell Mar 17, 2022
c874d50
add all extensions to collection fragments
pjhartzell Mar 18, 2022
99b8ac9
add antimeridian_strategy to commands
pjhartzell Mar 18, 2022
1f082c8
update the example for the legacy comparison
pjhartzell Mar 18, 2022
3b1e78d
remove problematic docker job from CI
pjhartzell Mar 18, 2022
cc52805
chore: clean up .gitignore
pjhartzell Mar 21, 2022
8e1b93b
chore: change stactools dependency to point at commit hash rather tha…
pjhartzell Mar 21, 2022
856f972
chore: update docs
pjhartzell Mar 21, 2022
6de9fe1
chore: update changelog
pjhartzell Mar 21, 2022
33323af
fix: add tests/data-files/external back to .gitignore
pjhartzell Mar 21, 2022
605ed1f
chore: clean up changelong
pjhartzell Mar 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
z-dev
.git
.github
.gitignore
Expand Down
84 changes: 0 additions & 84 deletions .github/workflows/continuous-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,87 +56,3 @@ jobs:
run: conda env update -f environment.yml -n test
- name: Execute linters and test suites
run: ./scripts/cibuild
docker:
name: docker
needs:
- codecov
- python-matrix
permissions:
contents: read
packages: write
runs-on: ubuntu-latest
steps:
- name: Set up Python 3.x
uses: actions/setup-python@v2
with:
python-version: "3.x"
-
name: Checkout
uses: actions/checkout@v2
-
name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run rename script if in the template repo
run: |
if [[ "$GITHUB_REPOSITORY" == */template ]]; then
scripts/rename
fi
- name: Source build args
run: |
source ./docker_env
STACTOOLS_VERSION=$(python scripts/stactools-version.py)
echo "stactools_version=$STACTOOLS_VERSION" >> $GITHUB_ENV
echo "docker_workdir=$DOCKER_WORKDIR" >> $GITHUB_ENV
echo "docker_namespace_package_dir=$DOCKER_NAMESPACE_PACKAGE_DIR" >> $GITHUB_ENV
-
name: Docker meta main
id: meta-main
uses: docker/metadata-action@v3
with:
images: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
-
name: Build and push main
uses: docker/build-push-action@v2
with:
context: .
file: docker/Dockerfile
build-args: |
STACTOOLS_VERSION=${{ env.stactools_version }}
DOCKER_WORKDIR=${{ env.docker_workdir }}
DOCKER_NAMESPACE_PACKAGE_DIR=${{ env.docker_namespace_package_dir }}
# Don't try to push if the event is a PR from a fork
push: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
tags: ${{ steps.meta-main.outputs.tags }}
labels: ${{ steps.meta-main.outputs.labels }}
-
name: Docker meta dev
id: meta-dev
uses: docker/metadata-action@v3
with:
images: ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}
flavor: |
suffix=-dev
tags: |
type=ref,event=branch
type=ref,event=pr
-
name: Build and push dev
uses: docker/build-push-action@v2
with:
context: .
file: docker/Dockerfile-dev
build-args: |
STACTOOLS_VERSION=${{ env.stactools_version }}
DOCKER_WORKDIR=${{ env.docker_workdir }}
DOCKER_NAMESPACE_PACKAGE_DIR=${{ env.docker_namespace_package_dir }}
# Don't try to push if the event is a PR from a fork
push: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
tags: ${{ steps.meta-dev.outputs.tags }}
labels: ${{ steps.meta-dev.outputs.labels }}
33 changes: 29 additions & 4 deletions CHANGELOG.MD
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,38 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [Unreleased]

### Feature
NOTE: For backwards compatibility, all items under the Added, Changed, and Fixed headings below are ignored for Landsat 8 scenes unless `legacy_l8=False` is passed to the `create_stac_item` function.

- Option to use geometry from the corresponding USGS STAC Item in lieu of computation from the ANG metadata.
### Added

### Update
- Collection 2 Level-1 and Level-2 Items from additional instruments and platforms:
- Landsat 1-5 Multispectral Scanner System (MSS, Level-1)
- Landsat 4-5 Thematic Mapper (TM, Level-2)
- Landsat 7 Enhanced Thematic Mapper Plus (ETM+, Level-2)
- Landsat 8*-9 Operational Land Imager and Thermal Infrared Sensor (OLI-TIRS, Level-2)
- Collection 2 Level-1 and Level-2 STAC Collections
- Item links to corresponding USGS STAC Items
- Item and Collection DOI links (`scientific` extension) specific to the instrument
- Option to split or normalize Item geometries that cross the antimeridian
- Option to use Item geometry from the corresponding USGS STAC Item in lieu of computation from the ANG.txt metadata file
- `raster extension`


### Changed

- Decrease Asset verbosity by moving redundant `projection` extension and `gsd` information from the Assets to Item properties.
- Assets having a different `gsd` than specified in the Item properties now override the Item properties `gsd` with their own `gsd` value (i.e., the thermal bands).
- Update the `landsat` extension from `v1.1.0` to `v1.1.1`
- Include non-zero `off_nadir` angles for Landsat 8-9
- Switch to [common EO band names](https://github.com/stac-extensions/eo#common-band-names) for Asset keys to address existence of different band numbers for the same EO band between instruments or platforms
- Update repository to the stactools-packages [template contents](https://github.com/stac-extensions/template/commit/7dfc84fd70c4cf15d6035968323c0baf8da06dd8).


### Fixed

- Correct EPSG code for southern hemisphere scenes to [always use northern hemisphere codes](https://www.usgs.gov/faqs/why-do-landsat-scenes-southern-hemisphere-display-negative-utm-values)
- Remove incorrect link to USGS STAC browser page

- Repo brought up to date with the current stactools-packages template contents as of 2022-02-18.

## [0.2.0] - 2021-07-20

Expand Down
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,23 @@ Source: https://pubs.usgs.gov/fs/2015/3081/fs20153081_ver1.2.pdf

## Examples

### STAC objects
### STAC Collections and Items

- [Item](examples/item/LC08_L2SP_005009_20150710_02_T2.json)
- [Collection](examples/landsat-c2-l2/collection.json)
- [Item](examples/landsat-c2-l1/LM01_L1GS_001010_19720908_02_T2/LM01_L1GS_001010_19720908_02_T2.json)

### Command-line usage

To create the example STAC `Item`:
To create a STAC `Item`:

```bash
$ stac landsat create-item "tests/data-files/assets/LC08_L2SP_005009_20150710_20200908_02_T2_MTL.xml" "examples/item/LC08_L2SP_005009_20150710_02_T2.json"
$ stac landsat create-item --mtl tests/data-files/oli-tirs/LC08_L2SP_047027_20201204_20210313_02_T1_MTL.xml --output examples --usgs_geometry
```

Use `stac landsat --help` to see all subcommands and options.
To create a STAC `Collection` from a text file containing a list of Landsat scene XML metadata files:

```bash
$ stac landsat create-collection --file_list examples/c2-l2-file-list.txt --output examples/landsat-c2-l2 --id landsat-c2-l2 --usgs_geometry
```

The above `create-collection` command is exactly how the contents of the `examples/landsat-c2-l2` directory are generated.
6 changes: 3 additions & 3 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ WORKDIR $DOCKER_WORKDIR

# For caching purposes, install dependencies but remove the actual package
COPY pyproject.toml setup.cfg ./
COPY src/$DOCKER_NAMESPACE_PACKAGE_DIR/__init__.py src/$DOCKER_NAMESPACE_PACKAGE_DIR/
RUN pip install . \
&& rm -r /opt/conda/lib/python$PYTHON_VERSION/site-packages/$DOCKER_NAMESPACE_PACKAGE_DIR
# COPY src/$DOCKER_NAMESPACE_PACKAGE_DIR/__init__.py src/$DOCKER_NAMESPACE_PACKAGE_DIR/
# RUN pip install . \
# && rm -r /opt/conda/lib/python$PYTHON_VERSION/site-packages/$DOCKER_NAMESPACE_PACKAGE_DIR

COPY src ./src
RUN pip install .
8 changes: 4 additions & 4 deletions docker/Dockerfile-dev
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ WORKDIR $DOCKER_WORKDIR

# For caching purposes, install dependencies but remove the actual package
COPY pyproject.toml requirements-dev.txt setup.cfg ./
COPY src/$DOCKER_NAMESPACE_PACKAGE_DIR/__init__.py src/$DOCKER_NAMESPACE_PACKAGE_DIR/
RUN pip install -r requirements-dev.txt \
&& pip install . \
&& rm -r /opt/conda/lib/python$PYTHON_VERSION/site-packages/$DOCKER_NAMESPACE_PACKAGE_DIR
# COPY src/$DOCKER_NAMESPACE_PACKAGE_DIR/__init__.py src/$DOCKER_NAMESPACE_PACKAGE_DIR/
# RUN pip install -r requirements-dev.txt \
# && pip install . \
# && rm -r /opt/conda/lib/python$PYTHON_VERSION/site-packages/$DOCKER_NAMESPACE_PACKAGE_DIR

COPY . ./
RUN pip install -e .
Loading