[DAR-1679][External] Made darwin-py ignore the .v7/metadata.json properties manifest when reading JSON annotation files

[JBWilkie]

Problem

In a previous bug fix (DAR-1609) it was discovered that pulling a dataset release does not include the properties metadata file if one needs to be downloaded. This was fixed in that ticket. However, there are several functions in darwin-py that assume every JSON file in the annotations directory is annotation file. If there's a properties metadata file, this is no longer true, and this causes issues

Solution

• 1: Identified everywhere where darwin-py reads annotation files from the annotations directory
• 2: Ensure we ignore the .v7/metadata.json properties manifest in each case

Changelog

Made darwin-py ignore the .v7/metadata.json properties manifest when reading JSON annotation files

[linear]

DAR-1679 BUG: The presence of the .v7/metadata.json file breaks a few functions in darwin-py

[JBWilkie]

Since we're doing the same glob() statement repeatedly with logic to avoid /.v7/ in filepaths, I can abstract this into a function and then add a unit test if it's deemed worthwhile

[saurbhc]

            if "/.v7/" not in str(e)

Do we need this part here? 🤔 as glob("**/*.json") is anyways fetching only the json files?

[JBWilkie]

I think we do sadly, and I can reproduce this by instantiating a LocalDataset - The ** in glob() is recursively checking the annotations directory. Since it now will contain .v7/metadata.json, it picks this file up unless we tell it to ignore filepaths containing /.v7/

The code I used to reproduce this is below - Before running it, a dataset release has to be pulled using 0.8.59 so that the metadata file is present in the annotations directory

from pathlib import Path

from darwin.dataset import LocalDataset

my_dataset = LocalDataset(
    dataset_path=Path("/path/to/dataset"),
    annotation_type="bounding_box",
)