New data drop

Erotemic · May 22, 2024 · 526689a · 526689a
1 parent f9c303b
commit 526689a
Show file tree

Hide file tree

Showing 4 changed files with 61 additions and 49 deletions.
diff --git a/README.rst b/README.rst
@@ -81,6 +81,7 @@ Recent Updates
 Check back for updates, but because this is a personal project, it might take
 some time for it to fully drop.
 
+* 2024-05-21 - Slowing down release cycles. Still collecting images at roughly the same rate.
 * 2024-03-30 - New CID is ``bafybeibw5xqmdiycd7vw5qqdf3ceidjbq3cv4taalkc3ruu3qeqmqdy6sm``. This includes recent models that have been performing reasonably well.
 * 2024-02-29 - New CID is ``bafybeia2gphecs3pbrccwopg63aka7lxy5vj6btcwyazf47q6jlqjgagru``. Going to change this year to be 1/3 validation, next update will have a new split. Will also rework this README eventually.
 * 2024-02-22 - Added centralized Girder download link to increase accessibility of the data with an ok-ish pretrained model.
@@ -261,55 +262,57 @@ groups is the number of groups the before / after pair had a successful
 registration via the SIFT+RANSAC algorithm.
 
 
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| Date        | # Images | # Estimated Groups  | # Registered Groups   | # Annotated Images    |
-+=============+==========+=====================+=======================+=======================+
-| 2021-11-11  |  1935    |   ~616              | N/A                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2021-12-30  |  2088    |   ~728              | N/A                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-03-14  |  2471    |   ~954              | 658                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-04-02  |  2614    |  ~1002              | 697                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-04-16  |  2706    |  ~1033              | 722                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-06-20  |  2991    |  ~1127              | 734?                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-07-17  |  3144    |  ~1179              | 823                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-09-19  |  3423    |  ~1272              | 892                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2022-11-23  |  3667    |  ~1353              | 959                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-01-01  |  3800    |  ~1397              | 998                   | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-03-03  |  4105    |  ~1498              | 1068                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-04-16  |  4286    |  ~1559              | 1094                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-07-01  |  4594    |  ~1662              | 1154                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-08-22  |  4776    |  ~1723              | 1197                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-09-22  |  4899    |  ~1764              | 1232                  | 0                     |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-10-15  |  4981    |  ~1790              | 1255                  | 362                   |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-10-20  |  5019    |  ~1804              | 1266                  | 430                   |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-11-17  |  5141    |  ~1845              | 1304                  | 919                   |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-12-20  |  5249    |  ~1881              | 1337                  | 1440                  |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2023-12-31  |  5330    |  ~1908              | 1360                  | 1440                  |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2024-01-31  |  5533    |  ~1975              | 1411                  | 1964                  |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2024-02-29  |  5771    |  ~2054              | 1479                  | 1964                  |
-+-------------+----------+---------------------+-----------------------+-----------------------+
-| 2024-03-30  |  6019    |  ~2137              | 1549                  | 2133                  |
-+-------------+----------+---------------------+-----------------------+-----------------------+
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| Date        | # Images | # Estimated Groups  | # Registered Groups   | # Annotated Images    | CID                                                          |
++=============+==========+=====================+=======================+=======================+==============================================================+
+| 2021-11-11  |  1935    |   ~616              | N/A                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2021-12-30  |  2088    |   ~728              | N/A                   | 0                     | QmNj2MbeL183GtPoGkFv569vMY8nupUVGEVvvvqhjoAATG               |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-03-14  |  2471    |   ~954              | 658                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-04-02  |  2614    |  ~1002              | 697                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-04-16  |  2706    |  ~1033              | 722                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-06-20  |  2991    |  ~1127              | 734?                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-07-17  |  3144    |  ~1179              | 823                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-09-19  |  3423    |  ~1272              | 892                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2022-11-23  |  3667    |  ~1353              | 959                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-01-01  |  3800    |  ~1397              | 998                   | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-03-03  |  4105    |  ~1498              | 1068                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-04-16  |  4286    |  ~1559              | 1094                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-07-01  |  4594    |  ~1662              | 1154                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-08-22  |  4776    |  ~1723              | 1197                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-09-22  |  4899    |  ~1764              | 1232                  | 0                     |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-10-15  |  4981    |  ~1790              | 1255                  | 362                   |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-10-20  |  5019    |  ~1804              | 1266                  | 430                   |                                                              |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-11-17  |  5141    |  ~1845              | 1304                  | 919                   | bafybeie275n5f4f64vodekmodnktbnigsvbxktffvy2xxkcfsqxlie4hrm  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-12-20  |  5249    |  ~1881              | 1337                  | 1440                  | bafybeifkufkmmx3qxbvxe5hbskxr4gijkevcryxwp3mys2pqf4yjv2tobu  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2023-12-31  |  5330    |  ~1908              | 1360                  | 1440                  | bafybeihuem7qz2djallypbb6bo5z7ojqnjz5s4xj6j3c4w4aztqln4tbzu  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2024-01-31  |  5533    |  ~1975              | 1411                  | 1964                  | bafybeibxxrs3w7iquirv262ctgcwgppgvaglgtvcabb76qt5iwqgwuzgv4  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2024-02-29  |  5771    |  ~2054              | 1479                  | 1964                  | bafybeia2gphecs3pbrccwopg63aka7lxy5vj6btcwyazf47q6jlqjgagru  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2024-03-30  |  6019    |  ~2137              | 1549                  | 2133                  | bafybeibw5xqmdiycd7vw5qqdf3ceidjbq3cv4taalkc3ruu3qeqmqdy6sm  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
+| 2024-05-21  |  6373    |  ~2255              | 1640                  | 2252                  | bafybeidle54us5cdwpzzis4h52wjmtsk643gprx7nvvtd6g26mxq76kfjm  |
++-------------+----------+---------------------+-----------------------+-----------------------+--------------------------------------------------------------+
 
 
 For further details, see the `Datasheet <DATASHEET.md>`_.

diff --git a/requirements/runtime.txt b/requirements/runtime.txt
@@ -1 +1,7 @@
 ubelt>=1.3.3
+geowatch>=0.16.0
+vtool_ibeis>=2.3.0
+pyhesaff>=2.1.0
+pyflann_ibeis>=2.4.0
+plottool_ibeis>=2.4.0
+vtool_ibeis_ext>=0.1.1
diff --git a/shitspotter/cid_revisions.txt b/shitspotter/cid_revisions.txt
@@ -24,3 +24,4 @@ bafybeihuem7qz2djallypbb6bo5z7ojqnjz5s4xj6j3c4w4aztqln4tbzu
 bafybeibxxrs3w7iquirv262ctgcwgppgvaglgtvcabb76qt5iwqgwuzgv4
 bafybeia2gphecs3pbrccwopg63aka7lxy5vj6btcwyazf47q6jlqjgagru
 bafybeibw5xqmdiycd7vw5qqdf3ceidjbq3cv4taalkc3ruu3qeqmqdy6sm
+bafybeidle54us5cdwpzzis4h52wjmtsk643gprx7nvvtd6g26mxq76kfjm
diff --git a/shitspotter/gather.py b/shitspotter/gather.py
@@ -249,6 +249,8 @@ def main():
             # labelme_data = json.loads(fpath.read_text())
             imginfo, annsinfo = labelme_to_coco_structure(labelme_data)
             image_name = imginfo['file_name'].rsplit('.', 1)[0]
+            if image_name not in coco_dset.index.name_to_img:
+                continue
             img = coco_dset.index.name_to_img[image_name]
 
             # Construct the inverted exif transform