Can't reproduce pitts30k results #6
Comments
|
Sorry for late reply. The replace at row 120 in cct.py does not affect the result. I guess this failure is probably due to the inconsistency of numpy (or torch) versions. Please refer to requirements.txt |
|
Thanks! I reviewed requirements.txt and my versioning is actually correct as far as I can tell. certifi 2024.7.4 And also this is my conda environment list output if maybe it is relevant: Thanks |
|
Actually - maybe it's the scipy and scikit-learn versions? eval.py requires them but I couldn't find which version to install in the requirements.txt, so I installed: |
The version I installed is: |
|
Ok - installed these and still doesn't work. By the way - training (initialization) of msls backbone checkpoint on the pitts 30k dataset starts at ~50 and is at 22.62 after 10 epochs, I don't know if that's expected or not. Thanks |
|
It looks like the finetuned model may not load successfully as you run eval.py on Pitts30k. Have you tried evaluation on MSLS using By the way, why do you train the MSLS backbone on Pitts30k? If everything goes well, the results should not be reduced after the training. |
|
That's a typo I actually trained the Pitts30k backbone. I'd like to try using MSLS but I couldn't get the dataset in the right format - VPR-datasets-downloader/format_mapillary.py doesn't seem to make the correct format for eval.py How should I format MSLS? |
|
Running VPR-datasets-downloader/format_mapillary.py directly can get the correct format. I'm not sure what happened in your program. You can also use the model finetuned on MSLS to the test on other datasets, such as SPED and St Lucia, to confirm if there is a problem in the previous model or dataset. |
|
You're correct format_mapillary.py has no problems, I didn't understand it - Thanks. MSLS works for eval! I get: 2024-08-03 12:59:48 baseline test: R@1: 74.8, R@5: 87.5, R@10: 90.3, R@100: 96.2 Although for some reason the Helsinki images were missing from the zip file I downloaded... Thanks a lot! |
|
OK. The difference on MSLS between your results and our paper is because we used the official version of MSLS-val, which only contains 740 query images (a subset of that you used). You can refer to this issue to get results consistent with our paper. |
Hey,
I've installed the dependencies as specified in requirements.txt, and have downloaded the models as referenced from the readme, and downloaded the pitts30k after contacting Relja, and formatted it using the convinient VPR-datasets-downloader.
I then used eval.py with the following arguments:
python -m eval --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k
Only difference is I had to replace row 120 in model/cct/cct.py as follows:
Because the link in row 25 in model/cct/cct.py appears to be broken:
I am getting these results:
(hmgrphy) PS C:\dev\projects\DHE-VPR> python -m eval --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k
2024-07-28 13:45:19 python C:\dev\projects\DHE-VPR\eval.py --resume_fe=finetunedCCT14_pitts30k.torch --resume_hr=finetunedDHE_pitts30k.torch --datasets_folder=./datasets --dataset_name=pitts30k
2024-07-28 13:45:19 Arguments: Namespace(brightness=None, cache_refresh_rate=1000, contrast=None, criterion='triplet', dataset_name='pitts30k', datasets_folder='./datasets', device='cuda', efficient_ram_testing=False, epochs_num=1000, exp_name='default', freeze_te=5, horizontal_flip=False, hue=None, infer_batch_size=32,
l2='before_pool', lr=1e-05, majority_weight=0.01, margin=0.1, mining='partial', neg_samples_num=1000, negs_num_per_query=2, num_reranked_preds=32, num_workers=8,
optim='adam', patience=3, queries_per_epoch=5000, rand_perspective=None, random_resized_crop=None, random_rotation=None, recall_values=[1, 5, 10, 20], resize=[384, 384], resume_fe='finetunedCCT14_pitts30k.torch', resume_hr='finetunedDHE_pitts30k.torch', saturation=None, save_dir='default', seed=0, test_method='hard_resize', train_batch_size=4, train_positives_dist_threshold=10, trunc_te=8, val_positive_dist_threshold=25)
2024-07-28 13:45:19 The outputs are being saved in logs_test/default/2024-07-28_13-45-19
C:\dev\projects\DHE-VPR\dataset_geoloc.py:43: DeprecationWarning:
np.floatis a deprecated alias for the builtinfloat. To silence this warning, usefloatby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.float64here.Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.gallery_utms = np.array([(path.split("@")[1], path.split("@")[2]) for path in self.gallery_paths]).astype(np.float)
C:\dev\projects\DHE-VPR\dataset_geoloc.py:44: DeprecationWarning:
np.floatis a deprecated alias for the builtinfloat. To silence this warning, usefloatby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.float64here.Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
self.queries_utms = np.array([(path.split("@")[1], path.split("@")[2]) for path in self.queries_paths]).astype(np.float)
2024-07-28 13:45:22 Geoloc test set: < GeolocDataset, pitts30k - #gallery: 10000; #queries: 6816 >
this will not modify any behavior and is safe. When replacing
np.int, you may wish to use e.g.np.int64ornp.int32to specify the precision. If you wish to review your current use, check the release note link for additional information.Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
correct_bool_mat = np.zeros((geoloc_dataset.queries_num, max_recall_value), dtype=np.int)
2024-07-28 13:47:28 baseline test: R@1: 1.2, R@5: 4.7, R@10: 8.2, R@100: 39.3
Testing: 100%|██████████████████████████████████████████████████| 6816/6816 [15:17<00:00, 7.42it/s]
2024-07-28 14:02:47 test after re-ranking - R@1: 1.1, R@5: 5.2, R@10: 9.2, R@20: 14.8
Please let me know if I did anything wrong, and how to reproduce correctly.
Thank you,
And thanks for the great work.
The text was updated successfully, but these errors were encountered: