tests currently fail for extractall_unicode after a package update (#550

)

* tests currently fail for extractall_unicode after a package update
we need to deal with zip archives created under windows and unicode filenames differently. thus I have added another if branch

* black formatting

* added partial dependency of haydn in instruction

docs/source/contributing.rst

@@ -35,6 +35,7 @@ To install ``mirdata`` for development purposes:
           pip install .[tests]
           pip install .[docs]
           pip install .[dali]
+          pip install .[haydn_op20]
 
 
 We recommend to install `pyenv <https://github.com/pyenv/pyenv#installation>`_ to manage your Python versions

mirdata/download_utils.py

@@ -320,8 +320,13 @@ def extractall_unicode(zfile, out_dir):
             "cp437"
         ).decode(errors="ignore") != filename:
             filename_bytes = filename.encode("cp437")
-            guessed_encoding = chardet.detect(filename_bytes)["encoding"] or "utf8"
-            filename = filename_bytes.decode(guessed_encoding, "replace")
+            if filename_bytes.decode("utf-8", "replace") != filename_bytes.decode(
+                errors="ignore"
+            ):
+                guessed_encoding = chardet.detect(filename_bytes)["encoding"] or "utf8"
+                filename = filename_bytes.decode(guessed_encoding, "replace")
+            else:
+                filename = filename_bytes.decode("utf-8", "replace")
 
         disk_file_name = os.path.join(out_dir, filename)