Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lack of UTF-8 in EasyBuild filetools? #37

Closed
terjekv opened this issue Nov 15, 2020 · 6 comments
Closed

Lack of UTF-8 in EasyBuild filetools? #37

terjekv opened this issue Nov 15, 2020 · 6 comments
Labels
bug Something isn't working

Comments

@terjekv
Copy link
Member

terjekv commented Nov 15, 2020

Building on arm, I ran into this when building GCC:

== preparing...
ERROR: Traceback (most recent call last):
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/main.py", line 115, in build_and_install_softwa
re
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/framework/easyblock.py", line 3325, in build_an
d_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/framework/easyblock.py", line 3228, in run_all_
steps
    self.run_step(step_name, step_methods)
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/framework/easyblock.py", line 3083, in run_step
    step_method(self)()
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/framework/easyblock.py", line 2105, in prepare_
step
    rpath_include_dirs=self.rpath_include_dirs)
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/tools/toolchain/toolchain.py", line 875, in pre
pare
    self.prepare_rpath_wrappers(rpath_filter_dirs, rpath_include_dirs)
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/tools/toolchain/toolchain.py", line 1014, in pr
epare_rpath_wrappers
    cmd_wrapper_txt = read_file(rpath_wrapper_template) % {
  File "/cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/tools/filetools.py", line 195, in read_file
    txt = handle.read()
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2079: ordinal not in range(128)

"Resolved" via:

--- /cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/tools/filetools.py.orig  2020-11-15 00:40:40.386799659 -0000
+++ /cvmfs/pilot.eessi-hpc.org/2020.10/software/aarch64/generic/software/EasyBuild/4.3.1/lib/python3.7/site-packages/easybuild/tools/filetools.py       2020-11-15 00:46:12.608351561 -0000
@@ -191,8 +191,12 @@
     """Read contents of file at given path, in a robust way."""
     txt = None
     try:
-        with open(path, mode) as handle:
-            txt = handle.read()
+        if 'b' in mode:
+            with open(path, mode) as handle:
+                txt = handle.read()
+        else:
+            with open(path, mode, encoding="utf-8") as handle:
+                txt = handle.read()
     except IOError as err:
         if log_error:
             raise EasyBuildError("Failed to read %s: %s", path, err)
@@ -242,8 +246,12 @@
     # note: we can't use try-except-finally, because Python 2.4 doesn't support it as a single block
     try:
         mkdir(os.path.dirname(path), parents=True)
-        with open(path, mode) as handle:
-            handle.write(data)
+        if 'b' in mode:
+            with open(path, mode) as handle:
+                txt = handle.write(data)
+        else:
+            with open(path, mode, encoding="utf-8") as handle:
+                txt = handle.write(data)
     except IOError as err:
         raise EasyBuildError("Failed to write to %s: %s", path, err)
@boegel
Copy link
Contributor

boegel commented Nov 16, 2020

@terjekv Why not issue a pull request to easybuild-framework for that fix?

@terjekv
Copy link
Member Author

terjekv commented Nov 16, 2020

Because, to be perfectly honest, I'm not confident in the patch. I don't know if this is actually a solution or a hack that worked with the python version in question. Although, maybe I can push a patch to framework and see if the CI finds any problems?

@boegel
Copy link
Contributor

boegel commented Nov 16, 2020

@terjekv Yeah, that's part of why I suggested to issue a proper PR for it. It looks like a decent fix at first sight, but I need to see it in full context (and see if CI raises anything)

@terjekv
Copy link
Member Author

terjekv commented Nov 16, 2020

Right, I'll see about making a PR out of this.

From https://stackoverflow.com/questions/491921/unicode-utf-8-reading-and-writing-to-files-in-python I find:

Note that in Python 3, the io.open function is an alias for the built-in open function. The built-in open function only supports the encoding argument in Python 3, not Python 2.

Hrmpf.

@boegel
Copy link
Contributor

boegel commented Feb 16, 2021

This should be fixed by easybuilders/easybuild-framework#3565, to be included with EasyBuild v4.3.3...

@boegel boegel added the bug Something isn't working label Feb 16, 2021
@boegel
Copy link
Contributor

boegel commented Apr 18, 2021

This should be fixed since EasyBuild v4.3.3...

@boegel boegel closed this as completed Apr 18, 2021
Neves-P added a commit to Neves-P/software-layer that referenced this issue Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants