Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeDecodeError when docstrings contain ASCII art #99

Closed
rudolfbyker opened this issue Sep 6, 2022 · 2 comments · Fixed by #100
Closed

UnicodeDecodeError when docstrings contain ASCII art #99

rudolfbyker opened this issue Sep 6, 2022 · 2 comments · Fixed by #100

Comments

@rudolfbyker
Copy link
Contributor

rudolfbyker commented Sep 6, 2022

Describe the bug
When some of my docstrings contain ASCII art, I get a UnicodeDecodeError from griffe when trying to run mkdocs serve or mkdocs build.

To Reproduce
See https://github.com/rudolfbyker/repro-griffe-issue-99

Expected behavior
Docs are generated without errors.

Screenshots

PS C:\aa\griffe-bug-repro\docs\dev> mkdocs build
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: C:\aa\griffe-bug-repro\docs\dev\dist
ERROR    -  Error reading page 'test.md': 'charmap' codec can't decode byte 0x90 in position 105: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Users\rudol\anaconda3\envs\foo\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\rudol\anaconda3\envs\foo\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\rudol\anaconda3\envs\foo\Scripts\mkdocs.exe\__main__.py", line 7, in <module>
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocs\__main__.py", line 192, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocs\commands\build.py", line 292, in build
    _populate_page(file.page, config, files, dirty)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocs\commands\build.py", line 174, in _populate_page
    page.render(config, files)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocs\structure\pages.py", line 175, in render
    self.content = md.convert(self.markdown)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\markdown\core.py", line 264, in convert
    root = self.parser.parseDocument(self.lines).getroot()
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\markdown\blockparser.py", line 90, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\markdown\blockparser.py", line 105, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\markdown\blockparser.py", line 123, in parseBlocks
    if processor.run(parent, blocks) is not False:
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocstrings\extension.py", line 121, in run
    html, handler, data = self._process_block(identifier, block, heading_level)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocstrings\extension.py", line 195, in _process_block
    data: CollectorItem = handler.collect(identifier, options)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\mkdocstrings_handlers\python\handler.py", line 191, in collect
    loader.load_module(module_name)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\griffe\loader.py", line 137, in load_module
    module_name, package = self.finder.find_spec(module, try_relative_path)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\griffe\finder.py", line 117, in find_spec
    return module_name, self.find_package(top_module_name)
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\griffe\finder.py", line 149, in find_package
    if init_module.exists() and not _is_pkg_style_namespace(init_module):
  File "C:\Users\rudol\anaconda3\envs\foo\lib\site-packages\griffe\finder.py", line 287, in _is_pkg_style_namespace
    code = init_module.read_text()
  File "C:\Users\rudol\anaconda3\envs\foo\lib\pathlib.py", line 1237, in read_text
    return f.read()
  File "C:\Users\rudol\anaconda3\envs\foo\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 105: character maps to <undefined>

System (please complete the following information):

  • griffe version: 0.22.0
  • Python version: 3.8.12
  • OS: Windows 10
@rudolfbyker
Copy link
Contributor Author

I think specifying encoding='utf-8' wherever read_text() is called will solve the problem. The encoding argument is documented here: https://docs.python.org/3/library/functions.html#open

In text mode, if encoding is not specified the encoding used is platform-dependent: locale.getpreferredencoding(False) is called to get the current locale encoding. (For reading and writing raw bytes use binary mode and leave encoding unspecified.)

@rudolfbyker
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant