special character issues when dealing with openzepplin

[OliverLaiGHong]

Environment information

• OS: linux (Ubuntu 20.04.6 LTS
• Python Version: 3.11
• ape and plugin versions:

$ ape --version
0.7.19

$ ape plugins list
Installed Plugins
  alchemy      0.7.2
  ens          0.7.1
  etherscan    0.7.2
  foundry      0.7.4
  hardhat      0.7.2
  infura       0.7.3
  polygon      0.7.2
  solidity     0.7.3
  template     0.7.0
  tokens       0.7.1
  vyper        0.7.1

• Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

$ cat ape-config.yaml
name: play_ape
plugins:
  - name: polygon
  - name: ape-etherscan
  - name: foundry
  
default_ecosystem: polygon

polygon:
  default_network: mainnet
  mainnet:
    default_provider: geth
    
foundry:
  host: auto
  request_timeout: 20
  fork_request_timeout: 600
  fork:
    polygon:
      mainnet:
        upstream_provider: geth
        
geth:
  polygon:
    mainnet:
      uri: http://localhost:8545

dependencies:
  - name: OpenZeppelin
    github: OpenZeppelin/openzeppelin-contracts
    version: 5.0.1

solidity: 
  version: 0.8.23
  import_remapping:
    - "@openzeppelin=OpenZeppelin/5.0.1"

compile:
  include_dependencies: true

What went wrong?

I tried using solidity v0.8.23 and v0.8.25, and openzepplin v5.0.1 and v5.0.2, i keep getting the special character issues. when I removed everything the needs openzepplin, it got no problems.

Please include information like:

• what command you ran
  $ ape compile

• the code that caused the failure and the full output of the error received:
  after compiling all my own contracts and interfaces I obtained from dexes' websites, as soon as it starts working on openzepplin dependencies, this error occurs:

Traceback (most recent call last):                                                                                                                                                               
  File "/root/py_envs/ape/bin/ape", line 8, in <module>                                                                                                                                          
    sys.exit(cli())                                                                                                                                                                              
             ^^^^^                                                                                                                                                                               
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1157, in __call__                                                                                                    
    return self.main(*args, **kwargs)                                                                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                            
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1078, in main                                                                                                        
    rv = self.invoke(ctx)                                                                                                                                                                        
         ^^^^^^^^^^^^^^^^                                                                                                                                                                        
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/_cli.py", line 87, in invoke                                                                                                          
    return super().invoke(ctx)                                                                                                                                                                   
           ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                   
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1688, in invoke                                                                                                      
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                      
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_compile/_cli.py", line 54, in cli
    contract_types = cli_ctx.project_manager.load_contracts(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/manager.py", line 765, in load_contracts
    manifest = self.local_project.create_manifest(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 193, in create_manifest
    compiled_contract_types = self._compile(project_sources)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 270, in _compile
    return _compile_sources(project_sources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/project/types.py", line 259, in _compile_sources
    return self.compiler_manager.compile(srcs_to_compile)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape/managers/compilers.py", line 172, in compile
    compiled_contracts = compiler.compile(
                         ^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 518, in compile
    input_jsons = self.get_standard_input_json(contract_filepaths, base_path=base_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 479, in get_standard_input_json
    files_by_solc_version = self.get_version_map(contract_filepaths, base_path=base_path)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 761, in get_version_map
    self.get_import_remapping(base_path=base_path)
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 303, in get_import_remapping
    self._add_dependencies(manifest, sub_contracts_cache, builder)
  File "/root/py_envs/ape/lib/python3.11/site-packages/ape_solidity/compiler.py", line 330, in _add_dependencies
    cached_source.write_text(str(src.content))
  File "/usr/lib/python3.11/pathlib.py", line 1079, in write_text
    return f.write(data)
           ^^^^^^^^^^^^^
UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 1078: ordinal not in range(256)

How can it be fixed?

Sorry. I have no idea.

[linear]

APE-1744 special character issues when dealing with openzepplin

[fubuloubu]

I had noticed that interfaces/IERC4626.sol is using a utf-8 special character for quotes in a comment somewhere, not sure if this is the problem but that file has been like that for 2+ years so I wouldn't think so

[OliverLaiGHong]

@fubuloubu Actually, these utf-8 characters are seen across multiple files. i have tried to remove them but it was proven impractical. however, Ape was able to handle them about a month ago.

[antazoey]

sorry - accidentally closed

[antazoey]

I can reproduce this problem by doing:

path.write_text("“", encoding="latin-1")

so that is why specifying the encoding as utf8 everywhere fixes the problem.
(see attached 2 issues)

also - this is no longer a problem in this specific use-case anyway because we are using shutil.copytree() now as of #2123

so this issue will be closed in every way once the 2 attached PRs merge.