[BUG]: cupy, and by extension cudf changes the system's preferred encoding to ANSI_X3.4-1968

[dagardner-nv]

Version

23.07

Which installation method(s) does this occur on?

Docker, Conda, Source

Describe the bug.

Calling various cupy and cudf methods changes the system's preferred encoding from UTF-8 to ANSI_X3.4-1968.
cupy/cupy#7514
rapidsai/cudf#13085

The problem is any code called after this that requires reading a UTF-8 data source without explicitly setting the encoding will fail.

Minimum reproducible example

import locale

import cupy as cp

print(locale.getpreferredencoding()) # UTF-8
cpa =  cp.arange(0, 10)
print(locale.getpreferredencoding()) # ANSI_X3.4-1968

with open('models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt') as fh: # contains unicode chars
    contents = fh.read()

Relevant log output

UTF-8
ANSI_X3.4-1968

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
Cell In[1], line 10
      7 print(locale.getpreferredencoding()) # ANSI_X3.4-1968
      9 with open('models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt') as fh: # contains unicode chars
---> 10     contents = fh.read()

File ~/work/conda/envs/morpheus/lib/python3.8/encodings/ascii.py:26, in IncrementalDecoder.decode(self, input, final)
     25 def decode(self, input, final=False):
---> 26     return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1323: ordinal not in range(128)

Full env printout

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

This is a known issue with NVRTC (cupy/cupy#7514 (comment)) currently there are only two known work-arounds:

1. Setting LC_ALL="POSIX"
2. Explicitly set the encoding when opening a file ex: with open(vocab_path, encoding='UTF-8')