[BUG] salt-cp UnicodeDecodeError: 'utf-8'

[daro1337]

Description of Issue

Copying windows files produces UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 10: invalid start byte

ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 10: invalid start byte
Traceback (most recent call last):
  File "/usr/bin/salt-cp", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-cp')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 429, in salt_cp
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 53, in run
    cp_.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 145, in run
    ret = self.run_oldstyle()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 153, in run_oldstyle
    arg = [self._load_files(), self.opts["dest"]]
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 128, in _load_files
    files.update(self._file_dict(fn_))
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 117, in _file_dict
    data = fp_.read()
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 10: invalid start byte
Traceback (most recent call last):
  File "/usr/bin/salt-cp", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-cp')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 429, in salt_cp
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 53, in run
    cp_.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 145, in run
    ret = self.run_oldstyle()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 153, in run_oldstyle
    arg = [self._load_files(), self.opts["dest"]]
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 128, in _load_files
    files.update(self._file_dict(fn_))
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 117, in _file_dict
    data = fp_.read()
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 10: invalid start byte

Copying files created on linux machine working fine.

Setup

master:

Salt Version:
          Salt: 3002.5
 
Dependency Versions:
          cffi: 1.13.2
      cherrypy: 3.5.0
      dateutil: 2.5.3
     docker-py: 1.9.0
         gitdb: 2.0.0
     gitpython: 2.1.1
        Jinja2: 2.9.4
       libgit2: 0.28.4
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.0.2
        Python: 3.5.3 (default, Nov 18 2020, 21:09:16)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.1
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.1
 
System Versions:
          dist: debian 9 stretch
        locale: UTF-8
       machine: x86_64
       release: 4.9.0-14-amd64
        system: Linux
       version: Debian GNU/Linux 9 stretch

Minion:
`PS C:\salt> ./salt-minion --versions
Salt Version:
Salt: 3001.4

Dependency Versions:
cffi: 1.12.2
cherrypy: 17.4.1
dateutil: 2.8.0
docker-py: Not Installed
gitdb: 2.0.5
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 1.0.0
mysql-python: Not Installed
pycparser: 2.19
pycrypto: Not Installed
pycryptodome: 3.9.7
pygit2: Not Installed
Python: 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
python-gnupg: 0.4.4
PyYAML: 5.3.1
PyZMQ: 18.0.1
smmap: 2.0.5
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.1

System Versions:
dist:
locale: cp1250
machine: AMD64
release: 2016Server
system: Windows
version: 2016Server 10.0.14393 SP0`

Steps to Reproduce Issue

download file from windows in my case it's https://www.catalog.update.microsoft.com/Search.aspx?q=KB4340355
copy files to minion using:
salt-cp 'server' sqlserver2016kb4340355.exe "C:\\scripts\\sqlserver2016kb4340355.exe"

Workaround:

When i use --chunked file is successfully copied to windows but copying is very slow :(
salt-cp -C 'server' sqlserver2016kb4340355.exe "C:\scripts\sqlserver2016kb4340355.exe"

I tried to make a zip file at linux machine after download this file but it also throws errors

Versions Report

Master: 3002.5 @ debian 9 stretch
Minion: 3001.4 @ 2016Server 10.0.14393 SP0

[sagetherage]

@daro1337 Salt v3001.4 is vulnerable, you will want to upgrade to the latest 3001.7 from the tag or from the package repo for v3001 latest, likely that will not resolve the issue but simply giving you this detail :)

[twangboy]

I can confirm this bug. This seems to affect any file that is not pure text. Any .exe or .docx for example.
It currently reads the files as text instead of reading binary which is causing this error.

A raw text file works fine.

[xeacott]

Hi @daro1337 I checked this out, and for salt-cp it looks like there's basically two ways of handling a copy:

1. passing --chunked
2. not passing --chunked

The bug that you experienced is:

salt/salt/cli/cp.py

Line 108 in 5457218

with salt.utils.files.fopen(fn_, "r") as fp_:

and the fix is adding a b to read in as binary with salt.utils.files.fopen(fn_, "rb") as fp_:

however it does appear my salt-master has already returned during the reading process, which salt thinks the master has timed out because its busy reading in this file. It does look like salt-cp --chunked also allows you to define your own chunk size in bytes in the minion or master file:

salt/salt/config/__init__.py

Line 918 in 5457218

"salt_cp_chunk_size": int,

Let me know if any of this information is helpful or useful in any way to you; maybe I do need to submit a fix for that L108 however it does cause other non-desirable side-effects...

[daro1337]

@xeacott thanks for your debug.
Also I'll try salt_cp_chunk_size in salt-master config, I didn't knew about this option, I guess it's undocumented #58112 :)