az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character #6408

jodok · 2018-05-22T21:30:10Z

Describe the bug
We're running az acr build ... from our jenkins slaves that also run tests.
If the docker build process returns unicode characters, az throws an UnicodeDecodeError (full traceback below).
When running the build process interactively on my local machine i see the line that is causing the error: npm WARN deprecated babel-preset-es2015@6.24.1: 🙌 Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
I suspect the emoji is causing the error because the library colorama used in az handles the output different for a non-tty environment.

To Reproduce
trigger a az acr build with a Dockerfile that causes unicode output in e.g. jenkins (non-tty environment).

Expected behavior
i'd expect the build to succeed and the unicode characters returned as well

Environment summary
rpm installation, bash, non interactive.
version:

azure-cli (2.0.32)

acr (2.0.24)
acs (2.0.32)
advisor (0.5.1)
ams (0.1.0)
appservice (0.1.32)
backup (1.1.1)
batch (3.2.2)
batchai (0.2.2)
billing (0.1.8)
cdn (0.0.14)
cloud (2.0.13)
cognitiveservices (0.1.13)
command-modules-nspkg (2.0.1)
configure (2.0.15)
consumption (0.3.1)
container (0.1.23)
core (2.0.32)
cosmosdb (0.1.21)
dla (0.0.19)
dls (0.0.21)
dms (0.0.1)
eventgrid (0.1.12)
eventhubs (0.1.3)
extension (0.0.13)
feedback (2.1.1)
find (0.2.9)
interactive (0.3.20)
iot (0.1.20)
keyvault (2.0.21)
lab (0.0.22)
monitor (0.1.6)
network (2.1.0)
nspkg (3.0.2)
profile (2.0.24)
rdbms (0.2.2)
redis (0.2.13)
reservations (0.1.2)
resource (2.0.28)
role (2.0.23)
servicebus (0.1.2)
servicefabric (0.0.12)
sql (2.0.25)
storage (2.0.32)
vm (2.0.31)

Python location '/usr/lib64/az/bin/python'
Extensions directory '/var/lib/jenkins/.azure/cliextensions'

Python (Linux) 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context
this seems to be related: tartley/colorama#36

full output:

az acr build -r myreg --no-push false -t myreg.azurecr.io/repo/myapp:master -f ./Dockerfile .
Sending build context (43.398 KiB) to ACR.
Queued a build with build ID: aah
Waiting for a build agent...
time="2018-05-22T21:11:59Z" level=info msg="Running command docker login -u 00000000-0000-0000-0000-000000000000 --password-stdin myreg.azurecr.io"
Login Succeeded
time="2018-05-22T21:12:01Z" level=info msg="Untarring to /root/acr-builder/src"
time="2018-05-22T21:12:01Z" level=info msg="Running command docker build --pull -f ./Dockerfile -t myreg.azurecr.io/repo/myapp:master ."
Sending build context to Docker daemon  220.2kB

Step 1/8 : FROM node:carbon
carbon: Pulling from library/node
3d77ce4481b1: Already exists
534514c83d69: Already exists
d562b1c3ac3f: Already exists
4b85e68dc01d: Already exists
f6a66c5de9db: Already exists
7a4e7d9a081d: Already exists
49f75af57b99: Pulling fs layer
454abe4fb374: Pulling fs layer
454abe4fb374: Verifying Checksum
454abe4fb374: Download complete
49f75af57b99: Verifying Checksum
49f75af57b99: Download complete
49f75af57b99: Pull complete
454abe4fb374: Pull complete
Digest: sha256:19efc0ee886057870334b4b4b4a505ea8a158d135f592cc367ab8840b684f6c0
Status: Downloaded newer image for node:carbon
 ---> 037acc6a00bc
Step 2/8 : WORKDIR /usr/src/app
Removing intermediate container ade8e4293757
 ---> 5fa54edc3ae3
Step 3/8 : COPY package*.json ./
 ---> 295c9eeaf7fc
Step 4/8 : RUN npm install
 ---> Running in 6846edad6a09
npm WARN deprecatedERROR: 'ascii' codec can't encode character u'\U0001f64c' in position 29: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 332, in execute
    six.reraise(*sys.exc_info())
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 306, in execute
    result = cmd(params)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 167, in __call__
    return super(AzCliCommand, self).__call__(*args, **kwargs)
  File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 109, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 420, in default_command_handler
    result = op(**command_args)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 330, in acr_build
    return acr_build_show_logs(client, build_id, registry_name, resource_group_name, True)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 71, in acr_build_show_logs
    raise_error_on_failure=raise_error_on_failure)
  File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py", line 132, in _stream_logs
    print(flush.decode('utf-8', errors='ignore'))
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 40, in write
    self.__convertor.write(text)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 166, in write_and_convert
    self.write_plain_text(text, cursor, start)
  File "/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f64c' in position 29: ordinal not in range(128)

The text was updated successfully, but these errors were encountered:

DaveKlassen · 2018-07-18T00:26:52Z

I had this as well:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 38: ordinal not in range(128)

Has the fix been released yet?

jodok · 2018-07-20T20:25:01Z

i created a fix upstream, but unfortunately this isn't merged yet: tartley/colorama#167 - meanwhile we ended up patching the library in our own deployments - which works so far.

timja · 2018-10-05T16:13:03Z

Is there a work around we can do?
Any published released version?

We have an angular app that is failing to build because of this

troydai · 2018-10-05T16:16:12Z

Use option --no-format

timja · 2018-10-05T16:19:05Z

aah thanks I see, it was released in 2.0.46

troydai · 2018-10-05T16:25:11Z

You're welcome.

* suppress ACR builds logs due to Azure/azure-cli#6408 * update aks client image, use --no-format in ACR build

DaveKlassen · 2018-10-16T00:48:00Z

Using az v2.0.47 with --no-format didn't solve this for me

sajayantony · 2018-10-16T00:53:05Z

@DaveKlassen if this isn't fixing the specific case then might be worth considering reopening the issue.
Do you have a Dockerfile that is hitting this for a repro?

DaveKlassen · 2018-10-16T01:18:28Z

Its a really simple Dockerfile. However there is a line where I run pipenv like:

RUN pipenv clean
RUN pipenv sync --dev

It seems to be failing when we run pipenv to prepare the container. docker build doesn't cause this error... but I would prefer to use az acr build to have a tighter integration with the registry.

Also it seems that the error isn't always consistent. I have seen the error in three different forms:

UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 38: ordinal not in range(128)
UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 73: ordinal not in range(128)
UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2026' in position 1037: ordinal not in range(128)

northtyphoon · 2018-10-17T03:45:35Z

@DaveKlassen what is the base image used your Dockerfile?

DaveKlassen · 2018-10-17T20:18:37Z

The base image contains the following:

FROM centos:latest

ENV LANG en_US.utf8

# Install tools
RUN yum -y install epel-release
RUN yum -y install python-pip
RUN yum -y install python-devel
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools

RUN pip install pipenv==11.10.4

northtyphoon · 2018-10-18T06:41:50Z

@DaveKlassen, so far I am unable to repro the issue in python 2.7.15 (alpine3.7). RUN pipenv sync --dev does output the ellipsis … (u+2026) properly. What is your python and os version?
Also may I suggest you try on Azure Cloud Shell which has python 3.5.2 and latest az tool installed?

DaveKlassen · 2018-10-18T07:10:48Z

I have a specific Dev environment that cannot be changed, and cannot be disclosed (ie. v3.5 is not an option). When I have time I will reproduce the errata for you.

One hint here, did you place anything in the PipFile? I would try a bunch of random packages, since I am fairly sure this issue occurs during pipenv install/sync

northtyphoon · 2018-10-20T05:13:36Z

@DaveKlassen I did try a bunch of packages in PipFile. The ellipsis character just worked fine on my machine. I noticed you reached Microsoft customer service and they should start a thread with you. If possible, can you send your Dev environment and PipFile to the customer service representative? It will help us narrow down the issue quickly. Thanks in advance.

DaveKlassen · 2018-10-20T06:52:55Z

I have sent this data. Likely the only different part is the PipFile

jenden · 2019-02-20T23:55:29Z

I'm getting this error as well. When I run the az acr build command during a Jenkins build (on a server with azure-cli 2.0.47 and acr 2.1.6) the last line of output I get is Length: 22897802 (22M) [application/octet-stream] before failing with this error UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128).

However, I can run it on my local development environment using azure-cli 2.0.41 and acr 2.0.28. Here is the same section of output from my local build:

Step 5/11 : RUN cd /usr/src &&     wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz &&     tar xzf Python-3.7.2.tgz &&     rm -f Python-3.7.2.tgz
 ---> Running in 5f6aace0d8a4
--2019-02-20 20:28:51--  https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz
Resolving www.python.org (www.python.org)... 151.101.52.223, 2a04:4e42:d::223
Connecting to www.python.org (www.python.org)|151.101.52.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22897802 (22M) [application/octet-stream]
Saving to: ‘Python-3.7.2.tgz’

It looks like there is some problem encoding the ‘ character in the last line. Here is the full error from the Jenkins build:


        "ERROR: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128)", 

        "Traceback (most recent call last):", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py\", line 197, in invoke", 

        "    cmd_result = self.invocation.execute(args)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 369, in execute", 

        "    six.reraise(*sys.exc_info())", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 343, in execute", 

        "    result = cmd(params)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 182, in __call__", 

        "    return self.handler(*args, **kwargs)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 436, in default_command_handler", 

        "    result = op(**command_args)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/build.py\", line 126, in acr_build", 

        "    return stream_logs(client, run_id, registry_name, resource_group_name, no_format, True)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/_stream_utils.py\", line 63, in stream_logs", 

        "    raise_error_on_failure)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/command_modules/acr/_stream_utils.py\", line 127, in _stream_logs", 

        "    print(flush.decode('utf-8', errors='ignore'))", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 40, in write", 

        "    self.__convertor.write(text)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 141, in write", 

        "    self.write_and_convert(text)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 166, in write_and_convert", 

        "    self.write_plain_text(text, cursor, start)", 

        "  File \"/usr/lib64/az/lib/python2.7/site-packages/colorama/ansitowin32.py\", line 174, in write_plain_text", 

        "    self.wrapped.write(text[start:end])", 

        "UnicodeEncodeError: 'ascii' codec can't encode character u'\\u2018' in position 11: ordinal not in range(128)

majidaldo · 2021-11-07T16:43:36Z

on windows, set PYTHONIOENCODING=utf-8. i'm using the latest cli (py39, pip installed). so stupid we're dealing with this in 2021.

i didn't do too much investigation but i think it's more fundamentally with colorama.

try to fix colorama problem by this Azure/azure-cli#6408 (comment)

tjprescott added bug This issue requires a change to an existing behavior in the product in order to be resolved. Container Registry az acr labels May 23, 2018

jodok added a commit to jodok/colorama that referenced this issue Jun 1, 2018

handle unicode strings properly (tartley#36, Azure/azure-cli#6408)

6486095

jodok mentioned this issue Jun 1, 2018

handle unicode strings properly tartley/colorama#167

Closed

jodok added a commit to jodok/colorama that referenced this issue Jun 16, 2018

handle unicode strings properly (tartley#36, Azure/azure-cli#6408)

49f2f54

tjprescott assigned djyou Jul 19, 2018

ehotinger mentioned this issue Aug 10, 2018

az acr build UnicodeEncodeError from log output causes CLI error #7011

Closed

NatalieGong mentioned this issue Aug 27, 2018

[ACR] Add a flag for displaying build logs to fix Unicode Encode Error #7155

Merged

2 tasks

troydai closed this as completed in #7155 Aug 29, 2018

bsiegel added the Service Attention This issue is responsible by Azure service team. label Sep 26, 2018

timwebster9 added a commit to hmcts/cnp-jenkins-library that referenced this issue Oct 5, 2018

suppress ACR builds logs due to Azure/azure-cli#6408

9158dfb

timwebster9 mentioned this issue Oct 5, 2018

fix ACR build logs hmcts/cnp-jenkins-library#319

Merged

timwebster9 added a commit to hmcts/cnp-jenkins-library that referenced this issue Oct 5, 2018

fix ACR build logs (#319)

a8309a4

* suppress ACR builds logs due to Azure/azure-cli#6408 * update aks client image, use --no-format in ACR build

ixxie mentioned this issue Jan 30, 2019

Getting 'UnicodeEncodeError' on CentOS and Python 2.7 #8312

Closed

PatMyron mentioned this issue Mar 13, 2019

Getting Unicode #8777

Closed

haroldrandom added ACR bug This issue requires a change to an existing behavior in the product in order to be resolved. Service Attention This issue is responsible by Azure service team. labels Oct 25, 2019

haroldrandom unassigned djyou Oct 25, 2019

ghost added Container Registry az acr and removed ACR labels Jun 25, 2020

kaiserthe13th added a commit to kaiserthe13th/tr-lang that referenced this issue Mar 22, 2022

try to fix colorama problem

4a71075

try to fix colorama problem by this Azure/azure-cli#6408 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character #6408

az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character #6408

jodok commented May 22, 2018 •

edited

Loading

DaveKlassen commented Jul 18, 2018

jodok commented Jul 20, 2018

timja commented Oct 5, 2018

troydai commented Oct 5, 2018

timja commented Oct 5, 2018

troydai commented Oct 5, 2018

DaveKlassen commented Oct 16, 2018

sajayantony commented Oct 16, 2018 •

edited

Loading

DaveKlassen commented Oct 16, 2018 •

edited

Loading

northtyphoon commented Oct 17, 2018

DaveKlassen commented Oct 17, 2018

northtyphoon commented Oct 18, 2018

DaveKlassen commented Oct 18, 2018 •

edited

Loading

northtyphoon commented Oct 20, 2018

DaveKlassen commented Oct 20, 2018

jenden commented Feb 20, 2019

majidaldo commented Nov 7, 2021

az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character #6408

az acr build throws UnicodeEncodeError: 'ascii' codec can't encode character #6408

Comments

jodok commented May 22, 2018 • edited Loading

DaveKlassen commented Jul 18, 2018

jodok commented Jul 20, 2018

timja commented Oct 5, 2018

troydai commented Oct 5, 2018

timja commented Oct 5, 2018

troydai commented Oct 5, 2018

DaveKlassen commented Oct 16, 2018

sajayantony commented Oct 16, 2018 • edited Loading

DaveKlassen commented Oct 16, 2018 • edited Loading

northtyphoon commented Oct 17, 2018

DaveKlassen commented Oct 17, 2018

northtyphoon commented Oct 18, 2018

DaveKlassen commented Oct 18, 2018 • edited Loading

northtyphoon commented Oct 20, 2018

DaveKlassen commented Oct 20, 2018

jenden commented Feb 20, 2019

majidaldo commented Nov 7, 2021

jodok commented May 22, 2018 •

edited

Loading

sajayantony commented Oct 16, 2018 •

edited

Loading

DaveKlassen commented Oct 16, 2018 •

edited

Loading

DaveKlassen commented Oct 18, 2018 •

edited

Loading