Sphinx fails to distinguish open and close quotation marks

[cocoatomo]

Subject: On the translation project of Python document, smart quotes, the feature from docutils, fails to determine whether some quotation mark is open or close.

Problem

• docutils transforms "Title" to ”Title” (&#8221, T, i, t, l, e, &#8221), or 」Title」 with smart quotes enabled and language set to Japanese.

Procedure to reproduce the problem

git clone https://github.com/python-doc-ja/cpython-doc-intl.git
cd cpython-doc-intl
git checkout 87caf1ccad1457387bec57579fc742af3f2b703b

git clone https://github.com/python-doc-ja/py36-locale.git ./Doc/locale
cd ./Doc/locale
git checkout 6f42a6b453026fff60e18da1a1900a9db9593cfb

cd ..
sphinx-build -j4 -D language=ja -D locale_dirs=locale -D gettext_compact=0 -A versionswitcher=1 -E -d build/doctrees . ./_build/html

Then, open _build/html/bugs.html with your browser.

Error logs / results

Expected results

docutils transforms "Title" to 「Title」 (also "Type" to 「Type」),
or do not transforms "Title" at all.

Reproducible project / your project

• https://github.com/python-doc-ja/cpython-doc-intl
• https://github.com/python-doc-ja/py36-locale

Environment info

• OS: macOS 10.13
• Python version: 3.6.2
• Sphinx version: 1.6.4

[jfbu]

I can reproduce it with Docutils.

Test file index.rst:

.. comment
   https://github.com/python-doc-ja/py36-locale/blob/master/ja/LC_MESSAGES/bugs.po#L122

.. comment https://github.com/sphinx-doc/sphinx/issues/4142

Welcome to FOO's documentation!
===============================

Hello \"Title\" World

バグ報告フォームには幾つかのフィールドがあります。\"Title\" フィールドには、問題の説明を *非常に*

Process with rst2html.py --language "ja" --smart-quotes "on" index.rst test.html.

html file contains:

<p>Hello 「Title」 World</p>
<p>バグ報告フォームには幾つかのフィールドがあります。」Title」 フィールドには、問題の説明を <em>非常に</em></p>

Can you open ticket or ask for advice/explanation at https://sourceforge.net/p/docutils/bugs ?

[cocoatomo]

@jfbu OK, I will open a ticket and give an explanation later.
Thank you for the advice and the reproducible sample.

[cocoatomo]

I just have created a ticket.

Ticket URL: https://sourceforge.net/p/docutils/bugs/332/

[tk0miya]

I confirmed. Indeed, docutils expects a space before opening quotes.
So it can't use for CJK documents because they does not use spaces between words. AFAIK, a tokenizer is needed to do that.
So it should be disabled for these languages by default, I believe.

[jfbu]

Closing because bd13945 disables Smart Quotes for Japanese (refs: #4360; Sphinx 1.6.6)