Add README for Transformer #994
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guoshengCS
force-pushed
the
add-transformer-README
branch
7 times, most recently
from
0340f27
to
54ee859
Compare
guoshengCS
force-pushed
the
add-transformer-README
branch
from
54ee859
to
6724c9e
Compare
guoshengCS
force-pushed
the
add-transformer-README
branch
from
8a84245
to
a4d06ea
Compare
kuke
reviewed
Jun 29, 2018
| ```sh | ||
| perl multi_bleu.perl data/newstest2013.tok.de < prdict.tok.txt | ||
| ``` | ||
| 可以看到如下的评估结果。 |
There was a problem hiding this comment.
可以看到如下的评估结果 -> 可以看到类似如下的结果，因为用户自己训练这个评估结果并不确定
| ```sh | ||
| paste -d '\t' train.tok.clean.bpe.32000.en train.tok.clean.bpe.32000.de > train.tok.clean.bpe.32000.en-de | ||
| ``` | ||
| 此外还需要在词典文件中加上表示序列的开始、序列的结束和未登录词的3个特殊符号 `<s>` 、`<e>` 和 `<unk>` 。 |
|
|
||
| ### 数据准备 | ||
|
|
||
| 我们这里使用 [WMT'16 EN-DE 数据集](http://www.statmt.org/wmt16/translation-task.html)，同时参照论文中的设置使用 BPE（byte-pair encoding）[4]编码的数据，使用这种方式表示的数据能够更好的解决未登录词（out-of-vocabulary，OOV）的问题。用到的 BPE 数据可以参照[这里](https://github.com/google/seq2seq/blob/master/docs/data.md)进行下载，下载后解压，其中 `train.tok.clean.bpe.32000.en` 和 `train.tok.clean.bpe.32000.de` 为使用 BPE 的训练数据（平行语料，分别对应了英语和德语），`newstest2013.tok.bpe.32000.en` 和 `newstest2013.tok.bpe.32000.de` 等为测试数据（`newstest2013.tok.en` 和 `newstest2013.tok.de` 等则为对应的未使用 BPE 的测试数据），`vocab.bpe.32000` 为相应的词典文件（源语言和目标语言共享该词典文件）。 |
kuke
requested review from
shanyi15,
reyoung and
guochaorong
and removed request for
reyoung,
guochaorong and
shanyi15
|
format LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add README for Transformer