Skip to content

Latest commit

 

History

History

rnet

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
LICENSE
LICENSE
 
 
README.md
README.md
 
 
config.py
config.py
 
 
download.sh
download.sh
 
 
evaluate-v1.1.py
evaluate-v1.1.py
 
 
func.py
func.py
 
 
main.py
main.py
 
 
model.py
model.py
 
 
prepro.py
prepro.py
 
 
util.py
util.py
 
 

README.md

R-Net

Note: some bugs are fixed and test passed on with and .

Original Repository: HKUST-KnowComp/R-Net

Requirements

General

  • Python >= 3.4
  • unzip, wget

Python Packages

  • Tensorflow == 1.4.0
  • spaCy >= 2.0.0
  • tqdm
  • ujson

Usage

To download and preprocess the data, run

# download SQuAD and Glove
sh download.sh
# preprocess the data
python config.py --mode prepro

Hyper parameters are stored in config.py. To debug/train/test the model, run

python config.py --mode debug/train/test

To get the official score, run

python evaluate-v1.1.py ~/data/squad/dev-v1.1.json log/answer/answer.json

The default directory for tensorboard log file is log/event

Detailed Implementation

  • The original paper uses additive attention, which consumes lots of memory. This project adopts scaled multiplicative attention presented in Attention Is All You Need.
  • This project adopts variational dropout presented in A Theoretically Grounded Application of Dropout in Recurrent Neural Networks.
  • To solve the degradation problem in stacked RNN, outputs of each layer are concatenated to produce the final output.
  • When the loss on dev set increases in a certain period, the learning rate is halved.
  • During prediction, the project adopts search method presented in Machine Comprehension Using Match-LSTM and Answer Pointer.
  • To address efficiency issue, this implementation uses bucketing method (contributed by xiongyifan) and CudnnGRU. Due to a known bug #13254 in Tensorflow, the weights of CudnnGRU may not be properly restored. Check the test score if you want to use it for prediction. The bucketing method can speedup the training, but will lower the F1 score by 0.3%.

Performance

Score

EM F1
original paper 71.1 79.5
this project 71.07 79.51

Training Time (s/it)

Native Native + Bucket Cudnn Cudnn + Bucket
E5-2640 6.21 3.56 - -
TITAN X 2.72 1.67 0.61 0.35

Extensions

These settings may increase the score but not used in the model by default. You can turn these settings on in config.py.

Others