Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fluid benchmark support recordio reader #11121

Merged
merged 10 commits into from
Jun 7, 2018
Merged

Fluid benchmark support recordio reader #11121

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
47630a4
fluid benchmark support recordio reader
Jun 1, 2018
a28a462
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Jun 5, 2018
2e0d851
update readme
Jun 5, 2018
725ea3f
update
Jun 5, 2018
8d14b39
follow comments
Jun 6, 2018
8893cf1
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Jun 6, 2018
6fdd5de
update
Jun 6, 2018
29b32b8
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Jun 6, 2018
8370c5a
fix errors by comment
Jun 6, 2018
cd33057
add num_passes
Jun 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions benchmark/fluid/fluid_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,10 @@ def train(avg_loss, infer_prog, optimizer, train_reader, test_reader, batch_acc,
# FIXME(wuyi): For use_reader_op, if the current
# pass is not the last, the last batch of this pass
# is also equal to args.batch_size.
num_samples += len(args.batch_size)
if args.use_reader_op:
num_samples += args.batch_size
Copy link
Contributor

@chengduoZH chengduoZH Jun 6, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

args.batch_size is the batch size on each GPU now. So it should be num_samples += args.batch_size * args.gpus.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much! Done. Current know issue, if set --use_reader_op we must also set --no_test will fix this in next PR.

else:
num_samples += len(data)
train_losses.append(loss)
print("Pass: %d, Iter: %d, Loss: %f\n" %
(pass_id, iters, np.mean(train_losses)))
Expand Down Expand Up @@ -350,9 +353,6 @@ def train_parallel(avg_loss, infer_prog, optimizer, train_reader, test_reader,
if iters == args.skip_batch_num:
start_time = time.time()
num_samples = 0
# NOTE: if use reader ops, the input data is not splited to multiple cards
if args.use_reader_op and iters >= args.iterations / args.gpus:
break
if args.use_fake_data or args.use_reader_op:
try:
loss, = exe.run([avg_loss.name])
Expand All @@ -362,7 +362,10 @@ def train_parallel(avg_loss, infer_prog, optimizer, train_reader, test_reader,
loss, = exe.run([avg_loss.name], feed=feeder.feed(data))
if args.update_method == "pserver":
exe.bcast_params()
num_samples += len(data)
if args.use_reader_op:
num_samples += args.batch_size
else:
num_samples += len(data)
iters += 1
if batch_id % 1 == 0:
print("Pass %d, batch %d, loss %s" %
Expand Down