Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Able to print gradients in event_handler #3085

Merged
merged 1 commit into from
Aug 11, 2017

Conversation

typhoonzero
Copy link
Contributor

@typhoonzero typhoonzero commented Jul 27, 2017

Able to print gradients in event handle using v2 API to train.

Related: #3040
Fixes: #3211

def event_handler(event):
        if isinstance(event, paddle.event.EndIteration):
            if event.batch_id % 10 == 0:
                for p in parameters:
                    print "parameters:", parameters.get(p)
                    grad = parameters.get_grad(p)
                    print "gradients:", grad
                    print "gradients max abs:" max(grad.min(), grad.max(), key=abs)
            if event.batch_id % 100 == 0:
                print "Pass %d, Batch %d, Cost %f" % (
                    event.pass_id, event.batch_id, event.cost)

@@ -161,14 +161,14 @@ def train(self, reader, num_passes=1, event_handler=None, feeding=None):
self.__parameter_updater__.update(each_param)
cost_sum = out_args.sum()
cost = cost_sum / len(data_batch)
self.__parameter_updater__.finishBatch(cost)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a little confusing, why we move the batch_evaluator to the event_handler?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to call event_handler before finishBatch operations so we can get inner status before clearing them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks!

assert isinstance(val, api.Vector)
val = val.copyToNumpyArray()
return val
# else continue

raise RuntimeError("Unexpected branch")

def __getitem__(self, key):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to get_param?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

__getitem__ will called when doing param[k], it's an operator reload. Need to keep this the same as before.

@typhoonzero typhoonzero merged commit 01e9e44 into PaddlePaddle:develop Aug 11, 2017
@typhoonzero typhoonzero deleted the print_grad branch August 11, 2017 06:54
@lcy-seso
Copy link
Contributor

even though it copies the parameter from the c++ side, very helpful for debugging the training process and tuning the model, thank you very much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants