Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PReLU accumulates grad #2532

Merged
merged 1 commit into from
Jun 2, 2015
Merged

PReLU accumulates grad #2532

merged 1 commit into from
Jun 2, 2015

Conversation

shelhamer
Copy link
Member

Adapt the PReLU gradient computation to accumulation now that #1977 is in.

@tnarihi could you double-check this? I cherry-picked your commit and cleaned up comments. Thanks.

@shelhamer shelhamer added the bug label Jun 1, 2015
@tnarihi
Copy link
Contributor

tnarihi commented Jun 1, 2015

Looks good. Thanks Evan!
Also, in order to test if every layer is implemented for gradient accumulation I made the following chagne in my branch: tnarihi@7d45526
Please consider this as well.

jeffdonahue added a commit that referenced this pull request Jun 2, 2015
@jeffdonahue jeffdonahue merged commit b051ce4 into BVLC:master Jun 2, 2015
@shelhamer
Copy link
Member Author

@jeffdonahue could you decide what to do with tnarihi/caffe@7d45526 too since you and Takuya talked about the gradient checker? Thanks.

@jeffdonahue
Copy link
Contributor

LGTM, thanks @tnarihi i and @shelhamer. I'm also not sure of the best way to go with changing the gradient checker -- always checking accumulation is safer, but anyone who's developed a layer of their own with parameters may be confused as to why their implementation is suddenly failing the gradient checker. I guess ideally there would be a first pass that does the current gradient check and, only if that passes, also checks accumulation afterwards and fails with an error message pointing to failed accumulation if applicable. Seems like the code would get kind of ugly though...

@jeffdonahue jeffdonahue deleted the accum-prelu branch June 2, 2015 22:30
@jeffdonahue jeffdonahue restored the accum-prelu branch June 2, 2015 22:30
@shelhamer shelhamer deleted the accum-prelu branch August 25, 2015 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants