Gelu activation #187
Gelu activation #187
Conversation
| features, tf.math.add( | ||
| tf.constant(1.0f), tf.math.tanh( | ||
| tf.math.mul( | ||
| tf.constant(0.7978845608028654f), |
There was a problem hiding this comment.
Looks like optimization to avoid the pi transformation?
There was a problem hiding this comment.
The value is of sqrt(2/pi) to avoid a constant division i took this value. This is similar to tf.nn.gelu implementation. I could add in sqrt if you want?
There was a problem hiding this comment.
maybe better to add it in the comments, for the future code readers
|
Thanks for the contribution! Where did you take the test data for Gelu? |
zaleslaw
added
Review
|
Hey @zaleslaw I hope you are doing well, I took the test cases from tf.addons |
|
Ha-ha, @therealansh "I hope you are doing well" In this context looks like "hey, buddy, open your eyes, I hope you are able to read a code today":) |
|
@zaleslaw Haha, no no I meant to ask about your well being only. And as for the PR i'll add some comments and docs about it. |
|
@therealansh please merge your branch with master; you will be the following in the merge queue (after that, I'll run the TC) |
This PR adds Gelu Activation Function. closes #165