Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Orthogonal Initializer #84

Merged
merged 3 commits into from
Jun 9, 2021
Merged

Conversation

therealansh
Copy link
Contributor

@therealansh therealansh commented May 31, 2021

This PR adds Orthogonal Initializer (#51).

TODO:

  • Add Initializer implementation
  • Add support for export and import of layer in JSON format
  • Add Documentation
  • Add Tests

@therealansh
Copy link
Contributor Author

Hey @zaleslaw , How are we writing tests for initialisers that are using RandomGenerator in it. Like using a statelessRandom will generate different Operand everytime. How can we "expect" something out of randomness?

@zaleslaw
Copy link
Collaborator

zaleslaw commented Jun 1, 2021

It's a good question, it's very difficult to write good tests in ML/DL area, especially unit tests for calculations in TensorFlow due to the known "non-determinism" problem.

So, I don't know why, but "stateless"Random operand in TF has seed and it produces stable results from run to run, please have a look at the RandomNormalTest and RandomNormal internals.

So the recipe: to use seed if it's available and hope that it will be stable:(

@therealansh therealansh changed the title [WIP] Add Orthogonal Initializer Add Orthogonal Initializer Jun 1, 2021
@zaleslaw zaleslaw added the Review This PR is under review label Jun 1, 2021
@zaleslaw
Copy link
Collaborator

zaleslaw commented Jun 8, 2021

Hi, @therealansh did you get test data from the Keras framework?
Could you please merge with the master branch, I'll run the TC builds for it

@zaleslaw zaleslaw added LGTM PR reviewed and is ready to merge and removed Review This PR is under review labels Jun 8, 2021
@therealansh
Copy link
Contributor Author

Hi, @therealansh did you get test data from the Keras framework?
Could you please merge with the master branch, I'll run the TC builds for it

Yes i took the data from tf.keras.initializers.Orthogonal

@zaleslaw zaleslaw merged commit 6b1ad84 into Kotlin:master Jun 9, 2021
@zaleslaw zaleslaw linked an issue Jun 9, 2021 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LGTM PR reviewed and is ready to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Orthogonal initializer
2 participants