Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Fix a vital performance bug when run in graph mode deepray will… #34

Merged
merged 3 commits into from
Nov 6, 2023

Conversation

fuhailin
Copy link
Collaborator

@fuhailin fuhailin commented Nov 2, 2023

broadcast at the beginning of every execution

Description

Broadcast too many times
image

Brief Description of the PR:

Fixes # (issue)

Type of change

Checklist:

  • I've properly formatted my code according to the guidelines
    • By running Black + Flake8
    • By running pre-commit hooks
  • This PR addresses an already submitted issue for Deepray
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • This PR contains modifications to C++ custom-ops

How Has This Been Tested?

If you're adding a bugfix or new feature please describe the tests that you ran to verify your changes:
*

… broadcast at the beginning of every execution
@@ -708,7 +709,7 @@ def run_customized_training_loop(
# Training loop starts here.
self.current_step = self._first_steps = self.optimizer.iterations.numpy()

self.first_batch = True
self.first_batch = tf.Variable(True, trainable=False, dtype=tf.bool, name='first_batch')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating variable first_batch under init_scope would be better? Which makes sure this variable was created in eager mode to be compatible with python logical controlling code in any time, and also guarantees a safety from TF tape and other contexts.

with tf.init_scope():
    # Initialization runs with eager execution enabled
    # assert tf.executing_eagerly()
    tf.Variable(True, trainable=False, dtype=tf.bool, name='first_batch')

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@fuhailin fuhailin merged commit 7505906 into main Nov 6, 2023
9 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants