Error at "Save tf lite models for best and last checkpoint" #39
Comments
|
Can you show the output of the model.fit? Did you change any of the metrics? Please put code or error output between "```" at the beginning and end, so it is formated for easier reading. You can also use the <> button the comment field. |
|
Haven't changed anything. My bad!
|
|
What's your tensorflow and conda version? |
|
conda is 4.8.3 and tensorflow is 2.2.0 |
|
Try removing the custom metrics in the first cell of the training section. In the meantime I'll see if I can reproduce your error somehow. |
|
Hmmm... tried doing as you suggested and now i run into a new error (even if I set the custom metrics back).
|
|
You need to restart the kernel and run the notebook again. The plots with the custom metrics won't work. |
|
Okay when running with just I receive the error
When doing the best_index part. |
|
Yes. Because you don't have the angle metric now. You need to replace it with |
|
Can you try adding |
|
Thanks for all the help! First time working with machine learning :) Added |
|
Can you pass the two custom metrics with the custom_objects arg? If this does not work, you can also try the following workaround. In the cell that throws the error, replace with |
|
Okay now things are cooking op nicely. Redefined the function as you said, however i added custom_object as parameter.
Then i could pass the custom metrics as a dict.
Guess I could have changed the load_model in utils.py the same way and have gotten the same result. Thank you for the help! I will get started on teaching my robot some policies. |
|
Did the function inside the notebook work without passing custom objects? This would indicate it's an issue with the scope. Yes, passing the dictionary of custom objects and leaving the code in util should also work. Trying to figure out why you were facing the issue while others including myself don't. If we understand why, we can fix it for others that may come across this issue. Main reason why I prefer not to pass custom_object is in case people change the metric list. Then it's one more part in the code that needs to be changed. But maybe it makes sense to just pass all custom metrics, just in case. If ppl define new ones they will have to add them manually. |
|
Nope it didn't work without passing. Still getting the same error as before. Ah yes makes sense. So far I've tried running the code on my laptop and my workstation, both result in the same error. Both running on Ubuntu 18.04. |
|
Strange, I have been able to run it without problems on Ubuntu 18.04, also on Mac and Windows. Are you using a conda environment? If yes, what is the version of conda, tensorflow and jupyter notebook? |
|
Okay so i messed up my ubuntu partition yesterday and had to do a fresh install. Thought i would test out this one, before installing anything else. The error still occurred with a fresh install and fresh git clone. I am running conda version 4.8.3 |
|
I see. I'm using tf 2.0.0. Maybe this is a 'new feature'. |
|
Could you do me a favour and try to install this environment: |
|
Yup it seems to work now. So it looks lige a "new feature"... How ever it didn't use the gpu for the calculations as default, as it did with 2.2. |
|
If you want to use the GPU, you need to install |
Great, I will add a note then for people that want to use newer versions of tensorflow. |
I cant seem to get further than this command, and I have had no luck fixing it myself.
best_index = np.argmax(np.array(history.history['val_angle_metric']) \ + np.array(history.history['val_direction_metric'])) best_checkpoint = str("cp-%04d.ckpt" % (best_index+1)) best_model = utils.load_model(os.path.join(checkpoint_path,best_checkpoint),loss_fn,metric_list) best_tflite = utils.generate_tflite(checkpoint_path, best_checkpoint) utils.save_tflite (best_tflite, checkpoint_path, "best") print("Best Checkpoint (val_angle: %s, val_direction: %s): %s" \ %(history.history['val_angle_metric'][best_index],\ history.history['val_direction_metric'][best_index],\ best_checkpoint))It returns the error
`---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in
2 + np.array(history.history['val_direction_metric']))
3 best_checkpoint = str("cp-%04d.ckpt" % (best_index+1))
----> 4 best_model = utils.load_model(os.path.join(checkpoint_path,best_checkpoint),loss_fn,metric_list)
5 best_tflite = utils.generate_tflite(checkpoint_path, best_checkpoint)
6 utils.save_tflite (best_tflite, checkpoint_path, "best")
~/git/OpenBot/policy/utils.py in load_model(model_path, loss_fn, metric_list)
72
73 def load_model(model_path,loss_fn,metric_list):
---> 74 model = tf.keras.models.load_model(model_path,
75 custom_objects=None,
76 compile=False
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/save.py in load_model(filepath, custom_objects, compile)
188 if isinstance(filepath, six.string_types):
189 loader_impl.parse_saved_model(filepath)
--> 190 return saved_model_load.load(filepath, compile)
191
192 raise IOError(
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in load(path, compile)
114 # TODO(kathywu): Add saving/loading of optimizer, compiled losses and metrics.
115 # TODO(kathywu): Add code to load from objects that contain all endpoints
--> 116 model = tf_load.load_internal(path, loader_cls=KerasObjectLoader)
117
118 # pylint: disable=protected-access
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py in load_internal(export_dir, tags, loader_cls)
600 object_graph_proto = meta_graph_def.object_graph_def
601 with ops.init_scope():
--> 602 loader = loader_cls(object_graph_proto,
603 saved_model_proto,
604 export_dir)
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in init(self, *args, **kwargs)
186 self._models_to_reconstruct = []
187
--> 188 super(KerasObjectLoader, self).init(*args, **kwargs)
189
190 # Now that the node object has been fully loaded, and the checkpoint has
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py in init(self, object_graph_proto, saved_model_proto, export_dir)
121 self._concrete_functions[name] = _WrapperFunction(concrete_function)
122
--> 123 self._load_all()
124 self._restore_checkpoint()
125
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _load_all(self)
207 # loaded from config may create variables / other objects during
208 # initialization. These are recorded in
_nodes_recreated_from_config.--> 209 self._layer_nodes = self._load_layers()
210
211 # Load all other nodes and functions.
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _load_layers(self)
310
311 for node_id, proto in metric_list:
--> 312 layers[node_id] = self._load_layer(proto.user_object, node_id)
313 return layers
314
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _load_layer(self, proto, node_id)
335 obj, setter = self._revive_from_config(proto.identifier, metadata, node_id)
336 if obj is None:
--> 337 obj, setter = revive_custom_object(proto.identifier, metadata)
338
339 # Add an attribute that stores the extra functions/objects saved in the
~/anaconda3/envs/openbot/lib/python3.8/site-packages/tensorflow/python/keras/saving/saved_model/load.py in revive_custom_object(identifier, metadata)
776 return revived_cls._init_from_metadata(metadata) # pylint: disable=protected-access
777 else:
--> 778 raise ValueError('Unable to restore custom object of type {} currently. '
779 'Please make sure that the layer implements
get_config'780 'and
from_configwhen saving. In addition, please use 'ValueError: Unable to restore custom object of type _tf_keras_metric currently. Please make sure that the layer implements
get_configandfrom_configwhen saving. In addition, please use thecustom_objectsarg when callingload_model().`
Every thing else runs smoothly.
The text was updated successfully, but these errors were encountered: