training problems

[chenzhm01]

Normalizing targets
done
Solving...
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "

data (1, 600, 687, 3)
im_info (1, 3)
gt_boxes (4, 5)
gt_ishard (4,)
dontcare_areas (0, 4) []
2018-02-08 15:40:32.394938: W tensorflow/core/framework/op_kernel.cc:1158] Invalid argument: ValueError: could not broadcast input array from shape (4) into shape (0)
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1139, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1121, in _run_fn
status, run_metadata)
File "/usr/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: could not broadcast input array from shape (4) into shape (0)
[[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_net.py", line 116, in
restore=bool(int(args.restore)))
File "/home/commaai02/TF_Deformable_Net/lib/fast_rcnn/train.py", line 412, in train_net
sw.train_model(sess, max_iters, restore=restore)
File "/home/commaai02/TF_Deformable_Net/lib/fast_rcnn/train.py", line 264, in train_model
cls_prob, bbox_pred, rois = sess.run(fetches=fetch_list, feed_dict=feed_dict)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: ValueError: could not broadcast input array from shape (4) into shape (0)
[[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

Caused by op 'roi-data/PyFunc', defined at:
File "train_net.py", line 106, in
network = get_network(args.network_name)
File "/home/commaai02/TF_Deformable_Net/lib/networks/factory.py", line 29, in get_network
return VGGnet_train()
File "/home/commaai02/TF_Deformable_Net/lib/networks/VGGnet_train.py", line 17, in init
self.setup()
File "/home/commaai02/TF_Deformable_Net/lib/networks/VGGnet_train.py", line 80, in setup
.proposal_target_layer(n_classes,name = 'roi-data'))
File "/home/commaai02/TF_Deformable_Net/lib/networks/network.py", line 41, in layer_decorated
layer_output = op(self, layer_input, *args, **kwargs)
File "/home/commaai02/TF_Deformable_Net/lib/networks/network.py", line 372, in proposal_target_layer
[tf.float32,tf.float32,tf.float32,tf.float32,tf.float32])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/script_ops.py", line 198, in py_func
input=inp, token=token, Tout=Tout, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_script_ops.py", line 38, in _py_func
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1269, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): ValueError: could not broadcast input array from shape (4) into shape (0)
[[Node: roi-data/PyFunc = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], Tout=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], token="pyfunc_2", _device="/job:localhost/replica:0/task:0/cpu:0"](rpn_rois/_97, _arg_gt_boxes_0_3, _arg_gt_ishard_0_4, _arg_dontcare_areas_0_2, roi-data/PyFunc/input_4/_99)]]

[Zardinality]

There is a similiar issue stating it might be the missing of background class that causes the dimension problem. For debug purpose, you should print the shape of all inputs in this layer to track down the issue.