(WIP) Tfdsv4 carla mot

[jprokos26]

No description provided.

[jprokos26]

EvalGenerator is seemingly never reached as armory runs fine without this import, but flake8 complains about this line without this import:

armory/armory/utils/config_loading.py

Line 194 in 0d15a1b

return EvalGenerator(dataset, num_eval_batches=1)

EvalGenerator should be completely removed from this file as suggested by #1836 (comment)

[jprokos26]

This function is translated from the linked function below, however tensorflow does not support storing dictionaries as elements of a tensor (whereas previously numpy had no problem with this). As such the return of this preprocessing function cannot be a tensor containing lists of dictionaries (which contain tensors as values) since this is not supported even with RaggedTensors. From this I only see two viable options:

1. Keep the logic of this function as a preprocessing step and encode the key/value pairs of the dictionary using some encoding schema to be decoded during the next call
2. Move this function to be applied in CarlaMOT in the next call
3. Some tensorflow magical operations to get this to work without touching scenario code

Thoughts?

armory/armory/data/adversarial_datasets.py

Lines 1029 to 1072 in 48dcc04

	def mot_array_to_coco(batch):
	"""
	Map from 3D array (batch_size x detections x 9) to extended coco format
	of dimension (batch_size x frames x detections_per_frame)
	NOTE: 'image_id' is given as the frame of a video, so is not unique
	"""
	if batch.ndim == 2:
	not_batch = True
	batch = [batch]
	elif batch.ndim == 3:
	not_batch = False
	else:
	raise ValueError(f"batch.ndim {batch.ndim} is not in (2, 3)")
	output = np.empty(len(batch), dtype=object)
	for i, array in enumerate(batch):
	if not len(array):
	# no object detections
	output.append([])
	continue
	frames = []
	for detection in array:
	frames.append(
	{
	# TODO: should image_id include video number as well?
	"image_id": int(np.round(detection[0])),
	"category_id": int(np.round(detection[7])),
	"bbox": [float(x) for x in detection[2:6]],
	"score": float(detection[6]),
	# The following are extended fields
	"object_id": int(
	np.round(detection[1])
	), # for a specific object across frames
	"visibility": float(detection[8]),
	}
	)
	output[i] = frames
	if not_batch:
	output = output[0]
	return output

[jprokos26]

Note that the rest of the function works up until the NotImplementedError; after that point it either throws an error when trying to tf.convert_to_tensor(output) or when the function is mapped to the dataloader it complains that the output is not a valid return type (since it is a list of dictionaries).
Also the static hash table makes no difference compared to using a generic dictionary.

[lcadalzo]

Ideally (3) is best but I think (2) may be preferable to (1)

[jprokos26]

Agreed; assuming you don't have a recommendation of how to proceed with (3) I will implement (2).

[lcadalzo]

I do not, we've run into a similar issue with other datasets that has required modifying the scenario code