Welcome to Datumaro Discussions!

[zhiltsov-max]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we build together 💪.

To get involved, feel free to leave a comment below, and join an existing topic or start a new thread. Don't hesitate to share your opinion, ideas, questions and other thoughts on the project.

[siddrrsh]

Hi! Is it possible to use datumaro to combine 50 separate COCO file folders? If so, how would this work?

[zhiltsov-max]

Hi, you need to do the following (docs):

datum create
datum add path -f coco path1/
...
datum add path -f coco path50/
datum export -f coco [-- --save-images]

If datasets have mismatching categories (labels or their order), you'll need to use datum merge proj1 ... proj50 instead (docs).

[CyberCoder-X]

Hi, when is you planning to release version v1.x?

[zhiltsov-max]

@CyberCoder-X, there is no specific date planned yet. Could you describe what are your expectations from the 1.0, which are not fulfilled with the current situation? Technically, we are pretty close to the initially planned 1.0 vision described here with only few bits unimplemented.

[CyberCoder-X]

I develop my Importers on version v0.1.4 which seems be out of date, and I am planning to upgrade to latest version. The version 1.0 sounds like a milestone which stands for complete features, but it seems like most features was implemented on the v0.2 so I guess that version will work. By the way, does the v0.2 support import from remote file systems and parallel tasks?

[zhiltsov-max]

Speaking about the library API, we expect some changes in near future. We're continue working on stabilizing API and introducing new features, trying to keep backward compatibility, if possible. The 0.2 is more about CLI and Project API changes. Currently, we are working on extending the core API - the Dataset class and operations - and documenting it. My expectations about versions is that 0.3 with public API implemented is going to be an 1.0 candidate. Keep in mind that Python 3.6 is going to be EOL'ed this year, so future versions won't support it.

BTW, if you have anything available to be contributed, we are open for external PRs.

By the way, does the v0.2 support import from remote file systems

Not yet, the patch only introduced local versioning operation. There is a PoC implementation for remotes in #26, but now we don't have enough resources to finish this task, so it is postponed. Currently, I can't give time estimations when it will be continued.

and parallel tasks?

Could you explain this part in more detail? I'm not sure I understood this.

[AbdelsalamHaa]

Hi all,
@zhiltsov-max
I was wondering if Datumaro has a function or a collection of methods that can fix an imbalanced dataset.
my dataset is for detection and the presence of one class is way more than the others. So how can achieve a dataset that will only keep an N annotation for each class? For example, the images that don't contain the rare classes can be removed until each class has almost the same number of anotations.

Thank you very much in advance

[vinnamkim]

Hi @AbdelsalamHaa, Sorry for late response.
We have LabelRandomSampler dataset transform which is similar with your problem, but not the exact one:

datumaro/src/datumaro/plugins/sampler/random_sampler.py

Lines 110 to 113 in bb314a1

	class LabelRandomSampler(Transform, CliPlugin):
	"""
	Sampler that keeps at least the required number of annotations of
	each class in the dataset for each subset separately.|n

We will put your requirement in our backlog. Thanks!