File Classifier by Type (v2)

[ZanSara]

Very simple component that takes in a list of paths and splits such list into several output lists, one for each file type.

It could look like the following:

@component
class FileTypeClassifier:

    @component.input
    def input(self):
        class Input:
            paths: List[Union[str, Path]]
        return Input

    @component.output
    def output(self):
        class Output:
            ... as many extensions as defined in the __init__, like: "txt: List[Union[str, Path]]" ...
        return Output

    def __init__(self, extensions: List[str]):
        ... 

    def run(self, data):
        ... categorizes the files ...
        return self.output(txt=txt_files, pdf=pdf_files, wav=wav_files, ....)

---

Note: during the implementation of the above we realized that extracting the mimetype for a file is a non-trivial task to carry out cross-platform in Python. The mimetype library relies on the file extension. The "correct" library to use, python-magic, is non-trivial to install on Windows due to the need to ship libmagic with it, and there might be Python based alternatives polyfile or filetype, but we would need to check their respective accuracy.

Therefore we decided to still implement a simple FileExtensionClassifier that uses the file extension: it's very easy to use, very quick, and does not try to read the file, which can be a plus in some scenarios. We'll then also implement a "real" FileTypeClassifier using one of the libraries mentioned for more accurate results.

Tasks

Beta Give feedback

1. FileExtensionClassifier feat: Add FileExtensionClassifier to previews #5514
2. FileTypeClassifier based on either python-magic or polyfile or filetype

[vblagoje]

What's the best folder package for this component in preview @ZanSara? I put it in io but not sure how we want to structure packages under components. Suggestions welcome.

[ZanSara]

So far we don't have specific rules yet, but for this component we may make a package called classifiers. It may end up including all types of classifiers (query classifiers, document classifiers, etc etc) and imho that's ok. If it gets too big we'll think later how to split it.

[vblagoje]

This issue was completed with #5514 wasn't it @ZanSara ?