-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added integration test for four model types #89
Conversation
Addresses #84 |
Ah crap I forgot DKHate was gated |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One change otherwise looks good
So problem: It simply takes too long to download all models and then run them, even if we only take the four that are currently in the test. The Fasttext model is especially heavyweight, which is a bummer because I just figured out that it had a bug. The translate E5 also has a very hefty translation model, so I'm frankly not sure how we could pull this off, especially on Github. I will try to think of a smart solution. Any thoughts? @KennethEnevoldsen |
I think for one, we could drop |
With FastText I'm at a bit of a loss. I think maybe we could add the wiki models to the test as they have a smaller vocabulary?? |
I would exclude the translation model and potentially remove the fasttext (though ideally download a smaller one). One thing you can also do is have it reuse the cache dir of the models (but that is mostly to speed things up). |
I believe it should run in a reasonable timeframe now. |
Hmm very weird behaviour. Pybind is clearly installed and the fasttext install still doesn't recognize it, and I can't reproduce this on my own machine, something fishy going on. |
Maybe adding an actual task (not just dummy) with all-MiniLM could be a good idea, but I'm not entirely sure cause right now it tests most types of models and runs relatively fast, what do you think? |
dd8bc70
to
2cd74d4
Compare
For the integration test it would be nice if it is an MTEB task. |
Okay, how about we keep the dummy task for the heftier models (so that we can test whether they run at all). And then add LCC for a lighter model (like MiniLM)? |
Add a comment for it.
If this is still a problem I would remove fasttext from the integration test for now and then get this merged in and create an issue. |
It does work now fortunately, I fixed it by using the |
as long as it is an optional dependency I don't have too many issues. |
@KennethEnevoldsen is this fine with you as it stands or do you want me to change something? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only a few minor things, but otherwise feel free to merge
@KennethEnevoldsen tests are failing because on an assertion. Do you have an explanation? |
It is because of the score not being correct:
It is due to an incorrect merge I believe. I will mark the sections |
Integration test for: