Include micro averaging - Bleu Score

[Ishan-Kumar2]

Fixes #2178

Description:
Have included method to pass batch of inputs to update and allow macro averaging also.
[WIP] The tests need to be added

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[Ishan-Kumar2]

@sdesrozis could you have a look, if the structure is correct so that I can change the tests also.

[sdesrozis]

@sdesrozis could you have a look, if the structure is correct so that I can change the tests also.

@Ishan-Kumar2 I did a plenty of comments, sorry for that. I think we could improve a bit what you did. The main point is the use of local/class variables.

[sdesrozis]

Another remark. Micro and batch are two things which could be tackled separately. If you loop over the batch example, you can have a batch macro version (which is not done).

Micro average is the "correct" (I mean most common) bleu metric. This is naturally compatible with batch.

I suggest that we keep the actual update api. The batch should be added in another PR, what do you think ?

[Ishan-Kumar2]

@sdesrozis thank you for the helpful reviews, I will first make another PR with just the addition of batch (and corresponding tests) then continue here for micro.

[sdesrozis]

Do as you prefer. I can help on the development of this PR because it is very clear to me 😉 Let me know.

[Ishan-Kumar2]

@sdesrozis , I have merged the other PRs contents and made the changes you suggested.
I have retained the _corpus_bleu function for now, as it is used in the tests. Do you think having a wrapper function for calculating bleu directly is useful? Otherwise one would have to call both _n_gram_counter and _brevity_penalty_smoothing_macro if they want to quickly check bleu for 1 example. I can rename it _corpus_bleu_macro if we decide to keep it.

If the micro avg calculation method is correct, I can add tests for the same.

[sdesrozis]

I converted to draft since it misses the tests and maybe some discussion is still needed.

[sdesrozis]

@Ishan-Kumar2 Thanks a lot for the work, although a few last comments to address.

Moreover, the tests seems ko.

[sdesrozis]

@Ishan-Kumar2 Thank you very much ! LGTM !

[vfdev-5]

A nit comment, otherwise LGTM as well