add normalisation to confidence scores

[akelad]

Proposed changes:

• fixes https://github.com/RasaHQ/research/issues/57
• normalize softmax confidence scores

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[akelad]

@Ghostvv started a PR on this, few questions before I polish it up and request a review:

• I would lean towards making the default be True, what do you think? though then i don't think we can put it in a minor :D
• should we use a lower number than 10 for the scores? i know we discussed 5 as well

[Ghostvv]

I would use LABEL_RANKING_LENGTH as normalization length, otherwise it is not clear how to report other scores in there.
True by default - yes. Not sure about the version, it is not even model breaking. But it has a danger to break whole bot functionality.
Did you test, that the scores become more intuitive after normalization?

[akelad]

I would use LABEL_RANKING_LENGTH as normalization length, otherwise it is not clear how to report other scores in there.

as zero 😂

No i haven't tested yet, i still need to do that. just wanted to get what i had done on a PR first

[tmbo]

I don't think this shouldn't go into a patch release, please change the base to master. Next minor is next week, so that would be last chance this year

[Ghostvv]

@akelad could you please add normalization to EmbeddingPolicy as well?

[erohmensing]

Histogram for sara on a held out test set with no normalization, normalizaton of the top 10 intents, and normalization of the top 5 intents, in that order

[erohmensing]

@Ghostvv I think it is just as easy to make the number configurable as to make it a boolean -- i've implemented it as such here (same for embedding policy), where 0 is no normalization. with respect to how to report the other intents, do you think it makes sense just to always report the number of intents that we have normalized, if we normalized them (essentially doing I would use LABEL_RANKING_LENGTH as normalization length,, but in reverse )? that's also how i went about it here.

Missed that you wanted to have it default:true, but figured i can wait to change that once we decide what the default number should be. Included histograms for sara, scores definitely become more intuitive with some level or normalization, just don't know how far to go with it

[wochinge]

@Ghostvv @erohmensing What's the state with this? Are you still planning to get this into Rasa 1.6?

[Ghostvv]

we need to introduce it to response selector as well. Please add docs

[erohmensing]

Everything should be implemented -- could use a few tests, let me know if you have any idea how to go about those (could count the number of non-zero actions and the length of the intent output)? Also feel free to change how I've explained it because 🤷‍♂

Would really like to get this on the train 🚋

[Ghostvv]

Also please add tests, we could create a couple of classifiers/policies, where we set different ranking_length (including 0) and check that the output is how we like it

[Ghostvv]

sorry, accidentally reviewed old commit, but comments are still valid

[wochinge]

@erohmensing @Ghostvv I'm sorry, but this missed the release train.

outdated