Add a global not_intent parameter for the slot_mappings

[akelad]

Description of Problem:
Right now a Form ignores the confidence of an intent when filling a slot, the FallbackPolicy doesn't jump in. There's no option to configure that (see #256).
With 2.0, that behaviour changes, because the FallbackClassifier overrides the intent with nlu_fallback. So you can specify not_intent=nlu_fallback on an individual slot_mapping level to make sure the Form doesn't use a user message that is below the confidence threshold. It would be great to have a global option to this, so you don't have to define it on a per slot basis

Overview of the Solution:
We have a global parameter in the Form, where you can define that you would like slots to never be filled, if the intent doesn't pass the confidence threshold (and therefore has the intent nlu_fallback). We could even generalise this to have a global not_intent setting for other intents too, e.g. you might never want to fill any slots with chitchat

[akelad]

cc @Ghostvv

[wochinge]

@RasaHQ/production-squad Unsure about this one. Future or backlog?

[wochinge]

backlog it is

[wochinge]

This is actually a Rasa Open Source issue as we moved the forms from the SDK to Rasa Open Source with the release of Rasa Open Source 2.0

[Imod7]

So here are my thoughts (mostly questions) on this issue which I share with @RasaHQ/enable-squad since I do not have enough knowledge on this to completely understand the related concepts, correctly assess the issue and even do a proposal.

First some questions that come into my mind by reading the very detailed and clear description.

1. Why do we have the individual not_intent option per slot ? What purpose does it serve? I mean why we would like to define it in a "per slot" level since :
   • the functionality can be covered from the FallbackClassifier and when it gets activated then it could not fill all the slots that have lower confidence
   • or the slot filling could be covered since we could say that we do not fill all the slots that have intent chitchat.
2. Slightly similar to the previous question, as soon as the confidence of the intent is below the threshold, the FallbackClassifier "takes over" right? So that looks to me like a "global variable" functionality and by that, I mean a functionality that works like having a global variable. Or maybe not?
3. Assuming that I already understood the previous purposes, if we have a not_intent option set per slot, I definitely see the usefulness of having a global parameter. But again some other questions arise:
   • If we set a global not_intent variable and we also have individual ones which one prevails ?
   • If nothing is set what happens ?
4. Regarding the "slots to never be filled" part. Can this be covered with the FallbackClassifier functionality ? slot with lower confidence --> never fill its slot.

Regarding the code, I checked where a global not_intent parameter might fit and I saw that the related class is FormValidationAction but also the class Tracker (which maintains the state of the conversation) seems a good place to have it. Like having a method like def set_global_not_intent? I was also looking at the class Action?
Also, one question on the code : if forms moved to Rasa Open Source then why the forms.py file is in Rasa SDK?

I think that maybe it would be very helpful if someone could also outline different test cases, eg. in test case A, the actual result is X and the desired one is Y because of reason B.

Thank you so much in advance for any answer/comment/advice.

[wochinge]

Forms overrule everything else unless they previously rejected their execution (for the sake of simplicity you can just assume form == loop) (this e.g. happens when the user didn't fill any slot after the form requested a slot to be filled). This means that the FallbackClassifier prediction is logged onto the tracker, but doesn't affect the prediction of the form.

Why do we have the individual not_intent option per slot

Imagine a form wants to fill a slot number_of_tables with an entity number. Imagine further that some reason there is an intent tell_age. If the form asks the user How many tables do you need? and the user says I'm 30, the form would see an intent tell_age + an extracted entity of type number (with value 30). If you just have a slot mapping of type entity the form would then fill number_of_tables which obviously doesn't make sense. You can use intents or not_intents to precisely define to which intents a slot mapping should apply and hence avoid the case from above where an intent tell_age fills a slot number_of_tables. Does that make sense?

Regarding the "slots to never be filled" part. Can this be covered with the FallbackClassifier functionality ? slot with lower confidence --> never fill its slot.

The FallbackClassifier is a NLU component and has only little effect on forms. The only thing which is exposed to forms is the prediction of an intent nlu_fallback.

Regarding the code, I checked where a global not_intent parameter might fit and I saw that the related class is FormValidationAction but also the class Tracker (which maintains the state of the conversation) seems a good place to have it. Like

FormValidationAction is in the SDK but this feature needs to be in Rasa Open Source as the default slot mappings are all applied within Rasa Open Source. If the users uses a custom action to validate slots they can exclude certain intents by writing some Python Code lines.

I'd imagine something like the following:

1. Users provide this flag in the domain.yml
2. The Domain class reads the information from the file (somewhere around this code) and stores it within it's attributes.
3. The FormAction retrieves this information from the domain (e.g. here) and then applies them during the extraction of the slots (an alternative could be to convert the "global" not_intent thing to a not_intents flag for each intent. In this case you'd have just change code in the domain 🤔