How to make BlenderBot generate from an unfinished sentence?

[tungnt55]

Assume we have a conversation like follow:

Human: Can you tell me about your job?
BlenderBot: Sure, I'm happy to talk about it. I am working as ...

I want to use BlenderBot to continue to generate response that follows the phrase "I am working as ". What's the best way to do this?

[klshuster]

One way to do this via command line would be to have BB's "prompt" message in a temp_history key in the observed message:

from parlai.core.agents import create_agent_from_model_file

blenderbot = create_agent_from_model_file()
blenderbot.observe({'text': 'Can you tell me about your job?', 'temp_history': "\nSure, I'm happy to talk about it. I am working as", "episode_done": False})
response = blenderbot.act()['text']

The issue here is that we'll still have an EOS token at the end of the full contextual sentence. A more complex way to go about this, then, is to implement an agent that implements get_prefix_tokens; see e.g. this test agent, which forces the model's generation to begin with '4 3 2 1'. In your case, then, you'd implement get_prefix_tokens to return your own set of tokens. Then, you could just add --model my_model to your args to use your new agent.

Incorporating something like a "prompt" into the top-level message would be useful, however. Presumably, the agent would set its prefix tokens to whatever is in the 'prompt' key, or something like that.

[tungnt55]

Thank you for the reply! I tried the way with 'temp_history' but it didn't work well. The chatbot treats the 'prompt' text as part of the user's utterance.
Looks like the only viable option is to implement the agent as you said.

[klshuster]

Yeah, I was trying locally to see whether the temp_history would work but doesn't seem to be what you're looking for - prefix tokens is definitely the way to go

[tungnt55]

I have tried to implement the agent with get_prefix_tokens(). However, I got the following error:

File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_agent.py", line 2148, in act
    response = self.batch_act([self.observation])[0]
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_agent.py", line 2244, in batch_act
    output = self.eval_step(batch)
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/projects/blenderbot2/agents/blenderbot2.py", line 862, in eval_step
    output = super().eval_step(batch)
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/agents/rag/rag.py", line 300, in eval_step
    output = super().eval_step(batch)
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_generator_agent.py", line 901, in eval_step
    beam_preds_scores, beams = self._generate(
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/agents/rag/rag.py", line 684, in _generate
    gen_outs = self._rag_generate(batch, beam_size, max_ts, prefix_tokens)
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/agents/rag/rag.py", line 727, in _rag_generate
    return self._generation_agent._generate(
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_generator_agent.py", line 1241, in _generate
    n_best_beam_preds_scores = [b.get_rescored_finished() for b in beams]
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_generator_agent.py", line 1241, in <listcomp>
    n_best_beam_preds_scores = [b.get_rescored_finished() for b in beams]
  File "/home/tung/Desktop/ros/haru_chat_ws/src/tools/parlai/core/torch_generator_agent.py", line 1758, in get_rescored_finished
    assert (pred == self.eos).sum() == 1, (
AssertionError: TreeSearch returned a finalized hypo with multiple end tokens with score -2494879019720245248.00

I guess the tokens of input prompt is not one of the candidate tokens when doing beam search, so the final score is always set to the smallest negative number.

[klshuster]

how did you set the tokens exactly?

[tungnt55]

I created a new field in the BlenderBotFidAgent called prefix_text, which is the prompt text. Then, in the get_prefix_tokens(), I have the following lines:

prefix_toks = self.dict.txt2vec(self.prefix_text)
prefix_toks_batch = [prefix_toks for _ in range(bsz)]
return torch.LongTensor(prefix_toks_batch).to(dev)