Symbols and non-word characters

[JuanFF]

Hello,
I see a lot of special characters (e.g. >>) and annotations between brackets (e.g. [MUSIC]) in the transcripts. Is there any key with explanations on what these symbols are used for?
Thanks!

[jianfch]

It's likely the a way of mapping the speech that the model picked up from the raw text in transcripts of the training data which received no "significant standardization" as described in the paper.

[jongwook]

This is correct; we fine-tuned the model to make it less likely to produce those, but it may still sample those non-speech annotations. We have a mechanism to prevent sampling those tokens by default:

whisper/whisper/tokenizer.py

Lines 235 to 270 in 9e653bd

	@property
	@lru_cache()
	def non_speech_tokens(self) -> Tuple[int]:
	"""
	Returns the list of tokens to suppress in order to avoid any speaker tags or non-speech
	annotations, to prevent sampling texts that are not actually spoken in the audio, e.g.
	- ♪♪♪
	- ( SPEAKING FOREIGN LANGUAGE )
	- [DAVID] Hey there,
	keeping basic punctuations like commas, periods, question marks, exclamation points, etc.
	"""
	symbols = list("\"#()*+/:;<=>@[\\]^_`{|}~「」『』")
	symbols += "<< >> <<< >>> -- --- -( -[ (' (\" (( )) ((( ))) [[ ]] {{ }} ♪♪ ♪♪♪".split()
	# symbols that may be a single token or multiple tokens depending on the tokenizer.
	# In case they're multiple tokens, suppress the first token, which is safe because:
	# These are between U+2640 and U+267F miscellaneous symbols that are okay to suppress
	# in generations, and in the 3-byte UTF-8 representation they share the first two bytes.
	miscellaneous = set("♩♪♫♬♭♮♯")
	assert all(0x2640 <= ord(c) <= 0x267F for c in miscellaneous)
	# allow hyphens "-" and single quotes "'" between words, but not at the beginning of a word
	result = {self.tokenizer.encode(" -")[0], self.tokenizer.encode(" '")[0]}
	for symbol in symbols + list(miscellaneous):
	for tokens in [self.tokenizer.encode(symbol), self.tokenizer.encode(" " + symbol)]:
	if len(tokens) == 1 or symbol in miscellaneous:
	result.add(tokens[0])
	return tuple(sorted(result))
	def _get_single_token_id(self, text) -> int:
	tokens = self.tokenizer.encode(text)
	assert len(tokens) == 1, f"{text} is not encoded as a single token"
	return tokens[0]