absolute positional encoding [abs_pos]

[poyupaulchen]

Hi all,
First appreciate the team providing such a great tool.
Currently I would like to try some experiments on the positional encoding for conformer. As I saw in the coding (see below), if I configure absolute positional encoding using
self_attention_model: abs_pos,
it will call the 'MultiHeadAttention'

NeMo/nemo/collections/asr/parts/submodules/conformer_modules.py

Lines 124 to 127 in df5a395

	elif self_attention_model == 'abs_pos':
	self.self_attn = MultiHeadAttention(
	n_head=n_heads, n_feat=d_model, dropout_rate=dropout_att, max_cache_len=MHA_max_cache_len
	)

And it seems like 'pos_emb' is never applied to the self attention part.

NeMo/nemo/collections/asr/parts/submodules/multi_head_attention.py

Lines 121 to 147 in df5a395

	def forward(self, query, key, value, mask, pos_emb=None, cache=None):
	"""Compute 'Scaled Dot Product Attention'.
	Args:
	query (torch.Tensor): (batch, time1, size)
	key (torch.Tensor): (batch, time2, size)
	value(torch.Tensor): (batch, time2, size)
	mask (torch.Tensor): (batch, time1, time2)
	cache (torch.Tensor) : (batch, time_cache, size)
	returns:
	output (torch.Tensor): transformed `value` (batch, time1, d_model) weighted by the query dot key attention
	cache (torch.Tensor) : (batch, time_cache_next, size)
	"""
	key, value, query, cache = self.update_cache(key=key, value=value, query=query, cache=cache)
	if torch.is_autocast_enabled():
	query, key, value = query.to(torch.float32), key.to(torch.float32), value.to(torch.float32)
	# temporary until we solve this more gracefully
	with avoid_float16_autocast_context():
	q, k, v = self.forward_qkv(query, key, value)
	scores = torch.matmul(q, k.transpose(-2, -1)) / self.s_d_k
	out = self.forward_attention(v, scores, mask)
	if cache is None:
	return out
	else:
	return out, cache

Does it mean that 'abs_pos' implies 'no positional encoding'? or I miss something.
Thank you,
Paul