Encoder and Decoder has different TP_SIZE

[heavyrain-lzy]

Your question

1. I have a question about creating the pp groups when enabling context_parallel_size > 1 and encoder_tensor_parallel_size != tensor_parallel_size.

When enabling context_parallel, the input will be split symmetrically to balance the calculation. Using zip(cycle(e_ranks), d_ranks) is wrong.

Megatron-LM/megatron/core/parallel_state.py

Line 602 in 4ff9e66

# Map 1 encoder tp rank to several decoder tp ranks, because

2. Why do we use stack operator to calculate the sum of received tensor.

   Megatron-LM/megatron/core/pipeline_parallel/p2p_communication.py

   Line 402 in 46ca068

   return torch.stack(x, dim=0).sum(dim=0, dtype=torch.float32).to(x[0].dtype)