group null judge fix

[TimeYWL]

PR types

Bug fixes

PR changes

Others

Description

According to the api(paddle/distributed/communication/group.py:67:

def is_member(self):
    if self.rank < 0:
        return False
    if self.nranks < 2:
        return False
    return True

and the group build code:

if size > 1 and global_rank in ranks:
    rank = 0 if backend == 'heter' else ranks.index(global_rank)
    pg = _new_process_group_impl(
        backend,
        _default_store,
        rank,
        size,
        group_name,
        pg_options=None,
        group_id=gid,
    )
else:
    rank = -1
    pg = None

The code in PaddleNLP/paddlenlp/data/dist_dataloader.py, egg line 155:
if self.mp_group is not None and self.pp_rank == 0:
can not determine the existence of 'mp_group' correctly.

If there is no model_parallel, mp_group will be:
rank: -1, nranks: 1, id: 12, ranks: 0; name: _default_pg12
The broadcast_data_list() will cased error:

Traceback (most recent call last):
  File "run_pretrain.py", line 567, in <module>
    main()
  File "run_pretrain.py", line 549, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/workspace/PaddleNLP/paddlenlp/trainer/trainer.py", line 738, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/workspace/PaddleNLP/paddlenlp/data/dist_dataloader.py", line 181, in __next__
    data_list = broadcast_data_list(data_list, paddle.int64, self.mp_rank, self.mp_group, self.mp_src_rank)
  File "/workspace/PaddleNLP/paddlenlp/data/dist_dataloader.py", line 210, in broadcast_data_list
    paddle.distributed.broadcast(size_cuda, src_rank, group=comm_group).wait()
AttributeError: 'NoneType' object has no attribute 'wait'

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Merging #7122 (4d043d1) into develop (9c3f8a4) will decrease coverage by 0.01%.
Report is 4 commits behind head on develop.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop    #7122      +/-   ##
===========================================
- Coverage    59.64%   59.64%   -0.01%     
===========================================
  Files          563      563              
  Lines        82644    82645       +1     
===========================================
  Hits         49291    49291              
- Misses       33353    33354       +1     

Files	Coverage Δ
paddlenlp/data/dist_dataloader.py	14.78% <0.00%> (ø)

... and 4 files with indirect coverage changes

[DesmonDay]

LGTM