U-Net训练问题

[gxqnba]

数据检校全部通过，但是进行训练时，提示reader出错，报错如下：
Use multi-thread reader
2020-09-11 08:25:32,431-WARNING: Your reader has raised an exception!
Exception in thread Thread-2:
Traceback (most recent call last):
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1156, in thread_main
six.reraise(*sys.exc_info())
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1136, in thread_main
for tensors in self._tensor_reader():
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/data_feeder.py", line 195, in call
for each_sample in self.generator():
File "pdseg/train.py", line 207, in data_generator
for b in data_gen:
File "/home/aistudio/work/PaddleSeg/pdseg/reader.py", line 81, in generator
yield self.process_image(line, self.data_dir, self.mode)
File "/home/aistudio/work/PaddleSeg/pdseg/reader.py", line 253, in process_image
img, grt = aug.resize(img, grt, mode)
File "/home/aistudio/work/PaddleSeg/pdseg/data_aug.py", line 44, in resize
grt = cv2.resize(grt, target_size, interpolation=cv2.INTER_NEAREST)
TypeError: Expected Ptrcv::UMat for argument '%s'

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py:1070: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")

---

C++ Call Stacks (More useful to developers):

0 std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1 paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2 paddle::operators::reader::BlockingQueue<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor > >::Receive(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >)
3 paddle::operators::reader::PyReader::ReadNext(std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >)
4 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&)
5 std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
6 ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

---

Python Call Stacks (More useful to users):

File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2610, in append_op
attrs=kwargs.get("attrs", None))
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 1078, in _init_non_iterable
attrs={'drop_last': self._drop_last})
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 976, in init
self._init_non_iterable()
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/reader.py", line 608, in from_generator
iterable, return_list, drop_last)
File "/home/aistudio/work/PaddleSeg/pdseg/models/model_builder.py", line 142, in build_model
use_double_buffer=True)
File "pdseg/train.py", line 237, in train
train_prog, startup_prog, phase=ModelPhase.TRAIN)
File "pdseg/train.py", line 453, in main
train(cfg)
File "pdseg/train.py", line 466, in
main(args)

---

Error Message Summary:

Error: Blocking queue is killed because the data reader raises an exception
[Hint: Expected killed_ != true, but received killed_:1 == true:1.] at (/paddle/paddle/fluid/operators/reader/blocking_queue.h:141)
[operator < read > error]
W0911 08:25:32.449854 315 operator.cc:187] read raises an exception std::future_error, St12future_error
F0911 08:25:32.449900 315 exception_holder.h:37] std::exception caught, No associated state
*** Check failure stack trace: ***
@ 0x7f51bb26b7dd google::LogMessage::Fail()
@ 0x7f51bb26f28c google::LogMessage::SendToLog()
@ 0x7f51bb26b303 google::LogMessage::Flush()
@ 0x7f51bb27079e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f51be2237c8 paddle::framework::details::ExceptionHolder::Catch()
@ 0x7f51be2be72e paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync()
@ 0x7f51be2bc0cf paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp()
@ 0x7f51be2bc394 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data
@ 0x7f51bb2c8ca3 std::_Function_handler<>::_M_invoke()
@ 0x7f51bb0c74c7 std::__future_base::_State_base::_M_do_set()
@ 0x7f5233313a99 __pthread_once_slow
@ 0x7f51be2b8562 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7f51bb0c9924 _ZZN10ThreadPoolC1EmENKUlvE_clEv
@ 0x7f51f3d00421 execute_native_thread_routine_compat
@ 0x7f523330c6ba start_thread
@ 0x7f523304241d clone
@ (nil) (unknown)
Aborted (core dumped)

[wuyefeilin]

根据提示是在进行resize的时候标签的格式不正确， 检查一下， 或者在这一行之前打印看一下grt的类型