Add pil backend for vision transforms #28035

LielinJiang · 2020-10-16T15:26:28Z

PR types

New features

PR changes

APIs

Describe

添加了pil处理图片的后端
添加了参数keys，便于后续将这些算子扩展到其余组件

文档地址：http://gzhxy-inf-bce36a39de.gzhxy.baidu.com:8121/documentation/docs/en/api/paddle/vision/BaseTransform_en.html

paddle-bot-old · 2020-10-16T15:26:41Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

saxon-zh

LGTM

saxon-zh

LGTM

qingqing01 · 2020-10-19T02:11:08Z

python/paddle/tests/test_transforms.py

@@ -18,14 +18,21 @@
 import cv2


lazy import cv2 here

感谢提醒，在functional中改为了lazy_import，这个是单测，无需lazy_import。

qingqing01 · 2020-10-19T02:18:23Z

python/paddle/vision/transforms/functional.py

-    Accordding to the code (the type of flip), flip the input image
+
+def _is_numpy_image(img):
+    return isinstance(img, np.ndarray) and (img.ndim in {2, 3})


Maybe add note, not process the 4D image

done, thanks

qingqing01 · 2020-10-19T02:21:04Z

python/paddle/vision/transforms/functional.py

-            0 : Flip vertically
-            1 : Flip horizontally
+        pic (PIL.Image|np.ndarray): Image to be converted to tensor.
+        data_format (str, optional): Data format of img, should be 'HWC' or 


the comments of data_format need more clearly for input image or returned image :

"Data format of returned image, ... "

done, thanks

qingqing01 · 2020-10-19T02:42:08Z

python/paddle/vision/transforms/transforms.py

    """

-    def __init__(self, transforms=[]):
-        self.transforms = transforms
+    def __init__(self, keys=None, backend='pil'):


有个全局的切换接口觉得更易用，而不是每个接口都加backend, functional的实现有同样的问题

XiaoguangHu01 · 2020-10-19T02:25:42Z

python/paddle/tests/test_transforms.py

@@ -18,14 +18,21 @@
 import cv2


paddle默认不安装cv2的话，这里的cv2依赖哪个特定的版本呢？
import失败的时候，需要提示用户手动安装特定的cv2版本。

感谢提醒，在functional中改为了lazy_import，这个是单测，无需lazy_import。默认还是让用户使用最新版本的opencv。

XiaoguangHu01 · 2020-10-19T02:27:50Z

python/paddle/tests/test_transforms.py

    def setUp(self):
+        self.backend = self.set_backend()


get_backend

done, thanks

XiaoguangHu01 · 2020-10-19T02:28:06Z

python/paddle/tests/test_transforms.py

@@ -40,146 +47,207 @@ def setUp(self):
                        (400, 300, 3)) * 255).astype('uint8')
                cv2.imwrite(os.path.join(sub_dir, str(j) + '.jpg'), fake_img)

+    def set_backend(self):


get_backend

done, thanks

XiaoguangHu01 · 2020-10-19T02:40:59Z

python/paddle/tests/test_transforms.py

+
+
+class TestTransformsPIL(TestTransformsCV2):
+    def set_backend(self):


get_backend

done, thanks

XiaoguangHu01 · 2020-10-19T02:49:47Z

python/paddle/vision/transforms/functional.py



-@keepdims
-def resize(img, size, interpolation=1):
+def resize(img, size, interpolation='bilinear', backend='pil'):


这里的backend='pil'，建议增加backend状态，可以简化backend切换的过程，比如：
默认设置为backend=None，表示采用缺省设置，然后增加
transforms.set_default_backend(backend)
transforms.get_default_backend()
transforms._defaut_backend = 'pil'
这样的话，用户在可以避免大量手动切换代码，比如：
resize(..., backend=self.backend)

qingqing01

LGTM

qingqing01 · 2020-10-19T09:15:28Z

python/paddle/vision/transforms/transforms.py

+            # try:
+            data = f(data)
+            if data is None:
+                print('deubgggggggg:', f)


clean debug code

done, thanks

qingqing01 · 2020-10-19T09:31:15Z

python/paddle/vision/transforms/transforms.py

+    In the other cases, tensors are returned without scaling.
+
+    Args:
+        size (int|list|tuple): Desired output size. If size is a sequence like


to_tensor的init和call函数都没有size参数

updated, thanks

qingqing01 · 2020-10-19T09:46:52Z

python/paddle/vision/transforms/functional_pil.py

+    Pads the given PIL.Image on all sides with specified padding mode and fill value.
+
+    Args:
+        img (PIL.Image|np.array): Image to be padded.


下面实现支持np.array吗？

已经更新，functional_pil.py，functional_cv.py，functonal_tensor.pil还不暴露给用户。暂时可以先关注functional.py中的注释和示例

qingqing01 · 2020-10-19T09:47:32Z

python/paddle/vision/transforms/functional_pil.py

+            img = np.pad(img, ((pad_top, pad_bottom), (pad_left, pad_right)),
+                         padding_mode)
+
+        return Image.fromarray(img)


从实现来看，不支持np.array的输入，注意上面的comments是否正确

已经更新，functional_pil.py，functional_cv.py，functonal_tensor.pil还不暴露给用户。暂时可以先关注functional.py中的注释和示例

LielinJiang · 2020-10-19T10:28:22Z

经过讨论，backend参数实际上只起到double check的作用，如果用户传入的图片类型和backend不符就直接报错了，而且这样的check也会降低性能。最后决定去除backend参数，根据用户传入的图片类型，调用相应backend的算子。同时，在读取图片的地方，增加了set_image_backend和get_image_backend函数，来控制想要的图片类型。

qingqing01

LGTM.

utils.py may change to image.py or image_io.py

LielinJiang · 2020-10-19T16:05:30Z

LGTM.

utils.py may change to image.py or image_io.py

done, thanks

XiaoguangHu01

LGTM

saxon-zh

LGTM

jzhang533 · 2020-10-20T03:16:04Z

python/paddle/vision/image.py

+        return Image.open(path)
+    else:
+        cv2 = try_import('cv2')
+        return cv2.imread(path)


这段代码可以重新组织下，比如：

if backend == 'pil': ... elif backend == 'cv2': ... else raise ValueError

嗯嗯，这个后续修改一下。目前来看并不影响

jzhang533 · 2020-10-20T03:19:36Z

python/paddle/vision/transforms/functional.py

+def to_tensor(pic, data_format='CHW'):
+    """Converts a ``PIL.Image`` or ``numpy.ndarray`` to paddle.Tensor.
+
+    See ``ToTensor`` for more details.


最好写ToTensor的全路径

好的，文档会在下个pr优化一下

jzhang533 · 2020-10-20T03:27:02Z

python/paddle/vision/transforms/functional.py

+    if _is_pil_image(pic):
+        return F_pil.to_tensor(pic, data_format)
+    else:
+        return F_cv2.to_tensor(pic, data_format)


从np array转成tensor的功能是放在了functional_cv2.py 下呀。

是的，都是从input的数据类型来区分的

jzhang533 · 2020-10-20T03:42:16Z

python/paddle/vision/transforms/functional.py

+    if _is_pil_image(img):
+        return F_pil.pad(img, padding, fill, padding_mode)
+    else:
+        return F_cv2.pad(img, padding, fill, padding_mode)


这里的逻辑（及其他类似的对F_cv2 的调用逻辑），有些问题。
用户设置的backend是pillow的时候，传入的img是np array时，走到了lazy import cv2的分支里。

@XiaoguangHu01 @saxon-zh
麻烦看看这个问题，是否要改。

线下讨论解决

LielinJiang added 2 commits October 16, 2020 14:58

add pil backend

2f81c3b

update comment

e8d1f06

LielinJiang requested a review from qingqing01 October 16, 2020 15:26

LielinJiang added 4 commits October 17, 2020 00:57

fix errors

3bb04a8

add example codes

39dbd62

fix py2

5e935fb

fix unittest

9858af4

PaddlePaddle locked and limited conversation to collaborators Oct 17, 2020

PaddlePaddle unlocked this conversation Oct 17, 2020

fix docs

c84449d

PaddlePaddle locked and limited conversation to collaborators Oct 17, 2020

PaddlePaddle unlocked this conversation Oct 17, 2020

PaddlePaddle locked and limited conversation to collaborators Oct 18, 2020

PaddlePaddle unlocked this conversation Oct 18, 2020

LielinJiang added 4 commits October 18, 2020 08:30

fix docs, test=document_fix

b8196ba

fix docs, test=document_fix

2617942

fix docs, test=document_fix

1f6a7e6

fix docs, test=document_fix

8b963fe

LielinJiang requested review from saxon-zh and XiaoguangHu01 October 18, 2020 10:11

fix docs, test=document_fix

9209514

saxon-zh previously approved these changes Oct 18, 2020

View reviewed changes

fix docs, test=document_fix

2ea1485

LielinJiang dismissed saxon-zh’s stale review via 2ea1485 October 18, 2020 14:05

LielinJiang requested a review from saxon-zh October 19, 2020 01:46

saxon-zh previously approved these changes Oct 19, 2020

View reviewed changes

qingqing01 reviewed Oct 19, 2020

View reviewed changes

XiaoguangHu01 reviewed Oct 19, 2020

View reviewed changes

refine code

28a9697

LielinJiang dismissed saxon-zh’s stale review via 28a9697 October 19, 2020 07:30

qingqing01 reviewed Oct 19, 2020

View reviewed changes

LielinJiang added 2 commits October 19, 2020 10:15

clean code, fix coverage

a3da96d

update comment

eadc85b

LielinJiang added 4 commits October 19, 2020 11:37

add set_backend get_backend

d203c26

add set_backend get_backend

66c10a3

fix sample code

e537a56

add set_backend get_backend

b6e612e

LielinJiang requested review from qingqing01, saxon-zh and XiaoguangHu01 October 19, 2020 12:22

LielinJiang added 3 commits October 19, 2020 13:02

fix sample code

4df5b62

fix unittest

16267c3

add image load

4229e88

qingqing01 previously approved these changes Oct 19, 2020

View reviewed changes

LielinJiang requested a review from jzhang533 October 19, 2020 15:39

change utils to image

2eb23f0

LielinJiang dismissed qingqing01’s stale review via 2eb23f0 October 19, 2020 16:05

LielinJiang requested a review from qingqing01 October 19, 2020 16:05

XiaoguangHu01 approved these changes Oct 20, 2020

View reviewed changes

qingqing01 approved these changes Oct 20, 2020

View reviewed changes

saxon-zh approved these changes Oct 20, 2020

View reviewed changes

jzhang533 reviewed Oct 20, 2020

View reviewed changes

LielinJiang requested a review from raindrops2sea October 20, 2020 05:14

raindrops2sea approved these changes Oct 20, 2020

View reviewed changes

LielinJiang merged commit 74c8a81 into PaddlePaddle:develop Oct 20, 2020

LielinJiang mentioned this pull request Oct 20, 2020

[cherry-pick] Add pil backend for transforms api #28132

Merged



		class TestTransformsPIL(TestTransformsCV2):
		def set_backend(self):

Add pil backend for vision transforms #28035

Add pil backend for vision transforms #28035

Conversation

LielinJiang commented Oct 16, 2020 • edited Loading

PR types

PR changes

Describe

paddle-bot-old bot commented Oct 16, 2020

saxon-zh left a comment

Choose a reason for hiding this comment

saxon-zh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LielinJiang commented Oct 19, 2020

qingqing01 left a comment • edited Loading

Choose a reason for hiding this comment

LielinJiang commented Oct 19, 2020

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

saxon-zh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LielinJiang commented Oct 16, 2020 •

edited

Loading

qingqing01 left a comment •

edited

Loading