Prune python tensor #5596

daquexian · 2021-07-26T03:05:25Z

将 python 的 Tensor 类删除，逻辑转移到 c++

>>> import oneflow as flow
>>> x=flow.Tensor(1,2)
>>> type(x)
<class 'oneflow.Tensor'>
>>> y=x+x
>>> type(y)
<class 'oneflow.Tensor'>
>>>

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian · 2021-07-26T03:11:36Z

oneflow/compatible_single_client_python/framework/tensor.py

@@ -13,55 +13,24 @@
 See the License for the specific language governing permissions and
 limitations under the License.
 """
-from oneflow.compatible.single_client.core.job import (


用 oneflow/python/framework/tensor.py 的内容覆盖了这个文件的原内容。因为它们共用了一个 _oneflow_internal

daquexian · 2021-07-26T03:12:29Z

oneflow/extension/python/py_compute.cpp


 namespace oneflow {
 namespace pyext {

 namespace {
 static PyObject* py_kernels_dic = nullptr;

-void OFDataTypeToNumpyType(DataType of_data_type, int* out_numpy_type) {


这两个方法移动到了 oneflow/extension/python/numpy.h 里并做了修改

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian · 2021-07-26T03:41:22Z

oneflow/python/framework/tensor.py

+    Tensor.normal_ = _normal
+    Tensor.fill_ = _fill
+    Tensor._placement_scope = _placement_scope
+    Tensor.copy_ = _copy


c++ tensor 类的一些方法在 python 层注入，暂时不着急全部重写到 c++

Signed-off-by: daquexian <daquexian566@gmail.com>

oneflow/api/python/framework/device.cpp

oneflow/api/python/framework/tensor.cpp

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian · 2021-07-26T05:59:11Z

cmake/pybind11.cmake

@@ -1,7 +1,7 @@
 include(FetchContent)
-set(PYBIND11_TAR_URL https://github.com/pybind/pybind11/archive/v2.6.0.zip)
+set(PYBIND11_TAR_URL https://github.com/pybind/pybind11/archive/v2.7.0.zip)


升级 pybind11 把这个修复 gcc 4 编译错误的 pr 包含进来 pybind/pybind11#2956

wyg1997

Parameter的构造函数里是不是忘处理了？

wyg1997 · 2021-07-26T10:32:00Z

oneflow/api/python/framework/tensor.cpp

+    } else {
+      return MakeLocalTensorByNumpy(arg, desired_dtype, device, requires_grad);
+    }
+  } else {


这里需要和Pytorch那样，如果参数检察不正确，把所有的合法参数都打印出来给个提示吗？

感觉可以 😂

Signed-off-by: daquexian <daquexian566@gmail.com>

lixinqi · 2021-07-27T11:20:48Z

oneflow/api/python/framework/device.cpp

+  std::string type;
+  int device_id = -1;
+  ParsingDeviceTag(type_and_id, &type, &device_id).GetOrThrow();
+  if (device_id == -1) { device_id = 0; }
+  return MakeDevice(type, device_id);


记得新开一个issue，让新手对这里做优化。用thread_local作缓存。

lixinqi · 2021-07-27T11:22:38Z

oneflow/core/framework/tensor.h

+  }
+
+  bool requires_grad() const override { return tensor_->requires_grad(); }
+  bool is_leaf() const override { return tensor_->is_leaf(); }


是不是应该直接返回true呢？

是的，已修改，同时通过让 Parameter 继承 TensorIf，也实现了 Parameter 有独立的 grad_fn

lixinqi · 2021-07-27T11:23:51Z

oneflow/core/framework/tensor.h

+  user_op::TensorDesc* mut_tensor_meta() override { return tensor_->mut_tensor_meta(); }
+
+  Maybe<MirroredTensor> AsMirroredTensor() override {
+    if (const auto& mirrored_tensor = std::dynamic_pointer_cast<MirroredTensor>(tensor_)) {


这语法是c++11所能支持的吗？

是的，这个和 c++17 的 if (int x = 0; x < 1) 不是同一个语法

哦哦。我才注意到两者的区别，那我还把很多的宏写复杂了。

oneflow/core/framework/tensor.h

oneflow/api/python/framework/tensor.cpp

…d missing get_device method, remove out-dated methods in docs Signed-off-by: daquexian <daquexian566@gmail.com>

Signed-off-by: daquexian <daquexian566@gmail.com>

…into prune_python_tensor

Signed-off-by: daquexian <daquexian566@gmail.com>

github-actions · 2021-07-28T01:46:59Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 141.0ms (= 7050.6ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 126.3ms (= 6313.8ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 141.0ms / 126.3ms)

PyTorch resnet50 time: 84.7ms (= 4236.8ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.7ms (= 3683.9ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 84.7ms / 73.7ms)

PyTorch resnet50 time: 60.3ms (= 3016.7ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 49.2ms (= 2459.6ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.23 (= 60.3ms / 49.2ms)

PyTorch resnet50 time: 51.2ms (= 2558.2ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 45.7ms (= 2286.1ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 51.2ms / 45.7ms)

PyTorch resnet50 time: 44.0ms (= 2199.3ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 41.9ms (= 2094.6ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.05 (= 44.0ms / 41.9ms)

daquexian added 3 commits July 26, 2021 10:23

move tensor to c++

5da3203

Signed-off-by: daquexian <daquexian566@gmail.com>

Merge remote-tracking branch 'origin/master' into prune_python_tensor

0765924

Signed-off-by: daquexian <daquexian566@gmail.com>

align with master and fix export error

2369658

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian added enhancement system labels Jul 26, 2021

reformat

9b0d522

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian commented Jul 26, 2021

View reviewed changes

rename some functions

08e4409

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2021 03:19

daquexian force-pushed the prune_python_tensor branch from f423439 to 003787a Compare July 26, 2021 03:38

refine

ffa1047

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian force-pushed the prune_python_tensor branch from 003787a to ffa1047 Compare July 26, 2021 03:39

daquexian commented Jul 26, 2021

View reviewed changes

remove unused header

7a93ba0

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2021 03:53

oneflow-ci-bot removed their request for review July 26, 2021 05:16

lixinqi reviewed Jul 26, 2021

View reviewed changes

oneflow/api/python/framework/device.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/device.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

daquexian added 2 commits July 26, 2021 13:37

restore device repr and str

0c7a413

Signed-off-by: daquexian <daquexian566@gmail.com>

upgrade pybind11 to 2.7.0 for pybind/pybind11#2956

9596fd8

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian commented Jul 26, 2021

View reviewed changes

daquexian requested a review from oneflow-ci-bot July 26, 2021 06:00

lixinqi approved these changes Jul 26, 2021

View reviewed changes

oneflow-ci-bot removed their request for review July 26, 2021 07:09

wyg1997 reviewed Jul 26, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into prune_python_tensor

a87b836

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian force-pushed the prune_python_tensor branch from 0e80934 to a87b836 Compare July 27, 2021 01:19

impl parameter in c++, remove determine() call

11ebffd

Signed-off-by: daquexian <daquexian566@gmail.com>

oneflow-ci-bot removed their request for review July 27, 2021 11:15

lixinqi reviewed Jul 27, 2021

View reviewed changes

hjchen2 reviewed Jul 27, 2021

View reviewed changes

oneflow/core/framework/tensor.h Show resolved Hide resolved

hjchen2 reviewed Jul 27, 2021

View reviewed changes

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

daquexian added 4 commits July 27, 2021 20:57

let parameter derive from TensorIf to own a separate grad_fn_node, ad…

2f2a6e0

…d missing get_device method, remove out-dated methods in docs Signed-off-by: daquexian <daquexian566@gmail.com>

Merge remote-tracking branch 'origin/master' into prune_python_tensor

72a92ba

Merge remote-tracking branch 'origin/master' into prune_python_tensor

f654490

add test_parameter.py, fix requires_grad_ and get_device

1d43c60

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian requested a review from oneflow-ci-bot July 27, 2021 14:06

use oneflow cast instead of numpy cast

1ecdd29

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian added automerge and removed automerge labels Jul 27, 2021

restore format changes

63d683b

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 27, 2021 14:34

auto format by CI

0d52ea8

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 27, 2021 14:37

daquexian added the automerge label Jul 27, 2021

daquexian mentioned this pull request Jul 27, 2021

ddp #5008

Merged

oneflow-ci-bot removed their request for review July 27, 2021 16:02

daquexian added 3 commits July 28, 2021 09:00

fix tests involving _local_or_consistent_tensor

6a5b27b

Signed-off-by: daquexian <daquexian566@gmail.com>

Merge branch 'prune_python_tensor' of github.com:Oneflow-Inc/oneflow …

6080c68

…into prune_python_tensor

Merge remote-tracking branch 'origin/master' into prune_python_tensor

03fd902

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian requested a review from oneflow-ci-bot July 28, 2021 01:03

oneflow-ci-bot removed their request for review July 28, 2021 02:04

oneflow-ci-bot merged commit 3a2b338 into master Jul 28, 2021

oneflow-ci-bot deleted the prune_python_tensor branch July 28, 2021 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prune python tensor #5596

Prune python tensor #5596

daquexian commented Jul 26, 2021 •

edited

Loading

daquexian Jul 26, 2021

daquexian Jul 26, 2021

daquexian Jul 26, 2021

daquexian Jul 26, 2021

wyg1997 left a comment

wyg1997 Jul 26, 2021

daquexian Jul 27, 2021

lixinqi Jul 27, 2021

lixinqi Jul 27, 2021

daquexian Jul 27, 2021

lixinqi Jul 27, 2021

daquexian Jul 27, 2021

lixinqi Jul 27, 2021

github-actions bot commented Jul 28, 2021

Prune python tensor #5596

Prune python tensor #5596

Conversation

daquexian commented Jul 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wyg1997 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 28, 2021

daquexian commented Jul 26, 2021 •

edited

Loading