From 22edfeb01c38f98b72ce351bddbad9e6cedcabf4 Mon Sep 17 00:00:00 2001 From: lizechng Date: Wed, 30 Nov 2022 21:57:44 +0800 Subject: [PATCH] dynamic/static graph translation unified --- docs/guides/jit/debugging_en.md | 20 ++++++++++---------- docs/guides/jit/grammar_list_en.md | 6 +++--- docs/guides/jit/index_en.rst | 8 ++++---- docs/guides/performance_improving/amp_en.md | 4 ++-- 4 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/guides/jit/debugging_en.md b/docs/guides/jit/debugging_en.md index 366c7f796ca..cfb8d6cae49 100644 --- a/docs/guides/jit/debugging_en.md +++ b/docs/guides/jit/debugging_en.md @@ -1,8 +1,8 @@ # Error Debugging Experience -## 1、Dygraph to Static Graph Error Log +## 1、Dynamic Graph to Static Graph Error Log ### 1.1 How to Read the Error Log -The following is an example code of Dygraph-to-Static error reporting: +The following is an example code of Dynamic-to-Static error reporting: ```python import paddle import numpy as np @@ -28,7 +28,7 @@ The error log can be divided into 4 parts from top to bottom: - **Native Python error stack**: As shown in the first two lines, it represents a series of subsequent errors caused by the function `train()` called on line 145 of the `/workspace/Paddle/run_dy2stat_error.py` file. -- **The start flag of Dygraph-to-Static error stack**: `In transformed code`, represents the dynamic-to-static error message stack, and refers to the error message when the transformed code is running. In the actual scene, you can directly search for the `In transformed code` keyword, and start from this line and read the error log. +- **The start flag of Dynamic-to-Static error stack**: `In transformed code`, represents the dynamic-to-static error message stack, and refers to the error message when the transformed code is running. In the actual scene, you can directly search for the `In transformed code` keyword, and start from this line and read the error log. - **User code error stack**: It hides the useless error message at the framework level, and reports the error stack of the user code. We add a wavy line and HERE indicator under the error code to indicate the specific error location. We also expanded the error line code context to help you quickly locate the error location. As shown in the third part of the above figure, it can be seen that the user code that made the last error is `x = paddle.reshape(x, shape=[1, two])`. @@ -39,8 +39,8 @@ The error log can be divided into 4 parts from top to bottom: ### 1.2 Customized Display of Error Information -#### 1.2.1 Native error message without being processed by the Dygraph-to-Static error reporting module -If you want to view Paddle's native error message stack, that is, the error message stack that has not been processed by the Dygraph-to-Static error reporting module, you can set the environment variable `TRANSLATOR_DISABLE_NEW_ERROR=1` to turn off the dynamic-to-static error module. The default value of this environment variable is 0, which means that the module is enabled by default. +#### 1.2.1 Native error message without being processed by the Dynamic-to-Static error reporting module +If you want to view Paddle's native error message stack, that is, the error message stack that has not been processed by the Dynamic-to-Static error reporting module, you can set the environment variable `TRANSLATOR_DISABLE_NEW_ERROR=1` to turn off the dynamic-to-static error module. The default value of this environment variable is 0, which means that the module is enabled by default. Add the following code to the code in section 1.1 to view the native error message: ```python import os @@ -56,7 +56,7 @@ The C++ error stack is hidden by default. You can set the C++ environment variab ## 2、Debugging Method -Before debugging, **please ensure that the dynamic graph code before conversion can run successfully**. The following introduces several debugging methods recommended in Dygraph-to-Static. +Before debugging, **please ensure that the dynamic graph code before conversion can run successfully**. The following introduces several debugging methods recommended in Dynamic-to-Static. ### 2.1 Pdb Debugging pdb is a module in Python that defines an interactive Pyhton source code debugger. It supports setting breakpoints and single stepping between source lines, listing source code and variables, running Python code, etc. #### 2.1.1 Debugging steps @@ -84,7 +84,7 @@ pdb is a module in Python that defines an interactive Pyhton source code debugge (Pdb) ``` -- step3: Enter l, p and other commands in the pdb interactive mode to view the corresponding code and variables of the static graph after Dygraph-to-Static, and then troubleshoot related problems. +- step3: Enter l, p and other commands in the pdb interactive mode to view the corresponding code and variables of the static graph after Dynamic-to-Static, and then troubleshoot related problems. ``` > /tmp/tmpm0iw5b5d.py(9)func() -> two = paddle.full(shape=[1], fill_value=2, dtype='int32') @@ -147,7 +147,7 @@ def func(x): func(np.ones([1])) print(func.code) ``` -After running, you can see the static graph code after Dygraph-to-Static: +After running, you can see the static graph code after Dynamic-to-Static: ```python def func(x): x = paddle.assign(x) @@ -198,12 +198,12 @@ Variable: assign_0.tmp_0 - data: [1] ``` ### 2.4 Print Log -Dygraph-to-Static module records additional debugging information in the log to help you understand whether the function is successfully converted during the Dygraph-to-Static. You can call `paddle.jit.set_verbosity(level=0, also_to_stdout=False)` or set the environment variable `TRANSLATOR_VERBOSITY=level` to set the log detail level and view the log information of different levels. Currently, `level` can take values 0-3: +Dynamic-to-Static module records additional debugging information in the log to help you understand whether the function is successfully converted during the Dynamic-to-Static. You can call `paddle.jit.set_verbosity(level=0, also_to_stdout=False)` or set the environment variable `TRANSLATOR_VERBOSITY=level` to set the log detail level and view the log information of different levels. Currently, `level` can take values 0-3: - 0: No log - 1: Including the information of the dynamic-to-static conversion process, such as the source code before conversion and the callable object of conversion - 2: Including the above information and more detailed function conversion logs -- 3: Including the above information, as well as a more detailed Dygraph-to-Static log +- 3: Including the above information, as well as a more detailed Dynamic-to-Static log > **WARNING:** > The log includes source code and other information. Please make sure that it does not contain sensitive information before sharing the log. diff --git a/docs/guides/jit/grammar_list_en.md b/docs/guides/jit/grammar_list_en.md index 26e5ea8f357..4c9f6211c9b 100644 --- a/docs/guides/jit/grammar_list_en.md +++ b/docs/guides/jit/grammar_list_en.md @@ -48,9 +48,9 @@ If you encounter problems with @to_static, or want to learn about debugging skil #### principle : -While In the dynamic diagram, the code is interpreted and executed line by line, so the value of condition variables used by `if` are determined, which means that the logic branch of False will not be executed. +While in the dynamic graph, the code is interpreted and executed line by line, so the value of condition variables used by `if` are determined, which means that the logic branch of False will not be executed. -However, In the static graph, the control flow is realized through the `cond` operators. Each branch is represented by `true_fn` and `false_fn` respectively . Under this circumstance, the `false_fn` will be executed to build the computation graph. +However, in the static graph, the control flow is realized through the `cond` operators. Each branch is represented by `true_fn` and `false_fn` respectively . Under this circumstance, the `false_fn` will be executed to build the computation graph. When the condition variables in `If` are `Tensor`, `if-else` will be transformed to a `cond` operators. @@ -265,7 +265,7 @@ def sort_list(x, y): - Don't support get shape after a reshape operators. You may get a -1 in shape value. -For example, `x = reshape(x, shape=shape_tensor)` , then use `x.shape[0]` to do other operation. Due to the difference between dygraph and static graph, it is okay in dygraph but it will fail in static graph. The reason is that APIs return computation result in dygraph mode, so x.shape has deterministic value after calling reshape . However, static graph doesn’t have the value shape_tensor during building network, so PaddlePaddle doesn’t know the value of x.shape after calling reshape. PaddlePaddle static graph will set -1 to represent unknown shape value for each dimension of x.shape in this case, not the expected value. Similarily, calling the shape of the output tensor of those APIs which change the shape, such as expend, cannot be converted into static graph properly. +For example, `x = reshape(x, shape=shape_tensor)` , then use `x.shape[0]` to do other operation. Due to the difference between dynamic and static graph, it is okay in dynamic but it will fail in static graph. The reason is that APIs return computation result in dynamic graph mode, so x.shape has deterministic value after calling reshape . However, static graph doesn’t have the value shape_tensor during building network, so PaddlePaddle doesn’t know the value of x.shape after calling reshape. PaddlePaddle static graph will set -1 to represent unknown shape value for each dimension of x.shape in this case, not the expected value. Similarily, calling the shape of the output tensor of those APIs which change the shape, such as expend, cannot be converted into static graph properly. #### examples : diff --git a/docs/guides/jit/index_en.rst b/docs/guides/jit/index_en.rst index 99e270ced61..584cc2b99d2 100644 --- a/docs/guides/jit/index_en.rst +++ b/docs/guides/jit/index_en.rst @@ -1,14 +1,14 @@ ####################### -Dygraph to Static Graph +Dynamic to Static Graph ####################### -The imperative-style coding of PaddlePaddle takes advantage of flexibility, Pythonic coding, and easy-to-debug interface. In dygraph mode, code immediately executes kernels and gets numerical results, which allows users to enjoy traditional Pythonic code order. Therefore it is efficient to transform idea into real code and simple to debug. However, Python code is usually slower than C++ thus lots of industrial systems (such as large recommend system, mobile devices) prefer to deploy with C++ implementation. +The static graph mode of PaddlePaddle takes advantage of flexibility, Pythonic coding, and easy-to-debug interface. In dynamic graph mode, code immediately executes kernels and gets numerical results, which allows users to enjoy traditional Pythonic code order. Therefore it is efficient to transform idea into real code and simple to debug. However, Python code is usually slower than C++ thus lots of industrial systems (such as large recommend system, mobile devices) prefer to deploy with C++ implementation. Static graph is better at speed and portability. Static graph builds the network structure during compiling time and then does computation. The built network intermediate representation can be executed in C++ and gets rids of Python dependency. -While dygraph has usability and debug benefits and static graph yields performance and deployment advantage, we adds functionality to convert dygraph to static graph. Users use imperative mode to write dygraph code and PaddlePaddle will analyze the Python syntax and turn it into network structure of static graph mode. Our approach retains both the usability of dygraph and portability of static graph. +While dynamic graph has usability and debug benefits and static graph yields performance and deployment advantage, we adds functionality to convert dynamic graph to static graph. Users use imperative mode to write dynamic graph code and PaddlePaddle will analyze the Python syntax and turn it into network structure of static graph mode. Our approach retains both the usability of dynamic graph and portability of static graph. -We introduce the transformation of dygraph to static graph in the following links: +We introduce the transformation of dynamic graph to static graph in the following links: - `Supported Grammars `_ : Introduce supported grammars and unsupported grammars . diff --git a/docs/guides/performance_improving/amp_en.md b/docs/guides/performance_improving/amp_en.md index 69599323101..2abdb0a9ae1 100644 --- a/docs/guides/performance_improving/amp_en.md +++ b/docs/guides/performance_improving/amp_en.md @@ -362,7 +362,7 @@ if paddle.is_compiled_with_cuda(): The previous article introduced the method of single card (GPU) training in dynamic graph mode, which is similar to it, [distributed training documents](https://fleet-x.readthedocs.io/en/latest/paddle_fleet_rst/collective/collective_performance/amp.html) and [dynamic graph to static graph](../jit/index_cn.html) can start AMP in the same way. Next, it mainly introduces the methods of starting AMP training in static graph modes and the advanced usage of AMP training, such as gradient accumulation. -### 3.1 Gradient Accumulation in dygraph graph mode +### 3.1 Gradient Accumulation in dynamic graph mode Gradient accumulation means running a configured number of steps without updating the model variables. Until certain steps, use the accumulated gradients to update the variables. Limited by the size of the gpu memory, you may not be able to open a larger batch_size, you can increase batch_size by using gradient accumulation. @@ -411,7 +411,7 @@ In the above example, after `accumulate_batchs_num` batch training steps, with o ### 3.2. AMP in Static Graph -Paddle starts AMP training in Static Graph, the compute logic is similar to the dynamic diagram, except that the called interfaces are different. Paddle Static Graph provides a series of convenient APIs for AMP: ``paddle.static.amp.decorate``, ``paddle.static.amp.fp16_guard``. +Paddle starts AMP training in Static Graph, the compute logic is similar to the dynamic graph, except that the called interfaces are different. Paddle Static Graph provides a series of convenient APIs for AMP: ``paddle.static.amp.decorate``, ``paddle.static.amp.fp16_guard``. - ``paddle.static.amp.decorate``: Decorate the optimizer, add amp logic, and set the parameters of grad_scaler through this API. - ``paddle.static.amp.fp16_guard``: In AMP_O2 mode, the scope of float16 is controlled only in context manager ``fp16_guard``.