Implement std::error::Error for PyErr

CHANGELOG.md

@@ -1,5 +1,6 @@
 # Changelog
-All notable changes to this project will be documented in this file.
+All notable changes to this project will be documented in this file. For help with updating to new
+PyO3 versions, please see the [migration guide](https://pyo3.rs/master/migration.html).
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
@@ -17,7 +18,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Add `#[derive(FromPyObject)]` macro for enums and structs. [#1065](https://github.com/PyO3/pyo3/pull/1065)
 
 ### Changed
-- Exception types have been renamed from e.g. `RuntimeError` to `PyRuntimeError`, and are now only accessible by `&T` or `Py<T>` similar to other Python-native types. The old names continue to exist but are deprecated. [#1024](https://github.com/PyO3/pyo3/pull/1024)
+- Exception types have been renamed from e.g. `RuntimeError` to `PyRuntimeError`, and are now accessible by `&T` or `Py<T>` similar to other Python-native types. The old names continue to exist but are deprecated. [#1024](https://github.com/PyO3/pyo3/pull/1024) [#1115](https://github.com/PyO3/pyo3/pull/1115)
+  - Rename `PyException::py_err()` to `PyException::new_err()`.
+  - Rename `PyUnicodeDecodeErr::new_err()` to `PyUnicodeDecodeErr::new()`.
+  - Remove `PyStopIteration::stop_iteration()`.
 - Correct FFI definitions `Py_SetProgramName` and `Py_SetPythonHome` to take `*const` argument instead of `*mut`. [#1021](https://github.com/PyO3/pyo3/pull/1021)
 - Rename `PyString::to_string` to `to_str`, change return type `Cow<str>` to `&str`. [#1023](https://github.com/PyO3/pyo3/pull/1023)
 - Correct FFI definition `_PyLong_AsByteArray` `*mut c_uchar` argument instead of `*const c_uchar`. [#1029](https://github.com/PyO3/pyo3/pull/1029)
@@ -27,7 +31,13 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 - Change `PyIterator::from_object` to return `PyResult<PyIterator>` instead of `Result<PyIterator, PyDowncastError>`. [#1051](https://github.com/PyO3/pyo3/pull/1051)
 - `IntoPy` is no longer implied by `FromPy`. [#1063](https://github.com/PyO3/pyo3/pull/1063)
 - `PyObject` is now just a type alias for `Py<PyAny>`. [#1063](https://github.com/PyO3/pyo3/pull/1063)
-- Implement `Send + Sync` for `PyErr`. `PyErr::new`, `PyErr::from_type`, `PyException::py_err` and `PyException::into` have had these bounds added to their arguments. [#1067](https://github.com/PyO3/pyo3/pull/1067)
+- Rework PyErr to be compatible with the `std::error::Error` trait: [#1067](https://github.com/PyO3/pyo3/pull/1067) [#1115](https://github.com/PyO3/pyo3/pull/1115)
+  - Implement `Display`, `Error`, `Send` and `Sync` for `PyErr` and `PyErrArguments`.
+  - Add `PyErr::instance()` which returns `&PyBaseException`.
+  - `PyErr`'s fields are now an implementation detail. The equivalent values can be accessed with `PyErr::ptype()`, `PyErr::pvalue()` and `PyErr::ptraceback()`.
+  - Change `PyErr::print()` and `PyErr::print_and_set_sys_last_vars()` to take `&self` instead of `self`.
+  - Remove `PyErrValue`, `PyErr::from_value`, `PyErr::into_normalized()`, and `PyErr::normalize()`.
+  - Remove `PyException::into()` and `Into<PyResult<T>>` for `PyErr` and `PyException`.
 - Change `#[pyproto]` to return NotImplemented for operators for which Python can try a reversed operation. #[1072](https://github.com/PyO3/pyo3/pull/1072)
 - `PyModule::add` now uses `IntoPy<PyObject>` instead of `ToPyObject`. #[1124](https://github.com/PyO3/pyo3/pull/1124)
 

guide/src/exception.md

@@ -61,7 +61,7 @@ use pyo3::exceptions::PyTypeError;
 fn main() {
     let gil = Python::acquire_gil();
     let py = gil.python();
-    PyErr::new::<PyTypeError, _>("Error").restore(py);
+    PyTypeError::new_err("Error").restore(py);
     assert!(PyErr::occurred(py));
     drop(PyErr::fetch(py));
 }
@@ -76,7 +76,7 @@ If you already have a Python exception instance, you can simply call [`PyErr::fr
 PyErr::from_instance(py, err).restore(py);
 ```
 
-If a Rust type exists for the exception, then it is possible to use the `py_err` method.
+If a Rust type exists for the exception, then it is possible to use the `new_err` method.
 For example, each standard exception defined in the `pyo3::exceptions` module
 has a corresponding Rust type, exceptions defined by [`create_exception!`] and [`import_exception!`] macro
 have Rust types as well.
@@ -87,7 +87,7 @@ have Rust types as well.
 # fn check_for_error() -> bool {false}
 fn my_func(arg: PyObject) -> PyResult<()> {
     if check_for_error() {
-        Err(PyValueError::py_err("argument is wrong"))
+        Err(PyValueError::new_err("argument is wrong"))
     } else {
         Ok(())
     }
@@ -115,41 +115,32 @@ fn main() {
 [`Python::is_instance`] calls the underlying [`PyType::is_instance`](https://docs.rs/pyo3/latest/pyo3/types/struct.PyType.html#method.is_instance)
 method to do the actual work.
 
-To check the type of an exception, you can simply do:
+To check the type of an exception, you can similarly do:
 
 ```rust
 # use pyo3::exceptions::PyTypeError;
 # use pyo3::prelude::*;
-# fn main() {
 # let gil = Python::acquire_gil();
 # let py = gil.python();
-# let err = PyTypeError::py_err(());
+# let err = PyTypeError::new_err(());
 err.is_instance::<PyTypeError>(py);
-# }
 ```
 
 ## Handling Rust errors
 
-The vast majority of operations in this library will return [`PyResult<T>`](https://docs.rs/pyo3/latest/pyo3/prelude/type.PyResult.html),
+The vast majority of operations in this library will return
+[`PyResult<T>`](https://docs.rs/pyo3/latest/pyo3/prelude/type.PyResult.html),
 which is an alias for the type `Result<T, PyErr>`.
 
-A [`PyErr`] represents a Python exception.
-Errors within the PyO3 library are also exposed as Python exceptions.
-
-The PyO3 library handles Python exceptions in two stages. During the first stage, a [`PyErr`] instance is
-created. At this stage, holding Python's GIL is not required. During the second stage, an actual Python
-exception instance is created and set active in the Python interpreter.
+A [`PyErr`] represents a Python exception. Errors within the PyO3 library are also exposed as
+Python exceptions.
 
-In simple cases, for custom errors adding an implementation of `std::convert::From<T>` trait
-for this custom error is enough. `PyErr::new` accepts an argument in the form
-of `ToPyObject + 'static`. If the `'static` constraint can not be satisfied or
-more complex arguments are required, the
-[`PyErrArguments`](https://docs.rs/pyo3/latest/pyo3/trait.PyErrArguments.html)
-trait can be implemented. In that case, actual exception argument creation is delayed
-until a `Python` object is available.
+If your code has a custom error type e.g. `MyError`, adding an implementation of
+`std::convert::From<MyError> for PyErr` is usually enough. PyO3 will then automatically convert
+your error to a Python exception when needed.
 
 ```rust
-# use pyo3::{PyErr, PyResult};
+# use pyo3::prelude::*;
 # use pyo3::exceptions::PyOSError;
 # use std::error::Error;
 # use std::fmt;
@@ -170,11 +161,12 @@ until a `Python` object is available.
 # }
 impl std::convert::From<CustomIOError> for PyErr {
     fn from(err: CustomIOError) -> PyErr {
-        PyOSError::py_err(err.to_string())
+        PyOSError::new_err(err.to_string())
     }
 }
 
-fn connect(s: String) -> PyResult<bool> {
+#[pyfunction]
+fn connect(s: String) -> Result<bool, CustomIOError> {
     bind("127.0.0.1:80")?;
     Ok(true)
 }
@@ -195,6 +187,10 @@ fn parse_int(s: String) -> PyResult<usize> {
 
 The code snippet above will raise a `ValueError` in Python if `String::parse()` returns an error.
 
+If lazy construction of the Python exception instance is desired, the
+[`PyErrArguments`](https://docs.rs/pyo3/latest/pyo3/trait.PyErrArguments.html)
+trait can be implemented. In that case, actual exception argument creation is delayed
+until the `PyErr` is needed.
 
 ## Using exceptions defined in Python code
 
@@ -213,7 +209,7 @@ fn tell(file: &PyAny) -> PyResult<u64> {
     use pyo3::exceptions::*;
 
     match file.call_method0("tell") {
-        Err(_) => Err(io::UnsupportedOperation::py_err("not supported: tell")),
+        Err(_) => Err(io::UnsupportedOperation::new_err("not supported: tell")),
         Ok(x) => x.extract::<u64>(),
     }
 }

guide/src/migration.md

@@ -5,13 +5,88 @@ For a detailed list of all changes, see [CHANGELOG.md](https://github.com/PyO3/p
 
 ## from 0.11.* to 0.12
 
+### `PyErr` has been reworked
+
+In PyO3 `0.12` the `PyErr` type has been re-implemented to be significantly more compatible with
+the standard Rust error handling ecosystem. Specificially `PyErr` now implements
+`Error + Send + Sync`, which are the standard traits used for error types.
+
+While this has necessitated the removal of a number of APIs, the resulting `PyErr` type should now
+be much more easier to work with. The following sections list the changes in detail and how to
+migrate to the new APIs.
+
+#### `PyErr::new` and `PyErr::from_type` now require `Send + Sync` for their argument
+
+For most uses no change will be needed. If you are trying to construct `PyErr` from a value that is
+not `Send + Sync`, you will need to first create the Python object and then use
+`PyErr::from_instance`.
+
+Similarly, any types which implemented `PyErrArguments` will now need to be `Send + Sync`.
+
+#### `PyErr`'s contents are now private
+
+It is no longer possible to access the fields `.ptype`, `.pvalue` and `.ptraceback` of a `PyErr`.
+You should instead now use the new methods `PyErr::ptype()`, `PyErr::pvalue()` and `PyErr::ptraceback()`.
+
+#### `PyErrValue` and `PyErr::from_value` have been removed
+
+As these were part the internals of `PyErr` which have been reworked, these APIs no longer exist.
+
+If you used this API, it is recommended to use `PyException::new_err` (see [the section on
+Exception types](#exception-types-have-been-reworked)).
+
+#### `Into<PyResult<T>>` for `PyErr` has been removed
+
+This implementation was redundant. Just construct the `Result::Err` variant directly.
+
+Before:
+```rust,ignore
+let result: PyResult<()> = PyErr::new::<TypeError, _>("error message").into();
+```
+
+After (also using the new reworked exception types; see the following section):
+```rust
+# use pyo3::{PyErr, PyResult, exceptions::PyTypeError};
+let result: PyResult<()> = Err(PyTypeError::new_err("error message"));
+```
+
+### Exception types have been reworked
+
+Previously exception types were zero-sized marker types purely used to construct `PyErr`. In PyO3
+0.12, these types have been replaced with full definitions and are usable in the same way as `PyAny`, `PyDict` etc. This
+makes it possible to interact with Python exception objects.
+
+The new types also have names starting with the "Py" prefix. For example, before:
+
+```rust,ignore
+let err: PyErr = TypeError::py_err("error message");
+```
+
+After:
+
+```
+# use pyo3::{PyErr, PyResult, Python, type_object::PyTypeObject};
+# use pyo3::exceptions::{PyBaseException, PyTypeError};
+# Python::with_gil(|py| -> PyResult<()> {
+let err: PyErr = PyTypeError::new_err("error message");
+
+// Uses Display for PyErr, new for PyO3 0.12
+assert_eq!(err.to_string(), "TypeError: error message");
+
+// Now possible to interact with exception instances, new for PyO3 0.12
+let instance: &PyBaseException = err.instance(py);
+assert_eq!(instance.getattr("__class__")?, PyTypeError::type_object(py).as_ref());
+# Ok(())
+# }).unwrap();
+```
+
 ### `FromPy` has been removed
-To simplify the PyO3 public conversion trait hierarchy, the `FromPy` has been removed. In PyO3
-`0.11` there were two ways to define the to-Python conversion for a type: `FromPy<T> for PyObject`,
-and `IntoPy<PyObject> for T`.
+To simplify the PyO3 conversion traits, the `FromPy` trait has been removed. Previously there were
+two ways to define the to-Python conversion for a type:
+`FromPy<T> for PyObject` and `IntoPy<PyObject> for T`.
 
-Now, the canonical implementation is always `IntoPy`, so downstream crates may need to adjust
-accordingly.
+Now there is only one way to define the conversion, `IntoPy`, so downstream crates may need to
+adjust accordingly.
 
 Before:
 ```rust,ignore
@@ -85,7 +160,6 @@ let list_ref: &PyList = list_py.as_ref(py);
 # })
 ```
 
-
 ## from 0.10.* to 0.11
 
 ### Stable Rust

pyo3-derive-backend/src/from_pyobject.rs

@@ -78,7 +78,7 @@ impl<'a> Enum<'a> {
                 .map(|s| format!("{} ({})", s.to_string_lossy(), type_name))
                 .unwrap_or_else(|_| type_name.to_string());
             let err_msg = format!("Can't convert {} to {}", from, #error_names);
-            Err(::pyo3::exceptions::PyTypeError::py_err(err_msg))
+            Err(::pyo3::exceptions::PyTypeError::new_err(err_msg))
         )
     }
 }
@@ -263,7 +263,7 @@ impl<'a> Container<'a> {
         quote!(
             let s = <::pyo3::types::PyTuple as ::pyo3::conversion::PyTryFrom>::try_from(obj)?;
             if s.len() != #len {
-                return Err(::pyo3::exceptions::PyValueError::py_err(#msg))
+                return Err(::pyo3::exceptions::PyValueError::new_err(#msg))
             }
             let slice = s.as_slice();
             Ok(#self_ty(#fields))

src/buffer.rs

@@ -156,10 +156,10 @@ pub unsafe trait Element: Copy {
 fn validate(b: &ffi::Py_buffer) -> PyResult<()> {
     // shape and stride information must be provided when we use PyBUF_FULL_RO
     if b.shape.is_null() {
-        return Err(exceptions::PyBufferError::py_err("Shape is Null"));
+        return Err(exceptions::PyBufferError::new_err("Shape is Null"));
     }
     if b.strides.is_null() {
-        return Err(exceptions::PyBufferError::py_err(
+        return Err(exceptions::PyBufferError::new_err(
             "PyBuffer: Strides is Null",
         ));
     }
@@ -190,7 +190,7 @@ impl<T: Element> PyBuffer<T> {
             {
                 Ok(buf)
             } else {
-                Err(exceptions::PyBufferError::py_err(
+                Err(exceptions::PyBufferError::new_err(
                     "Incompatible type as buffer",
                 ))
             }
@@ -441,7 +441,7 @@ impl<T: Element> PyBuffer<T> {
 
     fn copy_to_slice_impl(&self, py: Python, target: &mut [T], fort: u8) -> PyResult<()> {
         if mem::size_of_val(target) != self.len_bytes() {
-            return Err(exceptions::PyBufferError::py_err(
+            return Err(exceptions::PyBufferError::new_err(
                 "Slice length does not match buffer length.",
             ));
         }
@@ -528,7 +528,7 @@ impl<T: Element> PyBuffer<T> {
             return buffer_readonly_error();
         }
         if mem::size_of_val(source) != self.len_bytes() {
-            return Err(exceptions::PyBufferError::py_err(
+            return Err(exceptions::PyBufferError::new_err(
                 "Slice length does not match buffer length.",
             ));
         }
@@ -564,7 +564,7 @@ impl<T: Element> PyBuffer<T> {
 
 #[inline(always)]
 fn buffer_readonly_error() -> PyResult<()> {
-    Err(exceptions::PyBufferError::py_err(
+    Err(exceptions::PyBufferError::new_err(
         "Cannot write to read-only buffer.",
     ))
 }

src/callback.rs

@@ -86,7 +86,7 @@ impl IntoPyCallbackOutput<ffi::Py_ssize_t> for usize {
         if self <= (isize::MAX as usize) {
             Ok(self as isize)
         } else {
-            Err(PyOverflowError::py_err(()))
+            Err(PyOverflowError::new_err(()))
         }
     }
 }
@@ -244,11 +244,11 @@ macro_rules! callback_body_without_convert {
             Err(e) => {
                 // Try to format the error in the same way panic does
                 if let Some(string) = e.downcast_ref::<String>() {
-                    Err($crate::panic::PanicException::py_err((string.clone(),)))
+                    Err($crate::panic::PanicException::new_err((string.clone(),)))
                 } else if let Some(s) = e.downcast_ref::<&str>() {
-                    Err($crate::panic::PanicException::py_err((s.to_string(),)))
+                    Err($crate::panic::PanicException::new_err((s.to_string(),)))
                 } else {
-                    Err($crate::panic::PanicException::py_err((
+                    Err($crate::panic::PanicException::new_err((
                         "panic from Rust code",
                     )))
                 }

src/class/basic.rs

@@ -9,7 +9,7 @@
 //! [typeobj docs](https://docs.python.org/3/c-api/typeobj.html)
 
 use crate::callback::{HashCallbackOutput, IntoPyCallbackOutput};
-use crate::{exceptions, ffi, FromPyObject, PyAny, PyCell, PyClass, PyErr, PyObject, PyResult};
+use crate::{exceptions, ffi, FromPyObject, PyAny, PyCell, PyClass, PyObject, PyResult};
 use std::os::raw::c_int;
 
 /// Operators for the __richcmp__ method
@@ -260,7 +260,7 @@ where
             ffi::Py_NE => Ok(CompareOp::Ne),
             ffi::Py_GT => Ok(CompareOp::Gt),
             ffi::Py_GE => Ok(CompareOp::Ge),
-            _ => Err(PyErr::new::<exceptions::PyValueError, _>(
+            _ => Err(exceptions::PyValueError::new_err(
                 "tp_richcompare called with invalid comparison operator",
             )),
         }

src/class/iter.rs

@@ -112,7 +112,7 @@ impl IntoPyCallbackOutput<*mut ffi::PyObject> for PyIterNextOutput {
     fn convert(self, _py: Python) -> PyResult<*mut ffi::PyObject> {
         match self {
             IterNextOutput::Yield(o) => Ok(o.into_ptr()),
-            IterNextOutput::Return(opt) => Err(crate::exceptions::PyStopIteration::py_err((opt,))),
+            IterNextOutput::Return(opt) => Err(crate::exceptions::PyStopIteration::new_err((opt,))),
         }
     }
 }

src/class/macros.rs

@@ -228,12 +228,10 @@ macro_rules! py_func_set {
                 let slf = py.from_borrowed_ptr::<$crate::PyCell<$generic>>(slf);
 
                 if value.is_null() {
-                    Err($crate::PyErr::new::<exceptions::PyNotImplementedError, _>(
-                        format!(
-                            "Subscript deletion not supported by {:?}",
-                            stringify!($generic)
-                        ),
-                    ))
+                    Err($crate::exceptions::PyNotImplementedError::new_err(format!(
+                        "Subscript deletion not supported by {:?}",
+                        stringify!($generic)
+                    )))
                 } else {
                     let name = py.from_borrowed_ptr::<$crate::PyAny>(name);
                     let value = py.from_borrowed_ptr::<$crate::PyAny>(value);
@@ -264,7 +262,7 @@ macro_rules! py_func_del {
                         .extract()?;
                     slf.try_borrow_mut()?.$fn_del(name).convert(py)
                 } else {
-                    Err(PyErr::new::<exceptions::PyNotImplementedError, _>(
+                    Err(exceptions::PyNotImplementedError::new_err(
                         "Subscript assignment not supported",
                     ))
                 }