Fix handling of ControlledGates in QPY (#8055)

* Fix handling of ControlledGates in QPY

This commit fixes the handling of ControlledGates in QPY. Previously the
extra parameters needed to reconstruct a custom controlled gate were not
encoded into the QPY payload. Fixing this required a version bump to the
QPY format to modify the payload for a custom instruction entry. Once we
added to the format the extra data required for a controlled gate, the
number of control qubits, the control state, and the base gate object,
the deserializer has enough information to recreate the custom
ControlledGate objects. However, fixing this exposed another bug with
standard library multicontrolled gates where they often didn't contain
sufficient data in the payload to reconstruct either.

Fixes #7999

* Fix test failure caused by missing condition

This commit fixes the qpy test failure. This was caused by the omission
of the classical condition on a controlled gate when reconstructing the
circuit in deserialization. Fixing this oversight fixes the test
failures.

* Add QPY version 5 payload format description

* Add release note

* Expand test coverage

* Only check controlled gate type key on version 5 or newer

* Fix mcu1 deserialization

* Add copy to avoid mutating list while iterating over it

* Fix compat test minimum version

This commit fixes the failing compat test which was incorrectly trying
to test ControlledGates with 0.20.2 generation. We should only run the
controlled gate tests starting with 0.21.0.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

qiskit/qpy/__init__.py

@@ -100,6 +100,82 @@
 by ``num_circuits`` in the file header). There is no padding between the
 circuits in the data.
 
+.. _qpy_version_5:
+
+Version 5
+=========
+
+Version 5 changes from :ref:`qpy_version_4` by changing two payloads the INSTRUCTION metadata
+payload and the CUSTOM_INSTRUCTION block. These now have new fields to better account for
+:class:`~.ControlledGate` objects in a circuit.
+
+INSTRUCTION
+-----------
+
+The INSTRUCTION block was modified to add two new fields ``num_ctrl_qubits`` and ``ctrl_state``
+which are used to model the :attr:`.ControlledGate.num_ctrl_qubits` and
+:attr:`.ControlledGate.ctrl_state` attributes. The new payload packed struct
+format is:
+
+.. code-block:: c
+
+    struct {
+        uint16_t name_size;
+        uint16_t label_size;
+        uint16_t num_parameters;
+        uint32_t num_qargs;
+        uint32_t num_cargs;
+        _Bool has_conditional;
+        uint16_t conditional_reg_name_size;
+        int64_t conditional_value;
+        uint32_t num_ctrl_qubits;
+        uint32_t ctrl_state;
+    }
+
+The rest of the instruction payload is the same. You can refer to
+:ref:`qpy_instructions` for the details of the full payload.
+
+CUSTOM_INSTRUCTION
+------------------
+
+The CUSTOM_INSTRUCTION block in QPY version 5 adds a new field
+``base_gate_size`` which is used to define the size of the
+:class:`qiskit.circuit.Instruction` object stored in the
+:attr:`.ControlledGate.base_gate` attribute for a custom
+:class:`~.ControlledGate` object. With this change the CUSTOM_INSTRUCTION
+metadata block becomes:
+
+.. code-block:: c
+
+    struct {
+        uint16_t name_size;
+        char type;
+        uint32_t num_qubits;
+        uint32_t num_clbits;
+        _Bool custom_definition;
+        uint64_t size;
+        uint32_t num_ctrl_qubits;
+        uint32_t ctrl_state;
+        uint64_t base_gate_size
+    }
+
+Immediately following the CUSTOM_INSTRUCTION struct is the utf8 encoded name
+of size ``name_size``.
+
+If ``custom_definition`` is ``True`` that means that the immediately following
+``size`` bytes contains a QPY circuit data which can be used for the custom
+definition of that gate. If ``custom_definition`` is ``False`` then the
+instruction can be considered opaque (ie no definition). The ``type`` field
+determines what type of object will get created with the custom definition.
+If it's ``'g'`` it will be a :class:`~qiskit.circuit.Gate` object, ``'i'``
+it will be a :class:`~qiskit.circuit.Instruction` object.
+
+Following this the next ``base_gate_size`` bytes contain the ``INSTRUCTION``
+payload for the :attr:`.ControlledGate.base_gate`.
+
+Additionally an addition value for ``type`` is added ``'c'`` which is used to
+indicate the custom instruction is a custom :class:`~.ControlledGate`.
+
 .. _qpy_version_4:
 
 Version 4
@@ -455,6 +531,8 @@
     struct {
         uint16_t name_size;
         char type;
+        uint32_t num_qubits;
+        uint32_t num_clbits;
         _Bool custom_definition;
         uint64_t size;
     }
@@ -470,6 +548,8 @@
 If it's ``'g'`` it will be a :class:`~qiskit.circuit.Gate` object, ``'i'``
 it will be a :class:`~qiskit.circuit.Instruction` object.
 
+.. _qpy_instructions:
+
 INSTRUCTIONS
 ------------