Introduce Qiskit native ParameterExpression serialization #13252
Labels
Milestone
Comments
mtreinish
added
type: feature request
mtreinish
added a commit
to mtreinish/qiskit-core
that referenced
this issue
Oct 1, 2024
Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in Qiskit#13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path.
github-merge-queue bot
pushed a commit
that referenced
this issue
Oct 2, 2024
* Workaround symengine serialization payload incompatibility In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine. To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not. Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us. * Handle schedules too This commit updates the schedule serialization path too, as it was also directly loading symengine expressions. The code handling the workaround is extracted to a standalone function which is used in both spots now instead of calling symengine directly. * Remove unused imports * Gracefully handle failure to parse historical symengine files During the discovery on the fix in this PR we discovered that setting the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in newer versions of Qiskit being unable to parse the QPY file for the same issue as being addressed here. This commit expands the logic to account for this and raise a useful warning. It also updates the release notes and documentation to document these limitations. * Cap upper version of symengine in requirements list Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in #13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path. * Fix release note upgrade section label * Rewrite support documentation * Fix mistakes in release note Co-authored-by: Matthew Treinish <mtreinish@kortar.org> --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Jake Lishman <jake@binhbar.com>
mergify bot
pushed a commit
that referenced
this issue
Oct 2, 2024
* Workaround symengine serialization payload incompatibility In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine. To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not. Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us. * Handle schedules too This commit updates the schedule serialization path too, as it was also directly loading symengine expressions. The code handling the workaround is extracted to a standalone function which is used in both spots now instead of calling symengine directly. * Remove unused imports * Gracefully handle failure to parse historical symengine files During the discovery on the fix in this PR we discovered that setting the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in newer versions of Qiskit being unable to parse the QPY file for the same issue as being addressed here. This commit expands the logic to account for this and raise a useful warning. It also updates the release notes and documentation to document these limitations. * Cap upper version of symengine in requirements list Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in #13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path. * Fix release note upgrade section label * Rewrite support documentation * Fix mistakes in release note Co-authored-by: Matthew Treinish <mtreinish@kortar.org> --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Jake Lishman <jake@binhbar.com> (cherry picked from commit fee9f77)
github-merge-queue bot
pushed a commit
that referenced
this issue
Oct 2, 2024
…13251) (#13255) * Workaround symengine serialization payload incompatibility (#13251) * Workaround symengine serialization payload incompatibility In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine. To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not. Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us. * Handle schedules too This commit updates the schedule serialization path too, as it was also directly loading symengine expressions. The code handling the workaround is extracted to a standalone function which is used in both spots now instead of calling symengine directly. * Remove unused imports * Gracefully handle failure to parse historical symengine files During the discovery on the fix in this PR we discovered that setting the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in newer versions of Qiskit being unable to parse the QPY file for the same issue as being addressed here. This commit expands the logic to account for this and raise a useful warning. It also updates the release notes and documentation to document these limitations. * Cap upper version of symengine in requirements list Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in #13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path. * Fix release note upgrade section label * Rewrite support documentation * Fix mistakes in release note Co-authored-by: Matthew Treinish <mtreinish@kortar.org> --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Jake Lishman <jake@binhbar.com> (cherry picked from commit fee9f77) * Fix symengine typo --------- Co-authored-by: Matthew Treinish <mtreinish@kortar.org> Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
ElePT
pushed a commit
to ElePT/qiskit
that referenced
this issue
Oct 3, 2024
) * Workaround symengine serialization payload incompatibility In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine. To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not. Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us. * Handle schedules too This commit updates the schedule serialization path too, as it was also directly loading symengine expressions. The code handling the workaround is extracted to a standalone function which is used in both spots now instead of calling symengine directly. * Remove unused imports * Gracefully handle failure to parse historical symengine files During the discovery on the fix in this PR we discovered that setting the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in newer versions of Qiskit being unable to parse the QPY file for the same issue as being addressed here. This commit expands the logic to account for this and raise a useful warning. It also updates the release notes and documentation to document these limitations. * Cap upper version of symengine in requirements list Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in Qiskit#13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path. * Fix release note upgrade section label * Rewrite support documentation * Fix mistakes in release note Co-authored-by: Matthew Treinish <mtreinish@kortar.org> --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Jake Lishman <jake@binhbar.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What should we add?
Right now QPY relies on symengine's native serialization format for serializing the internal symbolic expression used by the parameter expression. At the time this made a lot of sense because we could relying on symengine to manage how best to represent it's internal data model and it was also C++ relying on the cereal library so it was very fast. But as was recently discovered by the symengine 0.13 release the payload format is version specific, and symengine will error if you try to load a expression generated with a different symengine version. For the immediate term we were able to "workaround" this issue here: #13251 but this isn't a general solution as symengine could change the format between versions, that's why they have the check.
Moving forward for qiskit 1.3.0 we must come up with a qiskit native serialization for
ParameterExpressionfor QPY instead of relying on symengine or sympy to do it for us. This should be introduced in QPY version 13, this version mismatch is still a possibility for users of QPY version >=10 and < 13 when loading a historically generated payload, but in those cases the only option is to install a compatible version of symengine if the payload format is relying on it.The text was updated successfully, but these errors were encountered: