Skip to content

Commit

Permalink
Power cycle MTM2 cabinet document and navigation links done
Browse files Browse the repository at this point in the history
  • Loading branch information
JackieS-NL committed May 30, 2024
1 parent 1eb9915 commit 2fd942b
Show file tree
Hide file tree
Showing 3 changed files with 85 additions and 44 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
- If a file cannot include a title (surrounded by ampersands (#)), comment out the title from the template and include a comment explaining why this is implemented (in addition to applying the ``title`` directive).
.. Include one Primary Author and list of Contributors (comma separated) between the asterisks (*):
.. |author| replace:: *Name-of-Primary-Author*
.. |author| replace:: *Jacqueline Seronr*
.. If there are no contributors, write "none" between the asterisks. Do not remove the substitution.
.. |contributors| replace:: *List-of-contributors*
.. |contributors| replace:: *Te-Wei Tsai*

.. This is the label that can be used as for cross referencing this procedure.
.. Recommended format is "Directory Name"-"Title Name" -- Spaces should be replaced by hyphens.
Expand All @@ -20,12 +20,11 @@
.. To reference a label that isn't associated with an reST object such as a title or figure, you must include the link an explicit title using the syntax :ref:`link text <label-name>`.
.. An error will alert you of identical labels during the build process.
###################
#########################
Power Cycle MTM2 cabinet
###################
#########################


.. note::
This is a procedure template file that is associated with a template directory structure. This note should be deleted when the section is properly populated.

.. _Title-of-Procedure-Overview:

Expand All @@ -34,7 +33,16 @@ Overview

.. This section should provide a brief, top-level description of the procedure's purpose and utilization. Consider including the expected user and when the procedure will be performed.
This is a template for a procedure. It is performed by authorized and trained users.
This procedure mirrors the `Power-Cycle section under M2 handling errors`_ authored by Te-Wei, albeit presented in the format of Observatory Operations Documentation.

.. _`Power-Cycle section under M2 handling errors`: https://ts-m2gui.lsst.io/error-handling/error-handling.html#power-cycle

This procedure is the final option for resolving an MTM2 issue during the night. It's utilized when the M2 GUI consistently displays alarms, and the MTM2 cannot transition to a closed-loop status following the steps outlined in the MTM2 Recovery procedure.

.. need to add link to the corresponding and not yet in the site "MTM2 recovery procedure" page
.. test that works
.. :doc:`MTM2 Recovery </MainTel/MainTel-Non-standard-Operations/MTM2-Non-standard-Procedures/MTM2-Non-standard-Procedures-index>`
.. The correct webpage would be in M2 Troubleshooting so </MainTel/MainTel-Troubleshooting/MTCS-Troubleshooting/M2-Troubleshooting/MTM2-Recovery>`
.. _Title-of-Procedure-Precondition:

Expand All @@ -45,12 +53,26 @@ Precondition
.. It is preferred to include them as a bulleted or enumerated list.
.. If there is a different procedure that is critical before execution, carefully consider if it should be linked within this section or as part of the Procedure section below (or both).
- This is an example bullet of a precondition (Telescope azimuth must be at 0 degrees.)
.. note::
Ensure you have access to 1password and are a member of the Operators vault. If not, request access for the 1password Vault Operators via an `IT Jira ticket`_.

.. _`IT Jira ticket`: https://rubinobs.atlassian.net/jira/

- Consult with the hardware or system engineers to obtain approval before proceeding with this procedure.

- Ensure you are connected to the control room network as VPN cannot reach it.

- You should only do this procedure if:

- Following MTM2 Recovery procedure doesn't fix the issue, this means re-starting the control system fails to clear alarms and you are unable to set MTM2 to close-loop.

.. add link to MTM2 Recovery
- You cannot ping or ssh the M2 controller m2-crio-controller01.cp.lsst.org (on TMA) **m2-crio-controller02.cp.lsst.org** (on level 3).
- This is an example describing an exception (If the dome is closed, telescope elevation position is not important.)
- You see errors codes 6051 and 6088, or 6051 and 6052, and in some cases error 6056.
- This is another example bullet of a precondition (Notification to specified SLACK channel.)
- This example refers to a required action at the beginning of the procedure (It is critical the correct status of the equipment. This will be verified during :ref:`a critical step <Title-of-Procedure-Critical-Step-1>` in the procedure.)
- Know the location of the MTM2 cabinet according its installation, which is either in TMA or on level 3.

.. _Title-of-Procedure-Post-Condition:

Expand All @@ -62,68 +84,71 @@ Post-Condition
.. Please provide screenshots of the software status or relevant display windows to confirm.
.. Do not include actions in this section. Any action by the user should be included in the end of the Procedure section below. For example: Do not include "Verify the telescope azimuth is 0 degrees with the appropriate command." Instead, include this statement as the final step of the procedure, and include "Telescope is at 0 degrees." in the Post-condition section.
- This is an example bullet of a post-condition (Telescope azimuth is 0 degrees.)
- This is another example of a post-condition (This procedure leaves the telescope with the E-stop activated.)
- MTMM2 GUI displays no alarms, and MTM2 can transition to a close-loop status.


.. _Title-of-Procedure-Procedure-Steps:

Procedure Steps
===============

.. todo::
Include utilization of admonishments (caution, warning, etc.)

.. This section should include the procedure. There is no strict formatting or structure required for procedures. It is left to the authors to decide which format and structure is most relevant.
.. In the case of more complicated procedures, more sophisticated methodologies may be appropriate, such as multiple section headings or a list of linked procedures to be performed in the specified order.
.. For highly complicated procedures, consider breaking them into separate procedure. Some options are a high-level procedure with links, separating into smaller procedures or utilizing the reST ``include`` directive <https://docutils.sourceforge.io/docs/ref/rst/directives.html#include>.
This paragraph describes some general guidance for this procedure. This paragraph is optional depending on its usefulness and the complexity of the procedure.
Depending on the location of MTM2 the corresponding steps are:

In the case of a very straightforward procedure, a simple numbered list could be used, like as follows:
.. _If-MTM2-is-on-the-TMA:

If MTM2 is on the TMA
-----------------------

.. warning::
For this example, this step is critical.

.. _Title-of-Procedure-Critical-Step-1:
.. .. _`tea-pdu`: https://tea-pdu01.cp.lsst.org/ this link currently (May 30, 2024) get a "This site can't be reached"
#. This is Step 1 in a enumerated list. It happens to be very important.
#. Enter **tea-pdu**, this is the internet interface of the general power distribution (PDU).

#. This is Step 2.
- You can see a screenshot in :ref:`pdu-m2` figure below.

#. This is step 3.
#. Find the credentials in the **Operators Vault**, under the name PDU Utilities Cabinet. *Note*: The link to tea-pdu address is in this item.

#. Step 4 has two branches, but Step 5 is independent of Step 4.
#. Select **outlets** in the left panel of the window.

a. If Condition A, do the following action in :ref:`Condition A Instructions <Title-of-Procedure-Condition-A-for-Step-4>`.
#. Select and power :guilabel:`Off` M2 Cabinet.

b. If Condition B, do the following action in :ref:`Condition B instructions <Title-of-Procedure-Condition-B-for-Step-4>`.
#. Wait 30 seconds.

.. _Title-of-Procedure-Final-Step:
#. Turn the power :guilabel:`On` the M2 Cabinet.

#. Complete the procedure's final step.
.. _pdu-m2:

.. figure:: /MainTel/MainTel-Non-standard-Operations/MTM2-Non-standard-Procedures/_static/pdu_m2.png
:name: Outlet section at tea-pdu

.. _Title-of-Procedure-Condition-A-for-Step-4:
.. _If-MTM2-is-on-level-3:

Condition A for Step 4
If MTM2 is on level 3
----------------------

This is an example of a sub-section, used when Condition A applied. Complete the steps in this section:
#. Go to level 3.

#. Step 1 for Condition A.
#. Return to :ref:`Step 5 <Title-of-Procedure-Final-Step>` in the section above.
#. Disconnect the power cable.

.. _Title-of-Procedure-Condition-B-for-Step-4:
#. Wait 30 seconds.

Condition B for Step 4
----------------------
#. Reconnect the power cable.

This is an example of a sub-section, used when Condition B applied. Complete the steps in this section:
.. warning::
Be cautious when handling the power cable, as it high voltage.

#. Step 1 for Condition B.
#. Return to :ref:`Step 5 <Title-of-Procedure-Final-Step>` in the section above.
.. figure:: /MainTel/MainTel-Non-standard-Operations/MTM2-Non-standard-Procedures/_static/pending_cable_power_photo.png
:name: Cable power photo

.. _Title-of-Procedure-Troubleshooting:

Final steps:
----------------------

Connect the MTM2 control system to MTM2 CSC. Bare in mind that before requesting a connection you must wait for 5 minutes.

Troubleshooting
===============
Expand All @@ -132,6 +157,15 @@ Troubleshooting
.. If there is no content for this section, remove the indentation on the following line instead of deleting this sub-section.
No troubleshooting information is applicable to this procedure.
You can verify if the control system is running and waiting for the TCS/IP connection, by referring to the "Additional information/Details" section in the MTM2 Recovery page.

.. add link to the mentioned section above.
In the event that connecting fails try re-starting the M2GUI.

Note about TCP/IP connection:
-----------------------------
If, even after performing a power-cycle, you're still unable to ping the controller, you must contact IT support.

.. No troubleshooting information is applicable to this procedure.
- This is an example bullet (If the following error is given during :ref:`Step 5 <Title-of-Procedure-Final-Step>`, resolve it using a specified action.)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,11 @@ Simonyi Non-standard Operations
:titlesonly:
:glob:

StarTrackers-Non-standard-Procedures/StarTrackers-Non-standard-Procedures-index.rst
StarTrackers-Non-standard-Procedures/StarTrackers-Non-standard-Procedures-index.rst

.. toctree::
:maxdepth: 3
:titlesonly:
:glob:

MTM2-Non-standard-Procedures/MTM2-Non-standard-Procedures-index.rst

0 comments on commit 2fd942b

Please sign in to comment.