ci.git revamp (zephyrproject-rtos#2)

* updating readme, adding ci.wiki from gitlab

* Adding git secret protected hidden/ directory with teamforge zdevops-keys, cvondra@zephyr-ci.jf.intel.com access only, currently.

* Adding ci.git wiki files

* move functional directories into /src, removed skiplist + overdue README.md update

* Update README.md

* hide-hidden: clarify script operation comments

* Adding zabbix server install script to /src/zabbix

* Adding gpg key for connor.graydon@intel.com to hidden/ keyring

.gitignore

@@ -0,0 +1,3 @@
+.gitsecret/keys/random_seed
+hidden.tar
+hidden/

.gitsecret/paths/mapping.cfg

@@ -0,0 +1 @@
+hidden.tar:

README.md

@@ -1,21 +1,32 @@
-# Intel internal Zephyr CI
-## *operated by your friendly IAGS/FMOS Zephyr DevOps team*
-### mail PDL: FMOS_DevOps 
-
-## [DevOps Documentation Wiki](https://gitlab.devtools.intel.com/zephyrproject-rtos/ci/-/wikis/home)
-## [Zephyr DevOps Overview](https://intel-my.sharepoint.com/:p:/p/christopher_g_turner/EfZ2TF9ElydPjpGBEAKiUkwBiFt5LFBZPI2aGO_HZnP7Wg?e=Bxeeho)
-
-# CI Terminology
-* **Jenkins** - A popular open-source CI/CD tool. Zephyr DevOps operates several Jenkins master instances at zerobot2, zerobot-stg & zephyr-ci.
-* **Agent** - Jenkins terminology for a remote computer used for building, testing in a CI/CD pipeline. Any computer capable of executing commands over ssh can be a Jenkins agent.
-* **Test-agent** - Jenkins agent that's configured for device-testing
-* **Build-agent** - Jenkins agent that's intended for virtual (qemu-only) sanitycheck jobs, not device-testing
-* **Test-head** - A dedicated server that provides services for a set of agents, via test-network.
-* **Test-net** - Private network linking a Testhead with a set of agents, implemented as a star-topology around a Gigabit Ethernet switch
-* **PXE Boot** - Also known as "network boot", this service allows a computer to receive it's operating system via a network link, removing the requirement for a hard-disk
-* **PXE Boot Docker (pxeboot.docker)** - Zephyr DevOps-created tool that provides PXE boot services & automates generation of PXE boot initramfs payload
-* **Workspace** - A build directory created by Jenkins for a specific job. Test-agents can have multiple workspaces, one for each job.
-
-# Architecture
-
-![Zephyr CI Block Diagram](zephyrCI-block-diagrams-WW08-2021.png "zephyr CI block diagram WW08 2021")
+# Intel 1RTOS / Zephyr internal CI repo
+*a catch-all for DevOps services & documentation*
+
+**Contact: email to: FMOS_DevOps, cc: Vondrachek, Chris & Graydon, Connor**
+
+## hidden.tar.secret & accessing the hidden/ directory
+DevOps infrastructure secrets & private configuration data is stored encrypted as hidden.tar.secret with access controlled by a GPG keyring. 
+
+### To reveal contents of hidden.tar.secret into hidden/ :
+1. Your public GPG key must be enrolled in the keyring in this repo. 
+2. Use our automation script to decrypt hidden.tar.secret & decompress to hidden/
+
+	````trusted-gpg-user@ci.git/ $ ./reveal-hidden.sh````
+
+3. Access protected files at hidden/
+4. If any changes are made, you MUST run ./hide-hidden.sh to capture changes & encrypt
+
+### To hide the contents of hidden/ & stage hidden.tar.secret for commit:
+1. Your public GPG key must be enrolled in the keyring in this repo. 
+2. Use our automation script to tar & encrypt hidden/, and also stage the change for commit:
+
+	````trusted-gpg-user@ci.git/ $ ./hide-hidden.sh````
+
+3. Commit changes to hidden.tar.secret & push per usual
+
+## Useful Links
+### [ci.git/docs](docs/) - more DevOps documentation
+### [Zephyr DevOps Overview.pptx](https://intel-my.sharepoint.com/:p:/p/christopher_g_turner/EfZ2TF9ElydPjpGBEAKiUkwBiFt5LFBZPI2aGO_HZnP7Wg?e=Bxeeho) - Permalink for our most-often presented slide-deck
+
+# Block Diagram
+
+![Zephyr CI Block Diagram](zephyrCI-block-diagram-WW36-2021.png "zephyr CI block diagram WW36 2021")

docs/ci.wiki/DevOps-Engineering/Hypervisor-Operations.md

@@ -0,0 +1,35 @@
+# Zepyhr DevOps Hypervisor Operations
+
+## A. Summary
+
+DevOps operates a single VMware ESXI 6.7 hypervisor on jfsotc17 that is tasked with CI & test automation for Intel's internal Zephyr project efforts.
+
+## B. Accessing Hypervisor 
+
+The hypervisor is not directly connected to the Intel intranet. To access you must be connected to our secured TestNet or use SSH tunneling to expose the https services on your local machine:
+
+From remote:
+~~~~
+ssh -L 4430:192.168.0.254:443 zephyr-ci.jf.intel.com
+https://127.0.0.1:4430
+~~~~
+From TestNet (direct connection in lab):
+~~~~
+https://192.168.0.254:443
+~~~~
+
+The hypervisor is also accessible via SSH from TestNet for CLI operations
+
+### ACL 
+
+root account should not be used
+
+User accounts for DevOps engineers are created manually.
+
+## C. VM Control ( power on/off, reset )
+
+1. Notify users of reboot/downtime. If this is a production VM, clear operation with FMOS_DevOps
+2. Access ESXi UI per instructions in **B** above
+3. Select the VM instance you'd like to control
+4. Click the "Actions" gear & select operation. If the option you require is grayed-out, contact FMOS_DevOps for permissions.
+

docs/ci.wiki/DevOps-Engineering/Jenkins/Configuration-Spec.md

@@ -0,0 +1,71 @@
+# Zephyr DevOps Jenkins Configuration Specification
+**Purpose**
+This docs aims to serve as as standard-operating-procedure for deploying & configuring Jenkins for internal zephyr production use.
+
+**Target Audience**
+DevOps Engineers
+
+**Doc Change Process**
+* Minor changes & documentation improvements may be submitted by anyone. 
+* Major policy or configuration changes should be RFC'd @ FMOS_DevOps first.
+
+**NOTE THIS DOC IS WIP & CHANGE POLICY IS NOT ACTIVE**
+
+## Zephyr DevOps Jenkins Configuration Standard
+
+### 0. Jenkins Service Options + SSL Config
+
+#### Request/download .jks from https://certs.intel.com/aperture**
+
+#### Edit /etc/default/jenkins:
+
+    JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --httpsPort=8443 --httpsKeyStore=/srv/jenkins/ssl/<machine>.intel.com.jks --httpsKeyStorePassword=<passwd>
+
+#### Re-direct port 443 connections to 8443
+    sudo iptables -I INPUT 1 -p tcp --dport 8443 -j ACCEPT
+    sudo iptables -I INPUT 1 -p tcp --dport 8080 -j ACCEPT
+    sudo iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT
+    sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
+    sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
+    sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443
+    sudo apt-get install iptables-persistent
+
+### 1. System-Wide Environment Variables
+
+**Rule:** Don't use system-wide environment variables (those specified in the "Manage Jenkins" configuration). Env should always been handled in the pipeline code or job runners.
+
+**Exception:** Site or deployment specifics such as locale, for example:
+~~~~
+LANG=en_US.UTF-8
+PYTHONIOENCODING=UTF-8
+LANGUAGE=en_US:en
+LC_ALL=en_US.UTF-8
+~~~~
+
+### 2. Plugins
+
+**Rule:** In order to reduce DevOps cycles required for updates & overall CI execution risk, only install plug-ins from [approved list](https://gitlab.devtools.intel.com/zephyrproject-rtos/devops/infrastructure/ansible-playbooks/-/blob/current/jenkins-plugins.yaml)
+
+### 3. Users/Security
+
+**Option 1 - IT VAS + Jenkins PAM**
+
+If Jenkins is executing on a system with functional IT VAS, select 'Unix user/group database' as Jenkins Security Realm. This will restrict logins to accounts in /etc/passwd. This means that users wishing to access the Jenkins UI must have first logged in over SSH for VAS to pickup their idsid & create a /etc/passwd entry.
+
+**Option 2 - IT SAML**
+
+TBD...
+
+### 4. Jenkins Job Statuses
+
+Jenkins implements status conditions to represent the global status of a build: SUCCESS, UNSTABLE, FAILURE, NOT_BUILT or ABORTED. Zephyr DevOps maps these statues to Zephyr CI/Automation jobs as follows:
+
+**SUCCESS** - All tasks defined by job executed & returned success exit codes.
+
+**UNSTABLE** - All tasks defined by job executed but at least one step returned non-zero exit code.
+
+**FAILURE** - At least one task defined by job failed to execute.
+
+**ABORTED** - Job was aborted either by a user or timeout.
+
+**NOT_BUILT** - Unused currently.

docs/ci.wiki/DevOps-Engineering/Jenkins/Gitlab-Plugin.md

@@ -0,0 +1,42 @@
+# Zephyr DevOps Gitlab Plugin Info & Configuration
+
+**Purpose**
+This docs provides background information & configuration guidance for implementing Jenkins-Gitlab CI leveraging the gitlab plugin.
+
+**Target Audience**
+DevOps Engineers
+
+**Doc Change Process**
+* Minor changes & documentation improvements may be submitted by anyone. 
+* Major policy or configuration changes should be RFC'd @ FMOS_DevOps first.
+
+## 1. Gitlab Plugin Parameters & Jenkins Jobs
+
+RTM @ https://plugins.jenkins.io/gitlab-plugin/#parameter-configuration
+
+When a merge-request is opened on a project configured with Jenkins integration enabled, gitlab automatically transmits variables that specify the merge source repo & branch in the JSON webhook payload. DevOps also operates manually triggered jobs that allow user-provided parameters via the "Build with Parameters" option in Jenkins. 
+
+In order for these methods to coexist in the same job, we must support different paths for manual & automated execution: 
+1.)	automated trigger via gitlab plugin with gitlabSrcBranch & gitlabSrcRepo vars provided.
+2.)	manual trigger w/ user-provided srcBranch & srcRepo from “Build with Parameters” function.
+
+When the MRV starts, it first populates vars from the job parameters into srcRepo + srcBranch:
+~~~~
+//default to override values from Jenkins Job "Build with Parameters" dialog 
+def srcRepo="${env.overrideSourceRepo}"
+def srcBranch="${env.overrideSourceBranch}"
+~~~~
+
+Then we check for gitlab… vars & if set from a plugin trigger, use those instead
+
+~~~~
+//now override with gitlab-webhook supplied values, if they exist
+if (env.gitlabSourceBranch)
+{
+	echo "Triggered by gitlab merge-request webhook"
+	srcBranch="${env.gitlabSourceBranch}"
+	srcRepo="${env.gitlabSourceRepoSshUrl}"
+}
+~~~~
+
+At this point, the job can continue with srcBranch & srcRepo set correctly for either manual or automated triggers.

docs/ci.wiki/DevOps-Engineering/Python-Dependencies.md

@@ -0,0 +1,34 @@
+# Zephyr DevOps Python Dependency Method
+**Purpose**
+This doc describes how Python dependencies are managed on DevOps infrastructure.
+
+**Target Audience**
+DevOps Engineers
+
+**Doc Change Process**
+* Minor changes & documentation improvements may be submitted by anyone. 
+* Major policy or configuration changes should be RFC'd @ FMOS_DevOps first.
+## Overview
+
+Zephyr DevOps maintains separate Python dependency sets for each Zephyr build-environment. For example:
+
+v1.14-branch - west 0.6.3, cmake 13.3, located at /usr/local_v1.14-branch
+
+v2.5-branch - west <tbd>, cmake <tbd>, located at /usr/local_v2.5-branch
+
+master - west <latest>, cmake <latest>, located at /usr/local_master
+
+
+## Quick-start: Python dep install/update on DevOps VMs
+
+**0.** For production, schedule down-time for the VMs that you wish to update. For staging, simply clear update plans with other DevOps engineers via email or Teams chat.
+
+**1.** Confirm target environment is free of any existing Python packages installed under /usr/local.
+
+**2.** Run ansible playbook [nativeBuild02-pythonDeps.yaml](https://gitlab.devtools.intel.com/zephyrproject-rtos/devops/infrastructure/ansible-playbooks/-/blob/current/nativeBuild02-pythonDeps.yaml) with *'--limit=target.machine.intel.com'* to restrict actions to a single host.
+
+## Troubleshooting
+
+### West fails on "import west.main"
+
+This most often occurs on the Jenkins instances where users are likely to run 'sudo pip3 install <package>' which results in packages being installed under /usr/local & thus conflicting with packages ** **Make sure no depDon't use system-wide environment variables (those specified in the "Manage Jenkins" configuration). Env should always been handled in the pipeline code or job runners.