CALM-DSL changes for v3.8.1 #308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue Link: https://jira.nutanix.com/browse/CALM-45018 Description: 1. Make a get call to app and fetch it's response. `app_profile_config_reference` in response contains name of profile used for launching application. 2. Use this launch profile name and filter out all other profiles before decompiling. Subsequently filter out dependent packages, substrates, deployments related to removed profiles. 3. Results after decompiling with above approach contains only profile used for launch in decompiled bp.  Decompiled blueprint: ` """ Generated blueprint DSL (.py) """ import json # no_qa import os # no_qa from calm.dsl.builtins import * # no_qa from calm.dsl.runbooks import CalmEndpoint as Endpoint # Secret Variables BP_CRED_admin_PASSWORD = read_local_file("BP_CRED_admin_PASSWORD") # Credentials BP_CRED_admin = basic_cred( "admin", BP_CRED_admin_PASSWORD, name="admin", type="PASSWORD", default=True, ) class Service1(Service): pass class VM1(Substrate): os_type = "Windows" provider_type = "EXISTING_VM" provider_spec = read_provider_spec(os.path.join("specs", "VM1_provider_spec.yaml")) readiness_probe = readiness_probe( connection_type="POWERSHELL", disabled=True, retries="5", connection_port=5985, address="@@{ip_address}@@", delay_secs="60", ) @action def __pre_create__(): CalmTask.Exec.escript.py2( name="Pre_Create1", filename=os.path.join( "scripts", "Substrate_VM1_Action___pre_create___Task_Pre_Create1.py" ), target=ref(VM1), ) class Default_Package(Package): services = [ref(Service1)] class Default_deployment(Deployment): min_replicas = "1" max_replicas = "3" default_replicas = "1" packages = [ref(Default_Package)] substrate = ref(VM1) class Default(Profile): deployments = [Default_deployment] class clonefortest_dsl_bug2(Blueprint): services = [Service1] packages = [Default_Package] substrates = [VM1] profiles = [Default] credentials = [BP_CRED_admin] class BpMetadata(Metadata): project = Ref.Project("Common") ` (cherry picked from commit 04caa35e046e6da1d19bfba83b181d96d0671ce2)
Cleanup activity: Removing stale entries in cache (~/.calm/dsl.db) from previous run so that it doesn't impact next regression run. Test: Tested by running another successfull round of regression after clean up. Regression: http://erdinger.eng.nutanix.com/blue/organizations/jenkins/QA%2FDSL_Regression/detail/DSL_Regression/618/pipeline/ (cherry picked from commit 28350835adbb642fb6e7cdf11a2360be4378248d)
Azure provider spec test cases were failing due to deprecated image being used. Replaced it with valid image version. Impacted test case: `tests/cli/provider_plugins/azure/test_azure_create_spec.py` <img width="588" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/850ffb2a-0339-41e4-be32-9fb71a32c471"> (cherry picked from commit 9016c96a4df66e4611dd72c47aec8c67bef74e2a)
Issue Link: https://jira.nutanix.com/browse/CALM-45373 Description: 1. Command was failing when categories was used as field in function `get_group_data_value` 2. entity_value[0]["values"] was empty list and function was returning 0th index from empty list/ 3. Added extra guard condition to return empty string in case entity_value[0]["values"] is empty list. Tested the fix and calm get marketplace bps execute successfully with valid categories column:  (cherry picked from commit 00ccc7de7caefa95fe3858841c0dd3e974f613cc)
Guest customisation was failing with key error if title was not a key in
the optionDict during adding guest customisation.
Used optionDict.get("title") which solves the is
sue as it returns None if not found.
Also added support for XML file paths. If XML file paths are used then
it extracts content and adds it to the final spec, with verification of
flow verifying existence of the xml file. As shown below example which
uses dummy xml.
<img width="1188" alt="Screenshot 2023-10-18 at 4 02 51 PM"
src="https://github.com/ideadevice/calm-dsl/assets/138576769/f79b0176-6249-4e5f-bed3-d1e5ec2b8606">
Original flow where content can be directly mentioned and they get used
directly is also present. As shown below.
<img width="1188" alt="Screenshot 2023-10-18 at 4 01 57 PM"
src="https://github.com/ideadevice/calm-dsl/assets/138576769/23b298bc-6dbb-43e7-9c58-bc5615dc1f1d">
(cherry picked from commit ef857187346f05d9f5186effc62d9f10cab42d7f)
All basic contents of main readme.md is now shifted to documentation on nutanix dev community website (https://www.nutanix.dev/docs/self-service-dsl/). Therefore, cleaning up sections and redirecting users to main documentation for respective section. (cherry picked from commit 1229652586ad6907cdeea4f27ca2ebc16df4cbc5)
**PR Description** Summary: Automate_CALM_44434_Policy_ID_updated_in_approval_name_list Test Plan: Approval policy name updated with policy ID so updated the code Bugs: #CALM-44434 (cherry picked from commit c5e00319ef6ec49c841f378ade105f7e3e393106)
Issue Link: https://jira.nutanix.com/browse/CALM-17380 Description: 1. Added support for testing escripts, shell scripts, powershell scripts, python remote tasks using run-script cli command. 2. `calm run-script` command has following flags: -> `(-t/--type)` - type of script to be tested, supported values [escript, shell, powershell, python] -> `(-f--file)` - script file path to test -> `(-e/--endpoint)` - endpoint file path (required for shell, powershell, python scripts) to define target machine -> `-p/--project` - reference project to be used while running script. Defaults to current context project in DSL. 4. Following support is being added through improvement: -> Test run of escript file -> Test run of shell script file with basic auth linux endpoint -> Test run of shell script file with ssh key based linux endpoint -> Test run of powershell script file with basic auth windows endpoint -> Test run of python remote script file with basic auth linux endpoint -> Test run of python remote script file with ssh key based linux endpoint 5. Following support is not present through UI. Hence, not implementing in DSL. -> Python remote tasks don't run on windows endpoint. -> Windows endpoint only have basic auth and no ssh based authentication. 6. Added unit test to test `calm run-script` command. <img width="1136" alt="image" src="https://github.com/ideadevice/calm-dsl/assets/123161845/fce93efd-54de-40cb-84fc-c35664dfaf86"> Example description of Run Script CLI commands: 1. Command to test escript file (doesn't require endpoint) <img width="971" alt="Screenshot 2024-06-19 at 5 34 08 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/7e33488a-8593-410a-9379-f7cc12570821"> 2. Command to test shell script with linux endpoint having basic auth <img width="1112" alt="Screenshot 2024-06-19 at 5 34 57 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/9b2b3750-4dc3-4b5e-8d64-b31e7e1e2784"> 3. Command to test shell script with linux endpoint having ssh auth <img width="1120" alt="Screenshot 2024-06-19 at 5 35 25 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/69d79e51-62de-4ccd-8687-460ec74a3024"> 4. Command to test powershell script with windows endpoint <img width="1114" alt="Screenshot 2024-06-19 at 5 36 07 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/b0e552b6-85aa-4af3-bf61-8b4edcff3081"> 5. Command to test python script with linux endpoint <img width="1115" alt="Screenshot 2024-06-19 at 5 36 49 PM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/ec096501-eb9b-4cb1-b9bd-1dfc312994d9"> (cherry picked from commit 79e5140b04f01e59b97b030de8073f496203306b)
CALM-39463 : https://jira.nutanix.com/browse/CALM-39463 Summary: 1. Added a new flag(api key) to capture the api key file location containing (name, token) to authenticate saas instance via dsl 2. Successfull authentication via saas api token showing username used to login (taking port 443 by default when api key location is provided)  3. Successfull authentication of pc (backward compatibility check) <img width="1179" alt="Screenshot 2023-11-09 at 10 13 42 AM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/cabad2b2-f57e-4970-a325-0b48708b7920"> 4. Check set config command for saas api token <img width="1221" alt="Screenshot 2023-11-09 at 10 17 11 AM" src="https://github.com/ideadevice/calm-dsl/assets/123161845/ba8ce234-829b-47b2-9821-2ab441065dea"> 5. Error handling check for incorrect or badly formatted keyfile   6. If saas port is provided without api key location. Throw a warning to supply api key location alongwith authentication error.  (cherry picked from commit 542d1714e1d34e72c6fbde2435d12740b68806b8)
Resetting api_key_location to None if it's the case of PC login. (cherry picked from commit febe0951534a365da6a5e1b99dbbcd15cdcdbbb8)
…id and upload endpoint to default project (#415) As part of the changes brought on by: ideadevice/calm#3269, a valid account reference needs to be sent while creating VM endpoints. The test was sending just a name and kind as part of the account_reference in the VM EP create payload. Modified the test to send a valid account_reference and uuid in the payload. The tests also downloads and uploads the endpoint, but does it using the _internal project. But since _internal project has no accounts in it. the /endpoints/import_file was not seeding the account_uuid in the uploaded endpoint. Hence modified the test to upload the endpoint to default project instead of internal project. --------- Co-authored-by: Prabhat Dwivedi <prabhat.dwivedi@nutanix.com> (cherry picked from commit 18f153cbe0614b9a5fef68bb05c45bd23e24be4b)
Description: -> Create endpoint payload should have valid vm reference due to fix in https://jira.nutanix.com/browse/CALM-45554 -> Earlier we were using dummy vm references. From now onwards vm references will be checked for authenticity, therefore, passing valid vm reference before executing vm endpoint tests. (cherry picked from commit 92602e1d2743301c74bbe37a59d34ef80d84233b)
(cherry picked from commit 5f56704431c642b88f1913b5df9c9530e76e87d4)
Fixed Endpoint VM Tests to use the VMs created via test_dsl setup and to create the endpoints inside the default project instead of _internal project. Fixed vm_actions tests to handle scenarios of endpoints with incorrect UUID (VM doesn't exists) (cherry picked from commit 2fe9d3b9b13f4cbf2718d0c738ea0dc8086208bf)
…st API call (#497) Currently, DSL calls /accounts/`uuid`/vms/list API to resolve VM references during Vm Endpoints Compilation. Since by default this API returns all VMs in the account. If the user supplies a VM that he isn't authorized to access, Vm Endpoint compilation goes through. Made a change in VM Ref compile code to always pass project_uuid in accounts/`uuid`/vms/list payload so that compilation passes only when authorized VMs are present in the reference. --------- Co-authored-by: Prabhat Dwivedi <prabhat.dwivedi@nutanix.com> (cherry picked from commit 23af8454d3dc2e94551dfc5d81659c79ade1371b)
(cherry picked from commit 176d3903b932ed98f2ef4b8e117802621e9249c6)
…a (#518) Issue Link: https://jira.nutanix.com/browse/CALM-46837 RCA: Vm reference verification check in #497 uses project in current context to fetch valid vm attached to a project but due to metadata payload being constructed after endpoint payload compilation, function `get_cur_context_project()` returned project from config.ini. Fix: Construct metadata payload before compiling endpoint and runbook. Note: While creating endpoint we are constructing metadata payload first therefore we didn't hit this issue. Adding fixing in compile endpoint logic. (cherry picked from commit 5225c62b802dfca4564bd9c0a813c4bc457f33c1)
(cherry picked from commit 2390d3b12475099d95877f2f036249e95705ac06)
Co-authored-by: Prabhat Dwivedi <prabhat.dwivedi@nutanix.com> (cherry picked from commit ed6ff9c45bfa97ede90445dc348a3f5fe4eea366)
(cherry picked from commit 8c179825c96c4e5c3d8814ceb82d9515b49ea547)
Updated latest version info (cherry picked from commit a0774e41549c9982e9bae56325f98a83fb1b29d6)
…et an entity. (#534) Issue Link: https://jira.nutanix.com/browse/CALM-47277 Description: 1. `/list` api for a runbook returns only non deleted runbook when `all` paramter is set `True` in `get_runbook` 2. Ideally, deleted entity should also be returned with `all` parameter 3. Modified filter parameter using `get_states_filter` method which now returns all types of runbook with `all` parameter 4. Similar changes done to endpoint, blueprints, library tasks, scheduler job 5. Use case: in describe command we pass all parameter as True which means we can describe deleted entities as well. With this changes it funcitons properly now. Earlier for deleted entities no entity found error came.     (cherry picked from commit a2c16e952272a2c3440f82d40d1646cdafc59deb)
(cherry picked from commit 46ce17c500fb1b40b1e5fa1da5201632e8eba022)
abhijeetkaurav1st
approved these changes
Aug 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.