Support zos4 deployments

[AhmedHanafy725]

Which package/s are you suggesting this feature for?

Dashboard, grid_client

Is your feature request related to a problem? Please describe

There are now 2 different zos on the nodes (zos3, zos4), so the dashboard and grid client should identify the node zos version and use the corresponding workloads to deploy on them.

zos4 currently doesn't support wireguad, yggdrasil, or public ips. so it will require these filters from the grid proxy to be supported threefoldtech/tfgrid-sdk-go#1209

Describe the solution you'd like

• 2 new workloads should be added

  • network light
  • zmachine light

• Before starting the deployment, the grid client should check the node version and decide which workloads to be used to deploy the vm and network.

• Adjust the automatic and manual filters to respect the newly added queries on the gridproxy to suggest valid nodes for deployment.

[0oM4R]

• we just need to add the new network filters in the gridclient, and on manual filter we need to check if the node support the selected options
• we need to handle listing as well, how to list zos4 workloads

start with the workloads till we have filters ready on gridproxy

[maayarosama]

1. Work Completed:

• Added workloads for zmachinelight and network light
• Added Unit tests for both

2. Work in Progress (WIP):

• Trying to figure out why I can't import newly added workloads even after building
• Adding new filters in grid_client
• On deploying checking the node features

[xmonader]

Should we move it to in progress?

[maayarosama]

Work in Progress (WIP):

Added a helper method to match filters with features and I'll be adding more networks filters so the method works well

Added a Vmlight primitive class
Currently working on network light primitive class

[maayarosama]

Added a network light class in primitives, still need some adjustments

[maayarosama]

Network light class is ready, trying to deploy a zos4 machines and I get multiple different errors

[maayarosama]

Deployed a machine with new workloads and worked on the extracting features from filters method on grid client
Currently trying to add filters like mycelium, wiregaurd and planetary to the filtersoption in dashboard

[maayarosama]

Work Completed:

• Added mycelium and planetary to FiltersOption, still struggling with wiregaurd
• Added features to FiltersOptions
• Updated automated and manual selection so that features validations is taken to consideration

Work in Progress (WIP):

• Working on adding wiregaurd to FiltersOptions
• Updating the deploy method to support zos4 workloads

[maayarosama]

Work Completed:

• Added wiregaurd to FiltersOptions
• Updated the GetFeaturesFromFilters method
• Updated Networklight primitive class to include multiple needed methods
• Updated _createDeployment and create methods to support zos4 deployments

Work in Progress (WIP):

• Trying to deploy a zos4 and zos3 VM and I get the following error, My network is stable

       Maximum number of attempts exceeded
       Please check your internet connection and try again. If the problem persists, please contact our support.```

[maayarosama]

Work Completed:
I deployed a a zos3 machine successfully
Work in Progress (WIP):

Trying to deploy a zos4 machine but I get an error the node id isn't set on the network it's set to undefined

[maayarosama]

Work Completed:

• Adjusted the load method to get contracts and save nodeIds to an array
• Adjusted addNode method to check if node exists
• Adjusted nodeExsits method to check in node id is included in NodeIds array

Work in Progress (WIP):

• When call getFreeSubnet method in networklight primitive I get the following error as network instance isn't set, I'm currently trying to follow along the network primitives and trying to figure out and debug when to set the network instance
  TypeError: Cannot read properties of undefined (reading 'node_id')

[maayarosama]

Work in Progress (WIP):

• Am I correct in assuming that every time I try to deploy a zos4 machine , The IP value will always be undefined so the userIPsubnet will always be undefined
  

  tfgrid-sdk-ts/packages/grid_client/src/high_level/machine.ts

  Lines 207 to 212 in 8419d9b

  	let userIPsubnet;
  	let accessNodeSubnet;
  	//ip el machine nafse we validate the subnet in network and network light
  	if (ip) {
  	userIPsubnet = network.ValidateFreeSubnet(Addr(ip).mask(24).toString());

  
  so the subnet passed to the addNode method will always be undefined?
  

  tfgrid-sdk-ts/packages/grid_client/src/high_level/machine.ts

  Line 295 in 8419d9b

  const znet_workload = await network.addNode(nodeId, mycelium, description, userIPsubnet, myceliumNetworkSeeds);

• I get the following error while trying to deploy a zos4 instance

Failed to send request to twinId 9458 with command: zos.deployment.deploy, payload: 
{
   "version":0,
   "twin_id":79,
   "contract_id":169871,
   "expiration":0,
   "metadata":"{\"version\":4,\"type\":\"network-light\",\"name\":\"hello9test\",\"projectName\":\"vm/new9MY\"}",
   "description":"test deploying single VM with mycelium via ts grid3 client",
   "workloads":[
      {
         "version":0,
         "name":"hello9test",
         "type":"network",
         "data":{
            "subnet":"10.249.2.0/24",
            "mycelium":{
               "hex_key":"6ee79969926454b19b0862c8948f7e23ab186ac7005d4597862f34be29411d9f",
               "peers":[
                  
               ]
            },
            "ip_range":"10.249.0.0/24",
            "node_id":259
         },
         "metadata":"{\"version\":4}",
         "description":"test deploying single VM with mycelium via ts grid3 client"
      }
   ],
   "signature_requirement":{
      "requests":[
         {
            "twin_id":79,
            "required":false,
            "weight":1
         }
      ],
      "weight_required":1,
      "signatures":[
         {
            "twin_id":79,
            "signature":"fc4fe32a01767cd8b605583130f4457951baee1d26bbd7ffc2c3de22c259c072774e269259537d1acba6bd8c5f0bff9e41b5edea49a0df5a83dde6a75ece9d81",
            "signature_type":"sr25519"
         }
      ]
   }
}
0 invalid reservation type 'network'

[AhmedHanafy725]

Work in Progress (WIP):

• Am I correct in assuming that every time I try to deploy a zos4 machine , The IP value will always be undefined so the userIPsubnet will always be undefined
  so the subnet passed to the addNode method will always be undefined?

no, user can pick the vm ip, therefore the subnet could have a value

• I get the following error while trying to deploy a zos4 instance

Failed to send request to twinId 9458 with command: zos.deployment.deploy, payload: 
{
   "version":0,
   "twin_id":79,
   "contract_id":169871,
   "expiration":0,
   "metadata":"{\"version\":4,\"type\":\"network-light\",\"name\":\"hello9test\",\"projectName\":\"vm/new9MY\"}",
   "description":"test deploying single VM with mycelium via ts grid3 client",
   "workloads":[
      {
         "version":0,
         "name":"hello9test",
         "type":"network",
         "data":{
            "subnet":"10.249.2.0/24",
            "mycelium":{
               "hex_key":"6ee79969926454b19b0862c8948f7e23ab186ac7005d4597862f34be29411d9f",
               "peers":[
                  
               ]
            },
            "ip_range":"10.249.0.0/24",
            "node_id":259
         },
         "metadata":"{\"version\":4}",
         "description":"test deploying single VM with mycelium via ts grid3 client"
      }
   ],
   "signature_requirement":{
      "requests":[
         {
            "twin_id":79,
            "required":false,
            "weight":1
         }
      ],
      "weight_required":1,
      "signatures":[
         {
            "twin_id":79,
            "signature":"fc4fe32a01767cd8b605583130f4457951baee1d26bbd7ffc2c3de22c259c072774e269259537d1acba6bd8c5f0bff9e41b5edea49a0df5a83dde6a75ece9d81",
            "signature_type":"sr25519"
         }
      ]
   }
}
0 invalid reservation type 'network'

wasn't node 259 having an issue while testing the new workloads?

[maayarosama]

Work in Progress (WIP):

• 259 was down so I was blocked for a while waiting for zos4 nodes to be rebooted, trying to deploy on node 270 gives me the same error as above. While deploying on the node from the deployment script, the deployment is successful.
• While testing on the dashboard I noticed that no nodes were found after some debugging I noticed that sometimes the features has zmachine and zmachine-light. If we want to show both zos4 and zos3 nodes when it's applicable I suggest using two features array and doing two requests one for zos4 and one for zos3

[maayarosama]

Work in Progress (WIP):

• Updated GetFeaturesFromFilters method to not add machine, zmachine-light, network and network-light to the features array. Instead it adds mycelium or wireguard , etc.. to the array. Testing the feature is blocked on
• 🐞 [Bug]: Features filter gives 400 (Bad Request) tfgrid-sdk-go#1258

[maayarosama]

blocked on 0 invalid reservation type 'network' error

[maayarosama]

Work Completed:

• Fixed the invalid reservation error, The issue was in network light and VM light primitives as the workload type wasn't set correctly
• Deployed successfully on node 255, contract id is 172018

Work in Progress (WIP):

• seprating contracts for network light and zmachine light
• Documenting the new feature

Note:

• While trying to deploy on node 259 it takes a lot of time then fails, same thing happens with @rawdaGastan while trying to deploy from go client
  throw new TimeoutError(`Deployment with contract_id: ${contract_id} failed to be ready after ${timeout} minutes.`); ^ TimeoutError: Deployment with contract_id: 172007 failed to be ready after 10 minutes.

[maayarosama]

Work Completed:

• separating contracts for network light and zmachine light
• Adding a single_vm_zos4.ts script
• Documenting the deployment flow