[DataLake-job] Model validation fails with errors

[veronicagg]

Validating "examples" and "x-ms-examples" in /Users/Veronica/github/openapi-validation-tools/arm-datalake-job/swagger/job.json:
> Operation: Job_Build
> x-ms-examples
> Scenario: SubmitGetListCancelTestPOST341
> InnerErrors:
1. code: MINIMUM
message: Value 0 is less than minimum 1
path: degreeOfParallelism
description: the degree of parallelism used for this job. This must be greater than 0.

Model from swagger:

        "degreeOfParallelism": {
          "type": "integer",
          "minimum": 1,
          "default": 1,
          "format": "int32",
          "description": "the degree of parallelism used for this job. This must be greater than 0."
        },

Example from .net recording - shows response returns invalid value for property:

{
  "parameters": {
    "api-version": "2016-11-01",
    "expanded": "true",
    "parameters": {
      "name": "azure sdk data lake analytics job",
      "type": "USql",
      "degreeOfParallelism": 2,
      "properties": {
        "type": "USql",
        "script": "DROP DATABASE IF EXISTS testdb; CREATE DATABASE testdb;"
      }
    }
  },
  "responses": {
    "200": {
      "body": {
        "jobId": "f9d64a74-f687-4227-bfb6-8552cd27d4d3",
        "name": "azure sdk data lake analytics job",
        "type": "USql",
        "submitter": "adlsvc01@benwgoldoutlook.onmicrosoft.com",
        "degreeOfParallelism": 0,
        "priority": 0,
        "submitTime": "0001-01-01T00:00:00-08:00",
        "state": "Ended",
        "result": "Succeeded",
        "properties": {
          "resources": [],
          "runtimeVersion": "kobo_vnext_signed_5177484",
          "script": "DROP DATABASE IF EXISTS testdb; CREATE DATABASE testdb;",
          "algebraFilePath": "mafs://accounts/testdatalake12107/fs/system/compilationService/jobs/USql/f9d64a74-f687-4227-bfb6-8552cd27d4d3/2016/11/08/04/16/algebra.xml",
          "compileMode": "Semantic",
          "errorSource": "Unknown",
          "totalCompilationTime": "PT0S",
          "totalPausedTime": "PT0S",
          "totalQueuedTime": "PT0S",
          "totalRunningTime": "PT0S",
          "diagnostics": [],
          "type": "USql"
        }
      }
    }
  }
}

> Operation: Job_Build
		> Scenario: SubmitGetListCancelTestPOST332
			> InnerErrors:
			  1. code: INVALID_TYPE
			     message: Expected type array but found type null
			     path: stateAuditRecords
			     description: the job state audit records, indicating when various operations have been performed on this job.
			  2. code: INVALID_TYPE
			     message: Expected type array but found type null
			     path: errorMessage
			     description: the error message details for the job, if the job failed.
			  3. code: INVALID_TYPE
			     message: Expected type string but found type null
			     path: endTime
			     description: the completion time of the job.
			  4. code: INVALID_TYPE
			     message: Expected type string but found type null
			     path: startTime
			     description: the start time of the job.
			  5. code: INVALID_FORMAT
			     message: Object didn't pass validation for format date-time: Mon, 01 Jan 0001 00:00:00 GMT
			     path: submitTime
			     description: the time the job was submitted to the service.

Model from swagger:

        "stateAuditRecords": {
          "type": "array",
          "readOnly": true,
          "items": {
            "$ref": "#/definitions/JobStateAuditRecord"
          },
          "description": "the job state audit records, indicating when various operations have been performed on this job."
        },

        "errorMessage": {
          "type": "array",
          "readOnly": true,
          "items": {
            "$ref": "#/definitions/JobErrorDetails"
          },
          "description": "the error message details for the job, if the job failed."
        },

        "endTime": {
          "type": "string",
          "readOnly": true,
          "format": "date-time",
          "description": "the completion time of the job."
        },

        "startTime": {
          "type": "string",
          "readOnly": true,
          "format": "date-time",
          "description": "the start time of the job."
        },

        "submitTime": {
          "type": "string",
          "readOnly": true,
          "format": "date-time",
          "description": "the time the job was submitted to the service."
        },

> Operation: Job_Get
	> x-ms-examples
		> Scenario: SubmitGetListCancelTestGET526
			> InnerErrors:
			  1. code: INVALID_TYPE
			     message: Expected type string but found type null
			     path: stateAuditRecords,4,requestedByUser
			     description: the user who requests the change.
			  2. code: INVALID_FORMAT
			     message: Object didn't pass validation for format date-time: Fri, 20 Nov 2015 22:27:00 GMT
			     path: stateAuditRecords,4,timeStamp
			     description: the time stamp that the state change took place.
			  5. code: INVALID_TYPE
			     message: Expected type string but found type null
			     path: stateAuditRecords,2,details
			     description:  the details of the audit log.

Model from swagger:

        "requestedByUser": {
          "type": "string",
          "readOnly": true,
          "description": "the user who requests the change."
        },

        "timeStamp": {
          "type": "string",
          "readOnly": true,
          "format": "date-time",
          "description": "the time stamp that the state change took place."
        },

        "details": {
          "type": "string",
          "readOnly": true,
          "description": " the details of the audit log."
        }

> Operation: Job_Create
	> x-ms-examples
		> Scenario: CredentialCRUDTestPUT525
			> InnerErrors:
			  1. code: OBJECT_MISSING_REQUIRED_PROPERTY
			     message: Missing required property: script
			     path: properties
			     description: The common Data Lake Analytics job properties.		  

Model from swagger:

    "JobProperties": {
      "discriminator": "type",
      "required": [
        "script",
        "type"
      ],
      "properties": {
        "runtimeVersion": {
          "type": "string",
          "description": "the runtime version of the Data Lake Analytics engine to use for the specific type of job being run."
        },
        "script": {
          "type": "string",
          "description": "the script to run"
        },
        "type": {
          "type": "string",
          "description": "the job type of the current job (i.e. Hive or U-SQL)."
        }
      },
      "description": "The common Data Lake Analytics job properties."
    },

> Operation: Job_List
	> x-ms-examples
		> Scenario: SubmitGetListCancelTestGET640
			> InnerErrors:
			  1. code: OBJECT_MISSING_REQUIRED_PROPERTY
			     message: Missing required property: properties
			     path: value,1
			     description: The common Data Lake Analytics job information properties.
                   > Scenario: SubmitGetListCancelTestGET643
			> InnerErrors:
			  2. code: OBJECT_MISSING_REQUIRED_PROPERTY
			     message: Missing required property: type
			     path: value,1
			     description: The common Data Lake Analytics job information properties.
			  3. code: OBJECT_MISSING_REQUIRED_PROPERTY
			     message: Missing required property: name
			     path: value,1
			     description: The common Data Lake Analytics job information properties.

Extract from swagger model:

    "JobInformation": {
      "required": [
        "name",
        "type",
        "properties"
      ],

		> Scenario: SubmitGetListCancelTestGET631
			> InnerErrors:
			  1. code: INVALID_TYPE
			     message: Expected type string but found type null
			     path: nextLink
			     description: the link (url) to the next page of results.

Model from swagger:

    "JobInfoListResult": {
      "properties": {
        "value": {
          "type": "array",
          "readOnly": true,
          "items": {
            "$ref": "#/definitions/JobInformation"
          },
          "description": "the list of jobInfo items."
        },
        "nextLink": {
          "type": "string",
          "readOnly": true,
          "description": "the link (url) to the next page of results."
        }
      },
      "description": "List of jobInfo items."
    }

[lmazuel]

@begoldsm ?

[begoldsm]

@veronicagg this is an interesting issue. We do require a minimum degreeOfParallelism for submitting jobs, but for building jobs this field is ignored, and the service simply returns default(int), which happens to be zero. I can remove the restriction in our swagger specs, since our service will fix submissions with degreeOfParallelism < 1 to be = 1. Thoughts?

[amarzavery]

But you still need that constraint while submitting the job. May be documenting this in the description and removing it as a constraint is the best option. I am not sure if using allOf can solve this accurately.

[begoldsm]

That is fair, I can make that update.

[begoldsm]

PR out: #917

[begoldsm]

This should now be resolved with the merge of PR #917. @amarzavery or @veronicagg can you verify and close this issue (I do not have permission to do so).

[begoldsm]

@amarzavery and @veronicagg please verify and close this.

[kirthik]

@veronicagg - Could you please close the issue if this is fixed