Pattern Validation - Max Length Issue Fix #1426

samyakOO7 · 2024-11-14T11:50:48Z

Pattern validation was taking priority in response schema for stub and request schema for test over max length parameter due to random() method generating value more than default

Fixed the use case putting a constraint of max length on random() method so that it never generates value for than max length keeping the current logic intact

Test Spec File - validate-regex-spec.yaml.zip

Backward Compatibility Spec (v2.0.36 release check) -
validate-regex-spec.yaml.zip

Logic used -

Default length - 5 (remains intact)

If minimum and maximum is both supplied, pattern length would be in range of (min to max)

If minimum length is not supplied in spec -> and maximum is less than default (5), pattern would be of length (1 i.e. generex min value, max)
Say max = 4 is only supplied
pattern length would be in range (1-4)

If minimum length is not supplied in spec -> and maximum is greater than default (5), pattern would be of length (5 (default, max)
Say max = 15 is only supplied
pattern length would be in range (5-15)

if max is not supplied and min value exist we will assume to take pattern length from (min , generex max value)

Test for maximum length conditions :
Request : {minLength : 2, maxLength: 11}
Response : {minLength : 2, maxLength: 6}

Before Stub :

curl -X POST -H "Content-Type: application/json" -d '{ "number": "12000000" }' http://0.0.0.0:49272/validate
{
    "number": "873.13863"
}

Generated value : 9 digit

Before Test:
Details:

Scenario: POST /validate -> 200 has FAILED
In scenario "Validate the input number. Response: Input is valid"
API: POST /validate -> 200

  In scenario "Validate the input number. Response: Input is valid"
     API: POST /validate -> 200
   
       REQUEST.BODY.number Contract expected string with maxLength 2 but request contained "2.8837947"

After Issue Fixed Stub:

curl -X POST -H "Content-Type: application/json" -d '{ "number": "12000000" }' http://0.0.0.0:9000/validate
{
    "number": "8.9162"
}

After Issue Fixed Test:

Request:
POST /validate
Accept-Charset: UTF-8
Accept: */*
Content-Type: application/json

{
    "number": "3."
}

Response:
200 OK
Vary: Origin
X-Specmatic-Result: success
X-Specmatic-Type: random
Content-Length: 26
Content-Type: application/json
Connection: keep-alive

{
    "number": "3618.9"
}

Test for Minimum Length conditions:
Request : {minLength : 2, maxLength: 11}
Response : {minLength : 2, maxLength: 6}

Stub:
Before Fix (The output generated was defaulting to 5 as minimum for num<5 (breaching min validation):

curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:54811/validate
{
    "number": "2.866"
}
 curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:54811/validate
{
    "number": "235.4"
}
curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:54811/validate
{
    "number": "8.150"
}

After Fix (Min validation is respected):

curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "2."
}
 curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "5.1"
}
curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:54811/validate
{
    "number": "8.150"
}
 curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "39"
}

Test :
Before Fix:

Request:
POST /validate
Accept-Charset: UTF-8
Accept: */*
Content-Type: application/json

{
    "number": "6.7293812038"
} 

Response:
400 Bad Request
Vary: Origin
X-Specmatic-Result: failure
Content-Length: 411
Content-Type: text/plain
Connection: keep-alive

In scenario "Validate the input number. Response: Input is valid"
API: POST /validate -> 200

  >> REQUEST.BODY.number
  
     Contract expected string with maxLength 2 but request contained "6.7293812038"

After Fix :

Request:
POST /validate
Accept-Charset: UTF-8
Accept: */*
Content-Type: application/json

{
    "number": "9."
}
Response:
200 OK
Vary: Origin
X-Specmatic-Result: success
X-Specmatic-Type: random
Content-Length: 23
Content-Type: application/json
Connection: keep-alive

{
    "number": "407"
}

Some edge cases taken care of defaults -

 Request { maxLength = 4 }
Response { maxLength = 4 } 
min = null and max (4) < default (5) [pattern (min ‎ = Generex(min val = 1), max = 4)]

Stub-

curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "7"
} curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "3.47"
}

Test -

Request:
POST /validate
Accept-Charset: UTF-8
Accept: */*
Content-Type: application/json

{
    "number": "65.0"
}
Response:
200 OK
Vary: Origin
X-Specmatic-Result: success
X-Specmatic-Type: random
Content-Length: 23
Content-Type: application/json
Connection: keep-alive

{
    "number": "4.72”
}

Request { maxLength = 15 }
Response { maxLength = 15 }
min length = null and max length > default (5) [pattern (min ‎ = default(5), max = 15)]

Stub-

curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "8.915"
} curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "9.609"
}
curl -X POST -H "Content-Type: application/json" -d '{ "number": "1" }' http://0.0.0.0:9000/validate
{
    "number": "7.09024600"
}

Test -

Request:
POST /validate
Accept-Charset: UTF-8
Accept: */*
Content-Type: application/json

{
    "number": "4.22214"
} Response:
200 OK
Vary: Origin
X-Specmatic-Result: success
X-Specmatic-Type: random
Content-Length: 25
Content-Type: application/json
Connection: keep-alive

{
    "number": "2.648"
}

Clear outputs respecting validation schema are found with multiple test of min and max length

Thanks

- Removing comments and instead renaming variables for readability - Removing redundant rows in parameterised test for min and max length based String generation - Adding test for regex based generation - Extracting regex generation code to separate method

harikrishnan83

Looks good to me

harikrishnan83 · 2024-11-15T12:41:06Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

-            if(defaultExample == null)
-                return StringValue(Generex(regex.removePrefix("^").removeSuffix("$")).random(randomStringLength))
+            if(defaultExample == null) {
+                if (maxLength != null)


Is there a possibility that the regex pattern can generate a String that is less than minLength, should we handle that also?

harikrishnan83 · 2024-11-15T12:42:34Z

core/src/test/kotlin/io/specmatic/core/pattern/StringPatternTest.kt

@@ -240,4 +250,12 @@ internal class StringPatternTest {
    fun `string pattern encompasses email`() {
        assertThat(StringPattern().encompasses(EmailPattern(), Resolver(), Resolver())).isInstanceOf(Result.Success::class.java)
    }
+
+    @Test
+    fun `should fail to generate string when maxLength is less than minLength`() {


good that you have covered this scenario with a test

harikrishnan83 · 2024-11-15T12:45:25Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

+            if(defaultExample == null) {
+                if (maxLength != null)
+                    return StringValue(
+                        Generex(regex.removePrefix("^").removeSuffix("$")).random(


any reason we have to remove the ^ and $?

As these characters are not addressed by generex we are omitting it

harikrishnan83 · 2024-11-15T12:46:29Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

+                if (maxLength != null)
+                    return StringValue(
+                        Generex(regex.removePrefix("^").removeSuffix("$")).random(
+                            randomStringLength,


if randomStringLength is already based on minLength and maxLength why are we looking at maxLength again here?

Before it was only based on minLength

joelrosario · 2024-11-21T14:46:59Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

+            if (matches(it, resolver).isSuccess()) {
+                return it
+            }
+            throw ContractException("Schema example ${it.toStringLiteral()} does not match pattern $regex")


In this form, this condition is no more about regex matching.

Ideally, the resolver should be given a new MismatchMessages indicating that we are checking the schema example.

The matches method returns a result, and we could call result.throwOnFailure() instead of throwing this exception

joelrosario · 2024-11-21T14:47:29Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

-            minLength != null && 5 < minLength -> minLength
-            maxLength != null && 5 > maxLength -> maxLength
+            minLength != null && minLength > 0 -> minLength
+            maxLength != null && maxLength < 5 -> 1


This worked a little differently earlier. If maxLength < 4, use it, else use upto 5 chars. What's the reason we changed this?

Logic used -

Default length - 5 (remains intact)
If minimum and maximum is both supplied, pattern length would be in range of (min to max)
If minimum length is not supplied in spec -> and maximum is less than default (5), pattern would be of length (1 i.e. generex min value, max)
Say max = 4 is only supplied
pattern length would be in range (1-4)
If minimum length is not supplied in spec -> and maximum is greater than default (5), pattern would be of length (5 (default, max)
Say max = 15 is only supplied
pattern length would be in range (5-15)
if max is not supplied and min value exist we will assume to take pattern length from (min , generex max value)

We revamped this logic a bit as per the one given above

joelrosario · 2024-11-21T14:49:49Z

core/src/main/kotlin/io/specmatic/core/pattern/StringPattern.kt

+
+    private fun generateFromRegex(regexWithoutCaretAndDollar: String, minLength: Int, maxLength: Int?): String =
+        maxLength?.let {
+            Generex(regexWithoutCaretAndDollar).random(minLength, it)


Regexes supported by Generex may also have a length, e.g. a{10}. While we are at it, do we want to handle this in case it's greater than a maxLength or less than minLength?

We are putting the constraints to generate regex from min to max length, previously we only gave it the min length and we assumed it to be normal length, which caused it to overflow most of the time, out of max range

The logic written before was a bit tricky to understand but was failing lot of cases

joelrosario · 2024-11-21T14:51:39Z

core/src/test/kotlin/io/specmatic/core/pattern/StringPatternTest.kt

-    @MethodSource("lengthTestValues")
-    fun `generate string value of appropriate length matching minLength and maxLength parameters`(min: Int?, max: Int?, length: Int) {
+    @MethodSource("minLengthMaxLengthAndExpectedLength")
+    fun `generate string value as per minLength and maxLength`(min: Int?, max: Int?, expectedLengthOfGeneratedValue: Int) {


Use CsvSource instead of MethodSource, to keep the test data local to the test.

joelrosario · 2024-11-21T14:52:02Z

core/src/test/kotlin/io/specmatic/core/pattern/StringPatternTest.kt

+
+    @ParameterizedTest
+    @MethodSource("regexMinLengthAndMaxLengthAndExpectedLength")
+    fun `generate string value as per regex in conjunction with minLength and maxLength`(regex: String?, min: Int?, max: Int?, expectedLength: Int) {


Use CsvSource instead of MethodSource, to keep the test data local to the test.

* main: (124 commits) Fix schema regex and generation 2.0.43 Generate 4xx responses as per the schema in virtual-service be it any pattern containing a json object pattern Ignore a seed data entry which involves attribute selection during the seed data load stage of the virtual service Ignore a seed data entry with an id which is pre-existing in the cache of the virtual service Fix failing tests due to stale config. Add test for ExamplePreProcessor Add tests for Asserts. Cleanup on amber validation. 2.0.42 Changes for GoBack Added Details Pre Div Removed highlighting of specmatic errors in editor Editor syntax highlighting Fix full pattern validation with ListPattern. Add tests for partially valid examples. Better mismatchMessages for missing optional keys. better variable namings and partial handling Interactive Server partially valid examples. Modify result logic when all keys are mandatory. ...

- adding tests for contradicting length constraints and regex

Pattern Validation - Max Length Issue Fix

278d865

samyakOO7 requested review from joelrosario, nashjain, harikrishnan83, pranavgawri, yogeshnikam671 and StarKhan6368 November 14, 2024 11:51

Samy and others added 8 commits November 16, 2024 11:00

Min Length Fix Tested

f0661ca

Default case updated

a6b4f19

Test case fixed

0661182

Cleaned code- Removed Cognitive Complexity

63a8fad

Pattern Validation Logic for Minimum Size Regex

6b177cf

Pattern Validation Logic for Minimum Size Regex

80c5f51

Updated Default Example Match Condition To the Top

75079ad

harikrishnan83 removed request for nashjain, pranavgawri, yogeshnikam671 and StarKhan6368 November 19, 2024 04:40

harikrishnan83 approved these changes Nov 19, 2024

View reviewed changes

Samy added 4 commits November 21, 2024 14:44

File formatted

13e7d53

File formatted

36cdf4e

File formatted

4634e6a

Fixed test cases

5b4ae82

joelrosario requested changes Nov 21, 2024

View reviewed changes

Samy added 5 commits November 22, 2024 14:04

Changes for String Pattern throwOnFailure()

1f10df5

Changes for String Pattern Csv Source Tests

5ad03bd

Removed unwanted arguments

82b1e46

String Pattern Validation Updated

0adddd6

Specmatic String Pattern Tests

8453faf

Samy and others added 7 commits November 27, 2024 12:53

Specmatic String Pattern Tests Rectified

3ff35ec

Specmatic String Pattern Tests Rectified

ef5f8e0

Merged latest

0175b6b

Merged latest

aeb16e5

Merged latest

f39364b

- minor grammatical and spelling changes in tests

9a8f77a

- adding tests for contradicting length constraints and regex

samyakOO7 requested a review from joelrosario December 13, 2024 11:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pattern Validation - Max Length Issue Fix #1426

Pattern Validation - Max Length Issue Fix #1426

samyakOO7 commented Nov 14, 2024 •

edited

Loading

harikrishnan83 left a comment

harikrishnan83 Nov 15, 2024

harikrishnan83 Nov 15, 2024

harikrishnan83 Nov 15, 2024

samyakOO7 Nov 22, 2024

harikrishnan83 Nov 15, 2024

samyakOO7 Nov 22, 2024

joelrosario Nov 21, 2024

samyakOO7 Nov 26, 2024

joelrosario Nov 21, 2024

samyakOO7 Nov 21, 2024 •

edited

Loading

samyakOO7 Nov 21, 2024

joelrosario Nov 21, 2024

samyakOO7 Nov 21, 2024 •

edited

Loading

joelrosario Nov 21, 2024

joelrosario Nov 21, 2024

Pattern Validation - Max Length Issue Fix #1426

Are you sure you want to change the base?

Pattern Validation - Max Length Issue Fix #1426

Conversation

samyakOO7 commented Nov 14, 2024 • edited Loading

harikrishnan83 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samyakOO7 Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samyakOO7 Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samyakOO7 commented Nov 14, 2024 •

edited

Loading

samyakOO7 Nov 21, 2024 •

edited

Loading

samyakOO7 Nov 21, 2024 •

edited

Loading