Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add options to control caching of schemas #1018

Merged
merged 1 commit into from
Apr 16, 2024
Merged

Add options to control caching of schemas #1018

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,16 +461,19 @@ The following is sample output from the Hierarchical format.

### Schema Validators Configuration

| Name | Description | Default Value
|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------
| `pathType` | The path type to use for reporting the instance location and evaluation path. Set to `PathType.JSON_POINTER` to use JSON Pointer. | `PathType.DEFAULT`
| `ecma262Validator` | Whether to use the ECMA 262 `joni` library to validate the `pattern` keyword. This requires the dependency to be manually added to the project or a `ClassNotFoundException` will be thrown. | `false`
| `executionContextCustomizer` | This can be used to customize the `ExecutionContext` generated by the `JsonSchema` for each validation run. | `null`
| `schemaIdValidator` | This is used to customize how the `$id` values are validated. Note that the default implementation allows non-empty fragments where no base IRI is specified and also allows non-absolute IRI `$id` values in the root schema. | `JsonSchemaIdValidator.DEFAULT`
| `messageSource` | This is used to retrieve the locale specific messages. | `DefaultMessageSource.getInstance()`
| `locale` | The locale to use for generating messages in the `ValidationMessage`. | `Locale.getDefault()`
| `failFast` | Whether to return failure immediately when an assertion is generated. | `false`
| `formatAssertionsEnabled` | The default is to generate format assertions from Draft 4 to Draft 7 and to only generate annotations from Draft 2019-09. Setting to `true` or `false` will override the default behavior. | `null`
| Name | Description | Default Value
|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------
| `pathType` | The path type to use for reporting the instance location and evaluation path. Set to `PathType.JSON_POINTER` to use JSON Pointer. | `PathType.DEFAULT`
| `ecma262Validator` | Whether to use the ECMA 262 `joni` library to validate the `pattern` keyword. This requires the dependency to be manually added to the project or a `ClassNotFoundException` will be thrown. | `false`
| `executionContextCustomizer` | This can be used to customize the `ExecutionContext` generated by the `JsonSchema` for each validation run. | `null`
| `schemaIdValidator` | This is used to customize how the `$id` values are validated. Note that the default implementation allows non-empty fragments where no base IRI is specified and also allows non-absolute IRI `$id` values in the root schema. | `JsonSchemaIdValidator.DEFAULT`
| `messageSource` | This is used to retrieve the locale specific messages. | `DefaultMessageSource.getInstance()`
| `preloadJsonSchema` | Whether the schema will be preloaded before processing any input. This will use memory but the execution of the validation will be faster. | `true`
| `preloadJsonSchemaRefMaxNestingDepth` | The max depth of the evaluation path to preload when preloading refs. | `40`
| `cacheRefs` | Whether the schemas loaded from refs will be cached and reused for subsequent runs. Setting this to `false` will affect performance but may be neccessary to prevent high memory usage for the cache if multiple nested applicators like `anyOf`, `oneOf` and `allOf` are used. | `true`
| `locale` | The locale to use for generating messages in the `ValidationMessage`. | `Locale.getDefault()`
| `failFast` | Whether to return failure immediately when an assertion is generated. | `false`
| `formatAssertionsEnabled` | The default is to generate format assertions from Draft 4 to Draft 7 and to only generate annotations from Draft 2019-09. Setting to `true` or `false` will override the default behavior. | `null`

## Performance Considerations

Expand Down
19 changes: 14 additions & 5 deletions src/main/java/com/networknt/schema/DynamicRefValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;
import java.util.Collections;
import java.util.Set;
import java.util.function.Supplier;

/**
* {@link JsonValidator} that resolves $dynamicRef.
Expand All @@ -39,7 +41,7 @@ public DynamicRefValidator(SchemaLocation schemaLocation, JsonNodePath evaluatio
static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext validationContext, String refValue,
JsonNodePath evaluationPath) {
String ref = resolve(parentSchema, refValue);
return new JsonSchemaRef(new CachedSupplier<>(() -> {
return new JsonSchemaRef(getSupplier(() -> {
JsonSchema refSchema = validationContext.getDynamicAnchors().get(ref);
if (refSchema == null) { // This is a $dynamicRef without a matching $dynamicAnchor
// A $dynamicRef without a matching $dynamicAnchor in the same schema resource
Expand Down Expand Up @@ -73,9 +75,13 @@ static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext val
refSchema = refSchema.fromRef(parentSchema, evaluationPath);
}
return refSchema;
}));
}, validationContext.getConfig().isCacheRefs()));
}


static <T> Supplier<T> getSupplier(Supplier<T> supplier, boolean cache) {
return cache ? new CachedSupplier<>(supplier) : supplier;
}

private static String resolve(JsonSchema parentSchema, String refValue) {
// $ref prevents a sibling $id from changing the base uri
JsonSchema base = parentSchema;
Expand Down Expand Up @@ -153,14 +159,17 @@ public void preloadJsonSchema() {
SchemaLocation schemaLocation = jsonSchema.getSchemaLocation();
JsonSchema check = jsonSchema;
boolean circularDependency = false;
int depth = 0;
while (check.getEvaluationParentSchema() != null) {
depth++;
check = check.getEvaluationParentSchema();
if (check.getSchemaLocation().equals(schemaLocation)) {
circularDependency = true;
break;
}
}
if (!circularDependency) {
if (this.validationContext.getConfig().isCacheRefs() && !circularDependency
&& depth < this.validationContext.getConfig().getPreloadJsonSchemaRefMaxNestingDepth()) {
jsonSchema.initializeValidators();
}
}
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/com/networknt/schema/JsonNodePath.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,8 +227,8 @@ public boolean equals(Object obj) {
if (getClass() != obj.getClass())
return false;
JsonNodePath other = (JsonNodePath) obj;
return Objects.equals(parent, other.parent) && Objects.equals(pathSegment, other.pathSegment)
&& pathSegmentIndex == other.pathSegmentIndex && type == other.type;
return Objects.equals(pathSegment, other.pathSegment) && pathSegmentIndex == other.pathSegmentIndex
&& type == other.type && Objects.equals(parent, other.parent);
}

@Override
Expand Down
29 changes: 26 additions & 3 deletions src/main/java/com/networknt/schema/JsonSchemaFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,17 @@ protected JsonSchema newJsonSchema(final SchemaLocation schemaUri, final JsonNod
final ValidationContext validationContext = createValidationContext(schemaNode, config);
JsonSchema jsonSchema = doCreate(validationContext, getSchemaLocation(schemaUri),
new JsonNodePath(validationContext.getConfig().getPathType()), schemaNode, null, false);
preload(jsonSchema, config);
return jsonSchema;
}

/**
* Preloads the json schema if the configuration option is set.
*
* @param jsonSchema the schema to preload
* @param config containing the configuration option
*/
private void preload(JsonSchema jsonSchema, SchemaValidatorsConfig config) {
if (config.isPreloadJsonSchema()) {
try {
/*
Expand All @@ -302,7 +313,6 @@ protected JsonSchema newJsonSchema(final SchemaLocation schemaUri, final JsonNod
*/
}
}
return jsonSchema;
}

public JsonSchema create(ValidationContext validationContext, SchemaLocation schemaLocation, JsonNodePath evaluationPath, JsonNode schemaNode, JsonSchema parentSchema) {
Expand Down Expand Up @@ -471,7 +481,7 @@ public JsonSchema getSchema(final InputStream schemaStream, final SchemaValidato
public JsonSchema getSchema(final InputStream schemaStream) {
return getSchema(schemaStream, createSchemaValidatorsConfig());
}

/**
* Gets the schema.
*
Expand All @@ -480,6 +490,19 @@ public JsonSchema getSchema(final InputStream schemaStream) {
* @return the schema
*/
public JsonSchema getSchema(final SchemaLocation schemaUri, final SchemaValidatorsConfig config) {
JsonSchema schema = loadSchema(schemaUri, config);
preload(schema, config);
return schema;
}

/**
* Loads the schema.
*
* @param schemaUri the absolute IRI of the schema which can map to the retrieval IRI.
* @param config the config
* @return the schema
*/
protected JsonSchema loadSchema(final SchemaLocation schemaUri, final SchemaValidatorsConfig config) {
if (enableSchemaCache) {
// ConcurrentHashMap computeIfAbsent does not allow calls that result in a
// recursive update to the map.
Expand All @@ -500,7 +523,7 @@ public JsonSchema getSchema(final SchemaLocation schemaUri, final SchemaValidato
}
return getMappedSchema(schemaUri, config);
}

protected ObjectMapper getYamlMapper() {
return this.yamlMapper != null ? this.yamlMapper : YamlMapperFactory.getInstance();
}
Expand Down
19 changes: 14 additions & 5 deletions src/main/java/com/networknt/schema/RecursiveRefValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;
import java.util.Collections;
import java.util.Set;
import java.util.function.Supplier;

/**
* {@link JsonValidator} that resolves $recursiveRef.
Expand All @@ -47,11 +49,15 @@ public RecursiveRefValidator(SchemaLocation schemaLocation, JsonNodePath evaluat

static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext validationContext, String refValue,
JsonNodePath evaluationPath) {
return new JsonSchemaRef(new CachedSupplier<>(() -> {
return new JsonSchemaRef(getSupplier(() -> {
return getSchema(parentSchema, validationContext, refValue, evaluationPath);
}));
}, validationContext.getConfig().isCacheRefs()));
}


static <T> Supplier<T> getSupplier(Supplier<T> supplier, boolean cache) {
return cache ? new CachedSupplier<>(supplier) : supplier;
}

static JsonSchema getSchema(JsonSchema parentSchema, ValidationContext validationContext, String refValue,
JsonNodePath evaluationPath) {
JsonSchema refSchema = parentSchema.findSchemaResourceRoot(); // Get the document
Expand Down Expand Up @@ -150,14 +156,17 @@ public void preloadJsonSchema() {
SchemaLocation schemaLocation = jsonSchema.getSchemaLocation();
JsonSchema check = jsonSchema;
boolean circularDependency = false;
int depth = 0;
while (check.getEvaluationParentSchema() != null) {
depth++;
check = check.getEvaluationParentSchema();
if (check.getSchemaLocation().equals(schemaLocation)) {
circularDependency = true;
break;
}
}
if (!circularDependency) {
if (this.validationContext.getConfig().isCacheRefs() && !circularDependency
&& depth < this.validationContext.getConfig().getPreloadJsonSchemaRefMaxNestingDepth()) {
jsonSchema.initializeValidators();
}
}
Expand Down
33 changes: 22 additions & 11 deletions src/main/java/com/networknt/schema/RefValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;
import java.util.Collections;
import java.util.Set;
import java.util.function.Supplier;

/**
* {@link JsonValidator} that resolves $ref.
Expand Down Expand Up @@ -58,10 +60,10 @@ static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext val
String schemaUriFinal = resolve(parentSchema, refUri);
SchemaLocation schemaLocation = SchemaLocation.of(schemaUriFinal);
// This should retrieve schemas regardless of the protocol that is in the uri.
return new JsonSchemaRef(new CachedSupplier<>(() -> {
return new JsonSchemaRef(getSupplier(() -> {
JsonSchema schemaResource = validationContext.getSchemaResources().get(schemaUriFinal);
if (schemaResource == null) {
schemaResource = validationContext.getJsonSchemaFactory().getSchema(schemaLocation, validationContext.getConfig());
schemaResource = validationContext.getJsonSchemaFactory().loadSchema(schemaLocation, validationContext.getConfig());
if (schemaResource != null) {
copySchemaResources(validationContext, schemaResource);
}
Expand Down Expand Up @@ -89,12 +91,12 @@ static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext val
}
return schemaResource.fromRef(parentSchema, evaluationPath);
}
}));
}, validationContext.getConfig().isCacheRefs()));

} else if (SchemaLocation.Fragment.isAnchorFragment(refValue)) {
String absoluteIri = resolve(parentSchema, refValue);
// Schema resource needs to update the parent and evaluation path
return new JsonSchemaRef(new CachedSupplier<>(() -> {
return new JsonSchemaRef(getSupplier(() -> {
JsonSchema schemaResource = validationContext.getSchemaResources().get(absoluteIri);
if (schemaResource == null) {
schemaResource = validationContext.getDynamicAnchors().get(absoluteIri);
Expand All @@ -106,15 +108,21 @@ static JsonSchemaRef getRefSchema(JsonSchema parentSchema, ValidationContext val
return null;
}
return schemaResource.fromRef(parentSchema, evaluationPath);
}));
}, validationContext.getConfig().isCacheRefs()));
}
if (refValue.equals(REF_CURRENT)) {
return new JsonSchemaRef(new CachedSupplier<>(
() -> parentSchema.findSchemaResourceRoot().fromRef(parentSchema, evaluationPath)));
return new JsonSchemaRef(
getSupplier(() -> parentSchema.findSchemaResourceRoot().fromRef(parentSchema, evaluationPath),
validationContext.getConfig().isCacheRefs()));
}
return new JsonSchemaRef(new CachedSupplier<>(
return new JsonSchemaRef(getSupplier(
() -> getJsonSchema(parentSchema, validationContext, refValue, refValueOriginal, evaluationPath)
.fromRef(parentSchema, evaluationPath)));
.fromRef(parentSchema, evaluationPath),
validationContext.getConfig().isCacheRefs()));
}

static <T> Supplier<T> getSupplier(Supplier<T> supplier, boolean cache) {
return cache ? new CachedSupplier<>(supplier) : supplier;
}

private static void copySchemaResources(ValidationContext validationContext, JsonSchema schemaResource) {
Expand Down Expand Up @@ -235,14 +243,17 @@ public void preloadJsonSchema() {
SchemaLocation schemaLocation = jsonSchema.getSchemaLocation();
JsonSchema check = jsonSchema;
boolean circularDependency = false;
int depth = 0;
while (check.getEvaluationParentSchema() != null) {
depth++;
check = check.getEvaluationParentSchema();
if (check.getSchemaLocation().equals(schemaLocation)) {
circularDependency = true;
break;
}
}
if (!circularDependency) {
if (this.validationContext.getConfig().isCacheRefs() && !circularDependency
&& depth < this.validationContext.getConfig().getPreloadJsonSchemaRefMaxNestingDepth()) {
jsonSchema.initializeValidators();
}
}
Expand Down
Loading
Loading