OOM occurred when validating the response.

[shuqin]

/What's-different-in-2.0#error-handling | com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at io.reactivex.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:367) ~[rxjava-2.2.21.jar:?]
at io.reactivex.internal.observers.ConsumerSingleObserver.onSuccess(ConsumerSingleObserver.java:65) ~[rxjava-2.2.21.jar:?]
at io.vertx.reactivex.impl.AsyncResultSingle.lambda$subscribeActual$0(AsyncResultSingle.java:49) ~[vertx-rx-java2-gen-4.3.1.jar:4.3.1]
at io.vertx.reactivex.ext.web.client.HttpRequest$10.handle(HttpRequest.java:770) ~[vertx-rx-java2-4.3.1.jar:4.3.1]
at io.vertx.reactivex.ext.web.client.HttpRequest$10.handle(HttpRequest.java:767) ~[vertx-rx-java2-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:390) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:377) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:355) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:322) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:284) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.ext.web.client.impl.HttpContext.lambda$null$7(HttpContext.java:500) ~[vertx-web-client-4.3.1.jar:4.3.1]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:101) ~[vertx-core-4.3.1.jar:4.3.1]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:64) ~[vertx-core-4.3.1.jar:4.3.1]
at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:44) ~[vertx-core-4.3.1.jar:4.3.1]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.77.Final.jar:4.1.77.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[guava-30.1-jre.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.resolveJsonSchema(SchemaValidator.java:196) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.validate(SchemaValidator.java:158) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.interaction.response.ResponseValidator.validateResponseBody(ResponseValidator.java:158) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.interaction.response.ResponseValidator.validateResponse(ResponseValidator.java:95) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.lambda$validateResponse$4(OpenApiInteractionValidator.java:234) ~[swagger-request-validator-core-2.35.1.jar:?]
at java.util.function.Function.lambda$andThen$1(Function.java:88) ~[?:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.validateOnApiOperation(OpenApiInteractionValidator.java:265) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.validateResponse(OpenApiInteractionValidator.java:231) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.swagger.SwaggerUtils.validateResponse(SwaggerUtils.java:190) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.requestvalidation.RestResponseValidator.validate(RestResponseValidator.java:275) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.requestvalidation.RestResponseValidator.validateResponse(RestResponseValidator.java:181) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.requestvalidation.ExtendedRequestResponseValidator.validateResponse(ExtendedRequestResponseValidator.java:46) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.bedrock.external.rest.PostgRestHttpModule.lambda$handlePostgrestResponse$35(PostgRestHttpModule.java:589) ~[core-1.0-SNAPSHOT.jar:?]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at com.emc.bedrock.external.rest.PostgRestHttpModule.handlePostgrestResponse(PostgRestHttpModule.java:589) ~[core-1.0-SNAPSHOT.jar:?]
at com.emc.bedrock.external.rest.PostgRestHttpModule.lambda$getRouteHandler$32(PostgRestHttpModule.java:515) ~[core-1.0-SNAPSHOT.jar:?]
at io.reactivex.internal.observers.ConsumerSingleObserver.onSuccess(ConsumerSingleObserver.java:62) ~[rxjava-2.2.21.jar:?]
... 20 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.databind.node.ObjectNode.deepCopy(ObjectNode.java:56) ~[jackson-databind-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.databind.node.ObjectNode.deepCopy(ObjectNode.java:21) ~[jackson-databind-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.databind.node.ObjectNode.deepCopy(ObjectNode.java:59) ~[jackson-databind-2.14.2.jar:2.14.2]
at com.fasterxml.jackson.databind.node.ObjectNode.deepCopy(ObjectNode.java:21) ~[jackson-databind-2.14.2.jar:2.14.2]
at com.atlassian.oai.validator.schema.transform.SchemaDefinitionsInjectionTransformer.apply(SchemaDefinitionsInjectionTransformer.java:23) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.lambda$readAndTransformSchemaObject$6(SchemaValidator.java:219) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator$$Lambda$13223/0x0000000102de5c40.accept(Unknown Source) ~[?:?]
at java.util.Arrays$ArrayList.forEach(Arrays.java:4390) ~[?:?]
at com.atlassian.oai.validator.schema.SchemaValidator.readAndTransformSchemaObject(SchemaValidator.java:219) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.access$300(SchemaValidator.java:52) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator$1.load(SchemaValidator.java:111) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator$1.load(SchemaValidator.java:108) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[guava-30.1-jre.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[guava-30.1-jre.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.resolveJsonSchema(SchemaValidator.java:196) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.schema.SchemaValidator.validate(SchemaValidator.java:158) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.interaction.response.ResponseValidator.validateResponseBody(ResponseValidator.java:158) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.interaction.response.ResponseValidator.validateResponse(ResponseValidator.java:95) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.lambda$validateResponse$4(OpenApiInteractionValidator.java:234) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator$$Lambda$13213/0x0000000102de3440.apply(Unknown Source) ~[?:?]
at java.util.function.Function.lambda$andThen$1(Function.java:88) ~[?:?]
at java.util.function.Function$$Lambda$244/0x000000010038e440.apply(Unknown Source) ~[?:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.validateOnApiOperation(OpenApiInteractionValidator.java:265) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.atlassian.oai.validator.OpenApiInteractionValidator.validateResponse(OpenApiInteractionValidator.java:231) ~[swagger-request-validator-core-2.35.1.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.swagger.SwaggerUtils.validateResponse(SwaggerUtils.java:190) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.requestvalidation.RestResponseValidator.validate(RestResponseValidator.java:275) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]
at com.emc.cyclone.contexts.infrastructure.rest.requestvalidation.RestResponseValidator.validateResponse(RestResponseValidator.java:181) ~[cyclone-controlpath-1.0-SNAPSHOT.jar:?]

‌

[shuqin]

We are trying to upgrade swagger-request-validator from 1.5.0 to latest version 2.35.1. After running some test, got OOM.

Here are the JVM configuration:

-Xms512m
-Xmx750m

The openapi definition file has more than 4.1M, it’s a large file.

=============================================================

After analyzed the heap dump, found the heap memory was occupied by the jsonSchemaCache in SchemaValidator object and the cache in CachingProcessor object.

This two caches store many copies from the openapi definition schema, each one has more than 4.7M.

The maximum size of the jsonSchemaCache in SchemaValidator object is 100 and it seems it can’t be configured.

The maximum size of the cache in CachingProcessor object is 512 and it can be configured, but we just used the default value 512.

==============================================================

From above analysis, the 750m heap size is obviously not enough.

But I have some questions, why do we need to do deepCopy when trying to get openapi schema for every validation request.

private JsonNode readAndTransformSchemaObject(final Schema schema, final boolean forRequest,
                                              final boolean forResponse, final JsonNode definitions) {
    final ObjectNode schemaObject = Json.mapper().convertValue(schema, ObjectNode.class);
    final SchemaTransformationContext transformationContext = SchemaTransformationContext.create()
            .forRequest(forRequest)
            .forResponse(forResponse)
            .withAdditionalPropertiesValidation(additionalPropertiesValidationEnabled())
            // Use a copy of the definitions. The JsonSchema validation process might change them
            // in its validation process. On concurrent validations it might even lead to
            // ConcurrentModificationException.
            .withDefinitions(definitions.deepCopy())
            .build();

    transformers.forEach(t -> t.apply(schemaObject, transformationContext));

    checkForKnownGotchasAndLogMessage(schemaObject);
    return schemaObject;
}

‌

And in SchemaDefinitionsInjectionTransformer it did deepCopy again

@Override
public void apply(final JsonNode schemaObject, final SchemaTransformationContext context) {
    if (!(schemaObject instanceof ObjectNode)) {
        return;
    }

    ((ObjectNode) schemaObject).putObject(COMPONENTS_FIELD).set(SCHEMAS_FIELD, context.getSchemaDefinitions().deepCopy());
}

Can we avoid deep copying for every validation request? Can we just use a single copy to validate all the validation requests?

‌

[shuqin]

Can anyone help check this issue?

[shuqin]

Our upgrading process is blocked by this issue. Please help check it.

[James Navin]

Available in 2.38.0

Fix is to support configuring the allowed schema cache size if needed.