SPIKE [FAT-1652]: mod-inn-reach: Study existing solutions for mocking (mock servers) and choose the appropriate one

FAT-1652 - Getting issue details... STATUS

Capabilities of interest

deployment options (docker, command line etc)
mock definition/configuration
- automatic mocking from provided API spec
request verification
stateful behaviour
supporting of dynamic/conditional content
simulating faults

WireMock

WireMock is an HTTP mock server. At its core it is web server that can be primed to serve canned responses to particular requests (stubbing) and that captures incoming requests so that they can be checked later (verification). All of WireMock’s features are accessible via its REST (JSON) interface and its Java API.

Deployment

a) Standalone process

The WireMock server can be run in its own process. Once the standalone JAR downloaded it can be run simply by doing this:

$ java -jar wiremock-jre8-standalone-2.33.1.jar

Various command line options available, can be listed with --help command. Some useful options:

--port: Set the HTTP port number e.g. --port 9999. Use --port 0 to dynamically determine a port.

--verbose: Turn on verbose logging to stdout

--root-dir: Sets the root directory, under which mappings and __files reside. This defaults to the current directory.

b) Docker container

From version 2.31.0 WireMock has an official Docker image.

Start a single WireWock container with default configuration:

docker run -it --rm -p 8080:8080 --name wiremock wiremock/wiremock:2.33.1

The Docker image supports exactly the same set of command line arguments as the standalone version. These can be passed to the container by appending them to the end of the command e.g.:

docker run -it --rm -p 8443:8443 --name wiremock wiremock/wiremock:2.33.1 --https-port 8443 --verbose

Mock definition/configuration

a) Configuring via JSON over HTTP

A stub can be created by posting mapping to WireMock’s HTTP API:

$ curl -X POST \
--data '{ "request": { "url": "/get/this", "method": "GET" }, "response": { "status": 200, "body": "Here it is!\n" }}' \
http://localhost:8080/__admin/mappings/new

b) JSON file configuration

the JSON API can also be used via files. When the WireMock server starts it creates two directories under the current one: mappings and __files. JSON files containing multiple stub mappings can also be used.

c) Packaging the stubs into a standalone JAR

If you want to package your stubs into the standalone JAR, so you can distribute an executable JAR with all the stubs intact, you can do this using the --load-resources-from-classpath option.

You could then run the packaged JAR as:

java -jar custom-wiremock.jar --load-resources-from-classpath 'wiremock-stuff'

Which will load your files and mappings from the packaged JAR.

Request verification

The WireMock server records all requests it receives in memory (at least until it is reset). This makes it possible to verify that a request matching a specific pattern was received, and also to fetch the requests’ details.

Verifying and querying requests relies on the request journal, which is an in-memory log of received requests.

Like stubbing, verification also uses WireMock’s Request Matching system to filter and query requests.

To verify that a request matching some criteria was received by WireMock at least once:

Java

verify(postRequestedFor(urlEqualTo("/verify/this"))
        .withHeader("Content-Type", equalTo("text/xml")));

The criteria part in the parameter to postRequestedFor() uses the same builder as for stubbing, so all of the same predicates are available. See Stubbing for more details.

To check for a precise number of requests matching the criteria, use this form:

Java

verify(3, postRequestedFor(urlEqualTo("/three/times")));

Stateful behaviour

WireMock supports state via the notion of scenarios. A scenario is essentially a state machine whose states can be arbitrarily assigned. It starting state is always Scenario.STARTED. Stub mappings can be configured to match on scenario state, such that stub A can be returned initially, then stub B once the next scenario state has been triggered.

Java

@Test
public void toDoListScenario() {
    stubFor(get(urlEqualTo("/todo/items")).inScenario("To do list")
            .whenScenarioStateIs(STARTED)
            .willReturn(aResponse()
                    .withBody("<items>" +
                            "   <item>Buy milk</item>" +
                            "</items>")));

    stubFor(post(urlEqualTo("/todo/items")).inScenario("To do list")
            .whenScenarioStateIs(STARTED)
            .withRequestBody(containing("Cancel newspaper subscription"))
            .willReturn(aResponse().withStatus(201))
            .willSetStateTo("Cancel newspaper item added"));

    stubFor(get(urlEqualTo("/todo/items")).inScenario("To do list")
            .whenScenarioStateIs("Cancel newspaper item added")
            .willReturn(aResponse()
                    .withBody("<items>" +
                            "   <item>Buy milk</item>" +
                            "   <item>Cancel newspaper subscription</item>" +
                            "</items>")));

    WireMockResponse response = testClient.get("/todo/items");
    assertThat(response.content(), containsString("Buy milk"));
    assertThat(response.content(), not(containsString("Cancel newspaper subscription")));

    response = testClient.postWithBody("/todo/items", "Cancel newspaper subscription", "text/plain", "UTF-8");
    assertThat(response.statusCode(), is(201));

    response = testClient.get("/todo/items");
    assertThat(response.content(), containsString("Buy milk"));
    assertThat(response.content(), containsString("Cancel newspaper subscription"));
}

Dynamic/conditional content

a) Transforming Responses

Via WireMock’s extension mechanism it is possible to dynamically modify responses, allowing header re-writing and templated responses amongst other things.

There are two ways to dynamically transform output from WireMock. WireMock stub mappings consist in part of a ResponseDefinition. This is essentially a description that WireMock (sometimes) combines with other information when producing the final response.

ResponseDefinition objects are “rendered” by WireMock into a Response, and it is possible to interject either before or after this process when writing an extension, meaning you can either transform the ResponseDefinition prior to rendering, or the rendered Response afterwards.

A response transformer extension takes a Response in its transform method’s parameter list and returns a Response:

Java

public static class StubResponseTransformerWithParams extends ResponseTransformer {

        @Override
        public Response transform(Request request, Response response, FileSource files, Parameters parameters) {
            return Response.Builder.like(response)
                    .but().body(parameters.getString("name") + ", "
                            + parameters.getInt("number") + ", "
                            + parameters.getBoolean("flag"))
                    .build();
        }

        @Override
        public String getName() {
            return "stub-transformer-with-params";
        }
}

b) Response templating

Response headers and bodies, as well as proxy URLs, can optionally be rendered using Handlebars templates. This enables attributes of the request to be used in generating the response e.g. to pass the value of a request ID header as a response header or render an identifier from part of the URL in the response body.

When starting WireMock programmatically, response templating can be enabled by adding ResponseTemplateTransformer as an extension e.g.

Java

@Rule
public WireMockRule wm = new WireMockRule(options()
    .extensions(new ResponseTemplateTransformer(false))
);

The boolean constructor parameter indicates whether the extension should be applied globally. If true, all stub mapping responses will be rendered as templates prior to being served.

Command line parameters can be used to enable templating when running WireMock standalone:

--global-response-templating: Render all response definitions using Handlebars templates.

--local-response-templating: Enable rendering of response definitions using Handlebars templates for specific stub mappings.

Parameter values can be passed to the transformer as shown below (or dynamically added to the parameters map programmatically in custom transformers):

Java

wm.stubFor(get(urlPathEqualTo("/templated"))
  .willReturn(aResponse()
      .withBody("{{request.path.[0]}}")
      .withTransformers("response-template")
      .withTransformerParameter("MyCustomParameter", "Parameter Value")));

JSON

{
    "request": {
        "urlPath": "/templated"
    },
    "response": {
        "body": "{{request.path.[0]}}",
        "transformers": ["response-template"],
        "transformerParameters": {
            "MyCustomParameter": "Parameter Value"
        }
    }
}

These parameters can be referenced in template body content using the parameters. prefix:

<h1>The MyCustomParameter value is {{parameters.MyCustomParameter}}</h1>

Simulating faults

In addition to being able to send back any HTTP response code indicating an error, WireMock is able to generate a few other types of problem.

a) Delayed responses

Per-stub fixed delays – A stub response can have a fixed delay attached to it, such that the response will not be returned until after the specified number of milliseconds
Global fixed stub delays – A fixed delay can be added to all stubs
Per-stub random delays – A delay can be sampled from a random distribution. This allows simulation of more specific downstream latencies, such as a long tail.
Global random stub delays
Chunked Dribble Delay – In addition to fixed and random delays, response can be dribble back in chunks. This is useful for simulating a slow network and testing deterministic timeouts.

b) Bad responses

It is also possible to create several kinds of corrupted responses:

Java

stubFor(get(urlEqualTo("/fault"))
        .willReturn(aResponse().withFault(Fault.MALFORMED_RESPONSE_CHUNK)));

The Fault enum has the following options: EMPTY_RESPONSE, MALFORMED_RESPONSE_CHUNK, RANDOM_DATA_THEN_CLOSE,CONNECTION_RESET_BY_PEER

Karate Netty

Deployment

a) Standalone process

All of Karate (core API testing, parallel-runner / HTML reports, the debugger-UI, mocks and web / UI automation) is available as a single, executable JAR file. Look for the latest release on GitHub and scroll down to find the "Assets". And look for the file with the name: karate-<version>.jar

To start a mock server, the 2 mandatory arguments are the path of the feature file 'mocks' -m and the port -p

java -jar karate.jar -m my-mock.feature -m my-2nd-mock.feature -p 8080

To directly start a mock from within a Karate test script use the karate.start() API. The argument can be a string or JSON. If a string, it is processed as the path to the mock feature file, and behaves like the read() function.

So starting a mock from a Karate test is simple. This example also shows how conditional logic can be used effectively.

Background:
  * def port = karate.env == 'mock' ? karate.start('cats-mock.feature').port : 8080
  * url 'http://localhost:' + port + '/cats'

Mock definition/configuration

Writing a mock can get complicated for real-life API interactions, and most other frameworks attempt to solve this using declarative approaches, such as expecting you to create a large, complicated JSON to model all requests and responses. Karate's approach combines the best of both the worlds of declarative and imperative programming. Combined with the capability to maintain state in the form of JSON objects in memory, and Karate's native support for Json-Path, XML and embedded expressions

The Karate 'server' life-cycle is simple and has only 2 phases - the Background and Scenario.

Background is executed on start-up. You can read files and set-up common functions and 'global' state here. Note that unlike the life-cycle of 'normal' Karate, the Background is not executed before each Scenario.

Here's an example of setting up a function to generate primary keys which can be invoked like this: uuid()

Karate

Feature: stateful mock server

Background:
  * configure cors = true
  * def uuid = function(){ return java.util.UUID.randomUUID() + '' }
  * def cats = {}

Scenario: pathMatches('/cats') && methodIs('post')
    * def cat = request
    * def id = uuid()
    * cat.id = id
    * cats[id] = cat
    * def response = cat

Scenario: pathMatches('/cats')
    * def response = $cats.*

Scenario: pathMatches('/cats/{id}')
    * def response = cats[pathParams.id]

Scenario:
    def responseStatus = 404

A server-side Feature file can have multiple Scenario sections in it. Each Scenario is expected to have a JavaScript expression as the content of the Scenario description which we will refer to as the "request matcher".

On each incoming HTTP request, the Scenario expressions are evaluated in order, starting from the first one within the Feature. If the expression evaluates to true, the body of the Scenario is evaluated and the HTTP response is returned.

Request verification

Validate payloads if needed, using a simpler alternative to JSON schema:

Detailed description of schema validation is available at Schema Validation section of Karate framework.

Stateful behaviour

In-memory JSON and JsonPath solves for 'state' and filtering if needed:

Link to working example in git: demo-mock.feature

Dynamic/conditional content

Shaping the HTTP response is very easy - you just set a bunch of variables: response,responseStatus, esponseHeaders

With embedded-expressions, you can create dynamic responses with a minimum of effort:

Karate

Scenario: pathMatches('/v1/cats')
    * def responseStatus = 201
    * def response = { id: '#(uuid())', name: 'Billie' }

Karate easily reads JSON or XML from files. So when you have large complex responses, you can do this:

Karate

    * def response = read('get-cats-response.json')

Note that embedded expressions work even for content loaded using read()

Because of Karate's Java interop capabilities there is no limit to what can be done.

Simulating faults

Delayed responses can be defined by setting responseDelay variable which accepts a value in milliseconds:

Karate

Scenario: pathMatches('/v1/test')
    * def responseDelay = 4000

Summary

	WireMock	Comments
Deployment / Standalone process
Deployment / Docker		Кarate doesn't have official docker image, but an image can be easily created since there is a possibility to start standalone Java process
Mock definition
Request verification		The verification in WireMock can be done with standard matchers programmatically. Karate option is more intuitive as it allows to define something similar to JSON schemas.
Stateful behaviour		WireMock is able to simulate stateful behaviour using so called scenarios but Karate makes it more natural with an ability to support real variables with saved states between the request calls
Dynamic payload of responses
Simulating faults		Karate has limited support for delayed responses but because of simplified integration with Java/JS any sort of customization can be easily added to response processing
	– supported out-of-box – supported with addition effort

As the above tables states, both WireMock and Karate Netty provide the required functionality. But Karate Netty looks more suitable because the integration tests in Folio are already written with Karate and in general it has more advanced options for customization with direct support of Java/JS code inside Karate features (test scenarios).

So the winner is Karate Netty !

References

WireMock documentation: https://wiremock.org/docs/
Prism documentation: https://meta.stoplight.io/docs/prism/ZG9jOjYx-overview
Karate Netty documentation: https://github.com/karatelabs/karate/tree/master/karate-netty
BenTen project is a great example of the usage of Karate test-doubles: https://github.com/intuit/benten/blob/master/benten-mock/src/main/resources/benten-mock.feature
Comparison of API simulation tools: https://en.wikipedia.org/wiki/Comparison_of_API_simulation_tools