Testing Async systems with Karate Testing Framework

I have love and hate relationship with asynchronous systems. I love them because the flow is natural and doesn't waste time and resources. I hate them because it's difficult to test and debug. The most common asynchronous system that you would have used is Webhook type. Where you get a callback from an external system when something happens and then you will handle it. That is straight forward. Because you just need to test your part.

Recently I have been working on a system that accepts the requests. Send the ACK message immediately but the real response comes as a callback sometime later. And that response can vary on your request and many other business scenarious. I wanted to setup an automated system to test our system end to end.

First thing I wanted to do, was to mock this external system. Replace it with an almost real webservice built using Karate's mock service.

Karate is the only open-source tool to combine API test-automation, mocks, performance-testing and even UI automation into a single, unified framework. The BDD syntax popularized by Cucumber is language-neutral, and easy for even non-programmers. Powerful JSON & XML assertions are built-in, and you can run tests in parallel for speed.

Test execution and report generation feels like any standard Java project. But there’s also a stand-alone executable for teams not comfortable with Java. You don’t have to compile code. Just write tests in a simple, readable syntax - carefully designed for HTTP, JSON, GraphQL and XML. And you can mix API and UI test-automation within the same test script.

Look at the example code mock-server.feature below

Feature: Service mock server

Background:
* def uuid = function(){ return java.util.UUID.randomUUID() + '' }
* configure responseHeaders = { "Content-Type": "application/json" }
* configure cors = true

Scenario: pathMatches('/service') && methodIs('POST')
    * def response = read("responses/jsons/ack.json")
    * def responseStatus = 200
    * def tc = "test_case_1" 
    * def search = function(){java.lang.Thread.sleep(2000); karate.call("responses/features/service_callback.feature", {"req": request, "tc":"#(tc)"})}
    * eval new java.lang.Thread(search).start()


Scenario: pathMatches('/search') && methodIs('GET')
    * def responseStatus = 404

This is the main mock service provider. When started it serves at /service and POST only. As soon as it receives a request. It responds with an ACK message which are the contents of responses/jsons/ack.json

And then it sleeps for about 2 seconds. And calls another Karate feature responses/features/service_callback.feature. Which uses the parameters in the request to respond appropriately. It basically calls responses/jsons/service.json and passes the values from the request to it.

Feature: service response

Scenario: Case1: good response
    karate.log("CALLBACK case 1 response")
    def req = #(req)
    def tc = #(tc)
    Given url "http://localhost:5000"
    Given path "callback"
    Given request read("responses/jsons/service.json")
    Given header Accept = "application/json"
    Given header Content-Type = "application/json"
    And header "X-API-Key" = api_key
    When method post     
    Then status 200

The service json is parameterized or its a template

{
        "transaction_id": "#(req.transaction_id)",
        "message_id": "#(req.message_id)",
        "timestamp": "2020-08-19T06:41:54.079868Z",
        "version": "#(req.version)",
        "resposne" : "#(req.message)"
}

You can see here responses/jsons/service.json uses the values from the request req so it can send respond appropriately. I have not shown the usage of tc. Idea is to pass the test-case number so it can send appropriate response accordingly.

Starting the mock server is very straight forward.

#start the mock server on port 8080
java -jar ../karate.jar -p 8000 -m mock-server.feature -w

Then I wrote a regular testing feature in karate to test our system. So finally the whole End to End testing system looked like this.

Feature: This is will test the /request call. Call get_request_details to see if you have got response. Delay is 15 seconds

Background:
    * def api_key = "1245"
    * def pause = function(){ java.lang.Thread.sleep(15000); return 1; }
    * def uuid = function(){ return java.util.UUID.randomUUID() + '' }

Scenario: Case: on_search good response
    Given url "http://localhost:5000"
    Given path "request"
    Given header X-API-Key = api_key
    Given header Accept = "application/json"
    Given header Content-Type = "application/json"

    Given def transaction_id = uuid()    
    Given def message_id = uuid()
    Given def version = uuid()
    Given def message = "Coming to you Async"
    Given request read("../jsons/request.json")
    When method post     
    Then status 200

    #wait 
    Then def x = pause()

    Given url "http://localhost:5000"
    Given path "get_request_details"
    Given header X-API-Key = api_key
    Given request read("../jsons/get_request_details.json")
    Given header Accept = "application/json"
    Given header Content-Type = "application/json"
    When method post     
    Then status 200
    Then assert response.transaction_id ==  transaction_id
    Then assert response.message ==  message

Here the calback is through webhook. But it could have been a Queue based too. You get the idea. I have been enjoying E2E testing using Karate framework. It has a lot of hidden gems. I will blog more about them as we test this system. Let me know what do you use test such systems.