A few months ago, when I ported the QUIC patches from nginx mainline to APISIX and tried to test it, I found that test::nginx didn’t work very well. It uses the wrong listen directive parameter “http3” instead of “quic” (probably due to version differences).

So I was wondering if I could design a simple testing framework because the latest curl already fully supports HTTP/3.

Many testing frameworks prefer DSLs (Domain Specific Languages) such as test::nginx and Hurl because it is more user-friendly. And it looks more data-driven. However, this is not good for developers looking to extend its functionality as they would need to learn the language behind it and take the time to understand the core and extension APIs by reading the code base of this testing framework. In other words, DSL and the language behind it create a knowledge gap for developers because it is not intuitive and cannot map the functions corresponding to the DSL to the language itself. Especially when the framework itself lacks capabilities and needs to be expanded, this gap will bring significant learning costs.

Take test::nignx as an example, which is implemented in Perl. When I need to verify the response body and decrypt it by pre-shared key, The built-in function doesn’t help as it only supports regular expression matching. So I need to write a bunch of Perl code to do this. However, I’m not familiar with Perl, so this isn’t an easy task. As we all know, Perl is not a mainstream language and has a steep learning curve. At that time, I was thinking, how great it would be if I could write code in Python.

The expressive capabilities of test::nginx are limited, even though it already supports sending requests using curl, because it is not a scripting system (Of course, you can extend it using Perl modules, but due to the limitations of the framework design, this is a difficult task for most people).

1. It assumes your input is a text string, you cannot use other formats such as protobuf, and you cannot compress or encrypt text.

2. The only output post-processing supported is regular matching, so you cannot parse and verify other forms of content, such as decryption.

3. You cannot describe complex test procedures, such as sending three requests to trigger the request quota and verify the rate-limiting results.

4. You cannot make other types of requests such as GRPC, or SOAP.

In my practice, when I have such needs that exceed test::nginx’s capabilities, I have to delegate them to the lua part, which is very troublesome.

For example, when I need to parse and verify the correctness of the CAS Auth plugin, it involves a lot of client-side CAS protocol details, so I have to implement the entire logic in the lua part, which increases the difficulty of maintaining this mix. As we all know, neither Perl nor Lua are popular languages. ;-(

You can see the test case below involves additional lua lib development to test CAS. And, to accommodate test::nginx’s lack of response parsing, you have to do the conversion in the Lua environment. Quite annoying, at least to me, because I’ve written a lot of test cases like this over the years.

=== TEST 2: login and logout ok
--- config
    location /t {
        content_by_lua_block {
            local http = require "resty.http"
            local httpc = http.new()
            -- Test-specific CAS lua lib
            local kc = require "lib.keycloak_cas"

            local path = "/uri"
            local uri = "http://127.0.0.1:" .. ngx.var.server_port
            local username = "test"
            local password = "test"

            local res, err, cas_cookie, keycloak_cookie = kc.login_keycloak(uri .. path, username, password)
            if err or res.headers['Location'] ~= path then
                ngx.log(ngx.ERR, err)
                ngx.exit(500)
            end
            res, err = httpc:request_uri(uri .. res.headers['Location'], {
                method = "GET",
                headers = {
                    ["Cookie"] = cas_cookie
                }
            })
            assert(res.status == 200)
            ngx.say(res.body)

            res, err = kc.logout_keycloak(uri .. "/logout", cas_cookie, keycloak_cookie)
            assert(res.status == 200)
        }
    }
--- response_body_like
uri: /uri
cookie: .*
host: 127.0.0.1:1984
user-agent: .*
x-real-ip: 127.0.0.1

Similarly, Hurl also uses a DSL to describe tests. Its backend language is Rust. So in terms of scalability, it seems to be worse than test::nginx because it needs to compile the code. However, Rust’s ecosystem is much better than Perl’s. So you can write code easily. But anyway, you need to write the code in a different language than the DSL.

Is there a simple testing framework that can satisfy the following requirements?

1. No DSL, just simple shell scripts

2. Easy to extend and programming language agnostic

3. Can be used to initialize and test any server, not limited to nginx, such as envoy

Please visit my blog for more details:

http://luajit.io/posts/a-simple-http3-testing-framework-based-on-bash-and-curl/