https://blogs.atlassian.com/2013/07/http-client-performance-io/
This is an interesting post. Unfortunately, the comparison is based on file downloading, which is not exactly what most web services are built for. So I decided to run my own analysis based on typical client interactions with RESTful services, where the client sends a request and the server replies with a reasonable size of payload.
Clients compared
- Netty 4.0.8 (no connection pool)
- Apache HttpAsyncClient 4.0-beta4 (with connecton pool enabled)
- Apache HttpClient 4.3 (with connection pool enabled)
Test Setup
On the server side, a cluster of servers are set up in Amazon EC2 cloud. They all serve relatively static content where the only variation is size of the payload which is controllable from client's query parameter. Each server runs a Tomcat application with 300 threads.The client runs on my laptop, which is a Macbook Pro (2.4Ghz Intel Core i7) running JVM 1.6.0_51 with 2G bytes heap size. Different request batch sizes are tested. Each batch sends all requests in a loop and is run several times without JVM shutdown. The test result for one batch size is the average of all runs. The data from first run of a batch is ignored as it usually includes JVM, thread pool and connections warm up.
Metrics
Three metrics are collected or calculated from the test. - Time needed to send all requests. For thread pool based blocking client, a request is deemed as sent when it is dequeued from thread pool and handed off to the client. For Netty, a request is deemed as sent when the ChannelFuture is obtained. For Apache HttpAsyncClient, a request is deemed as sent when the Future<HttpResposne> object is obtained.
- Average latency. For NIO clients, a timer is started when a request is handed off to the client. For HttpClient, the timer is started when a request is enqueued to the thread pool. The timer is stopped when the full HTTP response is received.This time is accumulated for all requests and used to calculate the average.
- Throughput. This is simply the size of batch divided by the time to sent.
Results
Average latency (ms)
| Batch size | 100 | 500 | 1000 | 5000 |
|---|---|---|---|---|
| Apache HttpAsyncClient | 318 | 772 | 1090 | 5096 |
| Netty | 408 | 1010 | 1629 | 5433 |
| HttpClient with 50 threads | 227 | 733 | 1342 | 6792 |
| HttpClient with 200 threads | 183 | 477 | 850 | 3509 |
| HttpClient with 300 threads | 293 | 502 | 1010 | 4125 |
It is interesting to observe that the best latency is achieved by a blocking I/O client with reasonable size of thread pool. This is perhaps due to the fact that each socket gets a dedicated thread in blocking I/O while in NIO, only a handful of selectors need to poll hundreds of even thousands or sockets. On the other hand, as the size of thread pool increases, the latency gets worse.
The HttpAsyncClient has better latency compared with Netty because it has built-in connection pool support.
Throughput (requests per second)
| Batch size | 100 | 500 | 1000 | 5000 |
|---|---|---|---|---|
| Apache HttpAsyncClient | 2797 | 4629 | 5031 | 19569 |
| Netty | 3278 | 6872 | 10126 | 10460 |
| HttpClient with 50 threads | 353 | 404 | 401 | 370 |
| HttpClient with 200 threads | 5194 | 705 | 720 | 741 |
| HttpClient with 300 threads | 5882 | 715 | 603 | 642 |
Here, the NIO clients have a clear edge over blocking client. They can achieve 10 times better throughput than the blocking HttpClient.
Conclusion
I believe the performance advantage of NIO clients is on the throughput side. If your service requires a client throughput of over 1000 rps, consider using NIO client to avoid bottleneck. On the other hand, if the service only requires a couple hundred rps, using blocking HttpClient with reasonable size of thread pool is manageable and will get you better latency.HttpAsyncClient enjoys the benefit of a built-in connection pool. This proves to be crucial when the clients have to deal with thousands of concurrent requests. However, note that this is just a comparison of HTTP clients with relatively simple use cases, and Netty clearly have a wider support of protocols.
Further discussion and correction
10/29/2013After the publishing blog, I had a discussion with Apache HttpClient development lead Oleg Kalnichevski, who pointed to me that the throughput comparison may be flawed in that the sending time for the NIO clients is considerably shorter as it only includes the time to put requests into the processing queue. There is a separate comparison conducted by the HttpClient team that shows the HttpClient outperforms NIO clients:
http://wiki.apache.org/
Stayed tuned for results of further testing from my side.
