With the recent announcement of Facebook that their HHVM is now more and more compatible with most of the popular framework, I was intrigued to finally try it out. We're currently building a Symfony2 based application, which has pretty high performance requirements (but we can mostly achieve them with varnish), so I went and did some performance tests on that real-life app.
The application basically gets data from an ElasticSearch server and then transforms them to JSON. Nothing too fancy from the output perspective, but we use many features of Symfony2 and what was clear from the beginning is that the more data it gets from ElasticSearch the slower the requests get. Meaning that we loose a lot of time in the serialization part (we use JMSSerializer for that).
In short, the numbers were amazing. I also compared PHP 5.3 with APC against 5.5 with opcache, that alone gave some pretty decent improvements.
The setup I did the tests on was a QuadCore Intel i7-4770 CPU @ 3.40GHz server over at Hetzner with SSD disks and more than enough RAM. The actual application ran in a VirtualBox container with 4 CPUs and 2 GB of RAM. I used Apache Bench for the tests with the HHVM vagrant VM running Ubuntu 12.04, but I installed the pre-built HHVM from their repo in the end.
I made 3 different requests with different amount of data the script had to get from ElasticSearch. One was approx. 7kb in response, the other 80kb and the last 220kb. I didn't it with different concurrency settings (1-50) and ran Apache Bench with 500 requests per run. The figures below are averages over those 500 requests.
I didn't do performance measurements for PHP 5.3 with more than 6 concurrent requests, because I added 10,20,50 later and didn't repeat for 5.3, but the trend is clear nevertheless.
As you can see below, in general the longer the PHP request ran, the more we gained from HHVM, up to and sometimes more than 300% against PHP 5.3, and about 200% against PHP 5.5. Only switching from PHP 5.3 to PHP 5.5 can save you up to twice the time, as well. So it is very much worth upgrading to 5.5. I find both numbers pretty amazing.
If you really need every millisecond performance, considering HHVM is worth some investigation. These numbers are very promising. And I don't had to change one line of code in our application. As we don't use it yet in production, I can't say anything about stability. Obviously tools like New Relic are, at least for now, also out of the question with HHVM.
Requests per second, small response
The higher the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 52 | 84 | 81 | 157 % | 96 % | 163 % |
| 2 | 120 | 163 | 207 | 173 % | 127 % | 136 % |
| 3 | 154 | 219 | 285 | 185 % | 130 % | 142 % |
| 4 | 172 | 247 | 317 | 185 % | 129 % | 144 % |
| 5 | 186 | 256 | 342 | 184 % | 134 % | 138 % |
| 6 | 203 | 259 | 340 | 168 % | 132 % | 128 % |
| 10 | 253 | 443 | 175 % | |||
| 20 | 247 | 441 | 179 % | |||
| 50 | 258 | 380 | 147 % |
Requests per second, middle response
The higher the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 10 | 14 | 26 | 269 % | 184 % | 146 % |
| 2 | 19 | 29 | 56 | 295 % | 193 % | 153 % |
| 3 | 28 | 42 | 79 | 286 % | 186 % | 154 % |
| 4 | 34 | 51 | 92 | 271 % | 179 % | 151 % |
| 5 | 35 | 53 | 98 | 279 % | 186 % | 150 % |
| 6 | 35 | 50 | 101 | 285 % | 203 % | 140 % |
| 10 | 56 | 113 | 202 % | |||
| 20 | 53 | 105 | 197 % | |||
| 50 | 56 | 108 | 193 % |
Requests per second, large response
The higher the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 4 | 6 | 12 | 283 % | 179 % | 158 % |
| 2 | 8 | 12 | 25 | 333 % | 207 % | 161 % |
| 3 | 11 | 18 | 37 | 324 % | 208 % | 155 % |
| 4 | 15 | 22 | 46 | 307 % | 207 % | 148 % |
| 5 | 15 | 23 | 48 | 318 % | 208 % | 153 % |
| 6 | 15 | 24 | 51 | 339 % | 215 % | 158 % |
| 10 | 24 | 51 | 218 % | |||
| 20 | 24 | 49 | 208 % | |||
| 50 | 24 | 50 | 212 % |
Median response time in ms, short response
The lower the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 15 | 11 | 9 | 60 % | 82 % | 73 % |
| 2 | 16 | 12 | 10 | 59 % | 79 % | 75 % |
| 3 | 19 | 13 | 11 | 55 % | 81 % | 68 % |
| 4 | 23 | 16 | 13 | 54 % | 78 % | 70 % |
| 5 | 26 | 18 | 14 | 54 % | 78 % | 69 % |
| 6 | 29 | 22 | 17 | 57 % | 75 % | 76 % |
| 10 | 38 | 21 | 55 % | |||
| 20 | 78 | 43 | 55 % | |||
| 50 | 191 | 105 | 55 % |
Median response time in ms, middle response
The lower the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 98 | 61 | 32 | 33 % | 53 % | 62 % |
| 2 | 103 | 66 | 34 | 33 % | 52 % | 64 % |
| 3 | 107 | 66 | 36 | 34 % | 55 % | 62 % |
| 4 | 112 | 71 | 40 | 36 % | 56 % | 63 % |
| 5 | 138 | 89 | 49 | 36 % | 55 % | 64 % |
| 6 | 171 | 112 | 57 | 33 % | 51 % | 65 % |
| 10 | 174 | 83 | 48 % | |||
| 20 | 366 | 185 | 51 % | |||
| 50 | 872 | 451 | 52 % |
Median response time in ms, large response
The lower the better
| Concurrency | PHP 5.3 | PHP 5.5 | HHVM | HHVM vs 5.3 | HHVM vs 5.5 | 5.5 vs 5.3 |
|---|---|---|---|---|---|---|
| 1 | 236 | 149 | 71 | 30 % | 48 % | 63 % |
| 2 | 249 | 158 | 75 | 30 % | 47 % | 63 % |
| 3 | 253 | 161 | 77 | 30 % | 48 % | 64 % |
| 4 | 261 | 168 | 81 | 31 % | 48 % | 64 % |
| 5 | 327 | 211 | 98 | 30 % | 46 % | 65 % |
| 6 | 394 | 246 | 115 | 29 % | 47 % | 62 % |
| 10 | 419 | 189 | 45 % | |||
| 20 | 843 | 394 | 47 % | |||
| 50 | 2069 | 968 | 47 % |
I will have a look at memory consumption in a follow up post.