[Ur] Framework Benchmarks Round 9 Preview

[Adam Chlipala]

On 02/18/2014 11:19 PM, escalier at riseup.net wrote:
> Previews for Round 9 have been posted:
> http://www.techempower.com/benchmarks/previews/round9/
>    

There are also results for their older environment with fewer cores, but 
that run wound up with an older Ur/Web version installed that has a 
silly performance problem.  I expect that, when it's rerun with the 
proper version, Ur/Web will place substantially better than in the 
results linked directly above (where the server has 40 hardware threads).

The results on the beefy server are already pretty good, because they 
were run with the right version of Ur/Web.  There may be some debugging 
for concurrency scaling in Ur/Web's future, since apparently another 
entrant is able to achieve 10 times the HTTP throughput on tests with 
keepalive and pipelining.  However, in absolute terms, the results look 
pretty good.  I'll summarize them here:

Basic HTTP with no database access: hundreds of kRPS (thousands of 
requests per second), about 40% of the best entrant's throughput
Pipelined HTTP with no database access: a few more hundreds of kRPS than 
the last one, about 10% of best
Single database query per req.: about 100 kRPS, 40% or 60% of best (for 
two different benchmarks)
20 database queries per req.: about 10 kRPS, 75% of best
20 database queries & updates per req.: about 500 RPS, 10% of best

The update performance pays an especially big penalty for running inside 
bona fide serializable database transactions, which most entrants are 
not doing.  At the same time, 40 SQL calls in one request is way beyond 
usual real-world workloads, I think.

In the other tests that don't run afoul of this problem, Ur/Web never 
places worse than 23rd, out of about 100 entrants.  The best performance 
is on the test ("Fortunes") that does one database query and formats the 
results as HTML, which arguably comes closest to Ur/Web's intended use 
case.  Here Ur/Web places 4th out of 60 entrants for throughput, and it 
has the best latency average with a comfortable 0.1 ms lead. :)

I'll post an updated summary when the final results are out (I'm hoping 
with the right Ur/Web version used on the server with 8 hardware threads).