[Ur] TechEmpower Benchmarks

[Adam Chlipala]

On 12/11/2013 10:47 AM, escalier at riseup.net wrote:
> The final preview results have been posted:
>
> http://www.techempower.com/benchmarks/previews/round8/
>
> These is pretty much the official result set except that some anomalous
> results will be filtered out before the official posting.
>    

Thanks again for your help producing the benchmark code!  I'm glad to 
see no errors for _most_ of the benchmarks.  Two benchmarks (which both 
use SQL) have some errors, and I'm not sure if they'd all be fixed by 
the Postgres configuration change that you requested for the next round 
(which I was sad to see not included yet for this round).

I'm very curious about which niggling details are responsible for the 
performance gap with most of the serious frameworks.

> Obvious improvements that occur to me for next round:
>    - Create a quiet or verbose flag so that the all the stdout logging is
> avoidable (e.g. "Handling connection with thread #1. Serving URI
> /json....") The test runners require stdout to be piped to a file so it
> can't just be `/dev/null`ed.
>    

I've added a '-q' command-line argument for "quiet," to turn off routine 
logging.  Do you think this will actually help significantly?  I see 
less than a 1% difference in throughput comparing no logging, logging to 
/dev/null, and logging to a disk file.

>    - Create an alternate configuration that places Ur/Web behind a frontend
> like Nginx
>    

I'm now thinking this isn't necessary and probably would reduce 
performance, but I'd still be interested to see the results of an 
experiment.

I believe I've pinpointed the reason for Ur/Web's poor showing on the 
no-database benchmarks: we build a single executable, which includes 
page handlers that do and do not touch the database.  However, the 
runtime system is creating a new database transaction for every request, 
even those that don't need the database!  Running a simple application 
by itself brings Ur/Web eerily close to the performance showing up in 
the benchmark results (within 1,000 requests per second!), which makes 
me optimistic that this is actually the "theoretical maximum" given the 
hardware, OS, etc.

I'm going to work on not creating transactions for requests that don't 
need them.