[Ur] TechEmpower Benchmarks

[Adam Chlipala]

On 11/27/2013 04:50 AM, Maciek Starzyk wrote:
> Some preview benchmark results are at:
> http://www.techempower.com/benchmarks/previews/round8/
> (Also see https://groups.google.com/forum/?fromgroups=#!topic/framework-benchmarks/6sQYeMlYo9Y
> )
>
> Ur/Web results don't look too good :(
>    

There are a few issues, for which escalier (the author of the Ur/Web 
benchmark code) pointed me to some logs.

Here's his log for one of the basic tests:
     
https://github.com/TechEmpower/TFB-Round-8/blob/master/i7/preview-2/20131124092825/json/urweb/raw
My guess is that I was just making a newbie-ish mistake of not calling C 
listen() with a large enough backlog parameter, so that connections were 
being needlessly rejected.  Apparently no one had tested Ur/Web's basic 
HTTP servers at this level of concurrency before!

I've pushed a changeset that addresses the issue.  It's also easy to 
patch the source code directly in src/c/http.c, changing:
     int uw_backlog = 10;
to
     int uw_backlog = SOMAXCONN;

If this change isn't actually a good idea, I'd appreciate advice from 
more experienced BSD sockets hackers!  I expect it will immediately 
reduce dramatically the number of "errors" reported in the benchmark 
results.


The other log he pointed me to is
     
https://github.com/TechEmpower/TFB-Round-8/blob/master/i7/preview-2/latest/logs/urweb/err.txt
where we see errors like
> Fatal error: /usr/local/lib/urweb/ur/top.ur:365:17-365:18: Query failed:
> SELECT T_World.uw_Id, T_World.uw_RandomNumber FROM uw_Bench_world AS 
> T_World WHERE (T_World.uw_id = 7522::int8)
> ERROR: out of shared memory
> HINT: You might need to increase max_pred_locks_per_transaction.

A few other things seem to go wrong in the log, but I won't be surprised 
if they're just follow-on consequences of the first basic issue.

escalier, do you know if there's a way to request a higher value of the 
configuration setting mentioned in the hint?  I won't be surprised if 
most Postgres users in the benchmarks aren't using real serializable 
transactions, which would certainly improve performance, but goes 
against the Ur/Web philosophy of exposing clean semantics to programmers.

Alternatively, I would expect pretty low overhead to try switching to 
MySQL, which doesn't _have_ serializable transactions and so might give 
lower overhead. :P

Chris Double wrote:
> The Json code in meta does a large amount of string concatenation
> which I've found to be quite slow in Ur/Web. This could be part of the
> performance issue if it's doing a lot of 'toJson',
>    

Yeah, I was expecting to see some cost from this, but it's not clear to 
me if interpretive overhead of the more mainstream frameworks would be 
greater in practice.  I think Ur/Web's poor showing ATM is stemming 
mainly from the mundane issues discussed above.

(I might decide in the not-too-distant future to add a new compiler 
optimization to reduce the cost of this code pattern.  It's already 
implemented for traditional generated pages only.)