[Ur] Drop of several orders of magnitude in Techempower benchmarks

[Oisín Mac Fhearaí]

I tried cloning the latest version of the benchmarks to run the Urweb tests
locally, but sadly the Docker image fails to build for me (due to a problem
with the Postgres installation steps, it seems). I've opened an issue here:
https://github.com/TechEmpower/FrameworkBenchmarks/issues/4969 ... I also
asked for advice on how to track down the massive performance drop in the
Urweb tests. Hopefully they might have some thoughts on it. Sadly I'm
running things on a 9 year old laptop so it's hard to draw conclusions
around performance...

On Thu, 1 Aug 2019 at 13:23, Adam Chlipala <adamc at csail.mit.edu> wrote:

> I'm glad you brought this up, Oisín.  I was already thinking of appealing
> to this mailing list, in hopes of finding an eager detective to hunt down
> what is going on!  I can say that I can achieve much better performance
> with the latest code on my own workstation (similar profile to *one* of
> the several machines used by TechEmpower), which leads me to think
> something basic is getting in the way of proper performance in the
> benchmarking environment.
> On 7/31/19 8:06 PM, Oisín Mac Fhearaí wrote:
>
> I've noticed that Ur/web's performance benchmarks on Techempower have
> changed significantly between round 16 and 17.
>
> For example, in round 16, Urweb measured 323,430 responses per second to
> the "Fortunes" benchmark.
> In round 17 (and beyond), it achieved 4,024 RPS with MySQL and 2,544 RPS
> with Postgres.
>
> What could explain such a drastic drop in performance? The blog entry for
> round 17 mentioned query pipelining as an explanation for some of the
> frameworks getting much faster, but I don't see why Urweb's RPS would drop
> by a factor of 100x, unless perhaps previous rounds had query caching
> enabled and then round 17 disabled them.
>
> Can anyone here shed light on this? I built a simplified version of the
> "sql" demo with the 2016 tarball version of Ur (used by the round 16
> benchmarks) and a recent snapshot, and they both perform at similar speeds
> on my laptop.
>
> Oddly, the load testing tool I used (a Go program called "hey") seems to
> have one request that takes 5 seconds if I set it to use more concurrent
> threads than the number of threads available to the Ur/web program.
> Otherwise, the longest request takes about 0.02 seconds. This seems
> unrelated to the performance drop on the Techempower benchmarks, since the
> max latency is quite low there.
>
> _______________________________________________
> Ur mailing list
> Ur at impredicative.com
> http://www.impredicative.com/cgi-bin/mailman/listinfo/ur
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.impredicative.com/pipermail/ur/attachments/20190802/270810c5/attachment.html>