<div dir="ltr"><div>Update! The good news:</div><div>I was able to update the Dockerfile to build Ur/web from the latest release tarball (basically, using the old round 16 Dockerfile with a couple of small fixes like installing libicu-dev) and compare the benchmarks with the version installed with apt from the Ubuntu repo. The version built from the latest release was over ten times faster, even running on my old laptop.</div><div><br></div><div>The bad news:</div><div>The latest version of Ur appears to fail the "fortunes" test with the following diff (there is more, but this seems to explain it):</div><div><br></div><div>fortune: -<tr><td>6</td><td>Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen</td></tr> <br>fortune: +<tr><td>6</td><td>Emacs is a nice operating system, but I prefer UNIX. Tom Christaensen</td></tr> <br>fortune: @@ -17 +17 @@ <br>fortune: -<tr><td>12</td><td>フレームワークのベンチマーク</td></tr> <br>fortune: +<tr><td>12</td><td></td></tr> <br></div><div><br></div><div>It would seem that Unicode characters are being stripped from the output, causing the test to fail. I'm not familiar with exactly what the test is trying to do, and I don't know much about how Ur handles UTF-8.<br></div><div><br></div><div>If you can advise on how to fix this, I'd be happy to open a PR on the Techempower benchmarks repo with my changes.</div><div><br></div><div>Oisín<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 5 Aug 2019 at 19:43, Oisín Mac Fhearaí <<a href="mailto:denpashogai@gmail.com">denpashogai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Although I'm no closer to understanding why performance seems to have dropped in the benchmarks, thanks to a couple of comments on the Github issue I was able to find some more detailed logs of the test runs.</div><div><br></div><div>Fortunes, Round 16: <a href="https://tfb-logs.techempower.com/round-16/final/citrine-results/urweb/fortune/raw.txt" target="_blank">https://tfb-logs.techempower.com/round-16/final/citrine-results/urweb/fortune/raw.txt</a></div><div>Fortunes, Round 17: <a href="https://tfb-logs.techempower.com/round-17/final/citrine-results/20180903024112/urweb/fortune/raw.txt" target="_blank">https://tfb-logs.techempower.com/round-17/final/citrine-results/20180903024112/urweb/fortune/raw.txt</a></div><div><br></div><div>I'm amazed by the difference in request latencies:</div><div><br></div><div>Round 16:</div><div> Thread Stats Avg Stdev Max +/- Stdev<br> Latency 235.27us 140.45us 1.80ms 90.30%<br> Req/Sec 4.36k 148.24 4.89k 72.06%</div><div><br></div><div>Round 17:</div><div> Thread Stats Avg Stdev Max +/- Stdev<br> Latency 16.29ms 39.60ms 327.45ms 95.34%<br> Req/Sec 123.38 20.22 141.00 95.00%</div><div><br></div><div>In both cases, the web service is being hit by the "wrk" load tester, with the exact same parameters.</div><div><br></div><div>The only differences I can think of, then, are that the round 17 Ur/web Dockerfile installs urweb via the apt package manager, whereas the round 17 Dockerfile directly downloads an old tarball from 2016. But I've tested the latest Ubuntu version on my laptop and it performs almost exactly the same as the latest version from Git. So why does the round 17 benchmark have a max latency of 327 ms compared to under 2 ms in the previous round?</div><div><br></div><div>So confuse.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 2 Aug 2019 at 21:45, Oisín Mac Fhearaí <<a href="mailto:denpashogai@gmail.com" target="_blank">denpashogai@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I tried cloning the latest version of the benchmarks to run the Urweb tests locally, but sadly the Docker image fails to build for me (due to a problem with the Postgres installation steps, it seems). I've opened an issue here: <a href="https://github.com/TechEmpower/FrameworkBenchmarks/issues/4969" target="_blank">https://github.com/TechEmpower/FrameworkBenchmarks/issues/4969</a> ... I also asked for advice on how to track down the massive performance drop in the Urweb tests. Hopefully they might have some thoughts on it. Sadly I'm running things on a 9 year old laptop so it's hard to draw conclusions around performance...<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 1 Aug 2019 at 13:23, Adam Chlipala <<a href="mailto:adamc@csail.mit.edu" target="_blank">adamc@csail.mit.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I'm glad you brought this up, Oisín. I was already thinking of
appealing to this mailing list, in hopes of finding an eager
detective to hunt down what is going on! I can say that I can
achieve much better performance with the latest code on my own
workstation (similar profile to <i>one</i> of the several
machines used by TechEmpower), which leads me to think something
basic is getting in the way of proper performance in the
benchmarking environment.<br>
</p>
<div class="gmail-m_-1404699185956245187gmail-m_5782163309027588654gmail-m_-1629436119373177788moz-cite-prefix">On 7/31/19 8:06 PM, Oisín Mac Fhearaí
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I've noticed that Ur/web's performance benchmarks on
Techempower have changed significantly between round 16 and
17.</div>
<div><br>
</div>
<div>For example, in round 16, Urweb measured 323,430 responses
per second to the "Fortunes" benchmark.</div>
<div>In round 17 (and beyond), it achieved 4,024 RPS with MySQL
and 2,544 RPS with Postgres.</div>
<div><br>
</div>
<div>What could explain such a drastic drop in performance? The
blog entry for round 17 mentioned query pipelining as an
explanation for some of the frameworks getting much faster,
but I don't see why Urweb's RPS would drop by a factor of
100x, unless perhaps previous rounds had query caching enabled
and then round 17 disabled them.</div>
<div><br>
</div>
<div>Can anyone here shed light on this? I built a simplified
version of the "sql" demo with the 2016 tarball version of Ur
(used by the round 16 benchmarks) and a recent snapshot, and
they both perform at similar speeds on my laptop.</div>
<div><br>
</div>
<div>Oddly, the load testing tool I used (a Go program called
"hey") seems to have one request that takes 5 seconds if I set
it to use more concurrent threads than the number of threads
available to the Ur/web program. Otherwise, the longest
request takes about 0.02 seconds. This seems unrelated to the
performance drop on the Techempower benchmarks, since the max
latency is quite low there.</div>
</div>
</blockquote>
</div>
_______________________________________________<br>
Ur mailing list<br>
<a href="mailto:Ur@impredicative.com" target="_blank">Ur@impredicative.com</a><br>
<a href="http://www.impredicative.com/cgi-bin/mailman/listinfo/ur" rel="noreferrer" target="_blank">http://www.impredicative.com/cgi-bin/mailman/listinfo/ur</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>