[Ur] callbacks from C FFI (again)

[Sergey Mironov]

2013/12/21 Adam Chlipala <adamc at csail.mit.edu>:
> On 12/18/2013 02:46 PM, Sergey Mironov wrote:
>>
>> Hi. I've done some work regarding C callbacks, hook the app->handle
>> as you suggested. The patch is rather big, but I think it handles the
>> problem
>> nice now. Basically, it adds ./src/c/srvthreads.c which does two things:
>> a)
>> manages a number of permanently-running worker threads to handle
>> st_loopback_enqueue calls (see below), and b) manages a list of threads
>> created
>> by user.
>
>
> Could you remind me of your use case for this functionality?
>

>
> A few other comments on your patch:
>
> General C code shouldn't assume that the variable uw_application exists.
> Everything is implemented to work even when multiple distinct applications
> run in one Ur/Web process, though I never released the (working) code for
> doing that.
>
> Why does your modified uw_request_context() take both app and loggers as
> arguments, when the latter implies the former?

 I accept your points regarding uw_application and loggers, I'll try
to redesign this part of patch (as well as other parts of it)

> Why isn't it sufficient just to start a periodic task that runs one or all
> tasks from a queue of URI strings every second?  You would need only a tiny
> amount of FFI code (and no modifications to the base Ur/Web distribution)
> with that approach.

The reason (probably a bit too idealistic) is to avoid hardcoded
latencies, even as small as one second. But also let me describe a
use-case example I'm working on in a bit more details. The downloader
application (I've described it in the first letter of this thread) may
be asked to start a separate thread to accomplish a download. Each
thread spawns and monitors standalone process, and this may be a
long-running job. Upon completion, thread needs to kick the Ur/web
somehow to notify the user. The action may be different: application
may need to send them email, to send something into a channel, or
both. So here comes the loopback API I'm trying to implement.

I need a function which may be called from FFI-code to enqueue the
url. This function should be thread-safe and uw_context-free.
Basically it's signature may be as simple as  'void (*)(const char *
url)`. The question is how to implement it. I don't see a way to
combine it with periodic tasks without aggressive polling from the
task's side, that is why I'm still suggesting to keep a separate
thread (or a fixed number of threads) to handle the URL queue, just
like http.c does to handle incoming requests. But please point me to
another solution if it exists.

Also, I thought about the behavior of periodic tasks in cgi-mode. As
far as I understand the CGI, some higher-level server creates a
process for every single request and that means that Ur/Web runtime
terminates periodics unconditionally when the main thread ends.
Unlucky tasks may never have a chance to reach a commit point.
Probably, it is acceptable for normal UrWeb code because of it's
doesn't lead to the data losses, but FFI-specific storages can't be
used without care. This problem confused me and that is why I've
overcomplicated the API and made special efforts to join threads
correctly before quitting from main().

Generally, I feel that I need more time to develop the solution. I
keep my work in the urweb-callback repo here:

https://github.com/grwlf/urweb-callback

(it requires the UrWeb patch I've posted before, the approach works in
general despite of being too complex for now). By the way, I've tried
C++11-based FFI there, with lambdas and shared-pointers, it is fun!

Regards,
Sergey