Delimited Generators – A more natural API for JS generators
Comments
steve_adams_86
ilaksh
I don't normally use generators. I can't figure out how this syntax or functionality is in any way different from normal generators.
steve_adams_86
I think it's unclear because the examples are simple enough that you could easily reproduce them with generators.
I think the ball example (https://github.com/manuel/delimgen/blob/main/demo/balls.html) is maybe the most helpful to look at. It provides an abstraction over the wrapped generator which allows you to inject events into the generator. You could do this with generators alone, but this syntax lets you write it more like synchronous code (and has the added benefit of not being "coloured" by async/await). The `run` method essentially allows you to pass of handling of a continuous process so you don't need to worry about the orchestration of concurrent tasks. In this case, ball is able to move itself around the document as soon as it's yielded, and the implementation details of that behaviour are totally hidden from where the event is injected.
I'm still trying to wrap my head around how I'd use delimited continuation in more practical ways, so I might be misunderstanding quite badly. It seems to me that generators themselves are a form of continuation, and we often create delimited continuations when we implement programs with generators. In this case, the library essentially ties the loose ends by reifying the continuation as a function. In this case, the function is what provides the synchronous-like syntax as well.
neals
What do you use generators for?
steve_adams_86
That’s a great question.
They work really well for reusing encapsulated logic in predictable ways. If you yield generators from within generators, it’s very easy to anticipate the flow of how they will execute, even if it’s conditional. It’s also very easy to do it asynchronously.
Apart from that, you can write logic which is agnostic of the flow it’s in, which makes it so you can yield it from any flow as long as the logic suits it. This makes it so something like a request and retry flow can use the same generators as an exponential back off version of the same logic, or you could drop the retry generator into database access logic, file upload or retrieval, and so on. If the retry logic is generic enough, it’ll be totally reusable.
Another use case is creating enormous or potentially infinite data structures which you can easily read and even manipulate. You can generate a deck of cards with just the suits and ranks, for example. If you add orders of magnitude to that scale factor, it still works just as well.
You can also easily wrap generators in a class or otherwise to make the app-facing logic nicer to look at. For personal projects I don’t bother, but I’ve done this at various jobs where I knew a dead-simple wrapper would dramatically clean up and simplify working with the generator while providing the same capability. It really depends on how much or how little can be abstracted and if it’s worthwhile.
I know you could say “but I can do this with promises” or similar, and in some cases you can (definitely not massive or infinite data traversal), but the control of the flow with a generator is remarkably powerful. Sometimes that’s very useful.
mmastrac
This is cool, but hitting break on the single timeout example in Firefox suggests that this code is actually filling the stack up pretty quickly, once per iteration through the loop.
The concept is neat, though: it's basically handing off a number of callbacks. Under the hood I think you'll find that it's just a really expensive version of `Promise.then`.
pwpwp
This appears to be an artefact of the "async call stacks" feature.
To turn it off in Firefox, go to about:config and set "javascript.options.asyncstack" to false.
In Chrome, in Devtools enter ctrl+shift+p and search for "async stack traces".
pwpwp
IOW, there isn't any real stack growth, it just appears that way when running in the devtools with the "async call stacks" feature enabled.
kabes
Have a look at effection,which seems to share some ideas: https://frontside.com/effection
qudat
Further, there is a delimited continuation library that effection uses: https://github.com/thefrontside/continuation
cowboyd
JavaScript generators are unmined gold! We've done a lot of usage of JavaScript generators as delimited continuations; using them to implement the classic shift/reset operations in JavaScript. https://github.com/thefrontside/continuation
Built on those delimited continuations is structured concurrency for JavaScript (https://frontside.com/effection)
agumonkey
oh, reading his previous blog article I see that J. Shutt (he made kernel, reviving fexpr in a way) commented
https://axisofeval.blogspot.com/2011/08/notes-on-delimited-c...
and that he has a blogger (last entry in 2020 though) https://fexpr.blogspot.com/
jdougan
Sadly, Dr. John Shutt died awhile ago. There is an obit on Lambda the Ultimate, but LtU appears to be down at the moment.
https://web.archive.org/web/20230326092743/http://lambda-the...
andrewstuart
Is this a reinvention of async/await?
steve_adams_86
My understanding is that it provides a more ergonomic API for using generators as delimited continuations. It does provide a similar control flow to async/await in that it can make asynchronous code look more like synchronous code, but there's more to it than that.
eyelidlessness
Not so much a reinvention. Both generators and async functions share the property that they can suspend execution and yield to the event loop. Many early polyfills for async functions utilized generators for that reason.
andrewstuart
async/await is built on generators.
WorldMaker
In many, but not all, compilers, yes. But that's an implementation detail and doesn't "have" to be the case.
One slightly different perspective is that the implementations of generators and async/await are just very similar monad transformers of two different monads (Iterable/"List" versus Future/Promise/Task). If you've already built one of these monad transformers you can built the other on top of it because they share a lot of the basics of being a monad transformer. In most cases today's languages had generators long before they had async/await so often generators or generator-like output is an implementation detail of async/await. It's not hard to imagine a world where async/await was the more common early implementation and generators used async/await an implementation detail. The abstractions are definitely related in interesting ways even if it's mostly only Haskell and a few others that decided to take the abstraction all the way "to the top" in their syntax. (Haskell do-notation being the same for whether you are working with Iterables or Futures or anything else that fits the abstraction of a monad.)
eyelidlessness
> It's not hard to imagine a world where async/await was the more common early implementation and generators used async/await an implementation detail.
It actually is: Promises have implications for event loop scheduling that preclude async (which implicitly returns a Promise) from providing synchronous semantics. Everything else you’re saying is mostly right (I’ll leave it to other folks to nitpick the monadic details, and just note that nits are there for the picking), but I think it’s important not to totally obscure this essential detail when observing that they’re otherwise closely related concepts.
WorldMaker
> Promises have implications for event loop scheduling that preclude async (which implicitly returns a Promise) from providing synchronous semantics.
Those also generally are implementation details of the chosen Promise/Futures/Tasks transformer model. The monad itself says nothing about how things are plumbed, event loops are just one common/useful way to plumb it.
C# Tasks run synchronously by default, on the same thread, and there is no required event loop (but you may need to interoperate with the SynchronizationContext for one). (.NET for better and worse uses a heavier threading abstraction than microstasks/green-threads.) It's often a common misconception for .NET developers that Tasks are always async and run in other threads. You can write code with synchronous semantics in C# async/await. It requires care and understanding, but it is possible.
One of the reasons async/await is so complicated in Python is that you can bring your own plumbing. The most common plumbing tools that are winning are event loop based, but there are other options, some of which allow synchronous semantics. Similarly, Rust's async/await support is nearly as complicated, just with fewer competing third party library options.
Thinking about Futures/Promises/Tasks from a JS-heavy perspective (which I can presume by your preference for the name Promises) where everything has to be browser-mediated microtasks in an event loop makes it really easy to mistake this one monad tree (of microtasks on an event loop) for the overall forest of this large monad family (with a number of implementations under the hood).
eyelidlessness
I think you have completely misunderstood the point I was making. I was saying that you can’t implement JavaScript generators with JavaScript async/await, even in your hypothetical scenario where they’d been developed in the opposite order. The reason you can’t is because JavaScript async has specific semantics that are incompatible with JavaScript generators. It isn’t that my perspective is particularly skewed towards JavaScript, rather that the language semantics we’re discussing have semantics that you can’t generalize away entirely. And it isn’t so much that I don’t see the generalization you’re describing.
You’re presumably coming from a place where you think it’s helpful to educate about the generalization, and that’s fine. But I’d suggest that when someone is clearly informed enough on the topic to clarify a language-specific nuance that diverges from the generalization, it probably isn’t because they’re ignorant, and it’s probably going to come off as rather condescending to respond as if they are.
WorldMaker
The topic at hand started as a generalization: "imagine a world where". That alternative world might not even have a Javascript. If you meant to make your point about the specifics of Javascript, I did not catch that change in topic and desire to talk about specifics, sorry.
eyelidlessness
Maybe you can “imagine a world” where you joined a conversation that was already taking place, about an article that framed the topic. It’s weird, and still rather condescending, to double down by suggesting I changed the topic which I was already engaged in before your imagination set the topic you think we were discussing. No need to apologize! But maybe, for your own benefit, go back and read through the thread again without trying to retcon your correctness onto it.
WorldMaker
"Imagine a world" is a common rhetorical way to start to an aside designed to be a deep generalization/alternate universe thought experiment. Yes, it was an intentional change of topic towards an aside. When you started your comment by quoting that specific phrase I easily assumed that you were interested in following my aside into generalities and/or alternate universe thought experiments. I didn't retcon anything, I intentionally created a fork in the conversation with my aside and assumed that by engaging with it you wanted to follow me down that aside. There's a lot of fun things we could have talked about if you had, and I regret that that wasn't what you were looking for. I'm sorry that "imagine a world" wasn't a strong enough indicator of an aside to the thread above.
That said, you've now called me condescending twice and I don't appreciate ad hominem attacks and will not engage with you further on this thread. I hesitated to even write this comment. (I also don't engage by creating "sock puppet" accounts, and I see the other comment accusing me of "sock puppeting" as a third, further unwarranted, ad hominem attack in this thread.) I don't think I need to reread the thread to know what happened in the communication mistakes, but I would suggest that you try to refrain from using attacks to deal with what were simple communication mistakes in the future.
eyelidlessness
It isn’t an attack to say that something comes off as condescending. It’s an observation that it comes off that way. It was also a good faith effort to invite you to understand that observation, because I didn’t think that was your intent.
I didn’t accuse you of creating a sock puppet account. I observed that the other response came from a newly created account, with that single comment posted about the same time. That observation wasn’t addressed to you, it was addressed to that new account.
grddvbytdb
It was pretty obvious that the person was saying “hey generators and async/await are in some sense dual”. You’re unhinged.
eyelidlessness
Weird to create a whole new account just to insult me but sure, I’m unhinged.
pwpwp
Actually, I built this because I don't really understand async/await. Now that I understand generators (somewhat), I might be able to grok async/await ;-)
neals
So what are some issues when working with async/await?
eyelidlessness
The classic complaint is the “function color” problem; otherwise worded as: async is infectious. Anything that depends on async itself must become async (whether by keyword or by dealing directly with Promise APIs).
There may be other valid issues, but they’re a pretty sharp drop off from that first one in most cases that aren’t either extremely niche or highly subjective.
orf
I remember being able to do this using Twisted in Python 2.6 or so, using the same techniques.
At the time, the way to manipulate “deferred” results was to use the equivalent of “Promise.then”, but using generators as a quasi async/await seemed much more fluid.
I’m not sure what this gives you over actual async/await though, other than being an interesting thing to look at?
masklinn
Python's plans for async was originally to use generators (that's the original reason for `yield from` IIRC: an async function may need to yield / await multiple times before it "completes" and returns a value).
Ultimately async was introduced because:
- mixing async-generators and sync-generators was extremely confusing and difficult to wrap your way around especially if you needed to nest them (e.g. a generator yielding awaitables)
- lots of statements may need async for which generators are inconvenient and require manual desugaring and ad-hoc protocols with pure generators most notably asynchronous iterables
I feel a bit stupid. I use generators a lot, but I don’t understand what this offers as a convenience.
Maybe I don’t actually understand generators!
Edit: I’m realizing it’s that I don’t understand delimiting. This is really interesting. I’ll have to sit with it for a while for it to click, I think.
I have a feeling a use delimiting patterns with generators already, but I didn’t understand it as such.