Correct me if I'm wrong, but isnt any sort of async support non integral to rust? For example in something like Javscript you can't impliment your own async. But in C, C++ or Rust you can do pretty much anything you want.
So if in the future io-uring and friends become the standard can't that just be a library you could then use?
Similar to how in C you don't need the standard library to do threads or async.
I agree. Completion-based APIs are more high level, and not a good abstraction at the systems language level. IOCP and io_uring use poll-based interfaces internally. In io_uring's case, the interfaces are basically the same ones available in user space. In Windows case IOCP uses interfaces that are private, but some projects have figured out the details well enough to implement decent epoll and kqueue compatibility libraries.
Application developers of course want much higher level interfaces. They don't want to do a series of reads; they want "fetch_url". But if "fetch_url" is the lowest-level API available, good luck implementing an efficient streaming media server. (Sometimes we end up with things like HTTP Live Streaming, a horrendously inefficient protocol designed for ease of use in programming environments, client- and server-side, that effectively only offer the equivalent of "fetch_url".)
Plus, IOCP models tend to heavily rely on callbacks and closures. And as demonstrated in the article, low-level languages suck at providing ergonomic first-class functions, especially if they lack GC. (It's a stretch to say that Rust even supports first-class functions.) If I were writing an asynchronous library in Rust, I'd do it the same way I'd do it in C--a low-level core that is non-blocking and stateful. For example, you repeatedly invoke something like "url_fetch_event", which returns a series of events (method, header, body chunk) or EAGAIN/EWOULDBLOCK. (It may not even pull from a socket directly, but rely on application to write source data into an internal buffer.) Then you can wrap that low-level core in progressively higher-level APIs, including alternative APIs suited to different async event models, as well as fully blocking interfaces. And if a high-level API isn't to some application developer's liking, they can create their own API around the low-level core API. This also permits easier cross-language integration. You can easily use such a low-level core API for bindings to Python, Lua, or even Go, including plugging into whatever event systems they offer, without losing functional utility.
It's the same principle with OS and systems language interfaces--you provide mechanisms that can be built upon. But so many Rust developers come from high-level application environments, including scripting language environments, where this composition discipline is less common and less relevant.
> IOCP models tend to heavily rely on callbacks and closures
While perhaps higher level libraries are written that way, I can’t think of a reason why the primitive components of IOCP require callbacks and closures. The “poll for io-readiness and then issue non-blocking IO” and “issue async IO and then poll for completion” models can be implemented in a reactor pattern in a similar manner. It is just a question of whether the system call happens before or after the reactor loop.
EDIT: Reading some of the other comments and thinking a bit, one annoying thing about IOCP is the cancelation model. With polling IO readiness, it is really easy to cancel IO and close a socket: just unregister from epoll and close it. With IOCP, you will have to cancel the in-flight operation and wait for the completion notification to come in before you can close a socket (if I understand correctly).
Anyways, I've been playing around with implementing some async socket APIs on top of IOCP for Windows in Rust [1]. Getting the basic stuff working is relatively easy. Figuring our a cancellation model is going to be a bit difficult. And ultimately I think it would be cool if the threads polling the completion ports could directly execute the wakers in such a way that the future could be polled inline, but getting all the lifetimes right is making my head hurt.
Yes, you can encode state machines manually, but it will be FAR less ergonomic than the async syntax. Rust has started with a library-based approach, but it was... not great. Async code was littered with and_then methods and it was really close to the infamous JS callback hell. The ergonomic improvements which async/await brings is essentially a raison d'être for incorporating this functionality into the language.
For comparison, Haskell went with the library approach but has the syntactic sugar of the equivalent of `and_then` built into the language. (I am talking about Monads and do-notation.)
It's a bit like iterating in Python: for-loops are a convenient syntactic sugar to something that can be provided by a library.
It is a PhD level research problem to know if monads and do notation would be able to work in Rust. The people who are most qualified to look into it (incidentally: a lot of the same crew was who was working on async) believe that it may literally be impossible.
Yes, a number of people have suggested introduction of a general do notation (or its analog) instead of usecase-specific async/awayt syntax, but since Rust does not have proper higher kinded types (and some Rust developers say it never will), such proposals have been deemed impractical.
Can't Rust come up with a new syntax (that matches io_uring idea better) and deprecate the old one? Or simply replace the implementation keeping the old syntax if it's semantically the same?
It could, although I highly doubt that any deficiencies with the current implementation of async/await are so severe as to warrant anything so dramatic.
So if in the future io-uring and friends become the standard can't that just be a library you could then use?
Similar to how in C you don't need the standard library to do threads or async.