Jekyll2020-03-06T14:49:45+00:00https://blag.nemo157.com/feed.xmlblag.nemo157.comInside Rust’s Async Transform2018-12-09T00:00:00+00:002018-12-09T00:00:00+00:00https://blag.nemo157.com/2018/12/09/inside-rusts-async-transform<p>Update: Clarified that C# and JavaScript can be understood as similar to a CPS
transform, not that they are actually implemented as one.</p>
<p>As you likely know if you’re reading this post Rust has an upcoming async/await
feature being tested in nightly. Because of Rust’s unique features and
positioning fully understanding the implementation powering this syntax is very
different to understanding other well-known implementations (C# and JavaScript’s
being the ones I am familiar with). Instead of thinking of a <a href="https://en.wikipedia.org/wiki/Continuation-passing_style" title="Continuation-passing style">CPS</a>-like
transform where an async function is split into a series of continuations that
are chained together via a <code class="highlighter-rouge">Future::then</code> method, Rust instead uses a
generator/coroutine transform to turn the function into a state machine (C# and
probably most JavaScript implementations use a similar transform under the hood,
but as far as I’m aware because of the garbage collector these are
indistinguishable from the naive CPS transform they are normally described as).
For more detail on why Rust is taking this approach you should read <a href="https://github.com/rust-lang/rfcs/blob/master/text/2033-experimental-coroutines.md">eRFC 2033:
Experimental Coroutines</a>, that lays out the why’s much better than I
could here.</p>
<p>What I’m going to try and provide instead, is a look into how this actually works
today. What steps the compiler takes to turn an <code class="highlighter-rouge">async fn</code> into a normal
function returning a state machine that you <em>could</em> write if you wanted to (but
you definitely don’t).</p>
<!--more-->
<h2 id="the-setup">The Setup</h2>
<p>First, to provide a nice self-contained example I need to get some setup out of
the way. In normal async/await in Rust you will bring in the <a href="https://github.com/rust-lang-nursery/futures-rs"><code class="highlighter-rouge">futures</code></a>
library to handle most of the pieces I’m about to mention; but to be able to
provide links to running playgrounds of these examples (and so that you can see
all the gristle in these here sausages) I’m going to avoid that.</p>
<p>Also, I’m going to assume complete knowledge of soon to be stable Rust 2018 and
the basic futures API (including pinning). I will endeavour to explain all the
other nightly features being used, but I have been living in nightly-only for
far too long now and may forget some.</p>
<h3 id="nightly-features">Nightly Features</h3>
<p>There are only three nightly features needed for the basic setup here, these are
what I’m assuming complete knowledge of:</p>
<ul>
<li><code class="highlighter-rouge">async_await</code> to allow using async functions and blocks</li>
<li><code class="highlighter-rouge">futures_api</code> to allow using the basic <code class="highlighter-rouge">core::future</code> and <code class="highlighter-rouge">core::task</code> APIs</li>
<li><code class="highlighter-rouge">pin</code> to allow using the basic <code class="highlighter-rouge">core::pin</code> APIs</li>
</ul>
<h3 id="the-async-io">The “Async IO”</h3>
<p>To provide a slightly more realistic example I will use the following “trait”
for reading in data:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="nf">AsyncRead</span><span class="p">(</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">);</span>
<span class="k">impl</span> <span class="n">AsyncRead</span> <span class="p">{</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">)</span> <span class="k">-></span> <span class="n">AsyncRead</span> <span class="p">{</span>
<span class="nf">AsyncRead</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="k">-></span> <span class="k">impl</span> <span class="n">Future</span><span class="o"><</span><span class="n">Output</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'_</span> <span class="p">{</span>
<span class="n">async</span> <span class="k">move</span> <span class="p">{</span> <span class="k">self</span><span class="err">.</span><span class="mi">0</span><span class="nf">.clone</span><span class="p">()</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, futures_api)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
fn main() {
// Not enough pieces to give a decent example
let encrypted = [108, 97, 104, 104, 107];
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<p>This can’t be a real trait because it requires generic associated types to
declare the return value of <code class="highlighter-rouge">read_to_end</code>, so wherever you see <code class="highlighter-rouge">AsyncRead</code> in
the later function signatures just imagine <code class="highlighter-rouge">impl AsyncRead</code>.</p>
<p>You may also note a lack of errors. This is because adding in the additional
error handling paths in the last step is a <em>lot</em> of work and I’m lazy. They
don’t really show anything new either, just more matching and more states to
implement to handle the different execution paths.</p>
<h3 id="the-executor">The “Executor”</h3>
<p>To run a <code class="highlighter-rouge">Future</code> to completion you require an executor to run it on. First
there is the most basic API for an executor that can run a single future to
completion:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="n">block_on</span><span class="o"><</span><span class="n">F</span><span class="p">:</span> <span class="n">Future</span><span class="o">></span><span class="p">(</span><span class="k">mut</span> <span class="n">future</span><span class="p">:</span> <span class="n">F</span><span class="p">)</span> <span class="k">-></span> <span class="nn">F</span><span class="p">::</span><span class="n">Output</span><span class="p">;</span>
</code></pre></div></div>
<p>Then, this is the simplest implementation of that executor that simply spins
polling the future until it completes (there is an equivalent executor that
requires less lines of code, and no unsafety, but it does require use of
<code class="highlighter-rouge">std::sync::Arc</code> and I’m attempting to use the least powerful implementation of
everything here):</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="n">block_on</span><span class="o"><</span><span class="n">F</span><span class="p">:</span> <span class="n">Future</span><span class="o">></span><span class="p">(</span><span class="k">mut</span> <span class="n">future</span><span class="p">:</span> <span class="n">F</span><span class="p">)</span> <span class="k">-></span> <span class="nn">F</span><span class="p">::</span><span class="n">Output</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">core</span><span class="p">::{</span>
<span class="nn">pin</span><span class="p">::</span><span class="n">Pin</span><span class="p">,</span>
<span class="nn">ptr</span><span class="p">::</span><span class="n">NonNull</span><span class="p">,</span>
<span class="nn">task</span><span class="p">::{</span><span class="n">LocalWaker</span><span class="p">,</span> <span class="n">Poll</span><span class="p">,</span> <span class="n">UnsafeWake</span><span class="p">,</span> <span class="n">Waker</span><span class="p">},</span>
<span class="p">};</span>
<span class="k">struct</span> <span class="n">NoWake</span><span class="p">;</span>
<span class="k">impl</span> <span class="n">NoWake</span> <span class="p">{</span>
<span class="k">fn</span> <span class="nf">local_waker</span><span class="p">()</span> <span class="k">-></span> <span class="n">LocalWaker</span> <span class="p">{</span>
<span class="c">// Safety: all references to NoWake are never</span>
<span class="c">// dereferenced</span>
<span class="k">unsafe</span> <span class="p">{</span> <span class="nn">LocalWaker</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">NonNull</span><span class="p">::</span><span class="o"><</span><span class="n">NoWake</span><span class="o">></span><span class="p">::</span><span class="nf">dangling</span><span class="p">())</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">unsafe</span> <span class="k">impl</span> <span class="n">UnsafeWake</span> <span class="k">for</span> <span class="n">NoWake</span> <span class="p">{</span>
<span class="k">unsafe</span> <span class="k">fn</span> <span class="nf">clone_raw</span><span class="p">(</span><span class="o">&</span><span class="k">self</span><span class="p">)</span> <span class="k">-></span> <span class="n">Waker</span> <span class="p">{</span>
<span class="nn">NoWake</span><span class="p">::</span><span class="nf">local_waker</span><span class="p">()</span><span class="nf">.into_waker</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">unsafe</span> <span class="k">fn</span> <span class="nf">drop_raw</span><span class="p">(</span><span class="o">&</span><span class="k">self</span><span class="p">)</span> <span class="p">{}</span>
<span class="k">unsafe</span> <span class="k">fn</span> <span class="nf">wake</span><span class="p">(</span><span class="o">&</span><span class="k">self</span><span class="p">)</span> <span class="p">{}</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">lw</span> <span class="o">=</span> <span class="nn">NoWake</span><span class="p">::</span><span class="nf">local_waker</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="c">// Safety: `future` is a local variable which is</span>
<span class="c">// only ever used in this pinned reference</span>
<span class="k">match</span> <span class="k">unsafe</span> <span class="p">{</span> <span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">future</span><span class="p">)</span> <span class="p">}</span><span class="nf">.poll</span><span class="p">(</span><span class="o">&</span><span class="n">lw</span><span class="p">)</span> <span class="p">{</span>
<span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="k">=></span> <span class="k">break</span> <span class="n">value</span><span class="p">,</span>
<span class="nn">Poll</span><span class="p">::</span><span class="n">Pending</span> <span class="k">=></span> <span class="n">continue</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, futures_api, pin)]
pub mod executor {
use core::future::Future;
pub fn block_on<F: Future>(mut future: F) -> F::Output {
use core::{
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
fn main() {
let encrypted = executor::block_on(async { [108, 97, 104, 104, 107] });
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h2 id="the-asyncawait-implementation">The <code class="highlighter-rouge">async</code>/<code class="highlighter-rouge">await!</code> implementation</h2>
<p>Now that we have the setup out of the way, here’s the super simple <code class="highlighter-rouge">async fn</code>
we’re going to be expanding. This function takes in a reference to some async
IO, constructs a handle to some “random” one-time-pad, waits for both to
complete, then XORs the data and pad together to secure the data. It may seem
simple here, but once we get to the final stage you’re going to be glad I chose
something so simple.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="n">async</span> <span class="k">fn</span> <span class="nf">quote_encrypt_unquote</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pad</span> <span class="o">=</span> <span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]);</span>
<span class="nd">await!</span><span class="p">(</span><span class="n">data</span><span class="nf">.read_to_end</span><span class="p">())</span>
<span class="nf">.into_iter</span><span class="p">()</span>
<span class="nf">.zip</span><span class="p">(</span><span class="nd">await!</span><span class="p">(</span><span class="n">pad</span><span class="nf">.read_to_end</span><span class="p">()))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span>
<span class="nf">.collect</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, await_macro, futures_api, pin)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
pub async fn quote_encrypt_unquote(data: &mut AsyncRead) -> Vec<u8> {
// one-time-pad chosen by fair dice roll
let mut pad = AsyncRead::new(vec![4; 32]);
await!(data.read_to_end())
.into_iter()
.zip(await!(pad.read_to_end()))
.map(|(a, b)| a ^ b)
.collect()
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h2 id="expanding-asyncawait">Expanding <code class="highlighter-rouge">async</code>/<code class="highlighter-rouge">await!</code></h2>
<p>The first step on our journey into madness is to simply expand the <code class="highlighter-rouge">async fn</code>
into a normal function. This has three parts to it:</p>
<ol>
<li>Expanding the <code class="highlighter-rouge">await!</code> macro</li>
<li>Rewriting the function signature</li>
<li>Expanding the body</li>
</ol>
<h3 id="but-first">But first</h3>
<p>Before doing the actual <code class="highlighter-rouge">async</code>/<code class="highlighter-rouge">await!</code> expansion I want to slightly rewrite
the function from before. This will be a functionally equivalent function, but
by pulling out a few temporary variables the control flow between the different
transforms will be easier to follow. Mostly, having <code class="highlighter-rouge">await!</code> inside other
expressions will greatly complicate the upcoming generator transform.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="n">async</span> <span class="k">fn</span> <span class="nf">quote_encrypt_unquote</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pad</span> <span class="o">=</span> <span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="nd">await!</span><span class="p">(</span><span class="n">data</span><span class="nf">.read_to_end</span><span class="p">());</span>
<span class="k">let</span> <span class="n">pad</span> <span class="o">=</span> <span class="nd">await!</span><span class="p">(</span><span class="n">pad</span><span class="nf">.read_to_end</span><span class="p">());</span>
<span class="n">data</span><span class="nf">.into_iter</span><span class="p">()</span><span class="nf">.zip</span><span class="p">(</span><span class="n">pad</span><span class="p">)</span><span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span><span class="nf">.collect</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, await_macro, futures_api, pin)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
pub async fn quote_encrypt_unquote(data: &mut AsyncRead) -> Vec<u8> {
// one-time-pad chosen by fair dice roll
let mut pad = AsyncRead::new(vec![4; 32]);
let data = await!(data.read_to_end());
let pad = await!(pad.read_to_end());
data.into_iter().zip(pad).map(|(a, b)| a ^ b).collect()
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h3 id="expanding-the-await-macro">Expanding the <code class="highlighter-rouge">await!</code> macro</h3>
<p>Currently <code class="highlighter-rouge">await!</code> is simply a normal macro <a href="https://doc.rust-lang.org/nightly/std/macro.await.html">defined in <code class="highlighter-rouge">std</code></a>. This
is unlikely to last, there are some requirements on it that I believe will
necessitate it moving into the compiler, but it makes things slightly simpler
here for now. We can expand this macro while still leaving the rest of the
<code class="highlighter-rouge">async fn</code> alone and still get something that compiles. Note the
<code class="highlighter-rouge">poll_with_tls_waker</code> function introduced here, I’ll come back to it later.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="n">async</span> <span class="k">fn</span> <span class="nf">quote_encrypt_unquote</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">core</span><span class="p">::{</span><span class="nn">pin</span><span class="p">::</span><span class="n">Pin</span><span class="p">,</span> <span class="nn">task</span><span class="p">::</span><span class="n">Poll</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">future</span><span class="p">::</span><span class="n">poll_with_tls_waker</span><span class="p">;</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pad</span> <span class="o">=</span> <span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">data</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span>
<span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span> <span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span> <span class="p">})</span>
<span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">pad</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">pad</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span>
<span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span> <span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span> <span class="p">})</span>
<span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">data</span><span class="nf">.into_iter</span><span class="p">()</span><span class="nf">.zip</span><span class="p">(</span><span class="n">pad</span><span class="p">)</span><span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span><span class="nf">.collect</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, await_macro, futures_api, pin, generators, gen_future)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
pub async fn quote_encrypt_unquote(data: &mut AsyncRead) -> Vec<u8> {
use core::{pin::Pin, task::Poll};
use std::future::poll_with_tls_waker;
// one-time-pad chosen by fair dice roll
let mut pad = AsyncRead::new(vec![4; 32]);
let data = {
let mut pinned = data.read_to_end();
loop {
if let Poll::Ready(x) = poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
let pad = {
let mut pinned = pad.read_to_end();
loop {
if let Poll::Ready(x) = poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
data.into_iter().zip(pad).map(|(a, b)| a ^ b).collect()
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h3 id="rewriting-the-function-signature">Rewriting the function signature</h3>
<p><code class="highlighter-rouge">async fn</code> does some slightly funky things to the function signature. The main
thing is just taking the return value (<code class="highlighter-rouge">R</code>) and wrapping it into <code class="highlighter-rouge">impl
Future<Output = R></code>, then the lifetime of the returned future is bound by the
lifetimes of all arguments. Currently if you have a function taking multiple
references you have to give it a single named lifetime for all those references
to use, but I believe the intention is for this to automatically work in the
future.</p>
<p>After re-writing the signature the body of the function can be wrapped in
an <code class="highlighter-rouge">async move { ... }</code> block to keep everything compiling with the exact same
semantics as before:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">quote_encrypt_unquote</span><span class="p">(</span>
<span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="k">impl</span> <span class="n">Future</span><span class="o"><</span><span class="n">Output</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'_</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">core</span><span class="p">::{</span><span class="nn">pin</span><span class="p">::</span><span class="n">Pin</span><span class="p">,</span> <span class="nn">task</span><span class="p">::</span><span class="n">Poll</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">future</span><span class="p">::</span><span class="n">poll_with_tls_waker</span><span class="p">;</span>
<span class="n">async</span> <span class="k">move</span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pad</span> <span class="o">=</span> <span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">data</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span>
<span class="p">})</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">pad</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">pad</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span>
<span class="p">})</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">data</span><span class="nf">.into_iter</span><span class="p">()</span><span class="nf">.zip</span><span class="p">(</span><span class="n">pad</span><span class="p">)</span><span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span><span class="nf">.collect</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, await_macro, futures_api, pin, generators, gen_future)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
use core::future::Future;
pub fn quote_encrypt_unquote(data: &mut AsyncRead) -> impl Future<Output = Vec<u8>> + '_ {
use core::{pin::Pin, task::Poll};
use std::future::poll_with_tls_waker;
async move {
// one-time-pad chosen by fair dice roll
let mut pad = AsyncRead::new(vec![4; 32]);
let data = {
let mut pinned = data.read_to_end();
loop {
if let Poll::Ready(x) =
poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
let pad = {
let mut pinned = pad.read_to_end();
loop {
if let Poll::Ready(x) =
poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
data.into_iter().zip(pad).map(|(a, b)| a ^ b).collect()
}
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h3 id="expanding-the-body">Expanding the body</h3>
<p>Finally to remove all <code class="highlighter-rouge">async</code> syntax from the function we can expand the <code class="highlighter-rouge">async
move { ... }</code> block into a generator and package this into a wrapper <code class="highlighter-rouge">Future</code>
with <code class="highlighter-rouge">from_generator</code> from <code class="highlighter-rouge">std::future</code>.</p>
<p><code class="highlighter-rouge">std::future::from_generator</code> is a counterpart to the <code class="highlighter-rouge">poll_with_tls_waker</code>
function mentioned earlier, these <em>must</em> be used together, the <code class="highlighter-rouge">Future</code> created
by <code class="highlighter-rouge">from_generator</code> will place the <code class="highlighter-rouge">&LocalWaker</code> passed in to <code class="highlighter-rouge">Future::poll</code>
into thread local storage, <code class="highlighter-rouge">poll_with_tls_waker</code> will then retrieve this to pass
in to sub-futures that are being <code class="highlighter-rouge">await!</code>ed on.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">quote_encrypt_unquote</span><span class="p">(</span>
<span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="k">impl</span> <span class="n">Future</span><span class="o"><</span><span class="n">Output</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'_</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">core</span><span class="p">::{</span><span class="nn">pin</span><span class="p">::</span><span class="n">Pin</span><span class="p">,</span> <span class="nn">task</span><span class="p">::</span><span class="n">Poll</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">future</span><span class="p">::{</span><span class="n">from_generator</span><span class="p">,</span> <span class="n">poll_with_tls_waker</span><span class="p">};</span>
<span class="nf">from_generator</span><span class="p">(</span><span class="k">static</span> <span class="k">move</span> <span class="p">||</span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pad</span> <span class="o">=</span> <span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">data</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span>
<span class="p">})</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">pad</span> <span class="o">=</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">pinned</span> <span class="o">=</span> <span class="n">pad</span><span class="nf">.read_to_end</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span><span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">pinned</span><span class="p">)</span>
<span class="p">})</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">yield</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">data</span><span class="nf">.into_iter</span><span class="p">()</span><span class="nf">.zip</span><span class="p">(</span><span class="n">pad</span><span class="p">)</span><span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span><span class="nf">.collect</span><span class="p">()</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(async_await, await_macro, futures_api, pin, generators, gen_future)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
use core::future::Future;
pub fn quote_encrypt_unquote(data: &mut AsyncRead) -> impl Future<Output = Vec<u8>> + '_ {
use core::{pin::Pin, task::Poll};
use std::future::{from_generator, poll_with_tls_waker};
from_generator(static move || {
// one-time-pad chosen by fair dice roll
let mut pad = AsyncRead::new(vec![4; 32]);
let data = {
let mut pinned = data.read_to_end();
loop {
if let Poll::Ready(x) =
poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
let pad = {
let mut pinned = pad.read_to_end();
loop {
if let Poll::Ready(x) =
poll_with_tls_waker(unsafe { Pin::new_unchecked(&mut pinned) })
{
break x;
}
yield
}
};
data.into_iter().zip(pad).map(|(a, b)| a ^ b).collect()
})
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<h2 id="expanding-the-generator">Expanding the generator</h2>
<p>Now that we have finished the <code class="highlighter-rouge">async</code> transform we have the much more
complicated generator transform to apply. This transform is implemented in
<code class="highlighter-rouge">rustc</code>’s MIR layer, so has much less direct equivalence to anything we can
implement manually in Rust’s surface language. I’m going to present a
manual implementation of a <code class="highlighter-rouge">Generator</code> that does the same thing as the one from
the previous snippet, using a sort of similar layout to what <code class="highlighter-rouge">rustc</code> would
generate, but since this is all very much unstable internals there’s no
guarantee that <code class="highlighter-rouge">rustc</code> will continue to generate something similar to this.</p>
<p>Also, rather than attempting to transform piece by piece like the last section
I’m going to first present the entire transformed generator, then pull out
pieces to explain from it. The following code block simply replaces everything
inside the <code class="highlighter-rouge">from_generator</code> call in the previous block.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">existential</span> <span class="k">type</span> <span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span><span class="p">:</span> <span class="n">Future</span><span class="o"><</span><span class="n">Output</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'a</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="n">read</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span> <span class="p">{</span>
<span class="n">read</span><span class="nf">.read_to_end</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">struct</span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="n">state</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span>
<span class="n">data_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">AsyncRead</span><span class="o">></span><span class="p">,</span>
<span class="n">pad_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">AsyncRead</span><span class="o">></span><span class="p">,</span>
<span class="n">pinned_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">>></span><span class="p">,</span>
<span class="n">data_2</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span><span class="p">,</span>
<span class="n">pinned_2</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">Generator</span> <span class="k">for</span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">type</span> <span class="n">Yield</span> <span class="o">=</span> <span class="p">();</span>
<span class="k">type</span> <span class="n">Return</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">;</span>
<span class="k">unsafe</span> <span class="k">fn</span> <span class="nf">resume</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="k">-></span> <span class="n">GeneratorState</span><span class="o"><</span><span class="nn">Self</span><span class="p">::</span><span class="n">Yield</span><span class="p">,</span> <span class="nn">Self</span><span class="p">::</span><span class="n">Return</span><span class="o">></span> <span class="p">{</span>
<span class="k">match</span> <span class="k">self</span><span class="py">.state</span> <span class="p">{</span>
<span class="mi">0</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">self</span><span class="py">.pad_1</span><span class="nf">.set</span><span class="p">(</span><span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]));</span>
<span class="k">self</span><span class="py">.pinned_1</span>
<span class="nf">.set</span><span class="p">(</span><span class="nf">read_to_end</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="o">*</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">()));</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">self</span><span class="nf">.resume</span><span class="p">()</span>
<span class="p">}</span>
<span class="mi">1</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">self</span><span class="py">.data_2</span><span class="nf">.set</span><span class="p">(</span><span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.get_mut</span><span class="p">()),</span>
<span class="p">)</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Yielded</span><span class="p">(());</span>
<span class="p">});</span>
<span class="k">self</span><span class="py">.pinned_2</span>
<span class="nf">.set</span><span class="p">(</span><span class="nf">read_to_end</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="o">*</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">()));</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
<span class="k">self</span><span class="nf">.resume</span><span class="p">()</span>
<span class="p">}</span>
<span class="mi">2</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">pad_2</span> <span class="o">=</span> <span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.get_mut</span><span class="p">()),</span>
<span class="p">)</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Yielded</span><span class="p">(());</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">read</span><span class="p">(</span><span class="k">self</span><span class="py">.data_2</span><span class="nf">.as_mut_ptr</span><span class="p">())</span>
<span class="nf">.into_iter</span><span class="p">()</span>
<span class="nf">.zip</span><span class="p">(</span><span class="n">pad_2</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span>
<span class="nf">.collect</span><span class="p">();</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="err">-</span><span class="mi">1</span><span class="p">;</span>
<span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Complete</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="p">}</span>
<span class="err">-</span><span class="mi">1</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled after completion"</span><span class="p">),</span>
<span class="err">-</span><span class="mi">2</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled after dropped"</span><span class="p">),</span>
<span class="n">_</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled with invalid state"</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">Drop</span> <span class="k">for</span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">fn</span> <span class="k">drop</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">match</span> <span class="k">self</span><span class="py">.state</span> <span class="p">{</span>
<span class="mi">0</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="mi">1</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="mi">2</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="err">-</span><span class="mi">1</span> <span class="k">=></span> <span class="p">{</span> <span class="cm">/* Everything already dropped in resume */</span> <span class="p">}</span>
<span class="err">-</span><span class="mi">2</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator dropped twice"</span><span class="p">),</span>
<span class="n">_</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator dropped with invalid state"</span><span class="p">),</span>
<span class="p">}</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="err">-</span><span class="mi">2</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="n">Self</span> <span class="p">{</span>
<span class="n">ManualGenerator</span> <span class="p">{</span>
<span class="n">state</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="n">data_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">pad_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">pinned_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">data_2</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">pinned_2</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a class="icon play" style="display: none" target="_blank" href="#" title="Run on playground"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M424.4 214.7L72.4 6.6C43.8-10.3 0 6.1 0 47.9V464c0 37.5 40.7 60.1 72.4 41.3l352-208c31.4-18.5 31.5-64.1 0-82.6z"></path></svg></a></p>
<script type="text/rust">#![feature(
async_await,
await_macro,
existential_type,
futures_api,
gen_future,
generator_trait,
generators,
maybe_uninit,
pin
)]
pub mod io {
use core::future::Future;
pub struct AsyncRead(Vec<u8>);
impl AsyncRead {
pub fn new(data: Vec<u8>) -> AsyncRead {
AsyncRead(data)
}
pub fn read_to_end(&mut self) -> impl Future<Output = Vec<u8>> + '_ {
async move { self.0.clone() }
}
}
}
pub mod executor {
use core::{
future::Future,
pin::Pin,
ptr::NonNull,
task::{LocalWaker, Poll, UnsafeWake, Waker},
};
struct NoWake;
impl NoWake {
fn local_waker() -> LocalWaker {
// Safety: all references to NoWake are never dereferenced
unsafe { LocalWaker::new(NonNull::<NoWake>::dangling()) }
}
}
unsafe impl UnsafeWake for NoWake {
unsafe fn clone_raw(&self) -> Waker {
NoWake::local_waker().into_waker()
}
unsafe fn drop_raw(&self) {}
unsafe fn wake(&self) {}
}
pub fn block_on<F: Future>(mut future: F) -> F::Output {
let lw = NoWake::local_waker();
loop {
// Safety: `future` is a local variable which is only ever used in this
// pinned reference
match unsafe { Pin::new_unchecked(&mut future) }.poll(&lw) {
Poll::Ready(value) => break value,
Poll::Pending => continue,
}
}
}
}
use self::io::AsyncRead;
use core::future::Future;
pub fn quote_encrypt_unquote(data: &mut AsyncRead) -> impl Future<Output = Vec<u8>> + '_ {
use core::{
mem::MaybeUninit,
ops::{Generator, GeneratorState},
pin::Pin,
ptr,
task::Poll,
};
use std::future::{from_generator, poll_with_tls_waker};
from_generator({
existential type ReadToEnd<'a>: Future<Output = Vec<u8>> + 'a;
fn read_to_end(read: &mut AsyncRead) -> ReadToEnd<'_> {
read.read_to_end()
}
struct ManualGenerator<'a> {
state: i32,
data_1: MaybeUninit<&'a mut AsyncRead>,
pad_1: MaybeUninit<AsyncRead>,
pinned_1: MaybeUninit<ReadToEnd<'a>>,
data_2: MaybeUninit<Vec<u8>>,
pinned_2: MaybeUninit<ReadToEnd<'a>>,
}
impl<'a> Generator for ManualGenerator<'a> {
type Yield = ();
type Return = Vec<u8>;
unsafe fn resume(&mut self) -> GeneratorState<Self::Yield, Self::Return> {
match self.state {
0 => {
// one-time-pad chosen by fair dice roll
self.pad_1.set(AsyncRead::new(vec![4; 32]));
self.pinned_1
.set(read_to_end(&mut *self.data_1.as_mut_ptr()));
self.state = 1;
self.resume()
}
1 => {
self.data_2.set(loop {
if let Poll::Ready(x) =
poll_with_tls_waker(Pin::new_unchecked(self.pinned_1.get_mut()))
{
break x;
}
return GeneratorState::Yielded(());
});
self.pinned_2
.set(read_to_end(&mut *self.pad_1.as_mut_ptr()));
ptr::drop_in_place(self.pinned_1.as_mut_ptr());
self.state = 2;
self.resume()
}
2 => {
let pad_2 = loop {
if let Poll::Ready(x) =
poll_with_tls_waker(Pin::new_unchecked(self.pinned_2.get_mut()))
{
break x;
}
return GeneratorState::Yielded(());
};
let result = ptr::read(self.data_2.as_mut_ptr())
.into_iter()
.zip(pad_2)
.map(|(a, b)| a ^ b)
.collect();
ptr::drop_in_place(self.pinned_2.as_mut_ptr());
ptr::drop_in_place(self.pad_1.as_mut_ptr());
ptr::drop_in_place(self.data_1.as_mut_ptr());
self.state = -1;
GeneratorState::Complete(result)
}
-1 => panic!("ManualGenerator polled after completion"),
-2 => panic!("ManualGenerator polled after dropped"),
_ => panic!("ManualGenerator polled with invalid state"),
}
}
}
impl<'a> Drop for ManualGenerator<'a> {
fn drop(&mut self) {
match self.state {
0 => unsafe {
ptr::drop_in_place(self.data_1.as_mut_ptr());
},
1 => unsafe {
ptr::drop_in_place(self.pinned_1.as_mut_ptr());
ptr::drop_in_place(self.pad_1.as_mut_ptr());
ptr::drop_in_place(self.data_1.as_mut_ptr());
},
2 => unsafe {
ptr::drop_in_place(self.pinned_2.as_mut_ptr());
ptr::drop_in_place(self.data_2.as_mut_ptr());
ptr::drop_in_place(self.pad_1.as_mut_ptr());
ptr::drop_in_place(self.data_1.as_mut_ptr());
},
-1 => { /* Everything already dropped in resume */ }
-2 => panic!("ManualGenerator dropped twice"),
_ => panic!("ManualGenerator dropped with invalid state"),
}
self.state = -2;
}
}
impl<'a> ManualGenerator<'a> {
fn new(data: &'a mut AsyncRead) -> Self {
ManualGenerator {
state: 0,
data_1: MaybeUninit::new(data),
pad_1: MaybeUninit::uninitialized(),
pinned_1: MaybeUninit::uninitialized(),
data_2: MaybeUninit::uninitialized(),
pinned_2: MaybeUninit::uninitialized(),
}
}
}
ManualGenerator::new(data)
})
}
fn main() {
let mut data = AsyncRead::new("hello".into());
let encrypted = executor::block_on(quote_encrypt_unquote(&mut data));
println!("Encrypted: {}", core::str::from_utf8(&encrypted).unwrap());
}</script>
<script type="text/javascript">(() => {
let me = document.currentScript
let playground = me.previousElementSibling
let linkContainer = playground.previousElementSibling
let link = linkContainer.children[0]
let snippet = linkContainer.previousElementSibling.children[0].children[0]
link.href = `https://play.rust-lang.org/?version=nightly&edition=2018&code=${encodeURIComponent(playground.text)}`
link.style.display = ''
snippet.style = 'position: relative;'
snippet.prepend(link)
})()
</script>
<p>(Unfortunately this playground is too large to be opened via URI, you can find
the fully running example for this <a href="https://github.com/Nemo157/blag.nemo157.com/blob/master/_posts/_async-transform/playgrounds/manual-generator.rs">on
GitHub</a>.)</p>
<h3 id="readtoend"><code class="highlighter-rouge">ReadToEnd</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">existential</span> <span class="k">type</span> <span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span><span class="p">:</span> <span class="n">Future</span><span class="o"><</span><span class="n">Output</span> <span class="o">=</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'a</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">read_to_end</span><span class="p">(</span><span class="n">read</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span> <span class="p">{</span>
<span class="n">read</span><span class="nf">.read_to_end</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The very first part of this code block is a little hack to give a name to an
unnameable type. If you look back at <code class="highlighter-rouge">AsyncRead::read_to_end</code> you’ll see that
the return type was declared as <code class="highlighter-rouge">impl Future<Output = Vec<u8>> + '_</code>.
Unfortunately we cannot easily store this type into the fields of a struct, so
we have this little hack using the <code class="highlighter-rouge">existential_type</code> feature to give ourselves
a name for the type.</p>
<h3 id="struct-manualgenerator"><code class="highlighter-rouge">struct ManualGenerator</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="n">state</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span>
<span class="n">data_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">AsyncRead</span><span class="o">></span><span class="p">,</span>
<span class="n">pad_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">AsyncRead</span><span class="o">></span><span class="p">,</span>
<span class="n">pinned_1</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">>></span><span class="p">,</span>
<span class="n">data_2</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span><span class="p">,</span>
<span class="n">pinned_2</span><span class="p">:</span> <span class="n">MaybeUninit</span><span class="o"><</span><span class="n">ReadToEnd</span><span class="o"><</span><span class="nv">'a</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Next we have the environment definition for the generator. This includes a
<code class="highlighter-rouge">state</code> variable to keep track of which yield point we are currently at, along
with fields for the upvars that were moved into the environment (<code class="highlighter-rouge">data_1</code>) and
the variables that live across yield points (<code class="highlighter-rouge">pad_1</code>, <code class="highlighter-rouge">pinned_1</code>, <code class="highlighter-rouge">data_2</code> and
<code class="highlighter-rouge">pinned_2</code>). Note that any variables that are only alive <em>between</em> yield points
are not stored in the environment, these will be normal variables on the stack
of the <code class="highlighter-rouge">resume</code> function (the final <code class="highlighter-rouge">pad</code> result variable, and the temporaries
used during polling and the final line).</p>
<p>Each of the fields are stored as <code class="highlighter-rouge">MaybeUninit</code> as some of them start
uninitialized and others will be dropped before the generator finishes. You
might already be able to notice one potential optimization here, <code class="highlighter-rouge">pinned_1</code> and
<code class="highlighter-rouge">pinned_2</code> contain the same type but have non-overlapping lifetimes. To keep
closer to the real transform I have kept these as separate fields, see
<a href="https://github.com/rust-lang/rust/issues/52924">rust-lang/rust#52924</a> for more details.</p>
<h3 id="state-0">State <code class="highlighter-rouge">0</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mi">0</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// one-time-pad chosen by fair dice roll</span>
<span class="k">self</span><span class="py">.pad_1</span><span class="nf">.set</span><span class="p">(</span><span class="nn">AsyncRead</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="mi">4</span><span class="p">;</span> <span class="mi">32</span><span class="p">]));</span>
<span class="k">self</span><span class="py">.pinned_1</span>
<span class="nf">.set</span><span class="p">(</span><span class="nf">read_to_end</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="o">*</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">()));</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">self</span><span class="nf">.resume</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now we get into the meat of the generator. First we have the code running from
the start of the closure until the beginning of the first <code class="highlighter-rouge">loop</code>. We don’t go
all the way to the yield point because it is a yield point at the end of a loop,
we need a state number representing the start of the loop so we can resume there
after yielding. In the real transform this would end with a <code class="highlighter-rouge">goto</code> to the start
of the loop, but it’s easier in the Rust surface syntax to just recurse into
<code class="highlighter-rouge">resume</code> again (we can’t stack overflow as we guarantee moving to a new state
before recursing and have a limited number of states).</p>
<p>This is a pretty straight-forward transform from the original code, just note
that we use <code class="highlighter-rouge">MaybeUninit::set</code> to store the variables into the environment
instead of <code class="highlighter-rouge">=</code> and call our hacky <code class="highlighter-rouge">AsyncRead::read_to_end</code> wrapper so that we
know the type of the result.</p>
<h3 id="state-1">State <code class="highlighter-rouge">1</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mi">1</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">self</span><span class="py">.data_2</span><span class="nf">.set</span><span class="p">(</span><span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.get_mut</span><span class="p">()),</span>
<span class="p">)</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Yielded</span><span class="p">(());</span>
<span class="p">});</span>
<span class="k">self</span><span class="py">.pinned_2</span>
<span class="nf">.set</span><span class="p">(</span><span class="nf">read_to_end</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="o">*</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">()));</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
<span class="k">self</span><span class="nf">.resume</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>State <code class="highlighter-rouge">1</code> covers the body of the first loop and the small bit of code between it
and the start of the second loop. Here we can see the same sort of
straightforward transform as the initial state, with two new things to
highlight:</p>
<ol>
<li>
<p>The <code class="highlighter-rouge">yield;</code> statement is simply replaced with <code class="highlighter-rouge">return
GeneratorState::Yielded(());</code>. Because the yield is at the end of the loop
the next statement to execute after resuming is the start of the loop again
so this is relatively simple. In a more complicated generator where there is
a yield in the middle of a loop you would need to split the body across
multiple states and bounce between them until complete.</p>
</li>
<li>
<p>The <code class="highlighter-rouge">pinned</code> variable is dropped at the end of its containing block. We have
to use <code class="highlighter-rouge">drop_in_place</code> here to avoid moving it before dropping, it doesn’t
matter for this variable but if we were awaiting some <code class="highlighter-rouge">!Unpin</code> futures here
then we would cause unsoundness if they were moved before <code class="highlighter-rouge">Drop::drop</code> was
called.</p>
</li>
</ol>
<h3 id="state-2">State <code class="highlighter-rouge">2</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mi">2</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">pad_2</span> <span class="o">=</span> <span class="k">loop</span> <span class="p">{</span>
<span class="k">if</span> <span class="k">let</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">=</span> <span class="nf">poll_with_tls_waker</span><span class="p">(</span>
<span class="nn">Pin</span><span class="p">::</span><span class="nf">new_unchecked</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.get_mut</span><span class="p">()),</span>
<span class="p">)</span> <span class="p">{</span>
<span class="k">break</span> <span class="n">x</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Yielded</span><span class="p">(());</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">read</span><span class="p">(</span><span class="k">self</span><span class="py">.data_2</span><span class="nf">.as_mut_ptr</span><span class="p">())</span>
<span class="nf">.into_iter</span><span class="p">()</span>
<span class="nf">.zip</span><span class="p">(</span><span class="n">pad_2</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)|</span> <span class="n">a</span> <span class="o">^</span> <span class="n">b</span><span class="p">)</span>
<span class="nf">.collect</span><span class="p">();</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="err">-</span><span class="mi">1</span><span class="p">;</span>
<span class="nn">GeneratorState</span><span class="p">::</span><span class="nf">Complete</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Another straightforward transform for the body of the second loop, final
statement, and implicit drops of all the closure variables. This is the first
time we use a variable by value, we have to use <code class="highlighter-rouge">ptr::read</code> to move the final
<code class="highlighter-rouge">data</code> variable out to call <code class="highlighter-rouge">into_iter</code> on it, since we have passed ownership of
it off to <code class="highlighter-rouge">Vec::into_iter</code> we don’t drop this variable, but do drop everything
else remaining alive before returning the final result.</p>
<h3 id="error-states">Error states</h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">-</span><span class="mi">1</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled after completion"</span><span class="p">),</span>
<span class="err">-</span><span class="mi">2</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled after dropped"</span><span class="p">),</span>
<span class="n">_</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator polled with invalid state"</span><span class="p">),</span>
</code></pre></div></div>
<p>We reserve two states for all generators: the generator has completed
successfully and the generator has been dropped. A generator should never be
polled after either of these states is reached, and should never be able to get
into any state we aren’t otherwise handling, so we panic if these are reached.</p>
<h3 id="drop"><code class="highlighter-rouge">Drop</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">Drop</span> <span class="k">for</span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">fn</span> <span class="k">drop</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">match</span> <span class="k">self</span><span class="py">.state</span> <span class="p">{</span>
<span class="mi">0</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="mi">1</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="mi">2</span> <span class="k">=></span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pinned_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_2</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.pad_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="k">self</span><span class="py">.data_1</span><span class="nf">.as_mut_ptr</span><span class="p">());</span>
<span class="p">},</span>
<span class="err">-</span><span class="mi">1</span> <span class="k">=></span> <span class="p">{</span> <span class="cm">/* Everything already dropped in resume */</span> <span class="p">}</span>
<span class="err">-</span><span class="mi">2</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator dropped twice"</span><span class="p">),</span>
<span class="n">_</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"ManualGenerator dropped with invalid state"</span><span class="p">),</span>
<span class="p">}</span>
<span class="k">self</span><span class="py">.state</span> <span class="o">=</span> <span class="err">-</span><span class="mi">2</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since we are using <code class="highlighter-rouge">MaybeUninit</code> for the fields we also need to implement <code class="highlighter-rouge">Drop</code>
for our generator manually, the compiler generated drop glue will not do
anything. We need to then check what state we are in and drop all the
variables that are still alive, before finally marking ourselves as dropped.</p>
<h3 id="manualgeneratornew"><code class="highlighter-rouge">ManualGenerator::new</code></h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">ManualGenerator</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="n">AsyncRead</span><span class="p">)</span> <span class="k">-></span> <span class="n">Self</span> <span class="p">{</span>
<span class="n">ManualGenerator</span> <span class="p">{</span>
<span class="n">state</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="n">data_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">data</span><span class="p">),</span>
<span class="n">pad_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">pinned_1</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">data_2</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="n">pinned_2</span><span class="p">:</span> <span class="nn">MaybeUninit</span><span class="p">::</span><span class="nf">uninitialized</span><span class="p">(),</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Finally we need a way to construct the generator from the upvars, this doesn’t
really get pulled out to a function in the real transform, but it’s easier to
see what’s happening if it is here.</p>Update: Clarified that C# and JavaScript can be understood as similar to a CPS transform, not that they are actually implemented as one. As you likely know if you’re reading this post Rust has an upcoming async/await feature being tested in nightly. Because of Rust’s unique features and positioning fully understanding the implementation powering this syntax is very different to understanding other well-known implementations (C# and JavaScript’s being the ones I am familiar with). Instead of thinking of a CPS-like transform where an async function is split into a series of continuations that are chained together via a Future::then method, Rust instead uses a generator/coroutine transform to turn the function into a state machine (C# and probably most JavaScript implementations use a similar transform under the hood, but as far as I’m aware because of the garbage collector these are indistinguishable from the naive CPS transform they are normally described as). For more detail on why Rust is taking this approach you should read eRFC 2033: Experimental Coroutines, that lays out the why’s much better than I could here. What I’m going to try and provide instead, is a look into how this actually works today. What steps the compiler takes to turn an async fn into a normal function returning a state machine that you could write if you wanted to (but you definitely don’t).