Asynchronous Programming in Rust

Based on "Asynchronous Programming in Rust: Learn asynchronous programming by building working examples of futures, green threads, and runtimes" by Carl Fredrik Samson

Additional Resources:

• Phil Opp - async rust

Chapter 1: Introduction to Multitasking

Multitasking Models

• Cooperative – Tasks voluntarily yield the CPU.
• Preemptive – CPU arbitrarily preempts tasks.

Core Concepts

• Concurrency: Managing multiple activities logically at the same time.
• Parallelism: Executing multiple activities physically at the same time.

Concurrency = working smarter; Parallelism = throwing more resources at the problem.

When to Use Concurrency (Async)

Ideal for situations where resources would otherwise sit idle while waiting.

Typical patterns:

1. I/O-bound work – Network calls, DB queries, etc.; keep the thread free, poll or await a notification.
2. Responsiveness – Prevent any single task from blocking the overall flow; interleave work so no task stalls too long.

When Concurrency Adds Little Value

CPU-bound, compute-heavy tasks that continuously use the processor give little gain from async alone.

When Parallelism Helps

• The problem can be partitioned into independent parts.
• Deploy multiple cores/threads to "throw more resources" at those parts, achieving real simultaneous execution.

Communicating with the Operating System

There is no way for you as a programmer to make your system optimally efficient without playing to the strengths of the operating system. You basically don't have access to the hardware directly. You must remember that the operating system is an abstraction over the hardware.

Communication with an operating system happens through what we call a system call (syscall). Most of the time, these calls are abstracted away for us as programmers by the language or the runtime we use.

A syscall is an example of something that is unique to the kernel you're communicating with, but the UNIX family of kernels has many similarities. UNIX systems expose this through libc.

CPU ↔ Memory

Most modern CPUs have a memory management unit (MMU). The MMU's job is to translate the virtual address we use in our programs to a physical address.

When the OS starts a process (such as our program), it sets up a page table for our process and makes sure a special register on the CPU points to this page table.

Most modern desktop CPUs are built with an OS in mind, so they provide the hooks and infrastructure that the OS latches onto upon bootup. When the OS spawns a process, it also sets its privilege level, making sure that normal processes stay within the borders it defines to maintain stability and security.

Yielding

In asynchronous programming, yielding is the act of a task voluntarily giving up control of the CPU when it can no longer make progress (e.g., waiting for a network response).

Chapter 2: How Programming Languages Model Asynchronous Program Flow

This chapter explores how different programming languages and libraries abstract asynchronous operations to provide programmers with efficient, expressive, and easy-to-use models for non-sequential tasks.

Threading Models

OS Threads (1:1 Threading)

• Kernel Managed: These threads are created and scheduled directly by the operating system.
• Resource Heavy: Each thread typically requires a large, fixed-size stack (often 8 MiB), which can lead to memory exhaustion if thousands of tasks are needed.
• Overhead: While they provide "parallelism for free" (true parallelism), they incur significant costs due to the time taken to create/discard threads and the performance hit of context switching.
• Context Switching: Switching between threads in different processes is particularly expensive because the OS must flush caches and switch address spaces to ensure security.

Green Threads / Fibers (M:N Threading)

• Runtime Managed: These are user-level threads handled by a language runtime rather than the kernel.
• Efficiency: Many "green" tasks (M) share a smaller pool of OS threads (N), allowing a system to run with only one OS thread if necessary.
• Implementation: They mimic OS threads by setting up a stack for each task and performing context switches by saving/restoring CPU registers.
• Flexibility: They are highly flexible and can support preemption, where the runtime stops a long-running task to allow others to progress.

Key Concepts

async fn

Syntax for defining asynchronous computations. An async fn does not execute immediately; instead, it returns a Future.

Task

A lightweight, non-blocking unit of execution. A task is similar to an OS thread, but rather than being managed by the OS scheduler, they are managed by the Tokio runtime (or other runtime). Another name for this general pattern is green threads, or rather, a task is "an instance of the green-thread model."

Executor (Runtime)

Component responsible for scheduling and polling tasks. In Tokio, the runtime maintains a pool of OS threads and schedules many tasks onto them, driving each task's underlying Future to completion.

Coroutine Implementation Models

A coroutine is a function that can pause its execution at certain points, hand control back to the caller (or a scheduler), and later resume exactly where it left off (anywhere - stackful, .await - stackless), keeping its local state alive between pauses.

Stackful Coroutines

• Dedicated Stack: Examples include fibers and green threads.
• Full Suspension: Because they own a complete call stack, execution can be suspended at any point in the program, including deep within nested function calls, while preserving the full state.
• Drawbacks: They require complex strategies for growable or segmented stacks to remain efficient, and calling foreign functions (FFI) can be difficult and slow.

Stackless Coroutines

• Shared Stack: They do not have their own stacks and must share the caller's stack.
• State Machines: They are implemented as state machines where the compiler saves only the minimal variables needed to resume execution at specific suspension points.
• Optimized Memory: Rust's implementation is highly memory-efficient; the size of a coroutine is determined by its largest single state rather than the total number of variables.
• Yield Points: They can only be suspended at predefined points (.await).

Asynchronous Abstractions

Callbacks

These store a pointer or closure to be executed after an operation finishes. While they have no context-switching overhead, they often lead to "callback hell" and make managing ownership and state sharing extremely difficult in languages like Rust that lack a garbage collector.

Promises / Futures

Lazy, state-machine representation of an asynchronous computation. A Future does nothing until it is polled by an executor. When polled, it either completes (Ready) or indicates it cannot yet make progress (Pending) or Rejected.

Async/Await

This syntax allows asynchronous code to look like sequential code. The compiler programmatically rewrites async functions into state machines; await points are where control is yielded to a scheduler.

The Mechanics of "Yielding to a Scheduler"

In asynchronous programming, yielding is the core of cooperative multitasking.

• Voluntary Control: A task voluntarily gives up control of the CPU when it reaches a point where it cannot progress (e.g., waiting for I/O).
• The Scheduler's Role: The scheduler acts as the "administrator" of tasks, deciding which task gets CPU time and when.
• Resume Mechanism: When a task yields and returns PollState::NotReady, it is moved to a pending queue. The scheduler then immediately starts polling a different ready task.

Practical Considerations

• Creation Overhead: Spawning OS threads is slow due to stack allocation and kernel bookkeeping; for many short-lived I/O tasks, this becomes a major bottleneck.
• Context Switching Costs: OS context switches involve saving/restoring registers and potentially flushing CPU caches, which can limit throughput in highly concurrent systems.
• Workload Suitability: For highly concurrent, I/O-bound workloads, green threads or stackless coroutines are generally superior as they reduce memory footprints and context-switching overhead.

Chapter 3: Understanding OS-Backed Event Queues, System Calls, and Cross-Platform Abstractions

How sockets, system calls, and event queues connect your code to the operating system and physical hardware.

Hardware (disk, network card, etc.)
        ↓
OS I/O event system (epoll / kqueue / IOCP)
        ↓
Async runtime event loop (Tokio reactor)
        ↓
Executor (task scheduler)
        ↓
Tasks (scheduled Futures)
        ↓
Future::poll()

Interacting with the Operating System

All interactions with the host operating system are performed through system calls (syscalls). Because the operating system is an abstraction over hardware, you do not have direct access to physical devices like network cards. To make these syscalls using Rust, you must use the language's Foreign Function Interface (FFI) to call external functions. On Linux and macOS, the system provides a version of libc, a C library that offers a consistent API for communicating with the OS regardless of the underlying platform architecture.

Sockets and Identifiers

A socket is an OS-provided abstraction that allows your program to interact with network hardware. When you request a socket via a syscall, the OS returns a platform-specific identifier used to track that communication channel:

• File Descriptors: On Unix-like systems (Linux and macOS), almost all I/O is modelled as a "file," so the socket identifier is an integer called a file descriptor.
• Socket Handles: On Windows, these identifiers are referred to as socket handles.

Efficiency and OS-Backed Event Queues

I/O operations depend on resources that the OS abstracts over, and these operations often involve significant delays (e.g., waiting for a remote server).

Avoiding Blocking Calls

Standard blocking syscalls suspend the current OS thread, storing its state while it sits idle until data arrives. To maximize efficiency, asynchronous runtimes avoid these and use non-blocking calls, which allow a thread to continue working after requesting an operation.

Event Queues

To track thousands of non-blocking operations efficiently, runtimes use OS-backed event queues. These queues allow the runtime to yield to the OS scheduler only when there is no work left to do, asking the OS to wake it when a specific event is ready.

Types of Event Queues

Different operating systems handle event notification in two primary ways:

• Readiness-based (epoll on Linux, kqueue on macOS): These queues notify you when an action is ready to be performed, such as when a socket has data waiting to be read.
• Completion-based (IOCP on Windows): The Input/Output Completion Port (IOCP) notifies you when an action has been completed, such as when the OS has already finished reading data into a memory buffer you provided.

The Hardware Connection

• Firmware Monitoring: A microcontroller on the network card runs firmware that polls for incoming electrical signals.
• Hardware Interrupts: When data arrives, the card sends a hardware interrupt (an electrical signal) to the CPU, which immediately stops its normal workflow to run an interrupt handler.
• Direct Memory Access (DMA): Modern hardware uses DMA to write data directly into main memory buffers set up by the OS. The CPU is only notified once the data is already in memory, making the process extremely efficient.
• Kernel Notification: The interrupt handler (managed by the device driver) informs the kernel that data is ready. The OS then updates the event queue (epoll, kqueue, or IOCP), which wakes your thread and allows the asynchronous runtime to progress the relevant task.

Clarification: Two Kinds of Polling

1. I/O event polling → OS-level (epoll)
2. Future polling → calling Future::poll()

Event polling wakes tasks; task polling drives futures.

Chapter 4: Create Your Own Event Queue

The primary focus of Chapter 4 is building a simple event queue using epoll (a Linux-specific system call) to understand the low-level mechanics of libraries like mio, which underpins much of the Rust async ecosystem.

Design Abstractions

The implementation mimics the mio API by dividing the event queue into two primary components:

• Poll: A struct representing the event queue itself. It contains the poll method, which parks the current thread and waits for the OS to report events.
• Registry: A handle used to register interest in specific events (like a socket becoming readable). By separating Registry from Poll, the design allows multiple threads to register events while a single thread remains blocked on the queue.

Core epoll System Calls

To communicate with the Linux kernel, the chapter defines four essential functions via Foreign Function Interface (FFI):

• epoll_create: Creates an epoll instance and returns a file descriptor.
• epoll_ctl: The control interface used to add, modify, or delete interests for specific file descriptors in the queue.
• epoll_wait: Blocks the calling thread until a tracked event occurs or a timeout is reached.
• close: Ensures the epoll file descriptor is properly released when finished.

Event Notification Modes

Understanding how the OS notifies the program about events is critical for avoiding bugs:

Level-triggered (Default)

An event is reported as long as the condition is true (e.g., there is data in a buffer). If the buffer isn't drained, the OS will notify the program repeatedly for the same event.

Edge-triggered (EPOLLET)

An event is reported only when the state changes (e.g., a buffer goes from empty to having data). This mode is more efficient but requires the programmer to fully drain the buffer; otherwise, they will never receive another notification for that source.

EPOLLONESHOT

A flag that disables a file descriptor after a single notification. This prevents the "thundering herd" problem where multiple threads might be woken up to handle the same event.

Implementation Details

• The Event Struct: Must be marked with #[repr(C, packed)]. This ensures the data structure matches the exact memory layout the Linux kernel expects, without any padding added by the Rust compiler.
• Bitmasks: Most system calls use bitflags (like 0x1 for EPOLLIN) combined via the OR (|) operator to send multiple configuration options in a single integer.
• Error Handling: Non-blocking operations often return a WouldBlock error. This is not a failure but a signal that the operation cannot progress right now, and the program should wait for a notification from the event queue.

Key Insight

Event polling is what makes async non-blocking.

Without it:

• Every task would need a blocking thread.
• You'd lose scalability.

With event polling:

• A few OS threads can handle thousands of tasks.
• Threads sleep only in epoll_wait, not per connection.

Chapter 6: Futures

A future is a representation of some operation that will be completed in the future.

The Poll-Based Approach

Async in Rust uses a poll-based approach in which an asynchronous task will have three phases:

1. The Poll Phase: A future is polled, which results in the task progressing until a point where it can no longer make progress. We often refer to the part of the runtime that polls a future as an executor.
2. The Wait Phase: An event source, most often referred to as a reactor, registers that a future is waiting for an event to happen and makes sure that it will wake the future when that event is ready.
3. The Wake Phase: The event happens and the future is woken up. It's now up to the executor that polled the future in step 1 to schedule the future to be polled again and make further progress until it completes or reaches a new point where it can't make further progress and the cycle repeats.

Leaf Futures vs Non-Leaf Futures

Leaf Futures

Leaf futures represent a resource such as a socket.

Example:

let mut stream = tokio::net::TcpStream::connect("127.0.0.1:3000");

Operations on these resources, such as reading from a socket, will be non-blocking and return a future, which we call a leaf future since it's the future that we're actually waiting on. (It's unlikely that you'll implement a leaf future yourself unless you're writing a runtime.)

Non-Leaf Futures

Non-leaf futures are the kind of futures we as users of a runtime write ourselves using the async keyword to create a task that can be run on the executor.

Mental Model of Async Runtime in Rust

A fully working async system in Rust can be divided into three parts:

• Reactor (responsible for notifying about I/O events)
• Executor (scheduler)
• Future (a task that can stop and resume at specific points)

Futures Execution Notes

1. Executor (Polling Phase)

• Holds a list of Futures.
• Polls a future and provides a Waker.
• The future returns:
  • Poll::Ready → finished.
  • Poll::Pending → not ready yet.
• After each result, the executor can poll another future.
• These hand-off points are yield points.

2. Reactor (Event Monitoring)

• Stores the Waker passed during polling.
• Monitors I/O events via an event queue (e.g., via epoll).
• Associates each event source with its Waker.

3. Wake-Up Mechanism

• When an I/O event occurs, the reactor calls Waker::wake().
• This notifies the executor to poll the future again (future is ready to make progress).

4. await and Yielding

async fn foo() {
    println!("Start!");
    let txt = io::read_to_string().await.unwrap();
    println!("{txt}");
}

• The await point yields control.
• It returns Poll::Pending (usually the first time) or Poll::Ready.

5. Design Properties

• Waker is executor-agnostic.
• Executors and reactors do not communicate directly.
• This design yields a flexible, zero-cost async abstraction.

I/O vs. CPU-Intensive Tasks in Async Rust

Non-Leaf Futures

Most async code you write is non-leaf: it contains await points that hand control back to the executor.

Example:

let non_leaf = async {
    let mut stream = TcpStream::connect("127.0.0.1:3000").await.unwrap(); // ← yield
    stream.write(get_dataset_request).await.unwrap();                 // ← yield
    let mut response = vec![];
    stream.read(&mut response).await.unwrap();                       // ← yield
    // CPU-intensive work
    let report = analyzer::analyze_data(response).unwrap();         // ← **no yield**
    stream.write(report).await.unwrap();                            // ← yield
};

• Yield points (await) return control to the runtime executor.
• Code between yield points runs on the same thread as the executor.

The Problem

The CPU-heavy analyzer::analyze_data blocks the executor thread. While it runs, the executor cannot poll other futures or accept new connections.

Typical Solutions

1. Spawn a Leaf Future (Thread Pool) — Most Common

   • Use spawn_blocking (or an equivalent API).
   • Executes the CPU-bound work on a dedicated thread from the runtime's thread pool.
   • Returns a future that resolves when the work finishes.
   • This is the standard approach in most async runtimes.

2. Supervisor + Executor Migration — Runtime-Dependent

   • The runtime monitors how long each task runs.
   • If a task blocks, the executor is moved to another thread so it can keep polling other futures.
   • Requires built-in support from the runtime; not all runtimes provide it.

3. Custom Reactor / Manual Thread Handling — Rare

   • Implement your own reactor that integrates with the runtime.
   • Perform the heavy computation manually (e.g., spawn separate OS threads).
   • Return an awaitable future that completes when the computation is done.
   • Mostly of academic or highly specialized interest.

Practical Recommendation

Most runtimes (Tokio, async-std, smol, etc.) provide spawn_blocking or similar APIs. Use it for any CPU-intensive or blocking operation that cannot be expressed as a non-blocking async call.

let report = tokio::task::spawn_blocking(move || {
    analyzer::analyze_data(response)
}).await.unwrap()?;

This sends the work to a runtime-managed thread pool, keeping the async executor free to handle I/O.

Takeaway

• Never perform long-running CPU work directly inside a non-leaf future; it blocks the executor.
• Offload such work to a dedicated thread (via spawn_blocking or equivalent).
• If you need a custom approach, ensure your runtime supports executor migration or build a compatible reactor.

Chapter 7: Async Function Transformation

As we'll see later, when your async functions are rewritten by the compiler, they're constructed in a way so that nothing you write in the body of an async function will execute before the first call to Future::poll.

async fn → coroutine that transforms Future into state machine.

Chapter 8: Introduction to Runtimes

Rust Runtime Essentials

Core Responsibilities

• Schedule tasks.
• Drive objects that implement the Future trait to completion.

Scheduling Control

• Taking over task scheduling is highly invasive.
• If you use a user-space scheduler, you generally cannot also rely on the OS scheduler without complex workarounds.
• Mixing the two can cause chaos and defeat the purpose of async programming.

Reactor and Executor

The part that tracks events we're waiting on and makes sure to wait on notifications from the OS in an efficient manner is often called the reactor or driver. It's an event loop, and in our case, it dispatches events to an executor.

The part that schedules tasks and polls them to completion is called the executor.

Since reactors will respond to events, they need some integration with the source of the event. If we continue using TcpStream as an example, something will call read or write on it, and at that point, the reactor needs to know that it should track certain events on that source.

For this reason, non-blocking I/O primitives and reactors need tight integration, and depending on how you look at it, the I/O primitives will either have to bring their own reactor or you'll have a reactor that provides I/O primitives such as sockets, ports, and streams.

The Waker Interface

The interface provided to signal the executor that it should wake up when an event that allows a future to progress has occurred is called Waker. It's no coincidence that this type is called Waker.

We have an Executor that schedules tasks and passes on a Waker when polling a future that eventually will be caught and stored by the Reactor. When the Reactor receives a notification that an event is ready, it locates the Waker associated with that task and calls Wake::wake on it.

This enables us to:

• Run several OS threads that each have their own executor, but share the same reactor.
• Have multiple reactors that handle different kinds of leaf futures and make sure to wake up the correct executor when it can progress.

Task Spawning

Often, you will pass in one top-level future first, and when the top-level future progresses, it will spawn new top-level futures onto our executor. Each new future can, of course, spawn new futures onto the Executor too, and that's how an asynchronous program basically works.

In many ways, you can view this first top-level future in the same way you view the main function in a normal Rust program. spawn is similar to thread::spawn, with the exception that the tasks stay on the same OS thread in this example. This means the tasks won't be able to run in parallel, which in turn allows us to avoid any need for synchronization between tasks to avoid data races.

Appendix: What Is Event Polling?

Event polling is how the runtime asks the OS:

"Tell me when this socket/file/timer is ready."

Platform-specific implementations:

• Linux: epoll
• macOS: kqueue
• Windows: IOCP

These APIs let the OS notify the runtime when I/O becomes ready.

How It Fits With Tasks and Futures

Let's walk through what happens when you do:

socket.read().await;

Step 1 — Task Polls the Future

The executor calls:

future.poll()

The socket is not ready yet. The future:

• Registers interest in the socket with epoll
• Stores a Waker
• Returns Poll::Pending

The task pauses.

Step 2 — Runtime Waits for Events

The Tokio runtime blocks on:

epoll_wait(...)

Now the OS watches the socket. No CPU wasted. No thread blocked per connection.

Step 3 — OS Signals Readiness

When data arrives:

• OS wakes the runtime
• Runtime finds which task was waiting
• Calls its stored Waker
• Task goes back into the run queue

Step 4 — Task Is Polled Again

Executor polls the future again:

future.poll()

Now the socket is ready. The future returns Ready(data). Task continues execution.

How Responsibilities Divide

Reactor + Executor Model (Tokio)

Tokio internally splits into:

• Reactor → handles event polling (epoll/kqueue)
• Executor → schedules tasks
• Scheduler → work stealing between threads

References

1. Tokio Task Documentation
2. Phil Opp - async rust
3. Samson, Carl Fredrik. Asynchronous Programming in Rust: Learn asynchronous programming by building working examples of futures, green threads, and runtimes.