For agents: start with the agent guide. Every page is Markdown by default; add Accept: text/html for HTML.

Queues, Pipelines, Workflows and K2 do different jobs

Each of these products moves work or data out of the request that started it. They differ in the unit they track. Choose by that unit, then check the delivery promise.¶

request ──┬─▶ Queue ──────▶ consumer: one job, ack or retry
          ├─▶ Pipeline ───▶ SQL ──▶ R2 table or files
          ├─▶ Workflow ───▶ step ─▶ step ─▶ sleep ─▶ step
          └─▶ K2 stream ──▶ subscription A (reads every record)
                       └──▶ subscription B (reads every record)

Queues track one job at a time

A queue holds messages for a consumer Worker⁶⁰. Cloudflare lists three uses: offload work from a request, send data from Worker to Worker, and buffer or batch writes. Storage options¶

• Delivery: at least once by default. Give each message a unique id and use it as an idempotency¹¹ key. Delivery guarantees¶
• Per message: a consumer can ack() or retry() each message in a batch. Batching and retries¶
• After retries: a message moves to a dead letter queue after max_retries, three by default. Without a dead letter queue, Cloudflare deletes it. Dead letter queues¶

Cloudflare’s rule: “Use Queues when each message is a unit of work that needs to be completed, retried, and tracked individually.” Storage options¶

In Alchemy, Cloudflare.Queues.Queue declares the queue and Cloudflare.Queues.Consumer attaches a Worker, with deadLetterQueue and settings.maxRetries. A Worker sends through Cloudflare.Queues.WriteQueue. Queue.ts Consumer.ts¶

See Dead letters need a return path¹⁵ for what the dead letter queue still needs: an owner and a redrive¹⁵.¶

Pipelines load rows into R2

Basin Pipelines is Cloudflare’s streaming ingestion product, and it has three parts. Alchemy names it Cloudflare.Pipelines. Basin Pipelines¶

1. A stream receives events over HTTP or a Worker binding²⁵.¶
2. A pipeline runs SQL over the stream.¶
3. A sink writes Iceberg tables to Basin Catalog, or Parquet and JSON files to R2.¶

Sinks deliver exactly once. Sinks Cloudflare lists clickstream analytics, telemetry⁸ and logs as uses. Storage options¶

The output is a table or a set of files in R2 that you query later.¶

How rat-stack uses it

Pipelines is the only one of the four that rat-stack declares. @rat-stack/basin builds the stream for both request analytics¹⁸ and join intake. The stream accepts no HTTP input, so only the Worker binding²⁵ writes to it.¶

export const basinStream = (name: string) =>
  Effect.gen(function* declareBasinStream() {
    const bucket = yield* Cloudflare.R2.Bucket(`${name}Bucket`, {}).pipe(
      retainInProd
    );

    const stream = yield* Cloudflare.Pipelines.Stream(`${name}Stream`, {
      http: { enabled: false },
    }).pipe(retainInProd);

    return { bucket, stream } as const;
  });

icebergSink then copies every row unchanged into one table:¶

    return yield* Cloudflare.Pipelines.Pipeline(`${name}Pipeline`, {
      sql: Output.interpolate`INSERT INTO ${sink.name} SELECT * FROM ${streamName}`,
    }).pipe(retainInProd);

Capture first⁸ explains why the copy stays raw.¶

Workflows run steps that resume

A Workflow chains steps, retries failed steps, and persists state for minutes, hours or weeks. It can sleep, and it can wait for an external event such as an approval. Workflows¶

Each step retries on its own, so make each step idempotent. Rules of Workflows A step that charges a card must first check whether the charge already happened.¶

In Alchemy, a Workflow body is an Effect³⁵. Cloudflare.Workflows.task runs one named durable step. Its result is persisted and replayed on retries. sleep, sleepUntil and waitForEvent pause the run. The same source describes Workflows as at-least-once. Workflow.ts¶

Choose a Workflow when the unit is a process with ordered steps and waits. Each instance has a unique ID and runs many steps. Rules of Workflows¶

K2 keeps a log that many readers replay

K2 is a durable log. Producers append records. Consumers read through subscriptions, and each subscription receives every record. K2 concepts¶

• Reading does not delete: records stay until retention ends. Retention goes up to 30 days, and longer by request. K2¶
• Replay: a new subscription can start at earliest and read records produced before it existed. Delivery guarantees¶
• Leases: a consumer holds a batch for five minutes, then acks, nacks, or loses it to redelivery. K2 concepts¶
• Limits of the promise: delivery is at least once per subscription. K2 acks whole batches, does not deduplicate, and does not order records across workers. Delivery guarantees¶

Cloudflare says to use K2 when what matters is “that every record is processed rather than the state of any one record.” Storage options¶

K2 is in public beta on the Workers Paid plan. K2 Alchemy declares it as Cloudflare.K2.Stream and Cloudflare.K2.Subscription. A stream can take an Effect Schema³⁰, so consumers receive decoded values. K2/Stream.ts¶

Choose by the unit you track

• One job that must finish: Queues. Track and retry each message.¶
• Rows you query later: Pipelines. Load them into R2 as a table.¶
• A process with steps and waits: Workflows. Resume from the last finished step.¶
• Events for several independent readers: K2. Replay them from the log.¶

Three of the four deliver at least once. Make every consumer and step safe to repeat. Retry needs idempotent effects¹⁵ shows how.¶

Where rat-stack does without them

Join intake must deliver each contact after the response. It uses no queue. It saves the contact with a pending-delivery marker, then adds its id to a sweep index in a separate call. A cron every five minutes redrives each pending id in that index.¶

That recovery has a documented gap. If the index write fails, the contact keeps its marker, but the sweep never finds it. Join intake spec The backstop works by id: rat-stack intake redeliver re-drives a delivery, and rat-stack intake erase re-drives an erase.¶

Durability¹⁰ shows the model tests that check it.¶

The marker plus its index entry is the recovery record. Cloudflare lists this job, offloading work from a request, as a Queues use. Storage options Request lifetime is not durability⁹ explains why a record is needed at all.¶

Next, read Durable work needs storage¹⁶, then the durability index³⁵.¶

Last changed 2026-11-08 in b8f4ccf. Source on GitHub. Change log.