Buffer
Accumulate arriving rows in a staging dataset and emit them as one batch when flushed or a bound is hit.
Collects rows across runs and releases them as one batch.
items appends to the staging dataset and flush drains it. Whichever of maxItems and maxAgeS is reached first also flushes; a null bound is no bound. A non-empty dedupeKey turns the append into a merge on those columns. This is one of the two module types that keep state between runs, so it is durable only.
At a glance
Module type
Buffer
Category
Data Processing
Default tier
durable
Runs on
durable
Starts a flow
no
Holds a body
no
Keeps state between runs
yes
Available in
every application type
Parameters
dedupeKey
string[] · required · defaults to []
Non-empty turns the append into a merge on these columns.
maxItems
number | null · optional · defaults to null
Flush once this many rows are staged. Null is no bound.
maxAgeS
number | null · optional · defaults to null
Flush once the oldest staged row is this old. Null is no bound.
Inputs
on
event
no
Implicit on every module that does not itself start a flow: wire an event here to make this module the start of what that event runs.
items
dataframe
no
Rows to add to the batch; batch stays empty until a flush, but count still reports.
flush
event
no
Releases the batch now; an event port, so a schedule's event out-port wires straight in.
Outputs
batch
dataframe
dataframe, signal
Everything staged since the last flush.
count
scalar<number>
scalar, json, text, signal
Rows waiting, emitted on every append — not the size of the batch, which only a flush produces.
Last updated