Inference as a Flow, Not a Request - Why Language Models Don’t “Run,” They Behave

Opening - Why Thinking of Inference as a Request Is a Mistake

When we use a language model, our experience is very simple:

we ask a question - and get an answer.

From the user’s perspective, this looks like one action: input → output.

But behind the scenes, there isn’t a single event, but a long sequence of small operations.

The model doesn’t “know” the answer when it starts. It doesn’t hold a ready-made answer. It builds it - piece by piece.

Every word it generates relies on everything said before it. And every such word requires computation, time, and memory.

When there’s one question, this still feels simple.

But once there are many questions in parallel - that assumption breaks.

Inference stops being “running a model,” and becomes a system dealing with:

• Shared time
• Shared memory
• And limited attention

And that’s exactly the point where you need to change your way of thinking.

Part 1 - Streaming Inference: Why an Answer Is an Ongoing Process

Let’s start with the most basic thing: how a language model even generates an answer.

It doesn’t write a whole paragraph. It doesn’t “think” and then print.

It performs one action over and over:

1. Looks at what’s already been said
2. Computes what the next token is
3. Adds it to the sequence
4. And then goes back to step 1

In other words: the answer is created in real time.

Given that, a simple question arises: why wait until the end?

If the answer is built gradually - it can also be consumed gradually.

That’s the meaning of Streaming Inference: not seeing an answer as something that appears at the end, but as a stream that opens immediately.

Once you understand that, you also understand that the system doesn’t need to “finish” to be useful.

It just needs to start moving.

Part 2 - Partial Results: Why a Partial Answer Isn’t a Compromise

There’s a natural tendency to think: “either there’s an answer - or there isn’t.”

But in the world of Inference, that’s a false dichotomy.

Suppose there are two requests:

• One simple
• One very complex

If the system behaves so that every request must reach 100% to be considered “ready” - the short one will suffer because of the long one.

But if the system is willing to produce a partial result:

• The short one will progress fast
• And the long one will continue in the background

A partial answer isn’t half an answer. It’s an answer at an earlier time.

And in most real systems - time matters no less than content.

The choice to show a partial result is a systemic choice: preferring flow over completeness.

Part 3 - Incremental Decoding: Why Progressing Slowly Is an Advantage

Now let’s dig a bit deeper into how the system works internally.

You can imagine two working styles:

1. Accumulate a lot of work - then execute it all
2. Perform a little work - and repeat that action over and over

The first approach creates “jumps.” There’s a stretch of time when the model isn’t working because it’s accumulating work, and then a stretch when it works, and it repeats. The second approach creates a rhythm - the model works all the time, at a steady pace.

Incremental decoding is based on the second approach.

The system doesn’t wait for a “big” moment. It doesn’t say: “when there’s enough - we’ll start.”

It starts immediately, and keeps progressing in small steps.

The result is a system that’s:

• Easy to stop
• Easy to combine
• Easy to slow down or speed up

In other words: a manageable system.

Part 4 - Time-Sliced Execution: Why a Request Doesn’t Get to Run Alone

When many requests are alive together, the biggest danger is this:

one request takes over the system and doesn’t let go.

Time-slicing solves exactly this.

Instead of letting one request run to completion, the system divides time into small chunks.

Every request gets:

• A bit of time
• And then makes room for others

None of them “finish” immediately, but all of them progress.

This might sound less efficient - but in practice, it’s the only way to keep a system responsive.

Summary - Inference Isn’t an Action. It’s an Ongoing Behavior.

If we connect all the parts together, a clear picture emerges:

Good Inference isn’t measured by how fast it finishes, but by how well it behaves over time.

A system that thinks in terms of requests tries to close things.

A system that thinks in terms of flow tries to stay stable, attentive, and alive.

Closing line:

Inference isn’t meant “to end.” It’s meant to keep moving - without falling.