Why Inference Is a System With a Short Memory

There’s an idea that feels very intuitive in computer systems: if we learn from the past - we’ll improve.

If something was slow before, we’ll adapt. If load rose, we’ll change behavior. If we saw a pattern - we’ll exploit it.

But in inference, this idea works against us.

Stable inference is a system with a short memory.

First, What Does “Memory” Mean Here

Not computer memory - behavioral memory.

That is:

• Decisions based on what happened before
• Adjustments that try to “learn” the system while it’s running
• Behavior changes based on recent history

It sounds smart. In practice - it’s dangerous.

History-Based Adjustments: A Common Temptation

Many mechanisms work like this:

• If there was load - lower the rate
• If it was quiet - raise the rate
• If requests took a long time - change parameters

The problem is that the system assumes one wrong thing: that the recent past represents the present.

And in inference - that’s almost never true.

Why “What Happened Before” Is Misleading

Inference operates in a changing world:

• Request types change
• Load arrives in waves
• Conditions change very quickly

What was true a second ago:

• Might be irrelevant now
• Or even harmful

An adjustment based on the past:

• Responds too late
• To conditions that already changed
• And creates unnecessary oscillation

Instead of stability, you get a seesaw.

The Analogy

Imagine driving using only the rearview mirror.

If there was a traffic jam a moment ago - you slow down. If the road was open a moment ago - you speed up.

But the road ahead keeps changing.

A good driver:

• Looks forward
• Responds to what’s there now
• And doesn’t try to “correct” what already passed

Inference needs to behave exactly this way.

Why Inference Should Respond Only to the Present

Inference happens in real time. It needs to:

• Respond to the current request
• To the current load
• And to the current state of the system

Not try to be too clever. Not try to guess trends. Not “remember” too much.

Long memory:

• Adds complexity
• Creates unpredictability
• And makes it harder to understand why the system behaves the way it does

Short Memory as a Foundation for Stability

A system with a short memory:

• Behaves consistently
• Responds immediately to what’s happening
• And doesn’t carry old decisions forward

It might sound less sophisticated - but it’s far more stable.

Summary

Inference isn’t meant to learn from the past. It’s meant to respond to the present.

The more a system “remembers,” the less predictable it tends to be.

And the more it focuses on what’s happening now - the simpler, calmer, and more stable it becomes.

In inference, short memory isn’t a weakness - it’s an advantage.