How a Small Idea Called a “Shortcut” Turned ResNet Into a Revolution

After seeing in the first post that deepening networks made them get confused, the simple question was: how can you add layers without losing stability? The answer turned out to be surprisingly simple.

The Shortcut - The Idea That Changed Everything

Instead of passing all information through every layer and forcing it to reshape it from scratch, ResNet proposed a subtle solution:

The original information doesn’t disappear - it flows forward and gets added back to the output of the new layers. In other words, the data travels along a “bypass path” and rejoins the flow at a later stage.

This is the famous shortcut - the Skip Connection. A new layer doesn’t have to “rewrite the information,” it can simply add a small improvement on top of it.

In practice, the meaning is simple: deep layers don’t destroy what already worked well in earlier stages.

Why Is This So Powerful?

Before ResNet, every layer was like a worker who receives a document and tries to rewrite it from scratch. If one of them made a mistake - the whole document suffered.

ResNet says:

• Take the original document.
• Add a small note on top of it.
• Send both of them forward together.

So even if a particular “note” isn’t good, the original document is still preserved, and the learning system remains stable.

An Analogy

Imagine a workshop where every craftsman adds a small fix to an item, but the original item always stays next to it for comparison. No one starts from scratch - everyone just refines what already exists.

This prevents degradation, and allows many improvements to accumulate over time.

The Shift in Thinking

Suddenly it became possible to build very deep networks - dozens, even hundreds of layers - without the model collapsing under its own weight.

This was a defining moment: what had been blocking progress wasn’t the hardware or the data - it was the right shape for the flow of information.

Conclusion

ResNet’s shortcut seems almost trivial, but it reminded the world of deep learning of a simple engineering principle: sometimes, for a system to go deeper, you need to give it a way to preserve what already works.

The next post in the series will dive into a structure that lets a deep network also be efficient - the Bottleneck Block.