If you can fluently understand people across northern China without explicitly learning to, then that means they speak “dialects” of your language.
But hundreds of millions of other people in China natively speak hundreds of other mutually unintelligible Chinese languages. Most of them are (at least) bilingual.
So, even though most people cannot understand some southern dialects when they hear it, they can fluently understand other dialects without explicitly learning to do so when it's written down. Even when you explicitly write down exactly what you say in the dialects (like the Cantonese version of Wikipedia pages), people can still understand it, without much difficulties. And yes, there're some words used differently from standard Chinese, but the difference is not much bigger than the difference between UK and US English.
For that reason, we define them as dialects not languages. We can "speak" other dialects but we can certainly "read" and understand them.
If you find that definition unsatisfying, sure you can call them different languages. We just call them dialects and that's not going to change.
But hundreds of millions of other people in China natively speak hundreds of other mutually unintelligible Chinese languages. Most of them are (at least) bilingual.
Here is a grossly simplified map showing major language groups: https://en.wikipedia.org/wiki/File:Map_of_sinitic_languages_...