On ARM there's a whole series of controller blocks that sit between the CPU caches and the outside world. Start looking up the AMBA (Advanced Microcontroller Bus Architecture) and you'll see how complicated it gets:

https://developer.arm.com/architectures/system-architectures...