After some time the CA will have signed a couple of thousands more
certificates which will need to be synced and downloaded to the
browser/app using them. What is actually stopping a MiTM from
intercepting that?
The fact that the website's certificate is signed is what prevents a MITM.
I will prove my point with a simple example. I don't know if that's exactly how modern SSL/TLS works. But the basic idea remains the same.
Step 1, Bob's browser
Bob has a CA's public key. The CA public key is well known and never changing, therefore it's built into Bob's browser/OS.
Step 2, Alice sets up SSL/TLS
Alice wants to make her website protected. Alice creates a private/public key pair and sends the public key to the CA.
The CA bundles Alice's public key, domain name, and some other information together in a single package. Then the CA encrypts that package with the CA's own private key and sends it back to Alice.
Now Alice has a signed certificate which contains her public key, domain name, and some other info. The certificate is encrypted(signed) with the CA's private key. It can be decrypted with the well known CA public key.
This step only needs to be done once.
Step 3, Bob connects to Alice
Bob conntects to Alice's website. Alice sends Bob her signed certificate. Bob gets the signed certificate and decrypts it using the CA's public key which he already knows. Now, bob checks if the domain name inside the certificate matches the one he is connecting to. Once done, Bob is confident he has Alice's public key and no MITM can occur.
(Assuming the CA isn't evil)
Step 4, Scar tries to intercept
Bob connects to Alice's website again. Alice sends Bob her signed certificate.
This time Scar who is in control of Bob's network is listening to the communication stream. Although he can intercept and decrypt Alice's certificate using the well known CA public key, He cannot modify the certificate and re-encrypt it because Scar doesn't have the CA's private key. The best he can do is corrupt the certificate, but that would alert Bob and the connection would drop.