On starup it checks for new mods & downloads them, it also checks your textures & items & gets them.
After that players are downloaded as you come across them, and if you don't have a mod then joining a room will make it download it.
The game doesn't download any temp guff, only stuff it uses.