Why does my rank go down even if i won a match?

Other players win matches at the same time as you win, and other players sometime climb the ladder faster than you, if they beat a better ranked player.

Lower rank number means higher rank.

oh yeah, i forgot about that. I just remembered it after i posted.

But will the rank of other players matter if they are lower or higher than me? the in-game rank doesnt matter in the belt right?

nope belt doesnt matter at all.

You will rank up faster if you beat better ranked players. Toribash now uses an ELO ranking system, and you can read more about it here.