C'est la vie.
As frustrating as it may be, it is perfectly legal for him to kick you from his private server at any time. Best thing you can do is return the favor if he ever joins your server. Perhaps stick with playing with people you know.
Next time PM a mod with things like that instead of making a topic.