Uhm... press b to show both of the tori's, if it shows yours, press b again.
Or was it a glitch when your opponent joins spectators?
were you fighting? did your opponent join spectators after first turn?
If he joins spectators in first turn, it will make your move glitch.