a good explanation of the modes can be found
on jens' site for the game. (if you don't want to be spoiled on secret / ending parts of the game, I wouldn't recommend going there.)
in short, Tournament Mode makes the game more challenging and more "fair for competition" - no ball saver at the start of a ball, no extra balls (you get points instead), and awards that are much less random. Challenge Mode only affects games of 2 or more players, and forces all players to share their progression through the game (like locked balls, lit modes, and such) instead of having each player progress through the game themselves.