...I'm not much of a DM, myself, but it only makes sense that there would be both "to hit" and "damage" modifiers to be considered where different sized opponents are concerned.
Using the example of Halfling vs Giant: The Halfling would certainly have an easier time hitting the giant (especially from range, I think) unless s/he's just a gods-awful shot. That being said, I'm fairly sure that the damage from a "halfling sized" weapon would be more or less negligible (depending on the specific type of giant, of course... that halfling weapon would do more damage to a hill giant as opposed to a storm giant, for instance).
Now, you flip that scenario around and put the giant on the offensive and you've got the complete opposite... Halflings being as small as they are, it's going to be a bit trickier for a giant to see/target them, right? So, the giant might not be as "accurate" when it comes to his attacks but, when they hit, look out! Where once stood a halfling, there is now but a quivering puddle of pudding in the footprint of a giant!
When it comes right down to it, I suppose, if you look at the halfling vs giant scenario, the giant's going to have the upper hand regardless of the modifiers (especially in a one on one situation)... In order to deal any significant damage to the giant, the halfling is going to have to expend quite a bit of energy and/or ammo... The giant, on the other hand, may pull the old "swiiing-and-a-miss" a few (dozen, hundred, thousand?) times depending on how small, agile, and/or crafty the halfling is but, when it comes right down to it, all the giant should need is one glancing blow and it's halfling hash, isn't it?
Now... take that same giant and have him lumber into a village full of li'l hairfoots and you can switch things up again... 100 halflings vs 1 Giant? Halflings stand a chance... but get ready for major casualties...