While no gun-smith, here's my take on your problem.
First, who made the gas-tube? There are tons of junk AR15 parts out there which don't conform to mil-spec requirements. You should insist on using mil-spec parts, IMO. (best not to purchase parts based on the price or which is cheapest, but more so on the reputation of the seller, stick with those that require compliance w/ mil-spec standards, Bushmaster parts for example or Colt)
Second, it is okay to use the proper sized drill bit to ream the gas block by hand preferably (to which the point above also applys) out to the proper dimension, not to enlarge it, but ream it to remove burrs.
These same principles should apply to the gas-key which the tube fits into, reaming to remove burrs using the proper drill bit (done by hand again) or sanding the tube as you mention.
Stick with quality parts, if yours are then I'd say your on the right path and not out of line with your proposed mods.
Mike
added: while the gas-tube should not be loose, it should be able to be positioned using small pliers or vise grips, without damaging the tube (by having to grip the tube too tightly)