I started with my brother around 5 or 6 or so, RWS air rifles, youth-sized, .177 caliber.
We would plink off the back deck, maybe 20 yards downrange/downhill.
Targets? "Combat Loss" GI Joes (early/mid-80's, the small ones, the size of my thumb now) and Matchbox cars. Open sights. I eventually upgraded to a scope, which made those tricky GI Joe headshots that much easier
The air rifles were locked away, and until we were 12 or so, were NEVER taken out without supervision, and my father was ALWAYS watching us.
The rifles were always broken open before we went downrange to check targets.
Actually the closest call we ever had (well, I ever had) was when we had our cousins over; I was what, 16? 17? Oldest cousin was a good 8+ years older than I...I was downrange checking something and walking back up, just as he decided to fire. I wasn't near his line of fire, but still, I spent a good 5 minutes screaming at him to hold fire while people were downrange. That's how well it had been drilled into me.
My brother's not very enthused about shooting, but I guess I picked up the bug in place of him.
I didn't own a real rifle until I bought one myself when I was 23 or so. Until then, airgun only. Yah, I know, late starter.
I highly suggest a single-shot, barrel-break, youth model air rifle, shooting .177 pellets.
Actually the only thing my dad DIDN'T make us do was clean them, because he didn't have a .177 brush or cleaning rod...that's something I wish he had drilled into me as well. You want to play, you gotta pay. Good thing to learn when you are a kid.
My $0.02...