For home self defense - the 20 ga will work and not penetrate walls. same with 12 ga.
For hunting - both will work as well.
I hunt with a 12 gauge and chase all upland birds, waterfowl, coyote, pig, bobcat etc... no large game as i live in CA and there is no such thing as shooting lanes, tree stands or feed plots and i dont bow hunt so it is all rifle shots for large game.
You dont need buckshot to shoot birds or rabbits as matter of fact the only thing I use buck for on the above is coyote and bobcat while pig is all slug! Shot patterning is all you need to shoot and kill the above effectively. I usually shoot 12ga 2 3/4" # 6 or 7 shot and have no problems retreiving the game after wards. Take a piece of cardboard and shoot it at 10, 15, 20 ,30 yards etc... to see where your pellets are hitting.
I have also shot all the above with a 20 ga except pig.
Kick WILL NOT be determined by gun but rather shot type, size etc... My stevens single 20 gauge with light loads kicks harder than my 12 ga extrema 4 with hot loads. However when comparing rounds in same gun the kick is ony related to the shell.
For self defense the 20 would work with buckshot i would not use a slug because I like to fill the hallway when I need to!
Turkey stick with the 12 in 3".