Funny you should ask that - my parents have a 38' IP, and I helped them out with the same question a few years back. There are several issues to consider. The first one is the tangle of legal issues you have to deal with. If your destination is a US-jurisdiction location (USVI, for instance) you're pretty much in good shape. If not, you have some problems. Most Caribbean destinations will allow shotguns as long as they stay on the vessel. Handguns and assault rifles are generally a no-no.
My personal feeling is that my safety on the high seas is the primary concern. When you're sailing, you don't have the ability to outrun pirates, so IMHO you need to be able to deter/disable/destroy them before it comes to an 'away boarders' situation. What they finally settled on was a DS FAL and a pair of Glock 21s. IMO, the 5.56 doesn't have the penetration ability to get into a vessel and really mess things up.
There are some basic precautions you can take to ensure a higher level of safety. Most modern marine radar sets have the capability of setting a proximity warning, sounding an audible alarm if a target closes closer than a preset range. Typically, when we're in blue-water locations, we set the 'anchor watch' as this setting is called, for about a 2 mile radius. This would give you ample time to wake up, arm yourself and figure out WTF is up.
To summarize, I believe a heavy assault rifle, and handguns with frangible ammunition (to avoid any hull-breach type incidents) is probably the best solution. You need to figure out a reasonably good way to hide them when you go into a port that might inspect, but as a sailboat you really aren't going to be subjected to a drug-enforcement level search most of the time...most drug runners choose a faster method of transportation.
YMMV.
QS