I thought I would try out the Colt 7.62x39 rifle I have, but don't have any magazines. The rounds don't fit very well in a 5.56mm magazine. What is my best bet for something with high capacity that will work reliably?

Yup, normal 223 mags can only hold/use 5 rounds or so before they don't work.

The best mags out there for the 7.62x39 AR is the "hybrid" magazines.  They have the top (neck and feed lips) of a Colt type 7.62x39 mag, but the bottom of an AK-47.  A few places made some before the ban.  They were welded together, but they work (so I have read).  The next best thing is the USA mag.  Wait.... before you run..... the USA mags are "hybrids" also.  They're the next best thing to the "other" hybrids.

Then there is also the drum mags that were made.  They work okay too.

I have a drum and mags, but I don't have an upper.  Plan to get one....

Search www.gunbroker.com or www.auctionarms.com and use "7.62x39 AR" or "USA AR" in the search.  The going prices so far are around 30+

If high-capacity isn't an issue (and when isn't it???), DPMS sells a 10-round synthetic unit that is extremely reliable.  I've also had good service out of MOST of the 7.62 USA mags I've bought (one or two clunkers over the years).  Please stay away from the flat-sided aftermarket mags that look like 30-round 5.56 mags - my experience with several "makes" of these has been negative without exception.  Good luck!