If they were to be pure silver, or high content silver alloys, rather than silver jackets, the main problem you'd have is sizing them, as cast silver shrinks substantially. Not enough to be a problem with artisnal "organic looking" jewelry, or in use as solder, but for bullets, if you used a steel or iron mold normally used for lead, the bullets may well just slide down the bore, and have low velocities and very poor accuracy, and probably won't engrave into the rifling well for spin, if at all.
A lot of silver casting that requires more precision is done in molds made of materials that have higher expansion coefficients, so that they expand either from pre-heating, or just from the molten silver pour, then shrink proportionally with the silver so the end result is closer to what was desired.
So you'd need molds that were oversize somewhat, and then one hell of a swagnig press that was strong enough to squeeze solid silver bullets down to size.
I'd say the idea to turn silver bullets on a lathe might be the easiest and the most precise. And I suppose you could try for long curly unbroken chips to make recovery/recycling easier.
That said, it ALL seems like a PITA, and buying them from a skilled silversmith might just be the easiest way to go.