Been a few years since I left, but two popular spots (i.e., no secrets here) were Hagelstein (marshy lakebed at the top of the windy, steep-ass FS road, off 97 at Hagelstein Park) and Balsam.. rock pits out on Balsam Drive, which takes off right where 140, 66, and 97 come together south of town. We usually went to Balsam just b/c it was quicker, but there was ALWAYS other folks there.
There's also a rock pit off, ahh, I think it was Hill rd... wherever the BLM rd you take to get up Stukel takes off at... anyway, I never got hassled there but I'm not sure of its status.
A couple other spots back in the woods, off 140 on the west side, but my memory isn't clear.
Sportsman's Park was pretty nice, though. I liked it. Plus I had a bunch of secret spots, mostly east of town out in the Fremont forest and further towards Lakeview. Then again, I didn't mind the longer drives.