I think the problem you're running into is you've got an old camera that likely won't perform well on autofocus past f/5.6 and your 300 f/4 + the 1.7 gives you a f/6.7 max aperture.
You can try focusing in live view, but even that may not work. You'd likely be better off with a 1.4x or getting closer to your subject.
That said, here's the general troubleshooting for autofocus issues.
You've already worked out the "rule of thumb" for this. 1/focal length + crop factor.
510 * 1.5 = 765mm effective focal length.
What I like to tell people is to also consider the resolution of your sensor, a high resolution sensor will be more sensitive to camera shake than a low resolution sensor.
The D300, I wouldn't be too worried about that.
So, handheld, you should be looking at about 1/800 or faster if your lens is not equipped with VR.
Using a tripod and proper long lens technique you should be able to shoot at practically any shutter speed you wish.
Subject motion and mirage and wind will be more of a factor. If nothing you're doing will get you sharp pictures, I'd start to suspect either the TC or an autofocus issue.