The fence can actually work in your favor.... leave 95% of it up. Then find some places you'd like to encourage deer movement and cut an opening.... Deer will use the opening....
Unless you have a couple hundred acres, deer are already moving through that fence. It's not keeping them out, its merely encouraging them to take easier routes.
If you want a sanctuary, fences won't achieve it. Take a decent 5-8 acre part of your land, preferably something thick with a high stem count, and flag tape it off. Declare that the sanctuary. And NEVER go into it. The reduction in human activity is what will turn it into a sanctuary. If it isn't thick with cover, consider hinge cutting some trees to increase ground level 'brushiness', let in light, and encourage more growth. Nice thick bedding areas, with numerous decent access trails in and out, really work to hold deer.