Quoted:
You need to fire your CPA and find a new one.
Basically, you stated the interest and taxes add up to $23K. Lets say, just as an example you make $100K per year. You don't pay taxes on the 100K, instead you deduct the $23K and pay taxes on $77K. The current federal tax rate for $77K is 25% (for married filing jointly). So you pay $19250 at $77K in taxes instead of $25000.
In other words, you pay out $23,000 to the bank and in property taxes in order to save $5750 in taxes. For some reason or another this doesn't sound like a very good plan to me. You could donate the $23,000 to a charitable organization and get the same tax deduction. Me personally, I had rather give it away than hand it over to the bank. Then again, I like to be in control. Also, there is a huge element of risk involved in borrowing money.
It isn't anywhere near that simple, first of all the personal income tax rate has little to do with the total tax burden on a business owner, you have no idea what his current facilities cost him or if they're adequate, if his business is growing or shrinking, you basically know nothing about this guys business except his CPA says it's time to buy a building and he should fire them? Neither of us can say whether he should buy a building or not, all he did was give us some hypothetical numbers and ask if he understood the way it worked correctly.
This is the area where a lot of small business owners destroy themselves, if you're not tax-efficient you're trying to swim upstream. You can significantly reduce your tax burden by investing in your business and then the growth pays you back a lot more than you would have gotten in the first place.
This is also one area where I think the GOP has tax policy all wrong, constantly trying to encourage business owners to pull capital out of their businesses, and discourage them from adding payroll, via the tax code.