You had a no-gift agreement. No reason for you to feel bad.
It kinda sounds like your brother wanted to put you in an uncomfortable situation, though.
Thank him for the gift, and apologize/explain that you didn't get him anything because of the no-gift arrangement you two had this year. Then ask him what he'd like to do next year, as far as giving gifts is concerned.