Currently my brother and I are both "trustee and settlor" on our trust
So far all of the stamps are in my name (i signed them)and listed as my address
Now he's ready to start buying NFA items
Do they all need to be kept at his place or does it only matter what is on the form1/4?

Are the existing stamps in your name, or the trust's name?  When your brother buys, will he be buying in his name, or the trust's name?  If "your" guns are owned by the trust, and "his" guns are owned by the trust, and both of you are trustees with current right of possession of trust property, and you're both in the same state, then either of you can have any of the guns.  If you're in different states, I believe you'll need to have an approved 5320.20 to cross state lines.

Sorry to clarify, the current stamps are in the trusts name, one is approved and 4 pending. Yes we are in the same state.