I would think brakes would generally tend to degrade slowfire benchrest-type accuracy, but there are a lot of variables.
If brakes do hurt accuracy, then those that are the least symmetrical probably hurt it the most. Say like where the bullet passes close to a flat solid surface on the bottom, while there is a big open hole on the top. Since the pressurized gases surpass the velocity of the bullet after it "uncorks" the muzzle, it seems reasonable that you wouldn't want an unbalanced situation with them bouncing off a surface on one side and not bouncing off a surface on the opposite side.
But in practice, it might well be that a particular brake might, by its mass, just happen to tune the vibrations of a particular barrel and so actually improve accuracy much more than it hurts it.