I am not sure what you mean by falling apart. Is the fish itself falling apart or is the breading falling off the fish?
When you pan fry you just want to brown each side and transfer to the oven on a rack for best results. This will keep the fish from being greasy and keep it crisp. You can however just cook it through in the pan, but if you are cooking several fillets then you will want to transfer to the oven. Pan frying requires high heat and small amounts of grease. Cooking multiple fillets will require one to clean the pan in between batches unless you want burnt pieces of batter sticking to your fillets. 2T of good cooking oil like Crisco will keep the smoke point of the oil high so you can fry at a higher temp. Olive oil has a low smoke point as do most animal fats, but one can add soybean or peanut to raise the smoke point.
If the fish itself is falling apart then it may be sticking to the bottom of the pan or old. Old fish can get mealy and it tends not to stay together well. Either that or you are not cooking at a high enough temp to allow the breading to seal over and crisp. Do not add the fish until the oil is hot. I will wait until the oil is just beginning to produce halos in the pan then add the fish.
If the breading is falling off you need to coat in flour or starch prior to dredging in the batter.
1) Dry/Wet/Dry is generally how 99% of things are fried or Dredge in flour/starch, then eggwash/buttermilk, then flour/crumbs/cracker/cornmeal.
2) Sometimes it goes wet/dry/wet if you are using a batter like buttermilk/ flour/ batter say for chicken. Hell, I"ve seen Wet/Dry/Wet/Dry for extra crispy.
3) Dry/Wet it depends for say a Tempura.
4) Or just Dry. A heavy coating of seasoned corn starch on a whole fish that has been scored will yeild a handsome meal out of the deep fryer.
5) Pan frying can be done with any method listed above or you can just coat with a seasoned flour or starch then sear each side and finish in the oven.
Dredging right into batter then frying will be a disaster. The batter will get blown off the fish as the moisture leaves the flesh. The flour or starch acts as a binder or glue to keep the final coating adheared to the fish.