That can be fairly involved, depends on how much access you have to the inside of the plenum. I'm going on the assumption that this is a standard upflow furnace in your basement. If you can remove the front of the plenum, or coil cabinet, you can usually remove the end plate on the A-coil.
At that point, just reach in and brush it off. But be careful, don't bend the fins, and don't cut yourself those fins are SHARP. I have the scars to prove it lol. If you can't access it that way then you will need to try going in from the top. Remove a register if there is one on the plenum, take a run down, or if there is room, cut in a register. Then use compressed air to blow down from the top.
But most likely your coil won't be plugged, because you did have a filter in there. Even plugged up that would stop most of the crud from reaching the coil.
Mike