I would have said "all of the above" but a couple regulatory agencies are actually necessary to some degree. They would be massively scaled back though. I would ask Congress for a bill to strip all the agencies of their regulation-writing authority, to rescind the entire CFR, and then to take up whatever parts they think need to stay as individual pieces of legislation, but with no guarantee that those bills would not be vetoed.
Until that happened, I would decline to appoint an Administrator of the EPA, or a Secretary of Education, or HUD, or Director of BATFE and accept the immediate resignation of all current political appointees in those agencies (and other that I'm forgetting right now), as well as the immediate retirement of all SES members in those organizations.
In fact, I think my basic position would be "If it isn't in the Constitution, my administration isn't doing it."