I know a guy that was both Special Forces and an Army pilot. His SF career was very short, his aviation career was not.
The guy I knew who did both was my father. I was about three when he went to flight school, and all I ever remember him doing in the Army was flying. I was 16 when I found out he had been a Green Beret. Some guy called the House, looking for Rusty (my fathers name), I told him he wasn't home, that he was in the field, and he asked me to take a not that Mr. XXXX had called, they hadn't seen each other since they graduated SF school, and he wanted to catch up. I told the guy that he must be looking for somebody else, my dad was a pilot, not an Infantryman, and he assured me he was looking for the right person, then seemed a little dismayed that I didn't remember him or my dad being a Green Beret.