Great problem!
We know that the moment generated by a guywire (anchored at an unknown height x) about the base of the flagpole must equal the moment generated by a 500 lb side force applied at the top of the pole. The moment equation that must be satisfied is:
where
where T is the tension force in the guywire.
Solving for T yields:
We can find a stationary point (which will either be a minimum or maximum) of this function by differentiating T with respect to x:
By setting this equal to zero, we can solve for x:
However, this equation does not have a solution for x in the finite interval [0,50]. By inspection, we can see that as x goes to infinity, this equation will approach 0. Thus, x=50 is the optimal value of the attachment point. This can be verified by numerically evaluating results of the T equation listed above for discrete values of x in [0,50]:
I think your intrepid mathematicians were a little off in their tension estimate- for a guywire anchored 9 feet away to generate a 500 lbs side force at an attachment point 50 feet up, the tension should be 500/sin(atan(9/50))=2822 lbs.