Actually if you check many state and local laws/ordinances about temporary/cardboard/plastic signage of any kind placed in the public domain/property, it's usually even illegal to do so...and not always just under the littering laws. The placing of signs like garage sales, puppies for sale, political campaigns, etc. all fall in the same category when they're nailed to a power/phone pole or stuck in the ground on a stake when the property is public right of way or just plain ol' public property is almost always illegal in most jurisdictions. Though not usually enforced with any gusto, it's still usually illegal.
So...removing those signs in those cases on public property is almost never illegal. Now, I would recommend you check local laws/ordinances to be sure, but most highway departments, refuse departments, street departments, police departments, etc., etc., would thank you. If those signs are in someone's yard or other private property, it goes without saying that it could be theft in most jurisdictions.