Good Luck!
Bittorrent is a remarkably adaptable protocol, that can function in very hostile environments. It can be configured to run on just about any port, and doesn't require specific Port forwarding back over the NAT to the local hosts. (Well, it is vastly more efficient and effective with port forwarding, but it will work without it.)
Today, you could use an IPS as the previous poster noted, to perform SPI (stateful packet inspection), or as it is known Deep Packet Inspection. This allows the IPS to look at not just the source or destination of a packet, but actually look at the contents and say 'this is part of an ftp download of an image and is OK' or 'this is part of a bittorrent protocol handshake and should be dropped'.
HOWEVER, SPI and Deep Packet tools utterly fail on Encrypted traffic. The payload is encrypted, so you can't look at it and make any meaningful conclusions. It might be part of a Word document someone is uploading to a website, or a part of an MP3 they are sending to another BT client. You can't tell. All of the modern BT clients support Encryption now, so most of your 'abusers' would simply turn this on (if it isn't turned on already) and your IPS would have no choice but to let the traffic pass.
You can't block all encrypted traffic, btw, since that is quickly becoming very standard for many websites. Any time you see SSL or HTTPS, that traffic would be encrypted.
There are some 'fuzzy' intelligent sensors that claim to detect the 'flow' of traffic to identify BT sessions, vs. other uses. You can imagine this like standing by the highway and picking a funeral procession out of the mass of cars. The pattern is distinctive. However, these are by no means foolproof, they are in the early stages, and the BT clients are already implementing 'countermeasures' to alter their traffic patterns to bypass these devices' logic.
All in all, it is a serious problem. The most common tactic I see these days is simply to cripple the Upload bandwidth. The RIAA and MPAA are less concerned with people Downloading the files, than they are with the folks who are Uploading them and Actively Sharing content. Most people using your WLAN network for average uses, could get by with a 512K DL, 128K UL, or even a 64K UL. But this would make the link virtually useless for anyone looking to UL large quantities of RIAA/MPAA content. They would probably look elsewhere for their connection.
Again, Good Luck!
FluxPrism