Unfortunately, there are a lot of people on the internet who, intentionally or unintentionally, hurt writers.
The best way to protect yourself is to be informed, and there are some great sites online that help with just that.
Writer's Beware is a group of conscientious writers who collect information about writing-related services and report on those that are disreputable or harmful to writers. Another must-read is their Thumbs-Down Agent List.
Preditors & Editors lists thousands of literary agents, with warnings and recommendations based on complaints received by authors.
First rule is never pay a literary agent. Money flows towards the author, not away. No matter how much the "literary agent" tells you that it will be money well spent because he will make you famous and wealthy beyond your imagination, don't believe him. If the literary agent says you'll get the money back right after publication, don't believe him.
Another rule is that if the literary agent has to advertise (either online or in print) they are more than likely a scam. Real literary agents have so many submissions already they do not need to pay for advertising.
Listings on QueryTracker are carefully screened to protect writers, but not all sites are as diligent or caring. Don't trust sites who throw up any and every name they can find. Don't trust sites who say they have "peer review" to filter out the bad listings. You don't know who those supposed peers are. He could be the scammer himself.
For further reading, see the QueryTracker forum - How to spot a scam literary agent.