Do you realize that your email password is probably your most sensitive piece of information? With your email address and password, I could manage to log into almost every web site you have an account on. Even if you use different passwords on all those other sites. I just have to click "I forgot my password" on those other sites, read your email for the 'reset password' link, and I've stolen your identity. Even credit card and bank web sites often have this "easy" password recovery feature. In fact since I have access to your email I could very likely discover most of the web sites you have membership on by searching through your email archives and get them all in just a few minutes. Have I got you thinking twice about giving away your email password to some site that wants to download your address book yet?
Never, never, never give someone or another web site your email password! It's just about the worst thing you can do. Facebook, LinkedIn, and other otherwise "reputable" sites love to ask you for your email address and password so they can offer you the convenience of spamming all your friends by looking up their email addresses from your email address book. All they ask for is your email address and password. Don't give it to them. Whether they are reputable sites or not, any person working at that company, in a moment of compromised integrity, could steal your password even long after the fact and rip off everything you own.
Do you still want to use Facebook, LinkedIn, or these other social networking sites and have them spam your friends? Fine (I hope I'm not in your address book!). Most of these sites also offer an option for you to manually export your address book to a .csv file which you can then upload to these social networking sites yourself without giving away your password. That's definitely a safer option.
It's less convenient than just typing in your password, true. There are upcoming technologies that will help solve that and make it safe and convenient for you to share your address book. But for now, do it the 'hard' way and save yourself a lot of trouble in the long run.
So protect your own identity by not sharing your password. Give away all your friends' privacy without asking them by giving away their email addresses to a social networking site that they may not want to join anyway. But that's an etiquette issue at least, rather than a hard-core security issue.