Theoretically, in a perfect world, speculation would reduce volatility. However, that assumes that all actors are rational and have equal access to useful information.
There really is no useful information about the future value of a Bitcoin. The future demand could be near zero (if Bitcoins become obsolete or are never widely adopted) or massive (if Bitcoins become popular, since their creation rate cannot be changed).
As a result, speculative bubbles are created frequently. The price shoots up for a day or so, and then when the bubble bursts, panic selling sets in. This creates exaggerated increases and decreases in the price. Also, for every savvy person who makes money on this volatility, there's a naive guy who lost some money. That probably turns a lot of people off as well, but most likely what it turns them off from is speculating.
But this actually does hurt Bitcoin adoption. One of the obstacles to Bitcoin adoption is volatility. It's hard to sell a product for X Bitcoins if the value of X Bitcoins can change drastically from day to day. A merchant could be forced to hold a large number of Bitcoins waiting for the prices to recover as their suppliers and employees expect to be paid in dollars.
The price of Bitcoin dropped from $11 to $9 in a day (8/25/2011), and stayed below $7 for a month. Imagine a merchant who accepted Bitcoins at $10.50 each and found them at $9 the next morning. Do they wait and just hope the prices go back up?
I should also point out that this doesn't mean speculation is "bad". It does, however, have some effects that may reasonably be considered bad. These reasons also are somewhat unique to Bitcoins -- these issues don't apply to speculation in more predictable commodities like oil which actually tends to reduce volatility.