Speculators can only influence the market in the short term. Sooner or later, usually sooner, the law of supply and demand takes over.
While what you say is true, speculators can move prices quite significantly in the short term.
Look at what the Hunt brothers did to silver in the late 1970s.