It's not as bad as OP makes out. Firstly, because although it's not well-publicised, apparently you get the lost points back if the answers are subsequently deleted. Which they very often will be if the downvotes are justified (since there'll likely be other downvoters, leading to more pressure on the OP to edit/delete).
Secondly, one might suppose that on average people with higher rep have a better understanding of what's "wrong enough" to justify downvoting (because they'll tend to be more knowledgeable about both the site ethos and the English language). And people with higher rep can afford to lose a few points; mostly it's just not an issue to them.
Thirdly, I think that ELL in particular needs to be aware that quite a lot of users may mistakenly think they know something when in fact they don't. In many contexts non-native speakers use English as a lingua franca, and obviously in some cases this will mean they routinely hear/use non-standard or "incorrect" phrasing so often it seems "normal".
That final point doesn't imply that I think non-native speakers should be discouraged from voting (down or up). Even though most "successful" answers tend to come from native speakers, many other users are perfectly capable of recognising "right" and "wrong". And of course we particularly need the votes of non-native speakers because they're often better able to identify an answer as "enlightening" in the context of learning English. Native speakers may already know usages they don't understand, but ELL isn't primarily here to provide interesting background knowledge for competent speakers. A good answer is one that helps someone learn English; only a learner can really tell if an answer is doing that for him.