No, we shouldn't. This is a topic that often comes up on different Stack Exchange and there's an FAQ entry on Meta Stack Exchange: Why isn't providing feedback mandatory on downvotes, and why are ideas suggesting such negatively received?
Forcing downvotes to be accompanied by comments or other feedback sounds like a good idea at first, and many here would like to see new users get all the info they need to ask questions that are a better fit! Contrary to popular opinion, most users here are nice and want to help, and don't enjoy "shooting down" newbies' questions with downvotes to make them feel bad and unwelcome.
However, downvotes are important for the health of a site, and mandating feedback for them would massively impede the way Stack Exchange currently works—to the point of potentially destroying it. It's just not feasible, for a number of very good reasons. That's why, although this gets suggested frequently (on average, a few times per week overall, both here and across per-site metas), it is declined and often downvoted* by meta users.
Not all reasons listed further down in the answer are applicable to English Language Learners, but most are.
Some people seem to downvote people for personal reasons.
If you feel you're the victim of serial downvoting and the system does not revert it overnight, you can always use a custom moderator flag and we'll look into it.