If you have second thoughts about an upvote, why is it not possible to simply remove it? Removing it now results in a downvote. It seems the algorithm is being too clever. Why not just decrement by one?
I would consider this a design flaw, if the behavior is intentional.
The two buttons are a little unintuitive because they aren't up/down controls for the score, they are two independent buttons: "add up-vote" and "add down-vote". Because you are only allowed one vote per post, adding an up-vote replaces any existing down-vote (and vice versa). It doesn't "remove" it.
To remove an up-vote or down-vote, click the same button again as a toggle. You can only reverse a vote within a set period of time - after that period, you can only change your vote if the post has been edited.
See: Cancel a vote so that the tally goes back to zero and not minus
