The two buttons are a little unintuitive because they aren't up/down controls for the score, they are two independent buttons: "add up-vote" and "add down-vote". Because you are only allowed one vote per post, adding an up-vote replaces any existing down-vote (and vice versa). It doesn't "remove" it.
To remove an up-vote or down-vote, click the same button again as a toggle. You can only reverse a vote within a set period of time - after that period, you can only change your vote if the post has been edited.
See: Cancel a vote so that the tally goes back to zero and not minus