The source of magic. There's a question for ya.
Could be the same no matter what. Could be a case of Good Spirits vs. Evil Demons. Could be God vs. Satan, in some obviously non-orthodox version of Judeo-Christianity. But I don't think it matters.
The source (if sentient, as opposed to some other realm of existence that may have endless banks of accessible power, or something like that) could be evil or good, yes, but the source and its output are two different things. A black mage might invoke the power of Satan and intentionally save the life of an innocent child, and that would be a good thing, and maybe this misguided (hypothetically) mage would even be one who has overall good intentions (mayhap not knowing he's working for Satan). Conversely, one who can use white magic (here I'll leave the source up in the air--perhaps it's fairies, dragons, or the Invisible Pink Unicorn that does it) may use said powers to heal an evil dictator, knowingly or even intentionally prolonging the suffering of those said dictator dictates. That would be bad.
This is why I most approve of fantasies--like Final Fantasy--that give both heroes and villains aptitude in both (or all, if other categories of magic come into play--Blue, Green, Arcane, Time, etc.) schools of magic.
Talking of the real-world distinction between White and Black Magic, though, it's clear that there's presumed sources in most cases. If we were to hypothetically assume such magic worked, then, how would we view the magic?
I would view it as a tool. If the result--that the user was driven from God, or something like that--was bad, then I guess that's wrong, but was it the magic that was wrong, or the person who used it? ...I'd blame the person before I even considered the magic.