So okay, I'm halfway through writing "Hunter," the 3rd book in the alternate history series, and I'm finding myself slipping away from the rigid rules of magic established for the series. Well, semi-rigid. Actually, fairly liquid and slippery, 'cause I keep seeing new and cool applications of the underlying source of magic. The temptation is always there to simply make it into the Force, or something akin to the Force, wherein anything is possible so long as you just concentrate hard enough.
Inventing good, new, and original magic systems is really hard. Most newbies just think their heroes should be able to blink and have anything they want appear. No rules, no limitations. Of course, if the Evil Overlord has no limitations, then there can't be any conflict or Heroic intervention, can there? His Exalted Evilness gets to do anything, can dominate the world effortlessly, and the fight is over before it begins. Even Homer, author of the world's first epic fantasy, knew this. The gods of Olympus needed to work through their pet humans, understanding well that when worship withers, so do the gods.
Magic needs rules, and I believe most gaming systems have very rigid rules in place to make sure the players aren't constantly running into arbitrary deus ex machina type solutions to puzzles and people airily throwing down impossible tricks. No Hero should be good at every aspect of the magic, nor able to simply invent new possibilities on the fly. If he can conjure fire at will, he should be miserably bad with water. Maybe he's creative enough to apply old magic in new ways, but it should be after some thought, or with at least the possibility of such application in his mind. Inspiration in crisis is wonderful, but it must be foreshadowed as a possibility first.
Magic should not come easily, and it should come with a price, either in a physical toll on the body or some unpleasantness in forcing the natural to bow to the unnatural. Maybe it's painful; perhaps it shortens the hero's life; perhaps the gifts are inborn and natural to the wielder, but were meant by nature for survival, and overuse sets nature out of balance and thus becomes actively counter-evolutionary. Maybe the magic systems are so intertwined that selfish manipulation results in unintended consequences, as in Tim Pratt's wonderful story where sucking the "cloud stuff" away indiscriminately lets the silver lining go flump onto unsuspecting people below. Whoops.
Consider the source of magic. Is it natural, rooted in elemental forces like fire, water, air? Is it mental, dependent on the strength of mind and will of the practitioners? Is it physical, dependent on proper placement of stones, brewing of potions, etc.? What happens when a practitioner is cut off from the source of his or her magic? Melanie Rawn did a good job with this in her Sunrunner series. A Sunrunner without light cannot exercise power, and if mentally running shafts of light when the sun goes down, will be left mindless forever. That is both powerful, consistent magic and logically limited.
Luke Skywalker, a mental practitioner, was mostly bounded by his own fears, his own inability to set aside logic to embrace the Force. He continually thought of it as something physical that must be stronger than the object to be overcome, instead of something that could be shaped to the desired strength. But Lucas's Force apparently has no outside limits apart from the mental will of the practitioner, and perhaps the strength of the physical vessel wielding it. This system is less logical, more prone to abuse as people display sudden new and unguessed-at powers.
It is well to think through how your magic works before ever putting your hero into a situation where he needs it. For every exercise of magic, think of the counterpoint the bad guy could use to negate it. For every time the hero goes out on a limb to use his magic, think of how you can saw the limb off behind him while he's exercising it. Give your Hero, at the outside, five things he can do well with magic. Give different powers to different people, or give them varying degrees of proficiency. Above all, set down the rules of logic for your magic system, and don't violate them; otherwise, your readers will rightly call foul.