Once, at the dawn of my roleplaying career, I had convinced a friend to try DMing, letting me be a player for once. It was, naturally, D&D. We had ourselves a decent enough adventure, and fought some robotic wolves at some point. An enjoyable experience. Later, I asked him what stats he used for the monsters. He said he used regular wolves, but wanted to make them tougher, so he didn’t track their hit points. Instead, they died after a few rounds, when he felt like the fight had gone on long enough.
The fight was a lie. The fight was exactly the same. We still stabbed the wolves, were bit in return and prevailed in the end. What changed? While individual rolls were a waste of time, the decision to fight did matter: we could have, perhaps, found a way to sneak past them. The starting point (decision to fight) and the destination (wolves dead) stayed the same, it’s the actual journey in the middle that turned out to have been meaningless. Except even that is not quite true. We still enjoyed rolling the dice, and imagining the hits and misses.
Don’t get me wrong, that was a terrible thing to do, but mainly because it violated the rules of the game that we thought we understood. One can easily imagine a system where a fight (or any other scene) only goes on as long as it’s interesting. In fact, that’s a backup rule or suggestion in many games: wrap things up and move on if players start getting bored. And in systems without such rules, people have invented tools like Combat Out for D&D 4e, ways to end a combat once the victor is determined or a specific condition is met.
Generalizing, many if not most games have the so-called rule zero: ignore the rules if that makes your game better. And on the flip side of that is the illusionism approach: the belief that the GM cannot cheat, and should be free to lie about the dice rolls and other mechanical elements to make the game better. This may involve helping a recurring villain get away, saving the party from an unlucky TPK or engineering one.
Then why even have rules? Why do we bother with mechanics if they inevitably get in the way of the experience we wish to have? Some people quite happily play freeform, after all. Others avoid using any rules for the majority of their game time, e.g. playing a social intrigue D&D game. In no particular order, and not necessarily exhaustively, rules provide impartiality, surprise, structure, and fun.
The fun part is the easiest to explain: by most definitions, you can’t have a game without rules. And we like games. We like to roll dice, to demonstrate our mastery of the rules and be rewarded for it. Then again, I already mentioned freeform games here, and I’m not particularly interested in declaring something Not A Roleplaying Game. A different kind of enjoyment can undoubtedly be had even without rules.
Rules allow players and GM alike to anticipate likely outcomes of their actions. They make it harder for a GM to play favorites with their players, and make it easier to be consistent. Rules also take on the blame. It wasn’t the GM that killed you, it were the dice. Rules make sure the game is fair. This is where the desire for game balance comes from: it is a problem if the players perceive, whether correctly or not, the game itself to not be fair.
Then again, there are entire playstyles dedicated to the idea of GM knowing best, like the aforementioned illusionism. “Rulings, not rules.” Indeed, some players reject the idea of rules governing some of their activities, such as social interactions or exploration.
By and large, the impartiality of rules is an illusion anyway, though a useful one. In all but the most restrictive games, the GM has leeway to interpret an event in a variety of ways, and to decide whether to apply mechanics at all. Fate is particularly noteworthy here: its rules govern narrative circumstances rather than the in-game situation. Meaning the GM (hopefully with cooperation of the players) chooses which rules to apply. Not to mention the ability to set up any kind of situation in the first place. It’s not against the rules to throw a tarrasque against a 1st level party, just not advised. Declaring that Rocks Fall, Everyone Die is legit, too.
Related to impartiality is the capacity of a game to surprise us, to choose the outcome for us. More than that, true surprise comes from unexpected outcomes rules can provide. Not just path A or B, but an entirely unforeseen path C opened up by a (un)lucky critical hit. 13th Age’s owlbear exemplifies this: most of the time it’s just a monster to be fought, but it could rip your arm off, forcing the story in a new, bloody, direction. Then again, the GM can certainly surprise players without any rules, and players are notorious for doing the same to the GM.
So far, everything the rules do, we can achieve otherwise. The last element left is structure, and this is, I believe, the main benefit mechanics provide. Roleplaying games are an interactive medium. A group creates their own game every time they sit down to play. We breathe life into it, use our imagination to create and live out stories of our characters. And while we do so, we have the rules, the mechanics, to fall back on. Any time we don’t know what to do, what to choose, we can lean on them. If you’ll forgive me waxing metaphorical for a moment, if a roleplaying game is a plant, its rules are the structure around which it entwines as it grows, as we play it. And just as a structure can offer support, it can be stifling.
We don’t need the rules. But with them, we can reach higher, go to places we wouldn’t have imagined otherwise. Constraints breed creativity, being forced to grow in a certain direction gives the plant a shape. Just be sure you picked the right rules for the game you wish to play.