The Anecdote of the Darwin Fence

I was speaking to some interested students recently about what exactly it was game designers do, and in particular what it entails for any developer working on an MMO. One of the subjects we got onto was how much work was involved in actually making the content. One one hand there is the common internet assumption that 'it can't be that hard', but by the same account, they noted there must be some complexity involved because, from the outside looking in, most content takes some time to produce.

One of the reasons is the players themselves...players are smart. I mean some of the them are really smart...and some of them have proven time and time again that they are smarter than us...

The students asked why that would be an issue.

I had the perfect story to illustrate why it might be...



The great beauty, and the complexity, of MMO gaming often lies in the sheer number of potential actions available to a player, which is then multiplied out by the number of players in any given scenario. So, from a development point of view, you might be surprised how quickly the first design phases actually are...I explained that in general we probably spend a small percentage of our time actually 'making'. That is to say, doing the manual labor, the input of database entries, creating scripts, assigning assets, or creating NPC placements in a world design tool...no, those things we can manage, usually quite efficiently...instead what we invariably spend a deal of our time doing, is learning from our own mistakes, and trying to figure out, and account for, the myriad of ways that you, the players, are going to find to break our stuff.

So rather than talking in theory I told them a little story of sorts, a true story no less, dating back to when I worked on Anarchy Online.

It is, I think at least, an interesting anecdotal example of the challenges we sometimes face, and how we, more often than not, learn by doing.

Before I start though, one disclaimer, a good few of the points I'll mention are simply to due with the age of the engine we used at the time. Parts of the AO code base date back to the late 90s, and many of the restrictions we had aren't really applicable to modern platforms (although some are I am sure, and similar problems probably haunt more than a few devs out there). So some of the technical limitations we were working around are no longer a consideration for devs these days (but they make for amusing war stories...)

I'll try and keep the AO specific jargon and lingo to a minimum so you needn't have played AO to make sense of the story that follows.

So let's talk about the Darwin fence...

Sector 13

Sector 13 was a new playfield we added to Anarchy Online back in 2006 or so. It asked the players to fight through a narrow canyon of alien invaders. It was content aimed at the higher level players and was designed to give players a challenge that couldn't be beaten by just 'zerging' the boss. It was during a period when we were trying our hardest to introduce new and interesting mechanics into the major encounters (something that has of course now become expected for the genre)

One of the things specific to AO that we had to mitigate by design was the ability for players to just bring lots of people, and then just have them constantly cycle in from the rez points. For us this meant we had to not have rez points in the zone, and find a way to block progression until certain goals had been completed. You basically had to stay alive to stay in the raid group, no rezzing, no returning, die and you had to sit out the rest of the encounter.

We also wanted to have these new zones (there were three of them) on a lock-out, but lacked a technical code solution for that, so we had to improvise. What we came up involved a security fence that would soon become infamous amongst AO players.

These 'fences' were objects that we could spawn in the game world that had an invisible 'collision sphere' around them. If a player entered that collision sphere the object would hit the player for a lethal amount of damage. So attempting to pass through them killed you - a simple and effective deterrent. We could then give the players a protective buff when we wanted to, which would allow them to be immune to that damage, and pass through.

So the set-up at the end of the playfield went like this - after a scripted event, where the players called down some support from NPC forces, they had to kill a mini-boss that could be found just before you encountered a set of these fences, which in turn blocked access to the final boss. Killing the mini-boss spawned a special device, a 'Biological transceiver' that, when clicked, provided the protective buff that would let you through the gate.

When these encounters were first introduced the raids that attempted them were usually around thirty to forty players in size. A sizable force to co-ordinate at the best of times. This meant that there would quite frequently be players who killed themselves on the fences because they forgot to pick up the protective buff.

The Darwin awards

This meant that players soon started to ridicule those who impaled themselves on these potential death traps. Raid leaders would bark orders and tell people to get the buff, and chastise people who didn't follow orders. It wasn't that hard to follow a simple instruction they thought.

Thus the fences started to be known as 'The Darwin Fences', since they weeded out those in your raid not smart enough to be deserve a crack at the boss loot.

Survival of the fittest.

You also couldn't help but notice that some players definitely seemed to enjoy watching their fellow players perish from perceived stupidity or impatience.

However, that is where our problems started...

Bugs, stacking orders, and malevolent influences

Pretty soon players started complaining that they had picked up the buff, but had still died when passing through the gates. They didn't like dying, and they certainly didn't appreciate being ridiculed by their peers for a bug of our creation.

'Of course you would say that' laughed those peers, and there actually wasn't a huge pressure to fix the issue, because, some people did indeed find it amusing to watch their power gaming peers make what appeared to be a 'newbie mistake'. The designers though went to work trying to establish if there was a problem that needed fixing.

Now, during beta testing, the buff itself had been mistakenly set so that a player could cancel it. So we checked to make sure the fix for that hadn't been reverted. Nope, not our culprit. All was good with the buff itself. It couldn't be cancelled by players, intentionally or otherwise.

We started to watch the raids more closely, and we could indeed see that some players did have the buff only to have it mysteriously disappear prior to them passing through the gate.

Now the first culprit was of our own making. In the RPG system for Anarchy Online, each spell effect, or buff, had a 'stacking order' - a numerical value that set the priority of that buff, so when two buffs from the same line of spells were cast, the game new to overwrite the weaker one with the stronger one. Our problem was that in a game as complex as AO there were a lot of of these 'spell lines' and we had overlooked one. In fact we had overlooked it because it was a buff within another buff that only effected one or two classes, and only occurred if they happened to use that ability (and it was one that wasn't specifically useful in the situation in question, but you should never presume what players will and won't do!). So a situation like that means, in particular when you also throw in the odd false report from the forums as well, that the issue often appears to be more random than it actually is. That is always a challenge because anything that sends you in the wrong direction in your hunt for a pattern (because that is what bug hunting is all about, pattern recognition) will invariably cost you time chasing down dead ends. So it took us a few days to work out that particular issue, but once done, we thought we had finally fixed it, and the resolution to that problem was rolled out...

...except that it kept happening...

...and we were out of our 'usual culprits' list when things like this happen.

We had now fixed all the instances that we thought we could find, but it was definitely still happening. We logged more hours watching players attempt the raids, and watched more players perish on the Darwin fence...some of them were indeed a result of the players impatience or not following their instructions...but there were also mysterious deaths. People who had correctly applied the protection buff to themselves, only for it to mysteriously vanish moments before they passed through the fence.

Doctor Evil

We were stuck...until we realized something. We had forgotten about one other way that buffs could be removed. There was an old PVP nano formula called Epsilon Purge (nanos were what equate to spells in AO) that could be used to remove low level buffs from an opposing player. Only one class had this nano - the doctor profession, and since it was only really applicable in PVP we hadn't really considered it as a possible cause here...after all, everyone on the raid was part of the raid, and all working towards the same goal....right?

In fact it turned out that several players who played doctors, usually those keeping the raid alive, were taking a little vindictive pleasure in dishing out a form of punishment to those who had criticized their healing skills. They were standing close to the fence as players went to pass through, and just before they passed the gate - bang - they hit them with the nano...even though they were part of the same raid...it was more amusing for them to see their fellow raider ridiculed for falling to the Darwin fence.

The nano only effected buffs that were below a certain threshold, but because we hadn't even considered someone would take that course of action, the protection buff in this case had been set to be below that threshold so as not to cram up the players available buff space (a limited resource that was called NCU in AO), so our efforts to make sure the buff wasn't intrusive (as NCU space was a resource very closely managed and optimized by AO players at that stage) we had opened them up to some sneaky revenge debuffing by their own raid healers!

The more evil of the designers wondered if we should leave the loophole in for the amusement factor, but this was one we ultimately had to change in order to protect players from themselves.

The double edged blade of RPG systems

It was an amusing example of the kind of factors that players can bring to complex RPG systems and infuriate (or amuse) designers. The ingenuity of players never ceases to amaze and that is the kind of thought process we have to go through with most content. While things are further along these days in terms of tools, and safeguards, we still spend a good deal of our time thinking 'so how will the sneaky buggers break it this time...'

So I always like this one as an example of why you should never underestimate your players...

...oh, and always be nice to your healers...





Comments

AmandaP said…
Awesome story :) Having healed in many a raid in different game I can see the attraction that doing that would have had. I hate people who criticize their healers!!
Driller said…
I never played AO but it sounds like such an awesome and deep game. Whenever people talk about it I always get the feeling it was fun back in the day, but maybe not if the healers in your raid were taking out their nerd rage on me!

Did the players ever figure out they were being targeted by their own?
Burmese said…
Did you ever plug that vulnerability, or can docs still do that to this day?
Craig Morrison said…
@Driller - I think some of them figured it out, although I am not sure all of them told on the culprits ;)

@Burmese it was fixed as I recall yes
Anonymous said…
Issue was fixed by raising the NCU cost of the protection buff. Some still do die, because they fail to notice they don't have sufficient NCU space available.
ForGwend said…
Awesome story :) Those were the days
Windguaerd said…
Wow, never knew that Craig. Been a while since we talked, how you doing in AoC?
Jenshae said…
I would have left it in ... and I would have used it for a few weeks. I wish I had been there, especially after the raid today, something like that would be such a treat.

I would, however, would have told the players what could happen and provided some sort of % or half measure protection against it that they would actively have to use.
Shareida said…
Cool story bro. I don't buy it. I've done enough s 13 raids, every time someone dies. Not even once since that hit the betas ive seen everyone coming throught alive. I also never seen anyone flagged dying.
Anonymous said…
It's stupid in itself to come to a raid flagged, so i'de just lol if ppl get hit by ep and carry on, specially when you can clearly see in the log that blabla executes epsilon purge in your ncu, all one needs to know.

Now, that RAGE removed it was bad, really bad since you need it to keep some runspeed.

EP part, hilarious :)
Craig Morrison said…
@Windguaerd Hey there :) Yup, they keep me busy and can't complain :)

@Shareida You'd be surprised, and this was right at the start of that content...although I am told the Darwin fences still claim the occasional victim ;)

@Anonymous yes, that was an annoying bug to have, the system for stacking lines was purely manual then, so it was easy for some stacking conflict to get missed due to the sheer number of abilities.
Target said…
Awesome story, I miss those days, when are you guys going to make AO2?