Thursday, November 6, 2008

Death of a WAR Server

There are a number of subjects I could write about today. I could write about WAR patch 1.05 and how it appears to be increasing the damage dealt by all classes across the board while simultaneously nerfing healing. Or I could write about the trials and tribulations of the server migration that took place for my guild earlier this week. In fact, I had a quite lengthy post planned on that subject for today except something else occurred last night that trumped pretty much any other subject for me.

Large scale RvR in Warhammer Online fails. It fails hard and it fails for exactly the reason you would think it would fail when several hundred people all show up in the same place. The server can’t handle it.

A little background information is required
On my old server, Eerie Downs, we were a low pop/med pop community. The interesting thing was that while Order was low pop, we owned everything in T3 and T4. In fact, it was actually a bit frustrating because our guild could never really form up a Warband to take anything because we owned everything – pretty much all the time.

I’d like to claim responsibility, but I can’t. The reason we owned everything was because of the Hate guild. These guys are organized and every single one of them is a vetted out player that goes through a lengthy application process. They are not a big guild, just a damn good guild. Formed back in the EQ days, they have stuck together through several MMOs.

You would expect the Destro to hate an Order guild that was that good (and arrogant) but the surprising thing was how many other Order guilds disliked them. Partly out of jealousy, but also out of the type of frustration that comes from what amounts to a one-sided battle with no objectives to take.

Server transfer shatters Eerie Downs community
When the free transfer struck our realm, Eerie Downs players were faced with two choices: Praag or Hochland. But the bigger question for most guilds (Destro and Order both) was where was Hate going to go? Guild and players who disliked Hate wanted to transfer away from them and those that liked Hate wanted to follow. Once they made the decision, everything else fell into place like dominos.

Hate decided on Hochland. Almost without exception, every single Destro guild chose Praag and every Order guild who disliked Hate also left to Praag. Our guild was pro-Hate and the idea of an organized Destro and organized Order guild facing off against each other just had too much appeal to ignore. The potential for epic battles was just too good an opportunity to pass up and considering that we had some friends in Hate already it only made sense to transfer to Hochland.

The existing Hochland community was described as the polar opposite of Eerie Downs: Destruction owned everything and was dominated by one or two organized Destro guilds. The largest and most dominant of which was a Destro guild named Ruin. The Ruin guild was described as a Zerg guild because of the sheer number of members, but they put together a good argument for how they like strategy that convinced Hate there might be more to them than just a zerg. Ultimately, they threw down a challenge to Hate who ultimately accepted it.

The most surreal thing about the transfer is how much the Hochland players courted the Eerie Downs players for transfers. A good Order guild is a hot commodity and the large Destro guilds on Hochland were desperate for a guild that could offer some challenge. Likewise, the Order guilds on Hochland were also desperate for a good Order guild because they were tired of being constantly beaten down.

Hate meets Ruin
The first couple of encounters between Ruin and Hate were interesting. On Eerie Downs, Hate didn’t have an alliance. They didn’t need one. They are extremely organized, they scout, they flank, and they hit objectives fast and hard.

Whenever Hate ran across Ruin in 2:1 or lower numbers, they trounced them. They took and held most of the objectives they were after and then Ruin started overwhelming them. Hate has less than 100 members. Ruin has more than 300. Anything less than 2:1 odds and Hate whooped Ruin. But as the night went on, Ruin numbers started getting huge. Particularly as the other Destro guilds got into the act.

By night’s end, Destro had won the day and regained control of most of the lost objectives and keeps.

Epic fight brewing
The thing to remember is that unlike on Eerie Downs, almost every single Order guild on Hochland was thankful to have a guild like Hate as part of the community. Hate formed an alliance with several of the larger Order guilds. And those guilds told some other guilds. And those guilds told some other guilds.

Until last night – there were at least 200 to 300 Order players fighting for objectives. And an equal or greater number of Destruction. I didn’t play DAoC, but when I think large scale RvR – this was exactly the thing I am talking about.

And it wasn’t just a Zerg fest either – all very organized under Hate. Warbands (not parties – Warbands!) with assignments hitting multiple objectives, flanking, scouts positioned at intervals at various points of interest. All very very cool stuff, except for one little thing…

Epic failure
The forces of Order formed at the Dragonwake encampment. From there, some warbands attacked each of the two keeps. The largest force circled around the back of the second keep and waited for the outer doors to break down. Once they did, the remaining group stormed the inner doors and we quickly took the keep. Meanwhile, most of the destruction was defending the other keep.

After gaining control of the further away keep, we circled back to help finish taking the other keep. There was a massive group on both sides. All our forces converging on the doors – all their forces defending.

Sounds epic right?

Except it took me 20+ seconds to mount up when we left the taken keep. Oh and every spell I cast took that long to cast. No real way to tell how much health anyone had to heal either since that was also delayed. Oh – and then the server crashed as we approached the door of the second keep. And no one could log on for five minutes. And when we did log in, we were several minutes farther back then when the server crashed.

Epic failure part 2
And so we gave up on Dragonwake simply because that many people in a single location wouldn’t work. So we went to Praag and split up our forces into several groups. These mini-groups of 2-3 warbands each then proceeded to take objective after objective and keep after keep. Destruction also split up forces to counter the attacks but couldn’t keep up with our organization and multiple keep/objective attacks.

So Destruction regrouped back into a large single force and attacked the center objective in Praag (the Square). We remerged back into a big group and…

No not a server crash again. Just a complete inability to cast more than one spell every 10 seconds and roots that last four times longer than the intended durations. And not just me – everyone. And this is particularly problematic when people are trying to HEAL. Group heals were impossible on my Archmage.

The crazy thing is that it wasn’t animations or graphical lag as I would have expected. For example, I could move around fine – I would just see people who weren’t really there. Or after my casting animation would end, nothing would happen. I would summon my horse, see the casting bar end and then nothing. If I moved, I got an interrupted error. If I didn’t move, then 10 to 15 seconds AFTER the casting bar ended, I would mount up.

Pretty much the definition of server lag.

Biggest issue facing Warhammer
This is a huge problem that needs to be addressed. As it stands right now, RvR just isn’t intended for more than a couple of warbands fighting against each other. Age of Conan got a bunch of flak for similar complaints about their end-game and it was much deserved. And so is this.

In my mind, this trumps all other possible issues with Warhammer and they need to derive some type of solution. The sad thing is that there might not be a solution. As with all other problems in Warhammer, how do you control player behavior? I mean if word-of-mouth says that the fight is in Dragonhawk, how do you stop a 1000 people on each side from showing up?

The thing is – this is only going to get worse as this type of thing becomes the “end game” for Warhammer Online. Right now, lots of people are leveling and doing it primarily in instanced servers, PQs or solo quests. What happens when all these people show up for the WAR and the server crashes?


Shalkis said...

Short version: Server load and bandwidth requirements go up exponentially as the number of combatants increases.

Long version: This happens in WoW as well, and probably in a lot of other games as well. The core of the problem is that the server has to send updates about every action to anyone who can be affected by said action. That includes showing the animation for said action.

Let's start with some examples.
Two people fighting, both performing one attack per second. Each attack generates a message to the server. Results from each attack have to be relayed to both sides so that their clients can keep track of the battle.

2 actions/s
6 messages/s

10 actions/s
110 messages/s

20 actions/s
420 messages/s

80 actions/s
6 480 messages/s

200 actions/s
40 200 messages/s

300 actions/s
90 300 messages/s

1 000 actions/s
1 001 000 messages/s

As you can see, large battles are quite unforgiving on the server. Based on empirical evidence, WoW can handle at most 300 players, and EvE recently had some successful 1000-person battles.

Of course, there are some measures that can be used to alleviate the situation. One of the most popular and effective is to limit the number of combatants, and that's exactly what the battlegrounds/scenarios are for.

Another is limiting by range or line-of-sight. A melee fighter a mile away is not going to affect you in any way, and you probably can't see him either. Of course, that's not going to help much if everyone is rushing a chokepoint.

Autoattacks help somewhat, and EvE's upcoming weapon grouping can cut down on traffic by up to a factor of eight. Instead of sending updates for every weapon (EvE's ships can have up to eight weapons), updates are sent once for each weapon group. If you have eight weapons that each do 100 damage per shot, grouping them results in one status message saying that they did 800 damage in total. This could also be accomplised server-side in fantasy games. For example, instead of sending messages like this:

You receive 100 damage from Foo's Stunning Blow.

You are stunned by Foo's Stunning Blow.

You receive 500 damage from Bar's Fireball.

You receive 400 damage from Baz's Death Curse.

The server could send just one message:
You receive 1000 damage from Foo, Bar and Baz. You are stunned.

The server still calculates the results of each attack separately, but can cut down bandwidth requirements.

Syri said...


Though I think part of those issues will be worked on... with the likely closing of some of the dead servers in the looming future, I am hoping they can reroute some of those resources to upgrading or clustering the other servers, and hopefully things like this will become possible. The thing to remember is that we are, in fact, still in the infancy stages of the game. Not even terrible twos... the game's not even reached the 3 month mark, let alone 6.

So I do think that large-scale rvr is something they will make possible, because if they don't, it'll kill the game. And rather than kill the game, I'm actually pretty sure they're already trying to figure out a solution. Though I also think it's still to early to tell whether or not the kind of action we're all spoiling for will ever actually take place.

Anonymous said...

Your web is great!! I learned something about WOW here!!!!