EA Crippled BioWare With Frostbite

At the beginning of April, Jason Schreier at Kotaku published a report on the troubled development of Anthem, detailing the weird problems the team had with leadership, personnel, and technology. It’s a long, sad, frustrating tale that helps explain why the game arrived in such a sorry state. It’s a good companion piece to an article he wrote two years ago on the development of the misbegotten Mass Effect: Andromeda.

The problems with Anthem and Andromeda are multifaceted and can’t be blamed on a single event or individual, but if you read both reports you can see a lot of similarities between the two. In particular, both games were apparently harmed by the requirement that they use EA’s Frostbite game engine. Apparently a lot of development time was lost trying to add features to Frostbite to make it support an open-world roleplaying game.

Part of what makes this so confusing is that the features BioWare struggled with seem fairly tame. Conversation trees, branching decisions, and saving a global world state are all things BioWare pulled off in Star Wars: Knights of the Old Republic way back in 2003. If BioWare had those features in 2003, then why did they struggle with implementing those same concepts over a decade later when they had more time and a bigger budget?

I was a programmer before I switched to complaining about things on the internet for a living, and so a lot of people are asking me what I think of the Frostbite situation. Was EA dumb? Is Frostbite a bad engine? Are the BioWare developers bad at programming? What caused this mess?

While I don’t like to defend EA, I will say that their initial idea was somewhat understandable. Before Frostbite, BioWare had been building their games with the Unreal Engine. That meant EA was handing over a cut of sales to rival company Epic. EA employs around 10,000 people. They have access to hundreds of talented programmers. It’s pretty reasonable for an accountant to look at the millions of dollars the company was sending to Epic in licensing fees and wondering why EA’s subsidiaries couldn’t switch to using an in-house engine. Not only would it save EA millions of dollars on every game the company shipped, but support for that engine could come from within the company. It would also make EA less dependent on a business rival.

Getting everyone onto the same in-house engine was a good idea. The problem is that Unreal is a generalized engine and Frostbite is not. Unreal has tools and frameworks for making shooters, roleplaying games, action games, puzzle games, and just about anything else a AAA studio might need. Frostbite was originally designed for cinematic linear first-person shooters and was laser-focused on that niche. This made it difficult to adapt it to other tasks.

An Example Might Help

Around the turn of the century, I worked for ActiveWorlds, a massively multiplayer (by the standards of late ‘90s technology) world with chat and creative building. It was basically a precursor to Second Life. The base technology had been designed in 1994, and we were trying to modernize it to keep up with what people had come to expect from an MMO.

At the time, we had a feature where you could click on a door to open it. The problem was that the door motion was handled entirely on the client side. If I clicked on the door, I’d be the only person to see it move. If I went through the doorway, other users would see my character glide through the closed door. It was confusing and frustrating for users.

We wanted to make it so that other people could see it when you opened the door. This seems like a very simple problem. If I click on a door to open it, my client should just send a message to all the nearby users so that they would see the door open.

That was easily added, and it worked well enough. The problem is, what happens when a third user shows up? I opened the door a minute before they arrived, and everyone present at the time got the message so they saw it happen. But this newcomer wasn’t around for the message, so they see the door closed. Different observers once again disagree on the state of the door. To the user, this feels like a bug.

My client could continue to send out the door-open message to everyone who arrived, but that would only work as long as I hung around. If I left and someone new showed up, we’d once again have a discrepancy between newcomers and people who observed the original event.

The only way to really fix this would be to have the server track the state of every door, every switch, and every active object in the world. But some of our game worlds were gigabytes in size. There’s no way that was going to fit into memory on the computers of 2002 without building some exotic specialized machines. We could add the door state to the database, but then that would mean making changes to the database format which involved complicated economic problems.

If you’ve ever played World of Warcraft, then you’ve probably noticed the odd way it handles dense populations. Instead of attempting to update the game with the position of hundreds of other players, the server just picks the dozen or so people closest to you. This is fine, since in most cases the client wouldn’t be able to render hundreds of other players anyway. This means that as you run around the city, you see an ever-shifting crowd that seems to exist only in your immediate vicinity. The trick here is that you might be one of the 12 closest people to me, but because of the distribution of other players, I might not be in your list of 12 closest people. I could see you, but you couldn’t see me. ActiveWorlds worked the same way. Once again, this made things like door-opening messages tough to distribute properly.

The point is that this door-opening idea looked trivial on the surface, but it was really hard to solve within the existing framework. If we were writing a brand new game from the ground up, then it wouldn’t be a big deal to make it so door behavior was propagated to all users. That idea would be incorporated into the design of the software at the start, and would be taken into account when working out how data would be stored and sent. The problem was that we needed to fit our solution into the existing code. We couldn’t crawl down into the guts of the server architecture and rewrite everything, because the bosses would laugh at us if we told them it would take six months to fix the “door problem.”

It’s the difference between making changes to the foundation of a house you plan to build, and making changes to the foundation of a house that has already been built and has people living in it. Both jobs give you the same final result, but one in monumentally more difficult than the other.

Getting Back to Frostbite

This is the terrible situation the BioWare team found themselves in. None of the features they needed were particularly complex by the standards of modern game programming, but the Frostbite engine was particularly ill-suited to adding them. Fixing this would require large structural changes to how the system worked, even if the features seem minor on the surface.

The whole point of using a third-party engine like Unreal or Unity is that your team doesn’t need to employ specialized engine programmers. You can build a programming team focused on gameplay and not waste money employing engine design wizards. That’s great if you’ve already got an engine that does everything you need, but if your engine needs to be redesigned, then you’ll find your team is ill-equipped for that type of work.

Worse, it seems like the various BioWare teams weren’t able to share their work. Dragon Age: Inquisition, Mass Effect: Andromeda, and Anthem all had to make their games using Frostbite using personnel that were more familiar with Unreal Engine. It might be possible to coordinate between these three teams and make a generalized engine that could suit BioWare’s existing template, but that kind of work takes additional overhead, cooperation, and executive support. Instead, each team hacked together their own version of the Frostbite engine that just did what their particular game needed.

It Really is EA’s Fault

While the desire to move to Frostbite might be understandable, it’s hard to argue the shift was worthwhile. This disruption to the development process was costly and time-consuming. It’s doubtful the savings on licensing fees offset that expense, and that’s before you take into account the terrible toll it took on sales.

This goes back to the problem I talked about last month: The people running EA are not game developers or software engineers. They don’t have the knowledge required to understand how to do something complex like a company-wide engine migration.

Someone apparently looked at Mass Effect and said, “This is a shooter, just like Battlefield. So why are we paying Epic Games for their engine when we could be using our own engine for free?” If they had a background in game development or programming, they would have understood the scope and scale of the problem they were about to create.

The way to properly handle something like this would be to create a specialized engine team that was just focused on building, extending, documenting, and supporting Frostbite for EA’s subsidiaries. This team would talk to the various studios, get a feel for everything their engine needed to be able to do, and then add those features to Frostbite. A project like this would take years and cost millions of dollars. It wouldn’t be able to pay for itself right away. But this is what’s required to make a AAA game engine work in today’s world.

Instead, EA just told everyone to switch to Frostbite and left them to work out the details on their own. The result was that teams needed to work on a game engine that was unfamiliar to them, and they needed to do specialized engine work without access to those kinds of specialists.They had to do all of this while trying to ship a game. The result was buggy games, long development times, a massive duplication of effort, and a lot of frustration for huge numbers of artists who had to do their jobs with unfamiliar tools that were constantly in flux.

Publishers are supposed to support their studios and make sure the development teams have what they need to make their games, so it’s tragic that EA’s support is more or less indistinguishable from sabotage. BioWare made a lot of questionable decisions during the development of Dragon Age: Inquisition, Mass Effect: Andromeda, and Anthem, and the studio deserves blame for getting itself into a bad position during the development of these games. But when BioWare was already struggling to stay afloat, EA threw the studio a cinder block and pointed it towards the deep end of the pool. Maybe it’s not fair to say that EA killed BioWare, but it’s totally fair to say that EA failed to fulfill their role as publisher.

BioWare fans — myself included — have been speculating if BioWare is about to be closed like so many EA subsidiaries before it. The truth is, it almost doesn’t matter at this point. If EA can’t give BioWare the support and leadership it needs to make good games, then leaving the studio running won’t do anyone any good. It will just result in more games like Anthem and Mass Effect: Andromeda.

Frostbite isn’t the problem, but it’s a symptom of the problem. The problem is a lack of leadership stemming from a lack of domain knowledge. There’s no way around this. EA needs programmers, former game designers, or producers in executive positions. Until the company has that, it will continue to squander potential and deliver frustrating, underwhelming products.