EQII: Rise of Kunark – Tradeskills Dev Diary #6

We polish off our behind the curtain look at EverQuest II: Rise of Kunark with the final set of four entries from tradeskills game designer Emily “Domino” Taylor. In the final entries she takes us from Halloween to yesterday, the very eve of the expansion’s launch.

A special thanks to Emily for her detailed articles.

Wednesday 31 October
Somehow October is entirely over already, although it feels as if it should still be summer. I’m pretty sure it’s Wednesday today although I don’t really recall what happened to Monday and Tuesday. On Monday the new content lock date was set: this coming Saturday evening. Delayed a week due to the San Diego brush fires (which the radio this morning informed me are still burning in areas, but just about under control now unless the winds suddenly pick up and do bad things). Nobody I know of at the office seems to have lost their home, nor been injured in the fire, and the people who were still evacuated last week are now back home. So all in all we’ve been very lucky.

Tonight being Hallowe’en, there are a series of children knocking at my door and asking for candy. It’s kind of fun; this is the first Hallowe’en I’ve actually spent in North America for over 10 years, and suddenly it’s a Big Deal. There were people at work in costume today, which seemed a bit weird to me to mix childrens’ games with a professional work environment, but is apparently normal in this country. Last time I was on this continent for Hallowe’en I was probably still trick-or-treating, or at least going to costume parties at university and not living anywhere that kids would come by. I did actually go to a costume party on Saturday, which was fun, and the first time I’ve been out past 3am for more years than I can remember. Tonight though I’m just fixing bugs and writing this, in between going to the door.

The new furniture graphics from Kunark objects were finished by the art folks at the start of this week, and so I spent yesterday and part of today making them into house items. Some of them are going to carpenters’ recipe books, some of them are being made into house items for the collection quest rewards (I offered to do this at the same time as I was making the carpenter furniture, since I’m probably the most familiar on the design team with making house items), and one’s going to be a city merchant purchasable item for level 80 guilds. It’s unfortunate, but there’s a strict limit on how many new house items can be added with an expansion, simply due to size restrictions. The entire size of the expansion can’t be over a certain size, and only a portion of that can be allocated to house items, so that limits the number of new graphics we can have. I never realized before, when I was just a player, the full reason behind why carpenters had been getting so few recipes in the new expansions. No other crafting class takes up that kind of download space but the carpenter stuff just eats it up. Fortunately I’d already prepared a bunch of other nice items from old world zones that will not add to the download size of the expansion, so carpenters should have more furniture in this expansion than they did in the last two.

Now that the remainder of the furniture is in, I’m just working on bugs from now on. I have to try and identify any bugs that actually require adding new content (new files, or new text that might need translating) and get those fixed by Saturday. Then there’s a few more days next week for other bugs. But basically, we’re supposed to aim to be done and making no more changes by the 7th, which is when they start testing final builds to go live with. That’s not a lot of time. It’s only 7 days in fact! Somehow I had been thinking I had more time for bug fixing … but time is flying by fast, and the week of fires ate up a lot of that in the confusion. So, nonessential stuff gets postponed to the next game update after the release. MissDoomCookie helpfully offered to handle any grammar/typo bugs that any of the team had outstanding, since she does regular coverage of the nightly typo reports already, so I passed a couple to her. Syzygy, who is actually on the coding team, came around and very generously offered to help out with any of my more straight forward bugs that wouldn’t require a lot of explanation for him to figure out what needed fixing. I had quite a lot left over from the removal of non-pristine products, things like some items not having had their crude qualities removed, or returning the wrong components. So I passed a ton of these on to him, which was a huge help and most wonderful of him to offer. Gallenite saw Syzygy’s bug fixes getting checked in tonight and just sent me a message saying “my god, either you have compromising pictures of him, or you’ve promised him more cookies than I can readily conceptualize”. Clearly I do need to do some serious cookie baking once time permits!

My tradeskill apprentice Olihin has taken the remaining quest bugs, since he did most of the quests that have outstanding bugs. He only works with the design team in the mornings, then does a full shift on Customer Service during the afternoon and evening. Even half a day having an extra person on tradeskills is a huge help though, and he’s been coming in after his normal Customer Service shift ends too on several occasions. He hopes to become a game designer some day, and I’ve been trying to give him tasks like planning and creating quests that will help him learn the skills he’s most likely to need for that. So he created the introductory faction quests for tradeskillers around Kunark, and is probably a better quest designer than me now since he’s done more of that. I’ve been taking the more boring stuff and stuff that requires writing perl scripts and such.

That leaves, as of tonight, 84 bugs still outstanding. Assuming no more are added (an unlikely assumption, alas) I need to do 12 bugs per day for the next 7 days, or about 1 bug per hour assuming 12 hour days for the next 7 days. I don’t know how realistic that is. Some will take less than 1 hour but some will take more, not quite sure how they’ll balance out in total. I don’t generally tend to be a particularly stressed-out or anxious person, but the volume of bugs remaining is definitely weighing on my mind. Last night I went to sleep at about 11, but woke up at 2. I’d been dreaming of nothing but bugs and people in beta having problems and not being able to do things, and I woke up remembering I’d meant to check in some bug fixes to writ quests before I went to bed, but I’d forgotten to. Then I couldn’t get back to sleep because all I could think about was all the bugs, so I gave up and got my laptop and spent about 2 hours fixing more bugs, before going to sleep again about 4 and waking up at 7 to get ready for work at 8. Not very restful! I hope I don’t spend the next 7 days doing the same, though I probably will. I’ve been dreaming about tradeskill bugs for most of the last month; that and coworkers doing random things. A month or two ago when I was wrestling with my perl scripts I dreamed Ilucide was actually being created from running a perl script that was compiling him up from code. Which was kind of disturbing…

Yesterday we did a lunch time play test of Karnor’s Castle. They ordered lunch in for us so we wouldn’t have to leave our desks, and could just play straight through. We took two groups in (it’s not instanced) and went in different directions. The quests aren’t in yet and there were some odd bugs with aggro that made it a bit difficult to pull, so we didn’t really get the feel for how it will be when those are done. Overall though it was fun, and the zone was very reminiscent of the old EQ1 Karnor’s and looks great. Jindrack was the tank for my group, playing a monk; I played a dirge as usual, which is my favorite class for grouping with.

I’ve been relaxing in the evenings lately by making little model slugs out of Fimo modelling polymer while watching a DVD. I watched “Flushed Away” a couple months ago, and that movie has cute little singing slugs all throughout it. In the DVD’s special features they had a little tutorial on how to make a slug. I’d bought the Fimo at least a month ago when I saw it on sale at Michaels, but it’s been sitting on the shelf for ages. In the last week or so though my eyes have been getting so tired from staring at the computer screen for so long that I’ve been having trouble focusing when I drive home in the dark. So when that happens I take a break from the computer screen and make a little slug or two. I actually used to make little model dragons out of Fimo and similar modelling substances, but they are a lot more work and require a lot of fine detail. Slugs about match how I feel at the moment. I’ve made about 8 so far, and started leaving them around the office on random people’s desks just to add a touch of surrealism to everyone’s life. Fyreflyte got one of the first; Feconix has one, Cat from the audio department has one, Tom in art who did the fantastic new graphic for tinkered fireworks has one.

Last night, following the play test, I made a slug that was feigning death, since at one point Jindrack used his group feign to save us all from adds on a pull. It’s very cute – lying on its back, little tail curled up, tongue lolling out. I stopped by Jindrack’s office today (Ilucide, Vhalen, and Jindrack share a corner office) when only Ilucide was in, and asked him if Jindrack would think I was completely insane if a feign deathed slug appeared on his monitor. His answer was “Probably, but I think he needs to see that,” so I left it on top of his monitor, and a non-feigned one for Vhalen too for good measure. Neither of them have said anything so far. Ilucide stopped by late this afternoon to ask Fyreflyte something about item names, and I asked him if Jindrack had noticed the slug.

“Yes,” he said, “he said ‘what the hell is that?'”

“And did you tell him,” I asked.

No, apparently he said he’d just denied knowing anything about anything. And he didn’t know if Vhalen had noticed at all. Men … So Jindrack probably either has no idea what it is and where it comes from, or possibly thinks I’m too dangerously insane to risk mentioning it. Hopefully he was amused at least. Slowly but surely as time permits, more slugs gradually invade the office, at least until I figure out how to lead my newly created slug armies to take over the world … or until I get bored of slugs and start making something else. Right now however I need to get back to bug fixing; three new ones have arrived just while writing this. I’m starting to dread that new email beep sound.

Saturday, 3 November

Today is final content lock, postponed from last week due to the fires in San Diego. Today at 9pm, to be exact. Currently it’s about 7:45pm, and I just got back from the office; there were still plenty of people there when I left, too. Lunch was ordered for us, and dinner too, and various people brought in goodies from muffins to beer to cinnamon buns to keep everybody going. Content lock means that after tonight, we are not supposed to add any new files or change any text that is visible to players. So we do have a few more days to fix bugs that don’t involve that — for example, tweaking the stats on armor, which is one thing still on my list. I went through my remaining bug list several times and I think I’ve fixed everything that involved new files and text changes, although sometimes you don’t know until you start fixing something. I hope I got all the text changes done anyway, as the translating team are going to be busy getting all the text properly translated for the international servers between now and release date.

Thanks in large part to the heroic efforts of many others, my bug list is miraculously down to only 20 items at the moment. Syzygy in particular has gone above and beyond the call of duty helping out, especially considering he’s not even in the same department! MissDoomCookie helpfully came up with some text for the faction reward letters for the Gorowyn tradeskill faction when neither Cronyn nor I had the time or inspiration to think of anything intelligible for them to say. Jindrack was a huge help untangling some complicated hail handlers today too. Hail handlers still give me headaches on a regular basis.

I was very grateful when Jindrack said he’d finished up his bugs and would be happy to help with the three hail handler bugs that were sitting on my list. The folks like him who write quests on a regular basis are much better than I am at spotting problems with hail handlers than I am. Even if they don’t seem to appreciate slugs feigning death; Jindrack still hasn’t said a word and the slug has been moved aside onto his filing cabinet. It may need to find a new home, poor rejected thing! I’m sure it will amuse someone somewhere. (Lyndro, Gallenite, Silverfrost, and MissDoomCookies have all acquired slugs of their own since I last wrote.)

One thing I’ve noticed very much in the past month is how helpful everybody on the team is. We all have different deadlines, but people are always offering to help out where they can, and I certainly owe thanks to many more people than I listed above for lending a hand here and there, not to mention to all of the QA team for being so thorough and catching so many bugs that I missed.

Where I was working before as an IT manager, I was really the only IT staff for a long time, and when I did get a contractor to help out, I never felt I could ask a contractor to work extra hours when they were being paid only on a 9 to 5 basis. The rest of the company was basically sales and marketing, and couldn’t have helped out for the most part even if they’d wanted to, which probably wouldn’t have occurred to them since it was my job to support them. So I’ve never been a part of a team this size who are all willing and eager to pull together and help out whoever needs a hand to accomplish a common goal, without worrying about whose job description is what and which department does which thing. It’s a pretty good feeling! Except of course for the part where I feel like I’m the one who’s needed the most help. But by the next expansion I hope I’ll be more on top of the game and in a better position to lend a hand rather than need it. I guess I did do a few things for people here and there — recipes for Srukin and Kander, collection quest reward furniture, stuff like that — but it’s mostly just been doing things for tradeskillers! Anyway it’s pretty amazing how everybody’s so willing to help out everybody else, even between different departments and different areas, it’s all just a bunch of people pulling together to make the game great, which is the one thing we all want most of all. It’s pretty cool, man, as they say here in California. I could get used to this!

Friday 9 November
Just a few days till release date, and things are looking pretty good. Tuesday was the last big push to fix issues, and I know that I, and probably many others, were in the office 15+ hours that day. In theory all we’re doing since then is fixing bugs, and in fact that theory is close to reality since we pushed so hard up to Tuesday getting things cleaned up. Since bugs are all we’re doing, we are starting to be able to work slightly less long hours, which is a nice change. As things have been calming down a bit, we’ve been doing more play tests of various zones for some last minute tweaking of difficulty levels and identifying any bugs that might have come up. Fyreflyte took advantage of the extra time to decorate our office with Christmas lights, making it look very festive and cheerful. Cat, our audio lead, has been so enthused about the little slug that ended up on her desk that she persuaded me to show her how to make them, and since we actually were able to leave work by 7 last night, she and Fyreflyte came over to my place for pizza and cheesecake, and we watched the DVD of “Flushed Away” while making random slugs. It was a nice break from bug fixing all evening!

The majority of my bugs this week have been fairly non serious stuff, the kind of things that should be fixed of course, but aren’t going to break anybody’s game if they didn’t get done in time for release. Things like having the wrong tint on some of the armor, or recipe names not exactly matching the item name. I also did a bit of tweaking of some of the crafted item stats, to make them a bit more desirable, and to address some of the concerns raised by the beta testers posting in the beta forums. There have been so many very helpful players taking the time to check all the items for their crafting classes and offer suggestions where things didn’t make sense to them, and identifying bugs where they found them, it’s been wonderful. Although in most cases QA catches the bugs also, things that aren’t outright bugs but just inconsistencies aren’t things that are necessarily readily apparent to someone who doesn’t know that crafting class inside out, so this kind of feedback is extremely valuable and I have definitely appreciated everyone who took the time to do that.

Monday 12 November
Tomorrow’s the big day; I was starting to think it would never arrive, but we’ve made it at last! The Rise of Kunark release, my very first expansion and the fourth expansion for EverQuest II. Of course, I’ve only seen the other three expansions as a player, not from the development side, but this one seems pretty solid to me and Gallenite has been telling us the same thing. Each day for the past week and a bit he’s emailed around a graph showing how many bugs for the expansion are still open for the dev team and for QA, and every day that number has fallen dramatically, until at the start of this weekend we were down to two digits, and by today I imagine there were almost none at all. (Of course, many of the QA queue bugs might not be bugs any longer, just waiting to be tested and signed off.) I know that in my own bug queue there’s been very little left for this expansion in the past few days, and mostly pretty minor things like spelling errors or inconsistencies in status reduction amounts in furniture, etc.

It being the weekend before release, we were all asked to come in on the weekend and ensure that any bugs that did come back from QA were promptly turned around. Normally we make changes and fix bugs and add things all day during the day, and then overnight an automatic process rebuilds stuff into a new version, which then goes to our internal servers in the morning so QA can test our changes on that. Over this weekend, however, we were turning over the builds much faster than usual, trying to get everything cleared up and tidied away. The dev team and coding team would fix their outstanding bugs, and then a new build would be run. This would take about 90 minutes to 2 hours and then QA would check out all the bugs we fixed, and either close them off or send them back. Then we would fix any bugs that got sent back (or, occasionally, new ones discovered) and once those were all checked in, another build, another round with QA, and so on. This went on for pretty much the whole weekend.

There was a lot of waiting around in between times while the builds were happening. Some people worked on new content for next month, some did play testing or chatted with players on the beta server, others who lived close went home in between. Since we’re in really good shape regarding number and severity of bugs it wasn’t overly stressful, mostly a lot of anticipating and waiting. I ended up going home during the build yesterday afternoon and baking some chocolate chip cookies to bring in today, which actually worked out pretty well for the folks who stayed late in the office, because when I had to go back in around 7pm last night to fix a bug that had come back, I was able to bring a batch of fresh baked, still-warm-from-the-oven cookies for the QA team and the remaining developers and coders to enjoy. The rest of the cookies came in today and were divided between QA, who have been doing such a fantastic job catching and checking stuff I’d missed, and the dev and code teams, who have also been working flat out of course.

Today the very last few remaining bugs were addressed, the update notes for the expansion were confirmed and given to the international community reps for translation into the various languages required, and from noon onward we were forbidden to change anything more without express permission. The live servers will go down at 5am PST tomorrow for the RoK updates, and folks from the Community team will be in from 6am to answer questions and concerns on the forums. Key members of the development and coding teams will be in early also just in case of any issues that come up and need addressing. By mid day we’ll probably know whether it’s all going to go smoothly or whether something will go horribly wrong — we’re all hoping for the “smoothly” eventuality, of course! And if all goes well, we’ll be back to working normal length days, which means I’ll have entire hours free in the evenings to do non-work things in. I’m going to have to figure out how on earth to fill all that time now, it’s been a while since I had that problem!

So now I’ve (almost) lived through my first expansion cycle, it’s time to contemplate what I have learned, and what I can improve for the next time around. Just knowing how it all works and what to expect will be a big benefit of course, and next time I’ll have a better idea of how to time things, such as how long to leave for recipe and item creation (minimum two weeks each, preferably more). I want to create some better spreadsheets and scripts for the recipe and item creation also; the ones I was using were old and outdated, and caused much extra work because in many cases nobody realized they were outdated until all the items (or recipes) were created and then later discovered to be missing this or that flag, or incorrect in some other way. Getting spreadsheets and scripts set up to accurately automate as much of it as possible should save a lot of time and make for a lot less QA work catching persistent errors introduced by faults in the spreadsheet or script. And I think I should start working on the next expansion, oh, about next week, just to be sure! Mainly I think a good lesson learned is that I did try to bite off a bit more than I could chew this expansion — trying to fix a lot of the more urgent tradeskill issues I wanted to address once I started the job, while at the same time planning an entire expansion, revising all the tradeskill reaction arts, removing subcombines, and implementing many new crafting quests, was probably a little on the ambitious side for my first expansion, even with the part time help of our tradeskill apprentice Olihin. All the combined bugs from all the large changes, while a very small percentage of the overall number of files checked in, still added up to quite a pile and without the help of people like Syzygy it would have been an uphill battle to get through them all in time. I think from now on I’ll aim for just one major change at a time, for the sake of my sanity. It’s hard not to want to do everything at once, when there’s so much I still want to do, and patience has never been one of my virtues … but I’ll try and take that lesson away from this expansion at least. EQII is three years old as of this month, and any changes worth doing can wait just a little bit longer to be done with as much attention and care as they deserve. And there are still many changes still to come…

I guess this will be my last dev diary entry for Rise of Kunark. It’s been interesting writing them, and I hope it’s been an interesting read for whoever may be reading them, and perhaps it’s given you a little more insight into just what goes on behind the scenes of a game expansion, and just how much work goes into each one. I haven’t even seen the whole cycle; preparation for the next expansion is going to be starting as soon as this one’s out the door (in fact, planning has already started). Things were already well underway for this expansion by the time I joined SOE at the end of April, so in the next few months I’ll get to see what the early stages of a new expansion look like.

If all goes well, perhaps you’ll be reading another expansion dev diary from me this time next year. See you then!