Well, I’ve been a bit busy. I posted an update video last week.
And I just posted this to the thread on the US forums.
Dear Blizzard,
I’m known online as Kurn. I’ve been playing World of Warcraft, off and on (though mostly on), since 2005. I have served as a guild officer in multiple guilds, a healing lead, a raid leader and a guild leader. I have run a blog about WoW and my adventures within it since 2007. For a time, it was the most popular blog dealing with holy paladin stuff, through much of Wrath of the Lich King and Cataclysm. I had two podcasts to do with World of Warcraft as well. Finally, I have a small YouTube channel with almost 300 videos, most of which have to do with WoW or other Blizzard properties, which was started in 2006.
To be clear, I’m sharing this information so that you can understand that I am a long-time player who cares about WoW and I am someone who has given back to the community — and continues to do so.
I am writing this post today because of the guild bank issues that continue to plague hundreds, perhaps thousands, perhaps tens of thousands of guild banks. There are other issues that also need to be discussed, such as guild bank logs, warband bank issues, void bank issues and problems with reputation, not to mention weekly quests not resetting, though I have much less information on those. As such, I’ll keep my discussion to the guild bank issues, while bearing in mind that there are many systems that have been adversely affected since prior to the launch of The War Within. I did want to note that apparently the warband bank issues are related to the visibility of the items, but this does not seem to be the case with guild bank items.
TIMELINE OF EVENTS
– On August 13/14 (depending on region), Patch 11.0.2 was released.
– In the patch notes for this release, it noted that Cross-Realm Guild bank functionality was now available.
– Shortly after this release, reports started to roll in that items were missing from guild banks.US thread in the Bug Report forum:
https://us.forums.blizzard.com/en/wow/t/guild-bank-missing-items-what-happened/1918643EU thread in the Customer Support forum:
https://eu.forums.blizzard.com/en/wow/t/guild-bank-items-gone/528709Of note, the US thread did not receive any responses from any Blizzard representatives, while the EU thread received several, mostly from Felranys, and many of these were apologetic in nature. These were mostly along the lines of “I’m sorry I don’t have any new news, details have not yet been shared with us”.
– On September 20th, Linxy posted threads in both the EU and US forums, which was the first the US regions heard anything from a Blizzard representative:
US: https://us.forums.blizzard.com/en/wow/t/guild-bank-missing-items-update/1963451
EU: https://eu.forums.blizzard.com/en/wow/t/guild-bank-missing-items-update/538628In it, they said:
Due to how some of the data was lost, we’ve reached a point where the result will be an incomplete restoration for some guilds, and we do not have a way to restore the remaining missing items for them.
The important thing to note here is that this heavily implies that some guilds will have an incomplete restoration, yet the rest will have a complete one. However, based on the responses since then, it seems that the reverse is true. Some guilds had an incomplete restoration, while the rest have had nothing restored.
For myself, my personal guild bank is the one that’s affected. I received the following, after losing hundreds of patterns and entire tabs being wiped clean:
2x Pink Mageweave shirts
1x Wildvine Potion recipe
1x Wildthorn Mail recipe
1x Schematic: Dark Iron RifleNot even the items shown in the bank logs themselves as being deposited (and not withdrawn) were restored. I’ve read hundreds of comments and hundreds more forum posts from others who are worse off than me.
– On September 23, the US thread in the Bug Report forum was closed without any response.
– On October 2, the EU thread was closed with this response from Yryadorne, directing people to the general EU thread:https://eu.forums.blizzard.com/en/wow/t/guild-bank-items-gone/528709/732
– Almost two months after the patch, restorations still seem to be going out, though I still have yet to read about anyone being fully restored. (As a point of data, I have not received anything since that one mail in late September.)
RESTORATION OF ITEMS… AND TRUST
In my 19 years of playing World of Warcraft, I have never, ever, ever run into a situation with my character or a guildmate’s character where Blizzard did not do the right thing. As any of my former guildmates will attest to, I have opened a ridiculous number of tickets over the years to ask for Blizzard to fix something. A guildmate of mine had lost her Vashj vial for the Vials of Eternity quest (which led to the Hand of A’dal title). I have screenshots of me standing in Mount Hyjal, arguing passionately for my guildmate, and getting stomped by the incoming boss because I was talking to a GM about it. Ultimately, the GM discovered that my guildmate had accidentally dropped the quest — which had deleted her vial. Totally get it. It was my guildmate’s fault. The GM did the right thing in telling me that she had just picked up the quest a few days ago. The collaboration and communication from the GM in question was outstanding.
Years later, there was a bug in terms of normal/heroic switching in Blackrock Descent in Cataclysm, which meant that my raiding guild was stuck in two separate instances of BRD. Once again, I reached out to Blizzard for help and, once again, they did so. We went off to do Bastion of Twilight and the next raid night, we were properly synchronized to the right Blackrock Descent instance.
Never, in my long history in WoW, have I been let down by Blizzard the way I have been when it comes to the guild bank items disappearing.
(Yes, I know the GMs and community managers are different than the developers, and that the GMs are limited in what they can do. I’m using the GMs as proxies for my experiences with Blizzard as a whole. They take a lot of crap from users and they deserve better than they get.)
As a player, it is gut-wrenching to look at empty bank tabs where hundreds of items should be. And my items weren’t even all that personal. They were items I collected over more than a decade of play, storing away for a variety of reasons. Some are no longer available in the game. Some are available, but extremely limited in number (dragonscales, for example). And it just flat-out sucks that they’re gone. Part of me wants to trust Blizzard when they say, “we do not have a way to restore the remaining missing items”.
As someone who works in tech, though, I can’t trust what Blizzard is saying here. Blizzard has been running this game for more than 20 years (when you count alpha and beta and such). As such, they know how important data integrity is. They know that every byte of data matters. And they probably have PETABYTES of data. One gigabyte is 1024 megabytes. One terabyte is 1024 gigabytes. One petabyte is 1024 terabytes. Blizzard’s data probably is in the realm of petabytes. Worse, that data is always changing. Anytime you vendor a grey or disenchant an item or buy something or upgrade something or put something in the bank or loot something, the data changes. And that’s just inventory. What about levelling up? There’s experience to think about, reputation, how much mana or focus or rage or whatever you have, plus how much health you have. The data is always changing. It is almost never static.
Guild banks are included in that. Pull out 10k gold, deposit a BOE, withdraw a stack of Black Lotuses, drop in a stack of Flasks of Distilled Wisdom. It shifts, it changes. It’s not quite as dynamic as a character’s inventory, but in a large guild, yeah, stuff can be withdrawn or deposited, including repair costs and such.
What I’m trying to illustrate here is that, particularly for larger guilds, the state of the guild bank NOW is likely to be very different from the state of the guild bank when the bug occurred.
Let’s say that Blizzard has a snapshot of all the data from right before that maintenance period. The amount of work to reconcile all the changes between Day 0 (when the bug happened) and Day 50-something (which is the period of time we’re at right now) is likely to be A LOT.
So when a Blizzard representative tells me that they “do not have a way to restore the remaining missing items”, that’s where I’m stuck. That’s why I’m still out here making a big deal about this. You want me to believe that the devs who have made (and maintained) this game for 20 years did not make a backup of the game’s data before adding a patch to production servers?
I don’t buy that. Data integrity is vital. Anyone who has ever touched a live database knows that you back it up and, ideally, have another backup for it in another physical location, just in case.
You want me to believe that the devs who have made (and maintained) this game for 20 years did not keep backups of the game’s data for 30/60/90 days after the patch?
Super unlikely to me. Storage is cheap. For a game that just launched an expansion that still has subscription fees, storage is a pittance.
And yes, I know there have been layoffs. I know Blizzard has been affected. I’m saying that at this point, 20 years in, I’m pretty sure that any organization would have very specific processes for maintenance during the application of patches. What if the patch explodes and destroys things? Then they roll back to the backup and try to figure out what went wrong on their dev instances and then they try again. But they always should have the opportunity to roll back.
I’m not saying they should roll back NOW, mind you. I’m just saying that the data absolutely must exist somewhere. It doesn’t make any sense at all that they wouldn’t have this data from mid-August, prior to 11.0.2. That’s where I’m stuck.
Would it be an absolutely incredible amount of work to reconcile the state of guild banks between mid-August and now? Yes. My opinion is that if they have the data (and they should have the data), they can do it.
As such, I think Blizzard is saying one thing to make us think that they cannot restore things when, in my opinion, they are CHOOSING not to do so. And I think that’s a mistake. I’m not just saying that because I’m affected and I will benefit from them restoring the banks. I’m saying it because we’ve already lost trust in Blizzard by virtue of having lost the items in the first place, the possibility that they are able to and choosing not to hits even harder.
Everything in this game is a line in a database. If they won’t restore guild banks, who’s to say that this couldn’t happen to other things? And who’s to say that they won’t restore those other things? I don’t want to lose my Hand of A’dal title. What about my Rhok’delar? What about my Benediction? What about my reputation with, say, Timbermaw Hold? Or the Argent Dawn? What about my Feat of Strength, The Fifth Element? If guild bank items can get wiped like this, what’s to stop something from wiping out these achievements or items or titles? And what’s to stop Blizzard from NOT restoring them?
I get it, they’re not obliged to restore anything. I’ve read their end-user license agreement (EULA). I maintain, though, that there’s a difference between being obliged to do something and choosing to do the right thing. Further, they probably can’t be honest with us, due to the drama that any admission that they don’t want to waste the time on restoring the guild banks would cause.
So Blizzard is in a tough spot right now. The restoration of items may never occur. So will our trust in Blizzard ever be restored?
For me, it won’t be, not as it stands. I need a real explanation or a full restoration, ideally both, before I can put any faith back into Blizzard. Should I ultimately decide to step away, I know you won’t miss my single subscription. I would hope, though, that you have someone counting the number of people affected by this who decide to cancel their subscriptions. I have a feeling there will be more than few.
Thank you for your time.
In other news, I tried about 10 pulls on Zekvir and got OBLITERATED, so that was hilarious.
I really do have other things I want to talk about — goldmaking, professions, how I’m finding the expansion. But it’s with true regret that I say that this guild bank thing is sapping my energy and enthusiasm to play.
That was extremely well written, Kurn.
Anyone who has ever touched a live database knows that you back it up and, ideally, have another backup for it in another physical location, just in case.
To pull a rabbit out of a hat, this is why Iron Mountain exists. Very powerful companies put their trust in both their backups as well as sites/companies such as Iron Mountain to hold their backups, and that a company that is now part of Microsoft is unable to fix guild banks because… reasons related to backups… that is a terrible look for that company (Blizzard). And by association, Microsoft itself.
If this were Verizon or Citibank that had a data loss issue with using Microsoft software/cloud services, you can bet that investors and customers would be up in arms. That Microsoft would be willing to shrug about data surrounding a billion dollar business that is XBox Game Studios, that doesn’t look good.
I am putting in terms of being a Microsoft problem, not a Blizzard problem, because it is Microsoft’s problem now. They own Activision Blizzard, and their reputations are now intertwined.
And yes, I’m sorry about this whole debacle sucking away interest in a game that you play in to escape for a while. That it is in an area you actually work in (IT) makes the situation worse, because not only is your stuff gone but you know the likely source of the problem and how to fix it, and… It’s like playing Retail WoW has turned –quite literally– into your work.
But hey, if you ever want to chat, you can find me (indirectly) through the blog; I do check the blog’s email address once a month on average. (Not that I get anything more than spam for the most part.)
Thanks, Redbeard. It’s taken me a couple of weeks (and two videos!) to get to the point where I actually was able to voice All The Things in a way I wanted to.
Blizzard saying that there is no way to restore things is such a cop-out answer. What happened to the data that means it’s not restorable? Is it that the data went *poof* or is it that it can’t be restored because they won’t put people on it? It’s maddening to not know which it is.
As to MicroBlizz, it’s true, they are definitely intertwined now, but I don’t know that a lot of people necessarily know that. They’ll learn it, as we go on and I’m sure that Microsoft will make us get Microsoft accounts instead of our Battle.net accounts and stuff like that, but do average folks know yet? Don’t know. My guess is no, because TWW launched wholly under Blizzard (Activision Blizzard and Activision Blizzard King, according to the credits). So I’m not sure their reputations are quite as intertwined as all that yet… but we’ll get there. If this doesn’t get resolved, the fumbled launch of TWW will be the point on the timeline people will look at and go “oh, that’s where it all changed” and then they’ll see the MS acquisition was less than a year before and then go “ohhhhhhh. Oh. Ohhh.” (IMHO)
As to work, I work in tech but I’m not fiddling with databases all day long. Thankfully, database stuff is only part of what I help customers with. More often than not, it’s “how do I upgrade my software without blowing shit up” and my first piece of advice is always “first, please take a backup, here are the instructions (link)”. It is frustrating because I’m sure they’d find scores of volunteers to help rebuild the data, and for free, but they’re going to keep it in-house. But like, I know how to use SQL. Looking at more recent job postings, though, SQL may not be all they’re using, and I’m less familiar with NoSQL type databases. So who knows how it’s all set up? I just know that if they have a backup, they could do it. The big changes were in 11.0, not 11.0.2. I’m sure that any data they have from 11.0.1, prior to the 11.0.2 patch, would work just fine as a reference. It’s just the reconciliation that gets harder and harder every. single. day. I’ve hardly used my guild bank since the warband bank came out because I was hit with another bug, too — being unable to withdraw more gold from my guild bank than I had on my character at any given time. So I moved all my gold (painstakingly) to the warband bank. But think of all the guild banks affected who are still using their bank now that the bug that caused the issue is now fixed, even though the resulting chaos is not. Every single day, that’s millions of transactions.
Frankly, were I in charge, I would give players 1 week’s notice and tell them that for the next month, the guild banks won’t work. Then I would spend those four weeks working on the guild bank issue and little else. But they won’t do that. They have the 20th anniversary stuff coming. They have bugs to squash, features to release, delves to fix, other expansions to work on.
It’s a thing about software (including games): sales needs dev to work on new stuff, because that’s how they get sales. Meanwhile, support needs dev to work on fixing old stuff, because that’s the stuff that’s bothering the older customers. And time after time, I see software companies failing to nicely balance these two needs. They either cater to the new or cater to the old and here, I think we’re seeing Blizzard/Microsoft cater to the new. NEW content, NEW season, NEW expansions. While people who have been here for 19 years, like myself, get tossed the occasional bone. “Here’s your two pink mageweave shirts! Enjoy!” 🙄
Anyhow, I appreciate the offer to chat. I’m mostly just down about this whole thing and when it becomes clear that I have done my best to champion the cause to no avail… I guess I’ll be done. We’ll see how it goes and how much energy I have for this nonsense. 😅
No worries; I work in IT as well, so I can appreciate your situation.
Blizz tends to focus only on the new, which is why they failed to see the interest in Vanilla for so long. (The “you think you do but you don’t” fiasco and all that.) Even now, they try periodically to “fix” Classic Era by making a few changes nobody asked for that get the Era community up in an arms; some they have reverted back, but others (notably the guild tab change to align it with Retail) they’ve held firm on.
It’s the “lalala I don’t HEEEAAR you!!” that I dislike about their behavior. If I wanted that sort of behavior out of an IT company, I’d go to Broadcom.
Oh, and Vid and a couple of other long time WoW bloggers know how to get in contact with me directly if you need to get a hold of me.