Sorry for the delay in getting the new 4.3 raid content up. Normal and Heroic Dragon Soul should now be recording correctly. Let me know if anything is amiss.

Thanks!

I’ve been away from the computer the past week. I should point out that only Cataclysm content is being recorded in the new crawler. Are you guys seeing issues where that content isn’t being updated?

The good news is that I ran it unattended all week and nothing blew up. The bad news is, as some have pointed out, is that it is not that fast.

Ideally, I will release a public version but I need to see how that all factors into the daily limits that Blizzard imposes on the API.

Thanks for all your comments on the crawler. I will be spending some time this week digging into these issues and hopefully I’ll have an update soon. Please continue to post feedback on any issues you are having.

UPDATE 1: I identified and fixed a bug where bosses that are tracked using an achievement’s criteria were not being recorded. Also, I have just forced a recrawl of all guilds crawled this past week.

Update: Should be ready this weekend soon!

Update 2: The program is done but I am waiting for authenticated access from Blizzard. Hopefully I’ll get it soon!

Update 3: I have a key! I’m going to be running the new crawler locally for the time being. Once I iron out the bugs I’ll release a public version. Thanks for your patience!

Update 4: I’ve been running the new crawler this past week. It’ll definitely be a big improvement on the quality of our data. I’m going to continue to run it privately but since it is much faster than the previous methods there may not even be a need to release a public version. Thanks for hanging in there during the transition!

Update 5: We’re currently exceeding our daily request limit on Battle.net. Blizzard has tripled the amount of requests we can make but this hasn’t taken effect just yet. This is why the crawler seems to stop working in the late afternoon. Request limits are reset at 7pm EST.

Follow @guildprogress! Thanks!

Update: I’ve released a new version of the crawler which will correctly collect Firelands data. The problem is due to Blizzard’s original armory servers not providing consistent 4.2 data. I have since updated the crawler to use the new armory. However, due to the quality of the data available, certain kill dates will not be accurate. This will be corrected when Blizzard’s long awaited API is released. Please continue you to report any problems and thank you to those who are running the crawler.

Update: Looks like some people in China are running bots that are automatically queueing random guilds for crawling and this is clogging up the crawler queue. I’ll do what I can to keep them out.