New WoW Crawler Completed!
by Xxav on August 7, 2011, 12:22 pm, under WoW Crawler, World of Warcraft
Now that Blizzard’s API is officially live, I am working on developing a new crawler program to take advantage of it. This should result in more accurate data as well as quicker updates. Stay tuned!
Update: Should be ready this weekend soon!
Update 2: The program is done but I am waiting for authenticated access from Blizzard. Hopefully I’ll get it soon!
Update 3: I have a key! I’m going to be running the new crawler locally for the time being. Once I iron out the bugs I’ll release a public version. Thanks for your patience!
Update 4: I’ve been running the new crawler this past week. It’ll definitely be a big improvement on the quality of our data. I’m going to continue to run it privately but since it is much faster than the previous methods there may not even be a need to release a public version. Thanks for hanging in there during the transition!
Update 5: We’re currently exceeding our daily request limit on Battle.net. Blizzard has tripled the amount of requests we can make but this hasn’t taken effect just yet. This is why the crawler seems to stop working in the late afternoon. Request limits are reset at 7pm EST.
July 7th, 2011 on 11:19 pm
i seen your post on the wow forums Xxav if ud like some help dev something send me a email and i have extracted all the info from pre 4.2 for achievements from wow i have them in cvs format if ud like them
July 13th, 2011 on 7:39 am
http://wow.guildprogress.com/US/Perenolde/The_Exiles/Cataclysm
There’s something wrong with the crawler – its posting achievement kills for my guild dated 5 months into the future… Its not November 2011 yet, is it?
Also the dates for our BoT kills are wrong.
July 13th, 2011 on 7:39 am
sorry, that should be BWD in the above… the dates are wrong. First kills were a few months ago, not last week.
August 9th, 2011 on 12:59 pm
“…but since it is much faster than the previous methods there may not even be a need to release a public version.”
Does this mean it can run the entire Guildprogress.com database of guilds in less than a week? Or even more often than once a week?
I know a lot of people like to get their progress updated 2-3 times a week. It’d be nice if we could get that without ever having to queue/force a guild crawl.
August 12th, 2011 on 10:12 am
Is this daily request limit from Blizz per IP or per app no matter how many people run it or where?
Based on the crawler page and recent crawls last hour, you’re only getting 40+ something per hour. That may be fast enough to handle the forced crawls, but that’s nowhere near enough to cover all the others that don’t get a forced queue.
Even if it ran 24/7, that would take over 2 months to crawl all the 65k+ guilds listed just under regular Cataclysm progression.
Yeah, we need a public version, to get more crawled.
that’s “only” 65k guilds. I see 108k guilds under Icecrown25 and 231k for Icecrown10.
August 14th, 2011 on 2:59 am
Please communicate!
How can we help you?
August 16th, 2011 on 11:44 pm
Since the new crawler went live, we haven’t been able to update our guild..
Crawler status says it’s running and later, timestamp is updated, but we’re still stuck at 2/7.
Also, even before API was updated, we sometines didn’t get updates here, for kills that was made in a 8/10 or 9/10 raid. It seems we need to be in a full 10/10 raid for the kills AND achievements to trigger..
However, Fore Play was done in a 10/10 raid, and we still haven’t been able to have it reflected here.
August 17th, 2011 on 2:18 am
I’ve queued/forced a crawl on all the top guilds on my server, and none of them actually got updated. Just the timestamp of last crawl changed, but it didn’t pick up any of their new kills, some of them dating back 3-4 weeks.
The new crawler may be crawling very fast, but it’s not really working since it’s not picking up the kills.
August 18th, 2011 on 8:15 am
I have to agree. Crawler seems to be broken because it doesn’t update the kills for my guild too. Been trying to force the update since yesterday but still nothing.
August 18th, 2011 on 7:24 pm
Have tried to update but crawler no longer working since updates. Achievements and Boss Kills no longer registering either 10 man or 25.
August 19th, 2011 on 8:19 am
The old public crawler no longer works. Since Blizz released it’s new API several weeks ago, Xxav had to write a new crawler. He did so, but has not released the new crawler to the public. Just as well, since IT IS NOT WORKING!!!!
He’s been running it himself, and it’s still running now, and he even updated above to say how fast it is. But it is still NOT updating kills.
August 20th, 2011 on 6:35 am
MikeW: I know. I’m not talking about the puplic crawler. I’m talking about the one Xxav is running. THAT one is not working.
August 21st, 2011 on 7:17 pm
i am from rexxar, and the new kills are not showing up when i update the guild haus der könige.