I just wanted to post a quick note to say we’ve changed the way the bot will be crawling. If you’re using either the manual or automatic pings, this will not affect you – we will still crawl these as they come in.
Otherwise, here are a few things to take note of:
- Unapproved blogs get crawled at night. If you submit a new blog, you’re going to have to wait at least a day for it to be approved (but probably longer).
- Crawling has been optimized so that we get through all the blogs within 90 minutes during off-peak hours. We’ll monitor to see if this remains similar throughout the day.
- If you’re a very active blogger, you will be prioritized over bloggers who post less frequently. This is an attempt to decrease the number of failed scrapes. We don’t want to scrape an inactive blog more than an active blog.
Also, we will probably be implementing some sort of “holding pattern” for inactive blogs. This will probably mean that, if you haven’t posted anything for a week or longer, we will only crawl once per night.
Again, if you’re using the pinging services, you shouldn’t have anything to worry about – we will still crawl them as they come in. This is probably the best way to get us your posts as quickly as possible. Another thing to take note of is, if you are using feedburner, pinging MAY not always work as quickly. This is because feedburner caches your feed and your latest post does not always appear immediately in your feedburner feed. If you are concerned about this, send over a link to your raw feed and we’ll replace the feedburner link with it.