Google have announced new configurations for the crawling and indexing of locale-adaptive pages over on the Webmaster Central Blog, which centre around locale-aware configurations for Googlebot.Â
This is big news for international websites that offer different content based on language or geographical locations, and could have a big impact on SEO.
What is a Locale-Adaptive Page?
A locale-adaptive page is a webpage which dynamically adapts in accordance with the language preference and perceived geographic location of the visitor. This means a webpage can offer differing content to users from different countries, despite it using the same URL. This may also mean that some countries have access blocked, or have different content available due to the language header.
What’s the Locale-Aware crawling announcement about?
Before now, Googlebot has crawled your website acting as a user accessing your website from the United States, which isn’t that helpful for websites based elsewhere. Not all variants of content which locale-adaptive pages offer may have been indexed completely. This is down to the fact Googlebot crawls with a default IP address that appears to be based in the USA, and  as Googlebot does not set an Accept-Language HTTP request header.
However, now Google are supporting crawling, indexing, and the ranking of locale-adaptive pages from elsewhere. This means that Googlebot now has locale-aware crawl configuration, so any content Google detects may adapt due to a user’s location and/or language will be indexed and crawled more completely.
Google describes the new configurations as follows:
- Geo-distributed crawling where Googlebot would start to use IP addresses that appear to be coming from outside the USA, in addition to the current IP addresses that appear to be from the USA that Googlebot currently uses.
- Language-dependent crawling where Googlebot would start to crawl with an Accept-Language HTTP header in the request.
What does the new Locale-Aware crawling announcement mean for international websites?
Essentially, now that Google may crawl your website from perceived IP addresses outside of the US (and languages other than US English), it means that Google will be crawling your website in more depth for content which may apply to searchers around the world.
Google recognises a variety of signals to determine whether your site serves locale-specific content, which they state includes:
- Serving different content on the same URL—based on the user’s perceived country (geolocation)
- Serving different content on the same URL—based on the Accept-Language field set by the user’s browser in the HTTP request header
- Completely blocking access to requests from specific countries
What do I need to do?
You don’t have to do anything. Locale-aware crawling is enabled algorithmically, based on whether Google detects your site will benefit from it.
However, as this is still a new feature, the Googlebot may not cover all combinations of languages and countries available; but at least it’s a step in the right direction. Despite the fact many websites use separate URLs for differing locales, this new announcement covers the websites where it does not make sense to do so.
What’s the best practice for international websites now?
Google still recommends that international websites should have entirely separate URLs for differing locales, and also that websites should use rel-alternative-hreflang annotations for them.
Google are still dedicated to user experiences, and they deem separate URLs as still being the best experience possible.
For more information, read the Webmaster Tools help section on Locale-aware crawling by Googlebot. Alternatively get in touch with the team here at Xanthos, or call us on  08450 740 068 if you have any concerns regarding your website.
