Google to start tracking content "behind" web forms?

According to a post in Google's Webmaster Central Blog, the company is experimenting with ways to start crawling content that is accessible via HTML forms. This will add to Google's index some of the content that was previously located in the "Deep Web," beyond the crawling capabilities of current spiders. According to estimates, tens of thousands terabytes of data are located in the Deep Web.

From the post: "In the past few months we have been exploring some HTML forms to try to discover new web pages and URLs that we otherwise couldn't find and index for users who search on Google. Specifically, when we encounter a element on a high-quality site, we might choose to do a small number of queries using the form... Needless to say, this experiment follows good Internet citizenry practices. Only a small number of particularly useful sites receive this treatment. (...) Similarly, we only retrieve GET forms and avoid forms that require any kind of user information. For example, we omit any forms that have a password input or that use terms commonly associated with personal information such as logins, userids, contacts, etc."

Google is already crawling parts of JavaScript and Flash files, undoubtedly in an attempt to get meaningful content and more pages out of the fashionable Ajax/rich internet applications. If their robot gets more intrusive, webmasters will need to look closer into their code and robots.txt file for ways to fine-tune the indexing process.