Dotbot

Our Purpose and Goal (From the Dotbot website)

Our purpose is rather simple. We want to make the internet as open as possible. Currently only a select few corporations have a complete and useful index of the web. Our goal is to change that fact by crawling the web and releasing as much information about its structure and content as possible. We plan on doing this in a manner that will cover our costs (selling our index) and releasing it for free for the benefit of all webmasters. Obviously, this goal has many potential legal, financial, ethical and technical problems. So while we can't promise specific results, we can promise to work hard, share our results, and help make the internet a better and more open space.

One can download the results of their crawls...

Download our index (Warning: File size is approximately 3.2 Gigabytes compressed and about 14 Gigabytes uncompressed)

— From their web site