"Spiders have been around since the early days of the Web, but what’s getting interesting is that as the data resources out on the Net get richer, programmers are building more specialized spiders–and here’s the really cool bit–sites built with spiders themselves are getting spidered, and spiders are increasingly combining data from one site with data from another. […] Eventually, these inefficient, brute-force spiders, built that way because that’s the only way possible, will give way to true Web services. The difference is that a site like Amazon or Google or MapQuest or E*Trade or eBay will not be the unwitting recipient of programmed data extraction, but a willing partner."
