Crawler

Software that follows links to discover new and updated pages on the web. Also called a bot or spider, for example Googlebot.