网络爬虫又被称作网络蜘蛛，网络机器人，主要用于收集互联网上的各种资源。

Web crawlers, also known as web spiders, web robots, are mainly used to collect various resources on the Internet. It is an important part of the search engine, and is a program that can automatically extract the contents of specific pages on the Internet. General search engine web crawler workflow: (1) Put the seed URL into the URL queue to be crawled; (2) Remove the URL from the URL queue to be crawled, read URL, DNS resolution, web page download, etc. Operation; (3) Put the downloaded web page into the downloaded page library; (4) Put the downloaded page URL into the crawled URL queue; (5) Analyze the URL in the crawled URL queue to extract new URL is placed in the queue to be crawled, and enters the next crawl cycle. <br>This topic studies the basic implementation of web crawlers, and the workflow of this topic: (1) crawl page code through URL; (2) use regular matching to obtain useful data on the page or useful URL on the page; (3) process the acquired The data or enter the next round of crawling cycle through the acquired new URL.

Web crawlers, also known as web spiders, are used primarily to collect resources on the Internet. It is an important part of the search engine, is a program that can automatically extract the content of a particular page on the Internet. General search engine network crawler workflow: (1) put the seed URL into the URL queue to be crawled; (2) take the URL to be crawled from the URL queue to be crawled, do read URL, DNS resolution, page download, etc.; (3) put the downloaded page page into the downloaded page library; (4) put the downloaded page crawl URL into the crawled URL queue; (5) analyze the URL in the URL to be crawled to be taken into the url to be crawled to be and go into the next crawl loop.<br>This topic studies the basic realization of network reptiles, the workflow of this topic crawler: (1) crawling page code through URL; (2) obtaining useful data on the page or useful URL on the page through regular matching; and (3) processing the obtained data or entering the next round of crawling cycle through the acquired new URL.

Web crawler, also known as web spider, is mainly used to collect various resources on the Internet. It is an important part of search engine and a program that can automatically extract the content of specific pages on the Internet. The workflow of general search engine web crawler: (1) put the seed URL in the URL queue to be crawled; (2) take the URL to be crawled out of the URL queue to be crawled, and do such operations as reading URL, DNS resolution, web page download; (3) put the downloaded Web page into the downloaded page library; (4) put the downloaded URL in the URL queue to be crawled; (5) analyze the URL in the URL queue to be crawled to extract a new URL The URL of is placed in the queue to be fetched and enters the next fetching cycle.<br>This paper studies the basic implementation of the web crawler, the work flow of the crawler: (1) grabbing the page code through the URL; (2) obtaining the useful data or URL on the page through regular matching; (3) processing the acquired data or entering the next round of grabbing cycle through the acquired new URL.