No matter how the Internet develops

No matter how the Internet develops, search engine has always maintained a very important position. Statistics show that: search engine has been the most frequently used network service on the Internet. Due to the explosive growth of information and data on the Internet, the current search engine's ability to obtain data can not meet people's needs. Keyword based search engine can only search according to keywords, that is to say, it can only search the information containing keywords, but can't distinguish the meaning of keywords according to semantics, and can't search the information data containing keyword synonyms, which leads to the development of search engine can't keep up with the development of Internet. The most direct performance is that it is difficult to find more comprehensive and accurate results for users in more and more Internet data. Therefore, the traditional search engine has been unable to adapt to the rapid development of Internet information technology, but, with the development of computer and Internet communication equipment, search engine has a large number of users, also has a good development prospect and economic value, so the world's information industry personnel and technology have great interest in the research and development of search engine We should pay more attention to it.Therefore, web crawler based on topic and semantic analysis and web crawler based on event driven are two major aspects of web crawler development in the future.