https://www.codementor.io/@fooyyll/what-is-a-web-crawler-and-how-does-it-work-12apr9j0yo