The company that I work for has a lot to do with SEO, SEM, PPC. There are such a thing as sitemaps and URL lists that are used in SEO for Search Engines to optimize your site easier and better.

We have a lot of sites which have some SEO development going on. To generate these URL lists is not a task for a human being to do and I didn’t want to develop a custom sitemap generator for each of our projects.

I have tried to search for a tool that could generate these URL lists given only a single root URL. It was a while ago so I cannot say that such tools don’t exist. I also like my tools to be simple and easily extended so I developed a tool of my own. It’s a simple crawler that only needs to know the start URL and the base URL and it can find all your URLs in no time. The tool remembers the pages it has already visited. So for example let’s say I wanted to know all my blog URL’s. I just feed the crawler two URLs and wait for the results. The tool also ignores any outside URLs and adds the base URL to relative links.

You can download the crawler or try it out for yourself.

Please read comments below if you are interested why the second input field is needed.

crawler, free, SEO, sitemap, Tools