The Developer Day | Staying Curious



Free Online SEO Sitemap Crawler Tool

The company that I work for has a lot to do with SEO, SEM, PPC. There are such a thing as sitemaps and URL lists that are used in SEO for Search Engines to optimize your site easier and better.

We have a lot of sites which have some SEO development going on. To generate these URL lists is not a task for a human being to do and I didn’t want to develop a custom sitemap generator for each of our projects.

I have tried to search for a tool that could generate these URL lists given only a single root URL. It was a while ago so I cannot say that such tools don’t exist. I also like my tools to be simple and easily extended so I developed a tool of my own. It’s a simple crawler that only needs to know the start URL and the base URL and it can find all your URLs in no time. The tool remembers the pages it has already visited. So for example let’s say I wanted to know all my blog URL’s. I just feed the crawler two URLs and wait for the results. The tool also ignores any outside URLs and adds the base URL to relative links.

You can download the crawler or try it out for yourself.

Please read comments below if you are interested why the second input field is needed.

RSS Feed

2 Comments for Free Online SEO Sitemap Crawler Tool

Alex | July 1, 2007 at 1:58 AM

why href field is required? can’t you just make one field that will take URL & that’s it?

Author comment by admin | July 2, 2007 at 5:57 PM

The problem is with the relative links like a href=”about.html”. Suppose your given start URL is How the crawler should figure out what the correct link would be. Is it or or ? It’s possible to fetch the base href from the meta tags but what if it’s not there?

Leave a comment!



Find it!

Theme Design by