http://www.thedeveloperday.com/web-crawler-tool/ Staying Curious Wed, 01 Feb 2012 07:05:15 +0000 hourly 1 http://wordpress.org/?v=3.3.1 http://www.thedeveloperday.com/web-crawler-tool/comment-page-1/#comment-6 admin Mon, 02 Jul 2007 14:57:51 +0000 http://www.thedeveloperday.com/web-crawler-tool/#comment-6 The problem is with the relative links like a href="about.html". Suppose your given start URL is www.someurl.com/blog/my-day/. How the crawler should figure out what the correct link would be. Is it www.someurl.com/blog/my-day/about.html or www.someurl.com/blog/about.html or www.someurl.com/about.html ? It's possible to fetch the base href from the meta tags but what if it's not there? The problem is with the relative links like a href=”about.html”. Suppose your given start URL is http://www.someurl.com/blog/my-day/. How the crawler should figure out what the correct link would be. Is it http://www.someurl.com/blog/my-day/about.html or http://www.someurl.com/blog/about.html or http://www.someurl.com/about.html ? It’s possible to fetch the base href from the meta tags but what if it’s not there?

]]> http://www.thedeveloperday.com/web-crawler-tool/comment-page-1/#comment-5 Alex Sat, 30 Jun 2007 22:58:50 +0000 http://www.thedeveloperday.com/web-crawler-tool/#comment-5 why href field is required? can't you just make one field that will take URL & that's it? why href field is required? can’t you just make one field that will take URL & that’s it?

]]>