{"id":965,"date":"2014-12-01T06:29:40","date_gmt":"2014-12-01T06:29:40","guid":{"rendered":"http:\/\/invisiblezero.net\/?p=965"},"modified":"2024-03-11T19:31:08","modified_gmt":"2024-03-11T19:31:08","slug":"php-crawl-websites-from-command-line-interface","status":"publish","type":"post","link":"http:\/\/ndthanh.com\/php-crawl-websites-from-command-line-interface\/","title":{"rendered":"PHP &#8211; Crawl websites from command line interface"},"content":{"rendered":"<p>Recently, i wrote a new crawler script to warn caches on some Magento websites. Today i&#8217;d like to share it with you, because i wrote it in a way that works with many websites other than Magento and many platforms.<\/p>\n<p>You can see the help content by running the crawler in command interface like below, make sure there is no sitemap.xml file or you have -help option as parameter in your command line.<\/p>\n<p><!--more--><\/p>\n<pre><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nphp -f iz_crawler.php\nUsage:  php -f crawler.php -- &#x5B;options]\n\n  -sitemap &amp;amp;lt;list of files&amp;amp;gt;     List of sitemap xml files, delimit by semicolon ; . Default is &#039;sitemap.xml&#039;\n  -website &amp;amp;lt;website&amp;amp;gt;           Website url for input. Will be ignored if -sitemap option selected or there is sitemap.xml file in the same directory with this crawler\n  -depth &amp;amp;lt;number&amp;amp;gt;              Set depth level. Default is 0\n  -interval &amp;amp;lt;number&amp;amp;gt;           Set scrap interval, measure in second(s). Defalt is 0\n  -exclude &amp;amp;lt;extensions&amp;amp;gt;        Exclude link extensions like png, css, js, etc... delimit by semicolon ; . Default is &amp;amp;quot;jpg;png;jpeg;pdf;7z;zip;rar;mp3;aac;mp4;apk;bat;tar;swf;iso&amp;amp;quot;\n  -verbose                     Display crawler output. Default is false\n  -help                        This help\n\n  Note: sitemap.xml default location is at root, and it will add initial urls for crawler, use -depth to make most use of sitemap.xml\n\n  Example : php -f crawler.php  -- -website http:\/\/www.google.com -depth 1 -interval 0.5 -verbose -exclude &amp;amp;quot;png;pdf;html&amp;amp;quot;\n<\/pre>\n<p>Because you can figure out a lot from the help content, so i will only show you how it looks here. i placed iz_crawler.php at the root directory of my website and execute this command &#8220;php -f iz_crawler.php &#8212; -verbose&#8221; :<\/p>\n<pre><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nphp -f iz_crawler.php -- -verbose\n<blockquote class=\"wp-embedded-content\" data-secret=\"5UfU4vyh3D\"><a href=\"https:\/\/ndthanh.net\/\">Home<\/a><\/blockquote><iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Home&#8221; &#8212; Thanh, Nguyen Duy\" src=\"https:\/\/ndthanh.net\/embed\/#?secret=5UfU4vyh3D\" data-secret=\"5UfU4vyh3D\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\nhttps:\/\/ndthanh.net\/tour-bac-ha-sapa-than-uyen-2013-2014\/\nhttps:\/\/ndthanh.net\/control-git-with-php\/\nhttps:\/\/ndthanh.net\/javascript-continuously-respectively-send-requests-to-server\/\nhttps:\/\/ndthanh.net\/magento-remove-redundant-attributes\/\n<blockquote class=\"wp-embedded-content\" data-secret=\"xMfY7GKxEV\"><a href=\"https:\/\/ndthanh.net\/contacts\/\">Contacts<\/a><\/blockquote><iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Contacts&#8221; &#8212; Thanh, Nguyen Duy\" src=\"https:\/\/ndthanh.net\/contacts\/embed\/#?secret=xMfY7GKxEV\" data-secret=\"xMfY7GKxEV\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\nhttps:\/\/ndthanh.net\/mac-osx-clean-duplicate-entries-in-open-with-menu\/\nhttps:\/\/ndthanh.net\/treasure-chest\/\nhttps:\/\/ndthanh.net\/philippines-2013\/\n<blockquote class=\"wp-embedded-content\" data-secret=\"7JVXHqadRZ\"><a href=\"https:\/\/ndthanh.net\/services\/\">Services<\/a><\/blockquote><iframe class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Services&#8221; &#8212; Thanh, Nguyen Duy\" src=\"https:\/\/ndthanh.net\/services\/embed\/#?secret=7JVXHqadRZ\" data-secret=\"7JVXHqadRZ\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\nhttps:\/\/ndthanh.net\/mu-cang-chai-2013\/\nhttps:\/\/ndthanh.net\/what-starts-with-f-and-ends-with-k\/\nhttps:\/\/ndthanh.net\/amazon-ec2-common-problems-theres-no-swap-space\/\nhttps:\/\/ndthanh.net\/ban-gioc-2013\/\nhttps:\/\/ndthanh.net\/wordpress-update-coreplugins-without-ftp\/\nhttps:\/\/ndthanh.net\/varnish-cache-server-guru-meditation-503\/\nhttps:\/\/ndthanh.net\/mysql-general-error-2006-mysql-server-has-gone-away\/\nhttps:\/\/ndthanh.net\/ninh-binh\/\nhttps:\/\/ndthanh.net\/ruby-on-rails-library-not-loaded-libmysqlclient-18-dylib-loaderror\/\nhttps:\/\/ndthanh.net\/compiling-latest-version-of-php-on-mac-os-x\/\nhttps:\/\/ndthanh.net\/magento-installing-extensions-via-command-line\/\nhttps:\/\/ndthanh.net\/unix-find-more-ssh-power-with-ssh-config\/\nhttps:\/\/ndthanh.net\/mac-os-x-quickly-switch-between-windows-of-an-application\/\nhttps:\/\/ndthanh.net\/magento-e-display-global-messages-when-you-have-full-page-cache-turned-on\/\nhttps:\/\/ndthanh.net\/technical-debt-in-software-development\/\nhttps:\/\/ndthanh.net\/mac-os-x-completely-disable-dashboard\/\n<\/pre>\n<p>Because i have the sitemap.xml file, so it will ready the sitemap file automatically. you can specify sitemap file(s) or scrape website by domain name and the crawler will extract link automatically.<\/p>\n<p>You can find download link below \ud83d\ude00<br \/>\n<a href=\"https:\/\/ndthanh.net\/wp-content\/uploads\/2014\/11\/iz_crawler.php_1.zip\">PHP Crawler from IZ<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently, i wrote a new crawler script to warn caches on some Magento websites. Today i&#8217;d like to share it with you, because i wrote it in a way that works with many websites other than Magento and many platforms. You can see the help content by running the crawler in command interface like below,&#8230;<\/p>\n<p><a class=\"read-more\" href=\"http:\/\/ndthanh.com\/php-crawl-websites-from-command-line-interface\/\">Read More<\/a><\/p>\n","protected":false},"author":1,"featured_media":1305,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"aioseo_notices":[],"views":14,"_links":{"self":[{"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/posts\/965"}],"collection":[{"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/comments?post=965"}],"version-history":[{"count":2,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/posts\/965\/revisions"}],"predecessor-version":[{"id":1307,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/posts\/965\/revisions\/1307"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/media\/1305"}],"wp:attachment":[{"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/media?parent=965"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/categories?post=965"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/ndthanh.com\/wp-json\/wp\/v2\/tags?post=965"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}