This PHP script crawls URL:s based on a sitemap. It is used to keep your cache warm by visiting all the pages in your sitemap at regular intervals. It supports sub-sitemap (Sitemap index).
Edit the file config.php and change the key parameter to a secret value. Upload the file onto your web host, preferably into its own folder (for example, /warm-cache)
Once you have uploaded this file onto your web host, you can visit the following URL to traverse a sitemap and visit all its URL:s:
http://example.com/warm-cache/warm.php?key=SECRET_KEY&url=http://example.com/sitemap.xml&sleep=0
key - Secret key, as entered in config.php (Required)
url - URL to the root sitemap, usually /sitemap.xml (Required)
sleep - Amount of time to sleep between each request in seconds. Used for throttling on slow hosts. (Optional, default is to not throttle.)
from - Number of the url to start with. (Optional, default is 0).
to - Number of the url to stop. Usefull to test some URLs on a heavy sitemap (Optional, default is till the end of the sitemap)
You will need to use CRON to schedule the crawls as often as you wish. Here's an example using cURL and crontab to crawl once every hour:
0 * * * * curl "http://example.com/warm-cache/warm.php?key=SECRET_KEY&url=http://example.com/sitemap.xml&sleep=0" >/dev/null 2>&1
If your host provides a CRON URL visiting function, all you need to do is enter the URL, as described in the "Usage" section.
The script will provide a JSON output with stats about the crawl, example:
{
"status": "OK",
"message": "Processed sitemap: http://example.com/sitemap.xml",
"count" : 4,
"duration": 9.5575199127197,
"log": [
"Processed sub-sitemap: http://example.com/post-sitemap.xml",
"Processed sub-sitemap: http://example.com/page-sitemap.xml",
],
"visited_urls": [
"http://example.com/page1/",
"http://example.com/page2/",
"http://example.com/page3/",
"http://example.com/page4/",
]
}
Also you can launch the script from the CLI to bypass the common errors of timeout (504) from an Nginx server.
php /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY
php /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY from=10 to=100
php /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY to=25
If you employ time-based static page cache, you can schedule your crawls to coincide with half the cache expiration time.
For example, if your expiration time is one hour (3600 seconds), you can schedule the crawls to take place every thirty minutes (1800 seconds).
If you have a lot of pages and few visitors, this may cause increased load on the server. For low-traffic deployments, use a long cache expiration time (24 hours or more) and invalidate cache when page content changes.
- SimpleXML
- allow_url_fopen in php.ini (Enabled on most hosts)
The plugin has been tested with the WordPress plugins Yoast WordPress SEO and Google XML Sitemaps. It should work with any sitemap which conforms to the sitemap standard.