middleman-robots
is an extension of Middleman. This can create robots.txt
.
This plugin support Middleman v3-stable and v4.
Add this line to your application's Gemfile:
gem 'middleman-robots'
And then execute:
$ bundle
Or install it yourself as:
$ gem install middleman-robots
# config.rb
activate :robots,
rules: [
{ user_agent: '*', allow: %w[/] }
],
sitemap: 'https://example.com/sitemap.xml'
Created robots.txt
:
User-Agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
You can use options, rules
[ {user_agent
(string), allow
(array), disallow
(array)}] and sitemap
. Like this:
# config.rb
activate :robots,
rules: [
{
user_agent: 'Googlebot',
disallow: %w[tmp/ /something/dir/file_disallow.html],
allow: %w[allow/ /something/dir/file_allow.html]
},
{
user_agent: 'Googlebot-Image',
disallow: %w[tmp/ /something/dir/file_disallow.html],
allow: %w[allow/ /something/dir/file_allow.html]
}
],
sitemap: 'https://example.com/sitemap.xml'
Created robots.txt
:
User-Agent: Googlebot
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html
User-Agent: Googlebot-Image
Disallow: /tmp/
Disallow: /something/dir/file_disallow.html
Allow: /allow/
Allow: /something/dir/file_allow.html
Sitemap: https://example.com/sitemap.xml
- Fork it ( https://github.com/yterajima/middleman-robots/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request