🚀 Intelligent search made easy
Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
Searchkick handles:
- stemming -
tomatoes
matchestomato
- special characters -
jalapeno
matchesjalapeño
- extra whitespace -
dishwasher
matchesdish washer
- misspellings -
zuchini
matcheszucchini
- custom synonyms -
qtip
matchescotton swab
Plus:
- query like SQL - no need to learn a new query language
- reindex without downtime
- easily personalize results for each user
- autocomplete
- “Did you mean” suggestions
- works with ActiveRecord, Mongoid, and NoBrainer
Searchkick 1.0 was just released! See instructions for upgrading
💬 Get handcrafted updates for new features
🍊 Battle-tested at Instacart
Install Elasticsearch. For Homebrew, use:
brew install elasticsearch
# start the server
elasticsearch
Add this line to your application’s Gemfile:
gem 'searchkick'
For Elasticsearch 2.0, use the version 1.0
and above. For Elasticsearch 0.90, use version 0.6.3
and this readme.
Add searchkick to models you want to search.
class Product < ActiveRecord::Base
searchkick
end
Add data to the search index.
Product.reindex
And to query, use:
products = Product.search "apples"
products.each do |product|
puts product.name
end
Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.
Query like SQL
Product.search "apples", where: {in_stock: true}, limit: 10, offset: 50
Search specific fields
fields: [:name, :brand]
Where
where: {
expires_at: {gt: Time.now}, # lt, gte, lte also available
orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
aisle_id: [25, 30], # in
store_id: {not: 2}, # not
aisle_id: {not: [25, 30]}, # not in
user_ids: {all: [1, 3]}, # all elements in array
category: /frozen .+/, # regexp
or: [
[{in_stock: true}, {backordered: true}]
]
}
Order
order: {_score: :desc} # most relevant first - default
All of these sort options are supported
Limit / offset
limit: 20, offset: 40
Searches return a Searchkick::Results
object. This responds like an array to most methods.
results = Product.search("milk")
results.size
results.any?
results.each { |result| ... }
Get total results
results.total_count
Get the time the search took (in milliseconds)
results.took
Get the full response from Elasticsearch
results.response
Boost important fields
fields: ["title^10", "description"]
Boost by the value of a field (field must be numeric)
boost_by: [:orders_count] # give popular documents a little boost
boost_by: {orders_count: {factor: 10}} # default factor is 1
Boost matching documents
boost_where: {user_id: 1}
boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000
boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}
Conversions are also a great way to boost.
Use a *
for the query.
Product.search "*"
Plays nicely with kaminari and will_paginate.
# controller
@products = Product.search "milk", page: params[:page], per_page: 20
View with kaminari
<%= paginate @products %>
View with will_paginate
<%= will_paginate @products %>
By default, results must match all words in the query.
Product.search "fresh honey" # fresh AND honey
To change this, use:
Product.search "fresh honey", operator: "or" # fresh OR honey
By default, results must match the entire word - back
will not match backpack
. You can change this behavior with:
class Product < ActiveRecord::Base
searchkick word_start: [:name]
end
And to search (after you reindex):
Product.search "back", fields: [:name], match: :word_start
Available options are:
:word # default
:word_start
:word_middle
:word_end
:text_start
:text_middle
:text_end
User.search params[:q], fields: [{email: :exact}, :name]
User.search "fresh honey", match: :phrase
Searchkick defaults to English for stemming. To change this, use:
class Product < ActiveRecord::Base
searchkick language: "german"
end
class Product < ActiveRecord::Base
searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
# or
# searchkick synonyms: -> { CSV.read("/some/path/synonyms.csv") }
end
Call Product.reindex
after changing synonyms.
Prepopulate English synonyms with the WordNet database.
Download WordNet 3.0 to each Elasticsearch server and move wn_s.pl
to the /var/lib
directory.
cd /tmp
curl -o wordnet.tar.gz http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz
tar -zxvf wordnet.tar.gz
mv prolog/wn_s.pl /var/lib
Tell each model to use it:
class Product < ActiveRecord::Base
searchkick wordnet: true
end
By default, Searchkick handles misspelled queries by returning results with an edit distance of one.
You can change this with:
Product.search "zucini", misspellings: {edit_distance: 2} # zucchini
To improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
Product.search "zuchini", misspellings: {below: 5}
If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
Turn off misspellings with:
Product.search "zuchini", misspellings: false # no zucchini
Search 🍨🍰 and get ice cream cake
!
Add this line to your application’s Gemfile:
gem 'gemoji-parser'
And use:
Product.search "[emoji go here]", emoji: true
Control what data is indexed with the search_data
method. Call Product.reindex
after changing this method.
class Product < ActiveRecord::Base
belongs_to :department
def search_data
{
name: name,
department_name: department.name,
on_sale: sale_price.present?
}
end
end
Searchkick uses find_in_batches
to import documents. To eager load associations, use the search_import
scope.
class Product < ActiveRecord::Base
scope :search_import, -> { includes(:department) }
end
By default, all records are indexed. To control which records are indexed, use the should_index?
method.
class Product < ActiveRecord::Base
def should_index?
active # only index active records
end
end
- when you install or upgrade searchkick
- change the
search_data
method - change the
searchkick
method
- App starts
There are three strategies for keeping the index synced with your database.
- Immediate (default)
Anytime a record is inserted, updated, or deleted
- Asynchronous
Use background jobs for better performance
class Product < ActiveRecord::Base
searchkick callbacks: :async
end
And install Active Job for Rails 4.1 and below. Jobs are added to a queue named searchkick
.
- Manual
Turn off automatic syncing
class Product < ActiveRecord::Base
searchkick callbacks: false
end
You can also do bulk updates.
Searchkick.callbacks(:bulk) do
User.find_each(&:update_fields)
end
Or temporarily skip updates.
Searchkick.callbacks(false) do
User.find_each(&:update_fields)
end
Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:
class Image < ActiveRecord::Base
belongs_to :product
after_commit :reindex_product
def reindex_product
product.reindex # or reindex_async
end
end
We highly recommend tracking searches and conversions.
Searchjoy makes it easy.
Product.search "apple", track: {user_id: current_user.id}
See the docs for how to install and use.
Searchkick can use conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.
You do not need to clean up the search queries. Searchkick automatically treats apple
and APPLES
the same.
Next, add conversions to the index.
class Product < ActiveRecord::Base
has_many :searches, class_name: "Searchjoy::Search"
searchkick conversions: "conversions" # name of field
def search_data
{
name: name,
conversions: searches.group(:query).uniq.count(:user_id)
# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
}
end
end
Reindex and set up a cron job to add new conversions daily.
rake searchkick:reindex CLASS=Product
Order results differently for each user. For example, show a user’s previously purchased products before other results.
class Product < ActiveRecord::Base
def search_data
{
name: name,
orderer_ids: orders.pluck(:user_id) # boost this product for these users
}
end
end
Reindex and search with:
Product.search "milk", boost_where: {orderer_ids: current_user.id}
Autocomplete predicts what a user will type, making the search experience faster and easier.
Note: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).
First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
class Book < ActiveRecord::Base
searchkick match: :word_start, searchable: [:title, :author]
end
Reindex and search with:
Book.search "tipping poi"
Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.
First, add a route and controller action.
# app/controllers/books_controller.rb
class BooksController < ApplicationController
def autocomplete
render json: Book.search(params[:query], {
fields: ["title^5", "author"],
limit: 10,
load: false,
misspellings: {below: 5}
}).map(&:title)
end
end
Then add the search box and JavaScript code to a view.
<input type="text" id="query" name="query" />
<script src="jquery.js"></script>
<script src="typeahead.js"></script>
<script>
$("#query").typeahead({
name: "book",
remote: "/books/autocomplete?query=%QUERY"
});
</script>
class Product < ActiveRecord::Base
searchkick suggest: [:name] # fields to generate suggestions
end
Reindex and search with:
products = Product.search "peantu butta", suggest: true
products.suggestions # ["peanut butter"]
Aggregations provide aggregated search data.
products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand]
products.aggs
By default, where
conditions apply to aggregations.
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size]
# aggregations for brandy wingtips are returned
Change this with:
Product.search "wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false
# aggregations for all wingtips are returned
Set where
conditions for each aggregation separately with:
Product.search "wingtips", aggs: {size: {where: {color: "brandy"}}}
Limit
Product.search "apples", aggs: {store_id: {limit: 10}}
Order
Product.search "wingtips", aggs: {color: {order: {"_term" => "asc"}}} # alphabetically
All of these options are supported
Ranges
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search "*", aggs: {price: {ranges: price_ranges}}
- Replace
facets
withaggs
in searches. Note: Stats facets are not supported at this time.
products = Product.search "chuck taylor", facets: [:brand]
# to
products = Product.search "chuck taylor", aggs: [:brand]
- Replace the
facets
method withaggs
for results.
products.facets
# to
products.aggs
The keys in results differ slightly. Instead of:
{
"_type":"terms",
"missing":0,
"total":45,
"other":34,
"terms":[
{"term":14.0,"count":11}
]
}
You get:
{
"doc_count":45,
"doc_count_error_upper_bound":0,
"sum_other_doc_count":34,
"buckets":[
{"key":14.0,"doc_count":11}
]
}
Update your application to handle this.
-
By default,
where
conditions apply to aggregations. This is equivalent tosmart_facets: true
. If you havesmart_facets: true
, you can remove it. If this is not desired, setsmart_aggs: false
. -
If you have any range facets with dates, change the key from
ranges
todate_ranges
.
facets: {date_field: {ranges: date_ranges}}
# to
aggs: {date_field: {date_ranges: date_ranges}}
Facets have been deprecated in favor of aggregations as of Searchkick 1.0. See how to upgrade.
products = Product.search "chuck taylor", facets: [:product_type, :gender, :brand]
p products.facets
By default, where
conditions are not applied to facets (for backward compatibility).
Product.search "wingtips", where: {color: "brandy"}, facets: [:size]
# facets *not* filtered by color :(
Change this with:
Product.search "wingtips", where: {color: "brandy"}, facets: [:size], smart_facets: true
or set where
conditions for each facet separately:
Product.search "wingtips", facets: {size: {where: {color: "brandy"}}}
Limit
Product.search "apples", facets: {store_id: {limit: 10}}
Ranges
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search "*", facets: {price: {ranges: price_ranges}}
Use the stats
option to get to max, min, mean, and total scores for each facet
Product.search "*", facets: {store_id: {stats: true}}
Specify which fields to index with highlighting.
class Product < ActiveRecord::Base
searchkick highlight: [:name]
end
Highlight the search query in the results.
bands = Band.search "cinema", fields: [:name], highlight: true
Note: The fields
option is required, unless highlight options are given - see below.
View the highlighted fields with:
bands.with_details.each do |band, details|
puts details[:highlight][:name] # "Two Door <em>Cinema</em> Club"
end
To change the tag, use:
Band.search "cinema", fields: [:name], highlight: {tag: "<strong>"}
To highlight and search different fields, use:
Band.search "cinema", fields: [:name], highlight: {fields: [:description]}
Additional options, including fragment size, can be specified for each field:
Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}
You can find available highlight options in the Elasticsearch reference.
Find similar items.
product = Product.first
product.similar(fields: ["name"], where: {size: "12 oz"})
class City < ActiveRecord::Base
searchkick locations: ["location"]
def search_data
attributes.merge location: {lat: latitude, lon: longitude}
end
end
Reindex and search with:
City.search "san", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}} # or 160km
Bounded by a box
City.search "san", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}
Boost results by distance - closer results are boosted more
City.search "san", boost_by_distance: {field: :location, origin: {lat: 37, lon: -122}}
Also supports additional options
City.search "san", boost_by_distance: {field: :location, origin: {lat: 37, lon: -122}, function: :linear, scale: "30mi", decay: 0.5}
Searchkick supports Elasticsearch’s routing feature.
class Business < ActiveRecord::Base
searchkick routing: true
def search_routing
city_id
end
end
Reindex and search with:
Business.search "ice cream", routing: params[:city_id]
Searchkick supports single table inheritance.
class Dog < Animal
end
The parent and child model can both reindex.
Animal.reindex
Dog.reindex # equivalent
And to search, use:
Animal.search "*" # all animals
Dog.search "*" # just dogs
Animal.search "*", type: [Dog, Cat] # just cats and dogs
Note: The suggest
option retrieves suggestions from the parent at the moment.
Dog.search "airbudd", suggest: true # suggestions for all animals
See how Elasticsearch scores your queries with:
Product.search("soap", explain: true).response
See how Elasticsearch tokenizes your queries with:
Product.searchkick_index.tokens("Dish Washer Soap", analyzer: "default_index")
# ["dish", "dishwash", "washer", "washersoap", "soap"]
Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search")
# ["dishwashersoap"] - no match
Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
# ["dishwash", "soap"] - match!!
Partial matches
Product.searchkick_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]
Product.searchkick_index.tokens("dieg", analyzer: "searchkick_word_search")
# ["dieg"] - match!!
See the complete list of analyzers.
Searchkick uses ENV["ELASTICSEARCH_URL"]
for the Elasticsearch server. This defaults to http://localhost:9200
.
Choose an add-on: SearchBox, Bonsai, or Found.
# SearchBox
heroku addons:add searchbox:starter
heroku config:add ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`
# Bonsai
heroku addons:add bonsai
heroku config:add ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
# Found
heroku addons:add foundelasticsearch
heroku config:add ELASTICSEARCH_URL=`heroku config:get FOUNDELASTICSEARCH_URL`
Then deploy and reindex:
heroku run rake searchkick:reindex CLASS=Product
Include elasticsearch 1.0.15
or greater in your Gemfile.
gem "elasticsearch", ">= 1.0.15"
Create an initializer config/initializers/elasticsearch.rb
with:
ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com"
To use signed request, include in your Gemfile:
gem 'faraday_middleware-aws-signers-v4'
and add to your initializer:
require "faraday_middleware/aws_signers_v4"
Searchkick.client =
Elasticsearch::Client.new(
url: ENV["ELASTICSEARCH_URL"],
transport_options: {request: {timeout: 10}}
) do |f|
f.request :aws_signers_v4, {
credentials: Aws::Credentials.new(ENV["AWS_ACCESS_KEY_ID"], ENV["AWS_SECRET_ACCESS_KEY"]),
service_name: "es",
region: "us-east-1"
}
end
Then deploy and reindex:
rake searchkick:reindex CLASS=Product
Create an initializer config/initializers/elasticsearch.rb
with:
ENV["ELASTICSEARCH_URL"] = "http://username:password@api.searchbox.io"
Then deploy and reindex:
rake searchkick:reindex CLASS=Product
For the best performance, add Typhoeus to your Gemfile.
gem 'typhoeus'
And create an initializer with:
require "typhoeus/adapters/faraday"
Ethon.logger = Logger.new("/dev/null")
Note: Typhoeus is not available for Windows.
Create an initializer config/initializers/elasticsearch.rb
with multiple hosts:
Searchkick.client = Elasticsearch::Client.new(hosts: ["localhost:9200", "localhost:9201"], retry_on_failure: true)
See elasticsearch-transport for a complete list of options.
Add the following to config/environments/production.rb
:
config.lograge.custom_options = lambda do |event|
options = {}
options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
options
end
See Production Rails for other good practices.
Prefer to use the Elasticsearch DSL but still want awesome features like zero-downtime reindexing?
Create a custom mapping:
class Product < ActiveRecord::Base
searchkick mappings: {
product: {
properties: {
name: {type: "string", analyzer: "keyword"}
}
}
}
end
To keep the mappings and settings generated by Searchkick, use:
class Product < ActiveRecord::Base
searchkick merge_mappings: true, mappings: {...}
end
And use the body
option to search:
products = Product.search body: {match: {name: "milk"}}
Note: This replaces the entire body, so other options are ignored.
View the response with:
products.response
To modify the query generated by Searchkick, use:
products =
Product.search "apples" do |body|
body[:query] = {match_all: {}}
end
To batch search requests for performance, use:
fresh_products = Product.search("fresh", execute: false)
frozen_products = Product.search("frozen", execute: false)
Searchkick.multi_search([fresh_products, frozen_products])
Then use fresh_products
and frozen_products
as typical results.
Note: Errors are not raised as with single requests. Use the error
method on each query to check for errors. Also, if you use the below
option for misspellings, misspellings will be disabled.
Reindex one record
product = Product.find 10
product.reindex
# or to reindex in the background
product.reindex_async
Reindex more than one record without recreating the index
# do this ...
some_company.products.each { |p| p.reindex }
# or this ...
Product.searchkick_index.import(some_company.products)
# don't do the following as it will recreate the index with some_company's products only
some_company.products.reindex
Reindex large set of records in batches
Product.where("id > 100000").find_in_batches do |batch|
Product.searchkick_index.import(batch)
end
Remove old indices
Product.clean_indices
Use custom settings
class Product < ActiveRecord::Base
searchkick settings: {number_of_shards: 3}
end
Use a different index name
class Product < ActiveRecord::Base
searchkick index_name: "products_v2"
end
Use a dynamic index name
class Product < ActiveRecord::Base
searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
end
Prefix the index name
class Product < ActiveRecord::Base
searchkick index_prefix: "datakick"
end
Change timeout
Searchkick.timeout = 15 # defaults to 10
Set a lower timeout for searches
Searchkick.search_timeout = 3
Change the search method name in config/initializers/searchkick.rb
Searchkick.search_method_name = :lookup
Eager load associations
Product.search "milk", include: [:brand, :stores]
Do not load models
Product.search "milk", load: false
Turn off special characters
class Product < ActiveRecord::Base
# A will not match Ä
searchkick special_characters: false
end
Use Okapi BM25 for ranking
class Product < ActiveRecord::Base
searchkick similarity: "BM25"
end
Change import batch size
class Product < ActiveRecord::Base
searchkick batch_size: 200 # defaults to 1000
end
Create index without importing
Product.reindex(import: false)
Lazy searching
products = Product.search("carrots", execute: false)
products.each { ... } # search not executed until here
Make fields unsearchable but include in the source
class Product < ActiveRecord::Base
searchkick unsearchable: [:color]
end
Reindex conditionally
Note: With ActiveRecord, use this feature with caution - transaction rollbacks can cause data inconsistencies
class Product < ActiveRecord::Base
searchkick callbacks: false
# add the callbacks manually
after_save :reindex, if: proc{|model| model.name_changed? } # use your own condition
after_destroy :reindex
end
Search multiple models
Searchkick.search "milk", index_name: [Product, Category]
Reindex all models - Rails only
rake searchkick:reindex:all
Turn on misspellings after a certain number of characters
Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
Note: With this option, if the query length is the same as prefix_length
, misspellings are turned off
Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
For large data sets, check out Keeping Elasticsearch in Sync. Searchkick will make this easy in the future.
This section could use some love.
describe Product do
it "searches" do
Product.reindex
Product.searchkick_index.refresh # don't forget this
# test goes here...
end
end
product = FactoryGirl.create(:product)
product.reindex # don't forget this
Product.searchkick_index.refresh # or this
- Change
search
methods totire.search
and add index name in existing search calls
Product.search "fruit"
should be replaced with
Product.tire.search "fruit", index: "products"
- Replace tire mapping w/ searchkick method
class Product < ActiveRecord::Base
searchkick
end
- Deploy and reindex
rake searchkick:reindex CLASS=Product # or Product.reindex in the console
- Once it finishes, replace search calls w/ searchkick calls
View the changelog.
Important notes are listed below.
- Added support for Elasticsearch 2.0
- Facets are deprecated in favor of aggregations - see how to upgrade
-
ActiveRecord 4.1+ and Mongoid 3+: Attempting to reindex with a scope now throws a
Searchkick::DangerousOperation
error to keep your from accidentally recreating your index with only a few records.Product.where(color: "brandy").reindex # error!
If this is what you intend to do, use:
Product.where(color: "brandy").reindex(accept_danger: true)
-
Misspellings are enabled by default for partial matches. Use
misspellings: false
to disable. -
Transpositions are enabled by default for misspellings. Use
misspellings: {transpositions: false}
to disable.
If running Searchkick 0.6.0
or 0.7.0
and Elasticsearch 0.90
, we recommend upgrading to Searchkick 0.6.1
or 0.7.1
to fix an issue that causes downtime when reindexing.
Before 0.3.0
, locations were indexed incorrectly. When upgrading, be sure to reindex immediately.
Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:
class Product < ActiveRecord::Base
searchkick settings: {number_of_shards: 1}
end
For convenience, this is set by default in the test environment.
Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.
- More features for large data sets
- Improve section on testing
- Semantic search features
- Search multiple fields for different terms
- Search across models
- Search nested objects
- Much finer customization
Everyone is encouraged to help improve this project. Here are a few ways you can help:
- Report bugs
- Fix bugs and submit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
To get started with development and testing:
git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
rake test