-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Provide a rake task for updating an Elasticsearch index schema in pla…
…ce when Search API configuration has changed This rake task has been added because at the moment even purely additive changes to elasticsearch schema configuration require a full reindex. Full reindexes require locking the index for significant periods of time so have to be run out of hours for production and can block integration for long periods. This is causing pain when developing applications that consume the Search API. The task is idempotent and Elasticsearch will not permit changes to the index that would attempt to change the type of already existing data. Any such changes result in a “Bad Request” error from Elasticsearch and these will be reported to the rake operator. In such cases a full re-index would be required.
- Loading branch information
Showing
6 changed files
with
107 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
class SchemaSynchroniser | ||
attr_reader :errors | ||
|
||
def initialize(index_group) | ||
@index = index_group.current | ||
end | ||
|
||
def sync | ||
@errors = @index.sync_mappings | ||
end | ||
|
||
def synchronised_types | ||
@index.mappings.keys.difference(@errors.keys) | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
require "spec_helper" | ||
|
||
RSpec.describe SchemaSynchroniser do | ||
before do | ||
clean_index_content("govuk_test") | ||
end | ||
|
||
it "synchronises the current Elasticsearch index schema with the schema defined by the search API" do | ||
index_group = search_server.index_group("govuk_test") | ||
mappings = index_group.current.mappings | ||
synchroniser = described_class.new(index_group) | ||
synchroniser.sync | ||
expect(synchroniser.synchronised_types).not_to be_empty | ||
expect(synchroniser.synchronised_types).to eq(mappings.keys) | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
require "spec_helper" | ||
|
||
RSpec.describe SearchIndices::Index do | ||
let(:base_uri) { "http://example.com:9200" } | ||
|
||
it "syncs mappings to elasticsearch and returns any failures" do | ||
mappings = { | ||
"generic-document" => { | ||
"properties" => { | ||
"new-field" => { "type": "text" }, | ||
}, | ||
}, | ||
"failing-document" => { | ||
"properties" => { | ||
"invalid-field" => { "type": "text" }, | ||
}, | ||
}, | ||
} | ||
stub = stub_request(:put, %r{#{base_uri}/govuk-abc/_mapping/generic-document}) | ||
.with(body: mappings["generic-document"]) | ||
|
||
error_body = { "error" => { | ||
"type" => "illegal_argument_exception", | ||
"reason" => "invalid mapping", | ||
} }.to_json | ||
failing_stub = stub_request(:put, %r{#{base_uri}/govuk-abc/_mapping/failing-document}) | ||
.with(body: mappings["failing-document"]) | ||
.to_return({ | ||
status: 400, | ||
body: error_body, | ||
headers: { "Content-Type" => "application/json" }, | ||
}) | ||
|
||
index = SearchIndices::Index.new(base_uri, "govuk-abc", "govuk", mappings, SearchConfig.default_instance) | ||
|
||
errors = index.sync_mappings | ||
|
||
assert_requested stub | ||
assert_requested failing_stub | ||
expect(errors).not_to be_empty | ||
expect(Elasticsearch::Transport::Transport::Errors::BadRequest.new("[400] #{error_body}").message).to eq(errors["failing-document"].message) | ||
end | ||
end |