Base module for connecting Odoo with external search engines. This addon is intended to be used as a base for other addons that implement specific search engines. It's designed to be easily extensible and modular.
Table of contents
This addon uses the native json python package provided by python. When a json for a record is recomputed, the new value is compared to the original one to see if an export to the search engine index is needed. This is done by comparing the md5 of the two json strings. This process when done on a large number of records can be slow when the json is large and complex. To speed up this process you can install the orjson package.
pip install orjsonA search engine is a system designed to store information in a way that makes it easy to find through search and analytics queries. The main difference between a search engine and a database is that a search engine is optimized for search and analytics queries, while a database is optimized for transactional and relational queries.
This addons is designed around 4 main concepts:
The search engine backend is used to define into Odoo the kind of search engine that will be used to index the data. It's main responsibility is to provide an instance of odoo.addons.search_engine.tools.adapter.SearchEngineAdapter that will be used to communicate with the search engine.
The search engine index is used to define into Odoo the index where the data will be indexed. An index is always linked to a search engine backend. The index provides methods to use to manage the lifecycle of the data put into the index for the records of a given model. To do so, it uses:
- The SearchEngineAdapter provided by the backend to communicate with the search engine.
- A ModelSerializer that is used to transform an odoo record into a dictionary that can be indexed into the search engine.
- A JsonValidator that is used to validate the data that is to be indexed into the search engine.
The RecordSerializer and IndexDataValidator are defined on the index itself. The current addon provides a default implementation only for the IndexDataValidator. You can find into the github repository search-engine An implementation of the RecordSerializer based on the jsonifier addon connector_search_engine_jsonifier.
The search engine indexable record is a mixin that is used to define the records that can be indexed into a search engine index. The mixin provides methods:
- To add a record to an index.
- To remove a record from an index.
- To mark the record into an index (the search engine bindings) as to be recomputed (This method should be called when modifications are made on the record that could impact the data that are indexed into the search engine. It will instruct the index that the record must be recomputed and re-indexed).
It also ensures that when the record is unlinked, it is removed from the indexes it was indexed into.
The search engine binding is a model that represents the link between an index and an indexable odoo record. It give you access to the data that are indexed into the search engine for the record. It's also used to manage the lifecycle of the data into the search engine. When a binding is created, it's marked as to be computed. Once the data are computed, the binding is marked as to be indexed. Once the data are indexed, the binding is marked as indexed. If the linked record is unlinked, the binding is marked as to be removed. Once the data are removed from the search engine, the binding is deleted.
The indexing lifecycle is based on the following steps:
- When a record is added to an index, a binding is created and marked as to be computed.
- A cron job scheduled every 5 minutes will look for bindings that are to be computed and for each of them will schedule a job to re compute the json data.
- When the json data is computed, the binding is marked as to be exported if the json is valid and is different from the one that has been computed last time.
- A cron job scheduled every 5 minutes will ensure the syncing with the search
engine. It will:
- look for bindings that are to be exported and for each of them will schedule a job to export the json data into the search engine. Once exported, the binding is marked as 'done'.
- look for bindings that are to be removed and for each of them will schedule a job to remove the data from the search engine. Once removed, the binding is deleted.
To keep in sync the data from your model instance and the data that are indexed into the search engine, you should call the method _se_mark_to_update on the mode instance when you make modifications that could impact the data that are indexed into the search engine.
- When the method _se_mark_to_update is called, the binding is marked as to be computed.
- From there, the same process as described above will be used to recompute the data and reindex them into the search engine.
When a model instance is unlinked, the binding is marked as to be removed. From there if will be processed by the job syncing the data with the search engine.
Note
In previous versions of this addon, there was no method to mark a record as to be recomputed. As a consequence, all the records were re-computed every day to ensure that the data in the search engine were up to date. This was a performance issue and consumed a lot of resources. If despite this, you want to recompute all the records every day, you can activate the cron jon Search engine: recompute all index and deactivate the one named earch engine: Generate job for recompute binding to recompute per index.
- Implement generic trigger for binding based on ir.export linked to the index (the aim is to set the binding to be updated if we modify a field configured in the exporter)
Bugfixes
Ensure that the record's model is compatible with the index's model before adding a new record to the index.
Before this change, the index would silently ignore records that were not compatible with the index's model. This could lead to unexpected behavior and errors when the record was later used to be serialized to JSON and exported to a search engine. (#177)
Lower memory consumption by disabling prefetch for the field 'data' on the binding model.
The field 'data' is a json field that is not used in the view or common management operations of the binding model. This json field can be very large. By disabling the prefetch, we avoid to overload the database and Odoo with useless data. (#179)
Bugfixes
Fix error when calling the methods export_record or delete_record from the se.binding model when called on a recordset with items from different se.backend.
The export and delete methods involves the use of a Backend Adapter to communicate with the target search engine. We then need to process the bindings by backend to call the correct adapter and ensure at same time a batch process of the requested operation for all the records linked to the same backend. (#173)
Bugfixes
Add missing description on the "se.binding.state.updater" model. As well as ensuring consistency in the model definition, this change removes a warning message from the server logs at registry load time.
Prevent warning message in server logs when running tests. (#172)
Bugfixes
- Fixes cache issue with the se_binding_ids field on the s.indexable.record model. When a binding is created or updated or deleted, the cache for the se_binding_ids field for referenced records is now invalidated. That way, the next time the field is accessed after such an operation, the value is recomputed to reflect the change. (#163)
Features
A new action Update state is now available on Search Engine Record objects. This action allows you to update the state of selected records on the tree view.
Add a smart button to quickly access to the bound records from the Search Engine Backend and Search Engine Record views. (#162)
Bugfixes
Fix Search Engine Binding form view. The fields data and error are now properly displayed and fit the width of the form.
Makes the Odoo's admin user a member of the Search Engine Connector Manager group. (#162)
TODO
Bugs are tracked on GitHub Issues. In case of trouble, please check there if your issue has already been reported. If you spotted it first, help us to smash it by providing a detailed and welcomed feedback.
Do not contact contributors directly about support or help with technical issues.
- Akretion
- ACSONE SA/NV
- Camptocamp
- Sébastien BEAU <sebastien.beau@akretion.com>
- Laurent Mignon <laurent.mignon@acsone.eu>
- Simone Orsi <simone.orsi@camptocamp.com>
- Raphaël Reverdy <raphael.reverdy@akretion.com>
This module is maintained by the OCA.
OCA, or the Odoo Community Association, is a nonprofit organization whose mission is to support the collaborative development of Odoo features and promote its widespread use.
This module is part of the OCA/search-engine project on GitHub.
You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.