Search Usage Analytics - tracking_log_session index does not exist yet - 2.11.1

[chrisbadley]

I have seen a previously raised issue regarding this, however the issue that I am having can be traced to an error in the indexing.

The elasticsuite_tracker_log_event table is being populated with front end visit information as expected, the cronjob is being attempted, but fails with the following error in the cron log:

main.ERROR: Cron Job elasticsuite_index_log_event has an error: Warning: Undefined array key "date" in /html/vendor/smile/elasticsuite/src/module-elasticsuite-tracker/Model/EventIndex.php on line 79.

Which is the line:

$date = substr($event['date'], 0, 7);

From this function:

public function indexEvents($events)
    {
        $bulk    = $this->indexOperation->createBulk();
        $indices = [];
        foreach ($events as $event) {
            if (isset($event['page']['store_id'])) {
                $date = substr($event['date'], 0, 7);
                $index = $this->indexResolver->getIndex(self::INDEX_IDENTIFIER, $event['page']['store_id'], $date);
                if ($index !== null) {
                    $event = $this->mappingEnforcer->enforce($index, $event);
                    $indices[$index->getName()] = $index;
                    $bulk->addDocument($index, $event['event_id'], $event);
                }
            }
        }
        if ($bulk->isEmpty() === false) {
            $this->indexOperation->executeBulk($bulk);
        }
        foreach ($indices as $index) {
            $this->indexOperation->refreshIndex($index);
        }
    }

This in turn makes the cron job fail and the index never appear.

Magento Version : 2.4.6
ElasticSuite Version : 2.11.1
Environment : Developer
Elasticsearch 7.17

Third party modules :
Mirasvit Cache Warmer
BSS Commerce Out of stock Notification
BSS Commerce B2B Registration
BSS Commerce Hide Price
Xtento Product Feed Export Module

[chrisbadley]

There is not date field in the 'data' column of the elasticsuite_tracker_log_event table, which I believe to be causing the issue. Any help?

[vahonc]

Hello @chrisbadley,

It looks like some of your event(s) do not have a date. Could you share the screenshot of the data from your elasticsuite_tracker_log_event table?

It's honestly hard to say right now what exactly caused your problem because there is always a factor in the specific settings of each environment or server. And we cannot be held responsible for what and how you do it.

If in your elasticsuite_tracker_log_event table there are events with no date you can try to delete them.

Also, I can try to check Elasticsearch log in /var/log/elasticsearch/ or Magento log, maybe you will find more information there.

BR,
Vadym

[romainruaud]

Are you using Fastly ? If yes, check this page : https://github.com/Smile-SA/elasticsuite/wiki/FAQ#im-using-fastly-and-i-cant-see-anything-on-the-elasticsuite--search-usage-dashboard

[andrewsurrey]

@romainruaud - Fastly isn't a factor here, I've seen the exact same issue on an instance with the same magento/elastisuite versions - with none of the custom extensions listed in the original post present, and no fastly.

[romainruaud]

Then check what do you have at this line when receiving an event (ie browsing a site page) : https://github.com/Smile-SA/elasticsuite/blob/2.11.x/src/module-elasticsuite-tracker/Controller/Tracker/Hit.php#L59

[chrisbadley]

Hello @chrisbadley,

It looks like some of your event(s) do not have a date. Could you share the screenshot of the data from your elasticsuite_tracker_log_event table?

It's honestly hard to say right now what exactly caused your problem because there is always a factor in the specific settings of each environment or server. And we cannot be held responsible for what and how you do it.

If in your elasticsuite_tracker_log_event table there are events with no date you can try to delete them.

Also, I can try to check Elasticsearch log in /var/log/elasticsearch/ or Magento log, maybe you will find more information there.

BR, Vadym

{"image":"h.png","page":{"store_id":"3","type":{"identifier":"catalog_product_view","label":"Catalog Product View (Any)"},"locale":"en_GB","domain":"111.111.111.111","category":{"id":"1072","label":"New","path":"1\/196\/1072","breadcrumb":""},"product":{"id":"45504","label":"Product Name","sku":"SKU1"},"site":"xxxxxxxx.com","url":"\/product-url","title":"Product Name","resolution":{"x":"1920","y":"1040"}},"session":{"uid":"946c2f5c-f88a-4f4f-915c-8e7862d23035","vid":"8e1b7bce-3de2-f025-ac07-08743fb55db5"}}

Thats what is actually being store in the table, I have changed some of the values for privacy reasons.

[chrisbadley]

Can no one shed any light on this, I have a cron log with 2GB of errors from this

[vahonc]

@chrisbadley,

Have you checked the Elasticsearch log?

Have you debugged what you have here: https://github.com/Smile-SA/elasticsuite/blob/2.11.x/src/module-elasticsuite-tracker/Controller/Tracker/Hit.php#L59?

Are you using a custom theme on the frontend?

Are you using the Magento Cloud environment?

BR,
Vadym

[chrisbadley]

Elasticsearch log has nothing of interest.

Yes I am using a custom theme, Weltpixel Pearl.

No, I am using Magento CE self hosted.

Has Elastic suite been tested on a vanilla Magento 2.4.6?

[vahonc]

@chrisbadley

Could you check this query in your database: SELECT * FROM cron_schedule where messages is not NULL;? Are there jobs with some error message?

BR,
Vadym

[chrisbadley]

@chrisbadley

Could you check this query in your database: SELECT * FROM cron_schedule where messages is not NULL;? Are there jobs with some error message?

BR, Vadym

Over 5000 entries of this:

Warning: Undefined array key "date" in /var/www/xxxxxxxxxxxx.com/html/vendor/smile/elasticsuite/src/module-elasticsuite-tracker/Model/EventIndex.php on line 79

Which what was what the original post was regarding.

[vahonc]

@chrisbadley,

It looks like there really is a certain bug here, especially when using Magento 2.4.6. Of course, we tested the compatibility of our module, but it seems that we somehow missed this problem. Apparently, this is my fault 🤦.

I will try to discuss this problem with the team tomorrow morning and prepare a PR and fixes to this.

BR,
Vadym

[chrisbadley]

@chrisbadley,

It looks like there really is a certain bug here, especially when using Magento 2.4.6. Of course, we tested the compatibility of our module, but it seems that we somehow missed this problem. Apparently, this is my fault 🤦.

I will try to discuss this problem with the team tomorrow morning and prepare a PR and fixes to this.

BR, Vadym

Can't catch them all, but as long as you are aware that half the battle dealt with.

[vahonc]

@chrisbadley,
@andrewsurrey

I have added fixes for this issue. It will be included in the next release, but in the meantime, you can take inspiration from my PR and add fixes manually or by creating a patch. I have tested this solution on my test environment and it fixes this problem.

I would be grateful for your feedback on whether this solution helped you too.

BR,
Vadym

[andrewsurrey]

@vahonc excellent thanks, I'll create a patch and test it out

[romainruaud]

@chrisbadley and/or @andrewsurrey can you confirm that Vadym's PR is fixing the issue on your side as well ?

If that's the case I'll merge & tag a minor version really soon.

Regards

[chrisbadley]

@chrisbadley, @andrewsurrey

I have added fixes for this issue. It will be included in the next release, but in the meantime, you can take inspiration from my PR and add fixes manually or by creating a patch. I have tested this solution on my test environment and it fixes this problem.

I would be grateful for your feedback on whether this solution helped you too.

BR, Vadym

I have just applied the patch, and cleared down the logs and truncated the cron schedule table, I will be back in an hour to give you the results

[chrisbadley]

@chrisbadley, @andrewsurrey

I have added fixes for this issue. It will be included in the next release, but in the meantime, you can take inspiration from my PR and add fixes manually or by creating a patch. I have tested this solution on my test environment and it fixes this problem.

I would be grateful for your feedback on whether this solution helped you too.

BR, Vadym

Sorry to be the bearer of bad news, but it breaks analytics > search usage page.

Error: Argument #2 ($array) must be of type ?array, string given in /var/www/xxxxxx.com/html/vendor/smile/elasticsuite/src/module-elasticsuite-tracker/Model/IndexManager.php:171

[vahonc]

@chrisbadle,

Probably you selected some date range in the date picker. Try to flush or delete your cache, maybe cookies in your browser.

[chrisbadley]

@chrisbadle,

Probably you selected some date range in the date picker. Try to flush or delete your cache, maybe cookies in your browser.

I ran flush cache from command line. It started working

[vahonc]

So, all is good now, and the tracking_log_session and event indices are exists?

[chrisbadley]

So, all is good now, and the tracking_log_session and event indices are exists?

Yes. although I did get a strange 503 error when trying to view the search usage page, but I that's nothing to do with this particular issue.

Thanks for your time.

[andrewsurrey]

@romainruaud fix is working well, thanks.

tested as a patch on Magento 2.4.6 + ESuite 2.11.1

[romainruaud]

I tagged a 2.11.1.1

Thank you everyone