[BUG] Autosuggest for german Umlaute not working in Ext:solr 11

[ulrike-cosmoblonde]

Hi,

I have been using the autosuggest feature in several projects on German content without any Umlaut issues with the solr extension v10.
When using the schema.xml from the solr extension version 10 in my solr instance, then autosuggest works for Umlaute.
But with the latest schema.xml shipped with the solr extension version 11, Umlaute are no longer working in autosuggest.
So entering "Künst" does not produce any suggest results, but entering "Kunst" does show e.g. Kunstler, kunstlich, etc.

When comparing the configuration for the field type textSpell (which is the recommended field type for autosuggest) in both schemas, the difference and hence the problem is the additional filter "solr.ASCIIFoldingFilterFactory" in the index and query analyzer.

<!-- Setup simple analysis for spell checking -->
<fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
<analyzer type="index">
	<tokenizer class="solr.StandardTokenizerFactory"/>
	<filter class="solr.LowerCaseFilterFactory"/>

	<filter class="solr.DictionaryCompoundWordTokenFilterFactory"
		dictionary="german/german-common-nouns.txt"
		minWordSize="5"
		minSubwordSize="4"
		maxSubwordSize="15"
		onlyLongestMatch="false"
	/>

	<!-- no synonyms here because we do not want to add them as spell suggestion -->
	<filter class="solr.ManagedStopFilterFactory" managed="${solr.core.name}"/>
	<filter class="solr.ASCIIFoldingFilterFactory"/>

	<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
	<tokenizer class="solr.StandardTokenizerFactory" />

	<filter class="solr.LowerCaseFilterFactory"/>

	<filter class="solr.ManagedSynonymGraphFilterFactory" managed="${solr.core.name}" />
	<filter class="solr.ManagedStopFilterFactory" managed="${solr.core.name}"/>
	<filter class="solr.ASCIIFoldingFilterFactory"/>

	<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>

When removing this filter in both analyzer and re-indexing all data, then Umlaute work again for autosuggest.

Can you please remove that filter for the textSpell type in the German schema.xml (solr/Resources/Private/Solr/configsets/ext_solr_11_1_0/conf/german/schema.xml)?

Possibly other language also need the removal of that filter definition.

Regards,
Ulrike

[dkd-kaehm]

@dkd-friedrich

Strange thing:
IMHO it should work, but despite <filter class="solr.ASCIIFoldingFilterFactory"/> in "textSpell" on query analyzers it doesn't.
What do you think? Maybe the order is wrong there?

ext-solr/Resources/Private/Solr/configsets/ext_solr_11_5_0/conf/german/schema.xml

Lines 195 to 225 in 39e0431

	<fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
	<analyzer type="index">
	<tokenizer class="solr.StandardTokenizerFactory"/>
	<filter class="solr.LowerCaseFilterFactory"/>
	<filter class="solr.DictionaryCompoundWordTokenFilterFactory"
	dictionary="german/german-common-nouns.txt"
	minWordSize="5"
	minSubwordSize="4"
	maxSubwordSize="15"
	onlyLongestMatch="false"
	/>
	<!-- no synonyms here because we do not want to add them as spell suggestion -->
	<filter class="solr.ManagedStopFilterFactory" managed="${solr.core.name}"/>
	<filter class="solr.ASCIIFoldingFilterFactory"/>
	<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
	</analyzer>
	<analyzer type="query">
	<tokenizer class="solr.StandardTokenizerFactory" />
	<filter class="solr.LowerCaseFilterFactory"/>
	<filter class="solr.ManagedSynonymGraphFilterFactory" managed="${solr.core.name}" />
	<filter class="solr.ManagedStopFilterFactory" managed="${solr.core.name}"/>
	<filter class="solr.ASCIIFoldingFilterFactory"/>
	<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
	</analyzer>
	</fieldType>

[dkd-friedrich]

@ulrike-cosmoblonde is right. The behaviour of the suggest query differs from a normal search query, but rather the suggestions are facet expressions.

However, I would suggest not to remove the filter completely, but to set the option preserveOriginal additionally. This option saves the original spelling and the simplified variant, e.g. "künstler" and "kunstler". This has the advantage that suggestions for a simplified spelling can also be found, especially with accents, so that suggestions are also available for "recherchée" with "recherchee".

Since this variant can lead to duplications, with "recherchée" and "recherchee" being suggested, I would suggest to include an additional spell field with the old simplified configuration, which can be used via TypoScript (plugin.tx_solr.suggest.suggestField) if required.

I'll prepare pull requests for 11.5 and 11.1, probably before the end of the week.

[dkd-friedrich]

@ulrike-cosmoblonde To fix this issue, I have adapted the Solr scheme, the original term is now retained. For the example here, this means that suggestions for "kunst" and "künst" are displayed.

The original less flexible variant can be used by switching to another field via plugin.tx_solr.suggest.suggestField = spellExact.

It would be helpful if you could test the adjustments as well.

[dkd-friedrich]

I'm closing this issue, bugfix will be part of upcoming versions 11.2 and 11.5.