Table of Contents
YuStemmer
Overview
Word stemming is usually applied to query and search systems. It enables them to return related results with similar meaning but slightly different spelling. As an example, the English stemmer returns “write” for “write”, “writes”, “writing”, and “writings”.
Stemmers are available for these languages:
- Arabic, Armenian, Basque, Catalan, Czech, Danish, Dutch, English, Finnish, French, German, German2 (different Umlaut handling), Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Latin, Lithuanian, Lovins (English), Nepali, Norwegian, Porter (English), Portuguese, Romanian, Russian, Serbian, Slovene, Spanish, Swedish, Tamil, Turkish.
YuStemmer is fully algorithmic. No extensive lookup dictionaries are needed. This results in small memory footprint and excellent performance.
YuStemmer was initially developed for the DISQLite3 Full Text Search (FTS) engine which is prepared to use it out of the box. Besides that, YuStemmer fits many other purposes.
YuStemmer is organized into different classes, each of them optimized for a particular string type and text encoding:
TYuStemmer
class:- ANSI text, 8-bit, usually in ISO-8859-1, unless otherwise noted.
- Some stemmers might not be available for this class.
TYuStemmer_8
class:- UTF-8 text, 8-bit.
- All stemmers are available for this class.
TYuStemmer_16
class:- UTF-16 text, 16-bit. This corresponds to Delphi's WideString / UnicodeString.
- All stemmers are available for this class.
Make sure to choose the stemmer class matching your string type and character set. Otherwise you will suffer a performance penalty caused by avoidable string conversions. In Delphi, such conversions usually happen implicitly and go unnoticed by most developers. Therefore, pay close attention here to make the most of YuStemmer!
Example – stem a single word
The Stem() method does the work for all of the above classes. It expects a single word and returns its stem. If there is no stem, the original word is returned unchanged.
function StemFrench(const AWord: AnsiString): AnsiString; var Stemmer: TYuStemmer; begin Stemmer := TYuStemmer_French.Create; Result := Stemmer.Stem(AWord); Stemmer.Free; end;
Example – stem multiple words in a TStringList
To improve performance when stemming a great number of words, it is safe to reuse the same instance of a stemmer class multiple times.
procedure StemItalian(const AWords: TStringList); var i: Integer; { TStringList is UTF-16 in Unicode Delphis. } Stemmer: {$IFDEF Unicode}TYuStemmer_16{$ELSE}TYuStemmer{$ENDIF}; begin {$IFDEF Unicode} Stemmer := TYuStemmer_Italian_16.Create; {$ELSE Unicode} Stemmer := TYuStemmer_Italian.Create; {$ENDIF Unicode} for i := 0 to AWords.Count - 1 do AWords[i] := Stemmer.Stem(AWords[i]); Stemmer.Free; end;