-iKdZddlmZmZmZmZddlZddlZej dddl m Z e dkr3ej dddl Z ddlZddlZddlZddlZddlZn$ddlm Z dd lmZdd lmZdd lmZdd lmZdd lmZGddZe dkrAeZgdZeD]1Zeeedeeejedeedeedejeeeedede edej!ede"ee"dee#ede$ede%ede&ede'ee(eee)ee*e+1dSdS)a\ Arabic Light Stemmer A class which provides a configurable stemmer and segmentor for arabic text. Features: ========= - Arabic word Light Stemming. - Root Extraction. - Word Segmentation - Word normalization - Default Arabic Affixes list. - An customizable Light stemmer: possibility of change stemmer options and data. - Data independent stemmer @author: Taha Zerrouki @author: Taha Zerrouki @contact: taha dot zerrouki at gmail dot com @copyright: Arabtechies, Arabeyes, Taha Zerrouki @license: GPL @date:2017/02/15 @version:0.3 )absolute_importprint_functionunicode_literalsdivisionNz ../support/__main__z../) normalize) stem_const) affix_const) roots_const)verb_stamp_const)arabicstopwordsceZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdCdZdZdZ d Z!d!Z"d"Z#d#Z$dCd$Z%d%Z&dCd&Z'dDd'Z(dDd(Z)dCd)Z*dEd+Z+d,Z,d-Z-dCd.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5gfd6Z6dFd9Z7dCd:Z8dGd;Z9dGd<Z:e;d=ZZ=e;d?Z>d@Z?e;dAZ@dBS)HArabicLightStemmera ArabicLightStemmer: a class which proved a configurable stemmer and segmentor for arabic text. Features: ========= - Arabic word Light Stemming. - Root Extraction. - Word Segmentation - Word normalization - Default Arabic Affixes list. - An customizable Light stemmer: possibility of change stemmer options and data. - Data independent stemmer @author: Taha Zerrouki @author: Taha Zerrouki @contact: taha dot zerrouki at gmail dot com @copyright: Arabtechies, Arabeyes, Taha Zerrouki @license: GPL @date:2017/02/15 @version:0.3 ctj|_tj|_tj|_tj|_tj |_ tj |_ tj |_tj|_tj|_t&j|_t-t/t0jt/t0jz|_d|_d|_d|_d|_d|_ d|_!d|_"g|_#tIj%dtHj&|_'|(|j|_)|*|j|_+dS)Nru [^\wً-ْ']+),r DEFAULT_PREFIX_LETTERSprefix_lettersDEFAULT_SUFFIX_LETTERSsuffix_lettersDEFAULT_INFIX_LETTERS infix_lettersDEFAULT_MAX_PREFIXmax_prefix_lengthDEFAULT_MAX_SUFFIXmax_suffix_lengthDEFAULT_MIN_STEMmin_stem_length DEFAULT_JOKERjokerDEFAULT_PREFIX_LIST prefix_listDEFAULT_SUFFIX_LIST suffix_listr ROOTS root_listsetlistr VERB_AFFIX_LISTNOUN_AFFIX_LISTvalid_affixes_listword unvocalized normalizedstarwordrootleftright segment_listrecompileUNICODE token_pat_create_prefix_tree prefixes_tree_create_suffix_tree suffixes_treeselfs Q/var/www/html/speakWrite/venv/lib/python3.11/site-packages/tashaphyne/stemming.py__init__zArabicLightStemmer.__init__Os')?(?'=!+!>!+!>):- %9%9$*#&d;+F&G&G${OjJkJk&k"l"l     $;RZHH!55d6FGG!55d6FGGc|jS)z return the prefixation letters. This constant take DEFAULT_PREFIX_LETTERS by default. @return: return a letters. @rtype: unicode. rr<s r>get_prefix_lettersz%ArabicLightStemmer.get_prefix_lettersp ""r@c||_dS)u set the prefixation letters. This constant take DEFAULT_PREFIX_LETTERS by default. @param new_prefix_letters: letters to be striped from a word, e.g.new_prefix_letters = u"وف":. @type new_prefix_letters: unicode. NrB)r=new_prefix_letterss r>set_prefix_lettersz%ArabicLightStemmer.set_prefix_lettersx1r@c|jS)z return the suffixation letters. This constant take DEFAULT_SUFFIX_LETTERS by default. @return: return a letters. @rtype: unicode. rr<s r>get_suffix_lettersz%ArabicLightStemmer.get_suffix_lettersrDr@c||_dS)u  set the suffixation letters. This constant take DEFAULT_SUFFIX_LETTERS by default. @param new_suffix_letters: letters to be striped from the end of a word, e.g.new_suffix_letters = u"ةون":. @type new_suffix_letters: unicode. NrJ)r=new_suffix_letterss r>set_suffix_lettersz%ArabicLightStemmer.set_suffix_lettersrHr@c|jS)z get the inffixation letters. This constant take DEFAULT_INFIX_LETTERS by default. @return: infixes letters. @rtype: unicode. rr<s r>get_infix_lettersz$ArabicLightStemmer.get_infix_letterss !!r@c||_dS)u  set the inffixation letters. This constant take DEFAULT_INFIX_LETTERS by default. @param new_infix_letters: letters to be striped from the middle of a word, e.g.new_infix_letters = u"أوي":. @type new_infix_letters: unicode. NrP)r=new_infix_letterss r>set_infix_lettersz$ArabicLightStemmer.set_infix_letterss/r@c|jS)z get the joker letter. This constant take DEFAULT_JOKER by default. @return: joker letter. @rtype: unicode. )r r<s r> get_jokerzArabicLightStemmer.get_jokers zr@cJt|dkr|d}||_dS)z set the joker letter. This constant take DEFAULT_JOKER by default. @param new_joker: joker letter. @type new_joker: unicode. rrN)lenr )r= new_jokers r> set_jokerzArabicLightStemmer.set_jokers) y>>A  !! I r@c|jS)z return the constant of max length of the prefix used by the stemmer. This constant take DEFAULT_MAX_PREFIX_LENGTH by default. @return: return a number. @rtype: integer. rr<s r>get_max_prefix_lengthz(ArabicLightStemmer.get_max_prefix_length %%r@c||_dS)a Set the constant of max length of the prefix used by the stemmer. This constant take DEFAULT_MAX_PREFIX_LENGTH by default. @param new_max_prefix_length: the new max prefix length constant. @type new_max_prefix_length: integer. Nr\)r=new_max_prefix_lengths r>set_max_prefix_lengthz(ArabicLightStemmer.set_max_prefix_length "7r@c|jS)z return the constant of max length of the suffix used by the stemmer. This constant take DEFAULT_MAX_SUFFIX_LENGTH by default. @return: return a number. @rtype: integer. rr<s r>get_max_suffix_lengthz(ArabicLightStemmer.get_max_suffix_lengthr^r@c||_dS)a Set the constant of max length of the suffix used by the stemmer. This constant take DEFAULT_MAX_SUFFIX_LENGTH by default. @param new_max_suffix_length: the new max suffix length constant. @type new_max_suffix_length: integer. Nrd)r=new_max_suffix_lengths r>set_max_suffix_lengthz(ArabicLightStemmer.set_max_suffix_lengthrbr@c|jS)z return the constant of min length of the stem used by the stemmer. This constant take DEFAULT_MIN_STEM_LENGTH by default. @return: return a number. @rtype: integer. rr<s r>get_min_stem_lengthz&ArabicLightStemmer.get_min_stem_lengths ##r@c||_dS)z Set the constant of min length of the stem used by the stemmer. This constant take DEFAULT_MIN_STEM_LENGTH by default. @param new_min_stem_length: the min stem length constant. @type new_min_stem_length: integer. Nrj)r=new_min_stem_lengths r>set_min_stem_lengthz&ArabicLightStemmer.set_min_stem_lengths 3r@c|jS)z return the prefixes list used by the stemmer. This constant take DEFAULT_PREFIX_LIST by default. @return: prefixes list. @rtype: set(). )r"r<s r>get_prefix_listz"ArabicLightStemmer.get_prefix_list r@cH||_||jdS)z Set prefixes list used by the stemmer. This constant take DEFAULT_PREFIX_LIST by default. @param new_prefix_list: a set of prefixes. @type new_prefix_list: set of unicode string. N)r"r8)r=new_prefix_lists r>set_prefix_listz"ArabicLightStemmer.set_prefix_list) +   !122222r@c|jS)z return the suffixes list used by the stemmer. This constant take DEFAULT_SUFFIX_LIST by default. @return: suffixes list. @rtype: set(). )r$r<s r>get_suffix_listz"ArabicLightStemmer.get_suffix_listrqr@cH||_||jdS)z Set suffixes list used by the stemmer. This constant take DEFAULT_SUFFIX_LIST by default. @param new_suffix_list: a set of suffixes. @type new_suffix_list: set of unicode string. N)r$r:)r=new_suffix_lists r>set_suffix_listz"ArabicLightStemmer.set_suffix_listrur@c|jS)z return the roots list used by the stemmer to validate roots. This constant take roots_const.ROOTS by default. @return: roots list. @rtype: set().  roots_listr<s r>get_roots_listz!ArabicLightStemmer.get_roots_lists r@c||_dS)z Set roots list used by the stemmer to validate roots.. This constant take roots_const.ROOTS by default. @param new_roots_list: a set of roots. @type new_roots_list: set of unicode string. Nr|)r=new_roots_lists r>set_roots_listz!ArabicLightStemmer.set_roots_lists )r@c|jS)z return the valid_affixes list used by the stemmer to validate affixes. This constant take valid_affixes_const.ROOTS by default. @return: valid_affixes list. @rtype: set(). r+r<s r>get_valid_affixes_listz)ArabicLightStemmer.get_valid_affixes_lists &&r@c||_dS)a Set valid_affixes list used by the stemmer to validate affixes.. This constant take valid_affixes_const.ROOTS by default. @param new_valid_affixes_list: a set of valid_affixes. @type new_valid_affixes_list: set of unicode string. Nr)r=new_valid_affixes_lists r>set_valid_affixes_listz)ArabicLightStemmer.set_valid_affixes_lists #9r@c||_dS)zw Set the word to treat by the stemmer. @param new_word: the new word. @type new_word: unicode. Nr,)r=new_words r>set_wordzArabicLightStemmer.set_word#s  r@c|jS)zf return the last word treated by the stemmer. @return: word. @rtype: unicode. rr<s r>get_wordzArabicLightStemmer.get_word*s yr@c|jS)u return the starlike word treated by the stemmer. All non affix letters are converted to a joker. The joker take by default DEFAULT_JOKER = "*". Exmaple: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتصربونني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_starword() أفت***ونني @return: word. @rtype: unicode. )r/r<s r> get_starwordzArabicLightStemmer.get_starword4s }r@c|dks|dkr|||n||_|jS)u return the root of the treated word by the stemmer. All non affix letters are converted to a joker. All letters in the joker places are part of root. The joker take by default DEFAULT_JOKER = "*". Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتصربونني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_starword() أفت***ونني >>> print ArListem.get_root() ضرب @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: root. @rtype: unicode. r) extract_root _choose_rootr0r= prefix_index suffix_indexs r>get_rootzArabicLightStemmer.get_rootEsM2 1    1 1   lL 9 9 9 9))++DIyr@ctj|jrtj|jS|js||j|}d|D}|}tt|j |}|r|}tt|j |}|r|}| |}|S)z" choose a root for the given word cg|] }|d Sr0).0ds r> z3ArabicLightStemmer._choose_root..ls///q6///r@) ris_stopr, stop_rootr3segmentget_affix_listr(filteris_root_length_validis_root most_common)r= affix_listroots roots_tmpaccepted accepted_roots r>rzArabicLightStemmer._choose_rootds  "49 - - 8",TY77 7  $ LL # # #((** //J/// t8)DDEE  9t|Y77@@  9((33 r@cTtjjrtjjSjsjj}fd|D}|sd}t j}n|\}}j||S)z" choose a stem for the given word cJg|]\}}||||f Sr) _verify_affix)rxyr=s r>rz3ArabicLightStemmer._choose_stem..s6KKKeq43E3Ea3J3JKQqEKKKr@r) rrr, stop_stemr3rrXget_left_rightr-)r=seg_listr1r2s` r> _choose_stemzArabicLightStemmer._choose_stem{s  "49 - - 8",TY77 7  $ LL # # #$KKKKxKKK 8D NNEE--h77KD%U ++r@c|jS)u return the normalized form of the treated word by the stemmer. Some letters are converted into normal form like Hamzat. Example: >>> word = u"استؤجرُ" >>> ArListem = ArabicLightStemmer() >>> stem = ArListem .light_stem(word) >>> print ArListem.get_normalized() استءجر @return: normalized word. @rtype: unicode. )r.r<s r>get_normalizedz!ArabicLightStemmer.get_normalizeds r@c|jS)u return the unvocalized form of the treated word by the stemmer. Harakat are striped. Example: >>> word = u"الْعَرَبِيّةُ" >>> ArListem = ArabicLightStemmer() >>> stem = ArListem .light_stem(word) >>> print ArListem.get_unvocalized() العربية @return: unvocalized word. @rtype: unicode. )r-r<s r>get_unvocalizedz"ArabicLightStemmer.get_unvocalizeds r@c|jS)u return the the left position of stemming (prefixe end position )in the word treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتصربونني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_starword() أفت***ونني >>> print ArListem.get_left() 3 @return: the left position of stemming. @rtype: integer. )r1r<s r>get_leftzArabicLightStemmer.get_lefts yr@c|jS)u return the the right position of stemming (suffixe start position )in the word treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتصربونني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_starword() أفت***ونني >>> print ArListem.get_right() 6 @return: the right position of stemming. @rtype: integer. )r2r<s r> get_rightzArabicLightStemmer.get_rights "zr@c|dks|dkr/|dkr|j}n|}|dkr|j}n|}|j||S|}|S)u return the stem of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتكاتبانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_stem() كاتب @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: stem. @rtype: unicode. r) stem_left stem_rightr-r)r=rrr1r2stems r>get_stemzArabicLightStemmer.get_stemsw, 1    1 1a~$a%#DJ/ /$$&&D r@cf|}|tjd}t|dkrCt jdtjtjtjd|j |}|S|ddt jtj|j |ddz}|j || dr9|ddt jtj|j |ddz}n%t jtj|j |}|j || dr9|ddt jtj|j |ddz}n%t jtj|j |}|S) zh Handle case of Teh as infix. The Teh can be Dal after Zain, and Tah after Dhad r[]Nuضطuزد) replacearaby TEH_MARBUTArXr4subTEHTAHDALr r, startswith)r=r/r1r2 newstarstemkey_stems r>_handle_teh_infixz$ArabicLightStemmer._handle_teh_infixsr  &&u'8<< x==A  &&eiiEIII!NPTPZ\ghhK ""1"obfUY KPQPRPRO&T&TT 9T%Z + +G 4 4 E%bqb/"&DJ TUTVTV*X*XXKK&DJ DDK Id5j ! , ,W 5 5 E%bqb/"&DJ TUTVTV*X*XXKK&DJ DDKr@c|j}|dkr|dkr||j|jS|j}|j}|dkr|}|dkr|}|jdkrPt jd|jt jd|j|||}| |||}n|jt|||z}|S)u{ return the star form stem of the treated word by the stemmer. All non affix letters are converted to a joker. The joker take by default DEFAULT_JOKER = "*". Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتكاتبانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_stem() كاتب >>> print ArListem.get_starstem() *ات* @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: stared form of stem. @rtype: unicode. rr[^r) r,r1r2rr4rrrr rrX)r=rrr/r1r2rs r> get_starstemzArabicLightStemmer.get_starstems09 !   q 0 0DIdj01 19DJEq  #q  $!R'' ff1C1CUEVEVEV%W:xU 355 #44[$NN "jXd5j-A)B)BB  r@cT|dkr|jd|jS|jd|S)u return the prefix of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتكاتبانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_prefix() أفت @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @return: prefixe. @rtype: unicode. rN)r-r1)r=rs r> get_prefixzArabicLightStemmer.get_prefixDs5 !  #JTYJ/ /#M\M2 2r@cT|dkr|j|jdS|j|dS)u  return the suffix of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتكاتبانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_suffix() انني @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: suffixe. @rtype: unicode. rN)r-r2)r=rs r> get_suffixzArabicLightStemmer.get_suffixZs5 !  #DJKK0 0#LMM2 2r@c|d||||gS)u return the affix of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتكاتبانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_affix() أفت-انني @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: in4teger. @return: suffixe. @rtype: unicode. -)joinrrrs r> get_affixzArabicLightStemmer.get_affixos<&yy$//,77  %%'(( (r@rc|||||||||||||dS)u return the affix tuple of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتضاربانني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_affix_tuple() {'prefix': u'أفت', 'root': u'ضرب', 'suffix': u'انني', 'stem': u'ضارب'} @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: affix tuple. @rtype: dict. )prefixsuffixrstarstemr0)rrrrrrs r>get_affix_tuplez"ArabicLightStemmer.get_affix_tuplesf(__\22__\22==|<<((|DD==|<<    r@c|dkrdS|||||S)uJ Stemming function, stem an arabic word, and return a stem. This function store in the instance the stemming positions (left, right), then it's possible to get other calculted attributs like: stem, prefixe, suffixe, root. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتضاربانني' >>> stem = ArListem.light_stem(word) >>> print ArListem.get_stem() ضارب >>> print ArListem.get_starstem() *ا** >>> print ArListem.get_left() 3 >>> print ArListem.get_right() 6 >>> print ArListem.get_root() ضرب @param word: the input word. @type word: unicode. @return: stem. @rtype: unicode. r)transform2starsrrr=r,s r> light_stemzArabicLightStemmer.light_stemsH6 3;;3 T""" T}}r@c@||_tj|}||_t jdtjztjtjz|}t jd|j |j d|j |}| |j }| |j }|dkrt||jdz }t!|dzt#||jz }|d|}|j||}||d}t jd|j z|j |}|jr#t jd|jz|j |}t jd|j z|j |}||z|z}| |j }| |j }|dkr%t|jt#|dz }|dkr|d|}|d kr"||jvr|dd }|d kr ||jv|dkr2t!t#|t#||jz }||d}|r||jvr|dd}|r ||jvt#|}t#|t#|z }|j||}|jr#t jd|jz|j |}||z|z}||_||_||_|||fS) ut Transform all non affixation letters into a star. the star is a joker(by default '*'). which indicates that the correspandent letter is an original. this function is used by the stmmer to identify original letters. and return a stared form and stemming positions (left, right) Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتضاربانني' >>> starword, left, right = ArListem.transformToStrars(word) (أفت*ا**انني, 3, 6) @param word: the input word. @type word: unicode @return: (starword, left, right): - starword: all original letters converted into a star - left: the greater possible left stemming position. - right: the greater possible right stemming position. @rtype: tuple. [%s]rrrrNz[^%s]rrr)r,rstrip_tashkeelr-r4r ALEF_MADDAHAMZAALEFrrr findrfindminrmaxrXrrr"r$rrr/)r=r,r1r2rrrs r>rz"ArabicLightStemmer.transform2starss, #D))vgu/0%+ej2H$OOvv$"5"5t7J7J7JK Tyy$$ 4:&& 199tT3A566DaT4+A!ABBE%4%[F9T%Z(D%&&\FVHT%88$*fMMF! Mvht'994:tLLVHT%88$*fMMF$;v%Dyy$$ 4:&& !88t-s4yy{;;D 199%4%[FB,,61A#A#AB,,61A#A#AqyyCKKT43I)IJJ%&&\F $V4+;;; $V4+;;;v;;DIIc&kk)E9T%Z(D ! Mvht'994:tLL$;v%D dE""r@c|||}d}t|dkr"||||_|jS|||}d}t|t|kr,t |D]\}}|||jkr||z }n|}||}t|dkr|||}||_|S)u return the root of the treated word by the stemmer. All non affix letters are converted to a joker. All letters in the joker places are part of root. The joker take by default DEFAULT_JOKER = "*". Example: >>> ArListem = ArabicLightStemmer() >>> word = u'أفتصربونني' >>> stem = ArListem .light_stem(word) >>> print ArListem.get_starword() أفت***ونني >>> print ArListem.get_root() ضرب @param prefix_index: indicate the left stemming position if = -1: not cosidered, and take the default word prefix lentgh. @type prefix_index:integer. @param suffix_index:indicate the right stemming position. if = -1: not cosidered, and take the default word suffix position. @type suffix_index: integer. @return: root. @rtype: unicode. rr)rrX _ajust_rootr0r enumerater normalize_root)r=rrrr0richars r>rzArabicLightStemmer.extract_roots2}}\<88 t99>>((t44DI9 $$\<@@ x==CII % %$T?? ! !4A;$*,,DLD !D""4(( t99>>##D(33D  r@c|s|St|dkrV|tjtj}|tjtj}|S|d}|dd}|tjtjfvrtj|z}n|tjkrtj|z}n||jkr*|tjtjfvr|tjz }n||jkr*|tjtjfvr|tjz }nS||jkrH||jkr=t|dkr ||dz }n|dtjz|dz}|S)z] If the root has only three or two letters, we complete it by another letter rrrNrr)rXrrrWAW ALEF_MAKSURAYEHr )r=r0rfirstlasts r>rzArabicLightStemmer._ajust_rootLsf K x==A  '' EI>>H''(:EIFFHO } UZ+ + +9t#DD ei  9t#DD dj Tej%)-D%D%D EI DD dj Te.@%)-L%L%L EI DD dj TTZ%7%78}}!!R Aw )472 r@ci}|D]1}|}|D]}||vri||<||}d|vr d|d|<*|di|d<2||_|jS)z Create a prefixes tree from given prefixes list @param prefixes: list of prefixes @type prefixes: list of unicode @return: prefixes tree @rtype: Tree stucture #)r9)r=prefixes prefixestreerbranchrs r>r8z&ArabicLightStemmer._create_prefix_treels  - -F!F & &v%%#%F4Lf}}&)s F##%cns )!!r@ci}|D]:}|}|dddD]}||vri||<||}d|vr d|d|<3|di|d<;||_|jS)z Create a suffixes tree from given suffixes list @param suffixes: list of suffixes @type suffixes: list of unicode @return: suffixes tree @rtype: Tree stucture Nrr)r;)r=suffixes suffixestreerrrs r>r:z&ArabicLightStemmer._create_suffix_trees  - -F!Fttt  & &v%%#%F4Lf}}&)s F##%cns )!!r@cZ|j}dg}d}|t|kr_|||vrUd|vr|||||vr|||}nn"|dz }|t|kr |||vU|t|krd|vr|||S)z lookup for prefixes in the word @param word: the given word @type word: unicode @return: list of prefixes starts positions @rtype: list of int rrr)r9rXappend)r=r,rleftsrs r>lookup_prefixesz"ArabicLightStemmer.lookup_prefixess# #d))mmQ6 1 1f}} QAw&  Q FA#d))mmQ6 1 1 s4yy==SF]] LLOOO r@cP|j}d}g}t|dz }|dkr`|||vrV|||z}d|vr||dz|||vr|||}nn|dz}|dkr |||vV|dkrd|vr||dz|S)z lookup for suffixes in the word @param word: the given word @type word: unicode @return: list of suffixes starts positions @rtype: list of int rrrr)r;rXr)r=r,rrrightsrs r>lookup_suffixesz"ArabicLightStemmer.lookup_suffixess# IIaK1ffaF**!WV^Ff}} ac"""Aw&  Q FA1ffaF** 66cVmm MM!A#    r@c||_tj||_t jdtjztjtjz|}| |}| |}|rt||_ nd|_ |rt||_nd|_tdt!|fg|_|D],}|D]'}||dzkr|j||f(-||j\|_ |_|jS)u generate a list of all possible segmentation positions (lef, right) of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'فتضربين' >>> print ArListem.segment(word) set(([(1, 5), (2, 5), (0, 7)]) @return: List of segmentation @rtype: set of tuple of integer. rrrr)r,rrr-r4rrrrrr rr1rr2r'rXr3addr)r=r,rr rjs r>rzArabicLightStemmer.segments9  /55vgu/0%+ej2H$OO$$T**%%d++  E DIIDI  VDJJDJ!SYY 011 2 2A 2 2!88%))1a&111 2 !% 3 3D4E F F 4:  r@c|jS)u return a list of segmentation positions (left, right) of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'فتضربين' >>> ArListem.segment(word) >>> print ArListem.get_segment_list() set(([(1, 5), (2, 5), (0, 7)]) @return: List of segmentation @rtype: set of tuple of integer. )r3r<s r>get_segment_listz#ArabicLightStemmer.get_segment_lists   r@c~|s|j}g}|D].\}}||||/|S)ug return a list of affix tuple of the treated word by the stemmer. Example: >>> ArListem = ArabicLightStemmer() >>> word = u'فتضربين' >>> ArListem.segment(word) >>> print ArListem.get_affix_list() [{'prefix': u'ف', 'root': u'ضرب', 'suffix': u'ين', 'stem': u'تضرب'}, {'prefix': u'فت', 'root': u'ضرب', 'suffix': u'ين', 'stem': u'ضرب'}, {'prefix': u'', 'root': u'فضربن', 'suffix': u'', 'stem': u'فتضربين'}] @return: List of Affixes tuple @rtype: list of dict. )r3rr)r=rrr1r2s r>rz!ArabicLightStemmer.get_affix_lists[ )(H # A AZT%   d224?? @ @ @ @r@nounrc|sdS|dkrUt|dkst|dkrdStj|vrdSt|dkr!|tjsdSt|dkr[|dtjtjfvr;|ddtjtjtjtjfvrdSnd|tjrE|ddtjtjtjtjtjfvrdStj |sdSn|d krt|d krdSd Sd S) z Test if the stem is acceptedFverbrrrNrT) rXrrrrrrNOONALEF_HAMZA_ABOVEr is_verb_stamp)r=rtagrs r> _valid_stemzArabicLightStemmer._valid_stem*sk 5 &==4yy1}}D A u"d**uTa (C(CuTaQEJ 3J(J(J"##;59eiUE["\\\ 5],, &+%)UZY^Ybdidz}B}GBH3H3Hu$2488 u  F]]4yyA~~u4tr@c||}||}d}|r|dz|z}|||}|tjvr?||d|r(|tjvr||drdSdS|tjvr||drdSdSdS)zB validate affixes against a list of valid affixes TrrrF)rrrr r)rr*r+)r=rrrrTAGaffixrs r>rz ArabicLightStemmer._verify_affixOs....  !3Jv%E]]<>>D 3338H8HfV\8]8]3K777Dr zArabicLightStemmer.normalizesQ 3;;49++3 S[[DII9D#8>>r@ct|sgS|j|}d|vr|d|S)z Tokenize text into words @param text: the input text. @type text: unicode. @return: list of words. @rtype: list. r)r7splitremove)r=textmylists r>tokenizezArabicLightStemmer.tokenizesF I^))$//Ff}} c"""Mr@c,|tjtjtjz}|tjd}|tjtj}tj|S) test if word is a rootr) rrrrrrrrnormalize_hamzars r>rz!ArabicLightStemmer.normalize_rootsb||E,ek5:.EFF||E-r22||E. ::$T***r@cNt|dkot|dkS)NrrrXrs r>rz'ArabicLightStemmer.is_root_length_valids#D Q/3t99a<0r@cd|D}|r|}ttt||jS)Nc8g|]}t|dk|S)rr,)rrs r>rz2ArabicLightStemmer.most_common..s#222!c!ffkkAkkkr@)key)rr'sortedcount)lsttrirootss r>rzArabicLightStemmer.most_commonsE22s222  C3vc{{##3333r@c||jvS)r))r&rs r>rzArabicLightStemmer.is_rootst~%%r@cn|sdSt|\}tfd|D}|fS)z4 get the max left and the min right rrc&g|] \}}|k |Srr)rrrls r>rz5ArabicLightStemmer.get_left_right..s"+++u!adddddr@)rr)ls_rr8s @r>rz!ArabicLightStemmer.get_left_rightsL  5 WW! +++++++ , ,!t r@Nr6)r)rr)rr)r)A__name__ __module__ __qualname____doc__r?rCrGrKrNrQrTrVrZr]rarerhrkrnrprtrwrzr~rrrrrrrrrrrrrrrrrrrrrrrrr8r:rr rrrrrr r' staticmethodrrrrrrr@r>rr5s)2HHHB###111###111"""///&&&777&&&777$$$333   333   333)))'''999">.,,,,    $&$$$$L6++++Z3333,3333*((((,    :###JR#R#R#h////b@"""."""24@(!(!(!\!!!"')*""""J7777x( ++\+11\144\4 &&&\r@r)uأفتضارباننيبالمكتبةu مزدهرةuكاتبu مضروبu مضاربu مردودu مطلوبuمشتتu مزتهرةu مضطربrAuمالبدرسمهu مكتوبu الآجالuبالبلدانu وفيهما1245Taha@rzroot:r1z left stemrr2 right_stemr r/rr r-),r? __future__rrrrr4syspathrpyarabic.arabyrr<r r r r r rrrARLISTEMwordlistr,rprintrrrrrrrrrrrrrrrrreprrrr@r>rOs4    zHOOE""""""!!!!!![[[[[[[[|$ z!!##HH**/*/D!!! fh''))*** h$%%% gx((**+++ fX&&(())) k(,--- h!!##$$$ h!!!$$%%% gh((**+++ l8./// h++--... h!!"%%&&& h&&(())) j(//11222 j..00111 k82244555 mH4466777 ht$$%%% h''))*** dd8**,,--....E0*/*/r@