-igdZddlZddlZddlZejdddlZddlmZ ddl Z ddl m Z  ddlZddlZn#ddlmZddlmZYnxYwGddejZdS)z Arabic stopword stemmer N.)stopwords_lexiconceZdZdZedZdZedZedZdZ edZ edZ d Z dd Z dd ZdZgfdZddZdZd S)stopword_affixercZtj|tj|_g|_tj|_tj|_ tj |_ tj |_ tj|_g|_tj|_tj|_t+tj|_g|_t+tj|_t+tj|_ tj|_g|_tj|_tj|_t5|_tj|_dSN) basic_affixer__init__SSCCOMP_PREFIX_LIST procleticsprefixesCONJ_SUFFIX_LISTsuffixesCOMP_SUFFIX_LIST encliticsSTOPWORDS_CONJUGATION_AFFIXaffixesCOMP_STOPWORDS_AFFIXEScliticsCOMP_PREFIX_LIST_TAGSprocletics_tags prefixes_tagsCONJ_SUFFIX_LIST_TAGS suffixes_tagsCOMP_SUFFIX_LIST_TAGSenclitics_tagslist%VOCALIZED_INDEX_COMP_PREFIX_LIST_TAGSkeys%VOCALIZED_INDEX_CONJ_SUFFIX_LIST_TAGS%VOCALIZED_INDEX_COMP_SUFFIX_LIST_TAGSr dictionary AJUSTMENTajustment_table)selfs W/var/www/html/speakWrite/venv/lib/python3.11/site-packages/alyahmor/stopword_affixer.pyr zstopword_affixer.__init__&s*#,,T444. , -6 1  #8 6!7sHMMOOPP SFKKMMNN cGLLNNOO"H F!G ,-- #}ctt|g}|r4|tjtjztjtjzfvr$|tjz}|||tjr,|ddtjz}|||}|S)u Generate the Stop stem variants according to the affixes. For example مدرستي = >مدرست+ي = > مدرسة +ي. Return a list of possible cases. @param stem: the input stem. @type stem: unicode. @param suffix_nm: suffix (no mark). @type suffix_nm: unicode. @return: list of stem variants. @rtype: list of unicode. N)setarYEHNOONWAWaddendswith ALEF_MAKSURA)stem suffix_nmpossible_stop_list possible_stopvalidated_lists r'get_stem_variantsz"stopword_affixer.get_stem_variantsZs"! "    2I"&27*:*,&27*:*<<< 26MM  " "= 1 1 1 ==  2 "I7M  " "= 1 1 1+r(c@tj|}|}tj|}|sC|ddtjtjtjfvrtj|rd}d|j|dvrtj|}n|}||fS)u$ Get the suffix variant to be joined to the word. For example: word = مدرس, suffix = ة, encletic = ي. The suffix is converted to Teh. @param word: word found in dictionary. @type word: unicode. @param suffix: second level suffix. @type suffix: unicode. @param enclitic: first level suffix. @type enclitic: unicode. @return: variant of suffixes (vocalized suffix and vocalized suffix without I'rab short mark). @rtype: (unicode, unicode) r*N متحركtags)r,strip_tashkeelstrip_lastharakar2r-ALEF is_harakar)r&wordsuffixenclitic enclitic_nm newsuffixsuffix_non_irab_marks r'get_suffix_variantsz$stopword_affixer.get_suffix_variantszs'11  "4(( tBCCy-  \&11 I D.v6v> > >#%#6y#A#A #, ...r(ctj|}tj|}|tjrE|tjtjzr|ddtjz}nP|tjtj zr9|tj rtjtj z}n|tjtj zre|tj rF|tj tj zstjtj z}nZ|tjtj zr.|tj tjzrd}|}||fS)u Get the enclitic variant to be joined to the word. For example: word = عن, suffix = , encletic = ني. The word and enclitic are geminated. @param word: word found in dictionary. @type word: unicode. @param enclitic: first level suffix. @type enclitic: unicode. @return: variant of suffixes (vocalized suffix and vocalized suffix without I'rab short mark). @rtype: (unicode, unicode) Nr:) r,r>r= startswithr.r1SUKUNSHADDAKASRAr-FATHAr2r/)rArC word_semi_vocword_nmenclitic_non_irab_marks r'get_enclitic_variantz%stopword_affixer.get_enclitic_variantsr"+D11 #D))   rw ' ' M,B,B27RXCU,V,V |bi/HH  BF!2 3 3  8N8Nrv8V8V y28+HH  BF!2 3 3  8N8Nr8_8_ ##BF2?$:;; 09rx/  BF!2 3 3  8N8NrvXZXaOa8b8b H"*///r(ctj|}tj|}tj|}tj|}|tjtjztjzr)|tjr |dd}|tjtjztjztjzr*|tjr |dd}nU|tjtjzr)|tjr |dd}|dkr6|tjr|ddtj z}|tj rk|ri|tj tj zr|ddtjz}n|ddtj ztjz}n|tj r|r|tjr|ddtjz}nj|tjr|ddtjz}n3tj|vstj|vr|ddtjz}|r(|dtjvrtj|}|tjrA|tjtjzrtj|}nl|tjtj zr@|tj tjzrtj|}|S)uz Get the word variant to be joined to the suffix. For example: word = مدرسة, suffix = ي. The word is converted to مدرست. @param word: word found in dictionary. @type word: unicode. @param suffix: suffix ( firts or second level). @type suffix: unicode. @return: variant of word. @rtype: unicode. NrIr:r*r)r,r>r=rJr?LAMr1rK TEH_MARBUTATEHr2r/r-HAMZADAMMA WAW_HAMZArM YEH_HAMZAKAFBEHHARAKATr.)rA procliticrB word_stem word_stem_nmr4proc_nms r'get_word_variantz!stopword_affixer.get_word_variants>'-- (.. %f-- #I..   "& 026 9 : : &w?O?OPRPV?W?W &!!"" I   "& 028 ;bf D E E &'JZJZ[][aJbJb &!!"" II  ! !"'BF"2 3 3 &8H8H8P8P &!!"" I ??y11".AA?!#2#/I   bo . . :9 :$$RVBO%;<< ?%crcNRW4 %crcNRV3bh>    ) ) :i :  ** :%crcNR\9 ""28,, :%crcNR\9 7""bf&7&7%crcNR\9   7fQi2:--+I66I   RW % % 7$--"(8J*K*K 7+I66II   rx"&0 1 1 7dmm!7#7# 7+I66Ir(cg}|}|j|id|D]#}|j|id|D]}|||\}} |||||z}||||\}} d||| |g} d||||g} d||||g} tj| } | | } | | } | | | | f%|S)a Join the stop and its affixes, and get the vocalized form @param stop: stop found in dictionary. @type stop: unicode. @param proclitic: first level prefix. @type proclitic: unicode. @param suffix: second level suffix. @type suffix: unicode. @param enclitic: first level suffix. @type enclitic: unicode. @return: vocalized word. @rtype: unicode. vocalizedr:-) rgetrrRrdrGjoinr,r=ajust_vocalizationappend)r&stopr`rBrCword_tuple_list suffix_voc proclitic_voc enclitic_vocencl_voc_non_inflectrFword_non_irab_markword_vocalized segmenteds r'vocalizezstopword_affixer.vocalizes "155iDDHHV_``" X" XM $ 3 7 7" E E I I+W_ ` `! X! X 595N5NtO[6]6]2 2,,T=&8BSTT483K3K*l44440 0&(WW"D*> M&O&O""$-z<)X!Y!Y HHmT:|%TUU -i88 %)%<%<=O%P%P"!%!8!8!H!H&&8JI'VWWWWC! XDr(c$fd|DS)ad Verify possible affixes in the resulted segments according to the given affixes list. @param word: the input word. @type word: unicode. @param list_seg: list of word segments indexes (numbers). @type list_seg: list of pairs. @return: list of acceped segments. @rtype: list of pairs. c~g|]9}dd|d|ddgv7|:S)rgNrrI)ri).0s affix_listrAs r' z1stopword_affixer.verify_affix..GsW   xxeqted1Q455k233zAA AAAr()rAlist_segrzs` `r' verify_affixzstopword_affixer.verify_affix;s4         r(ct|tr|d}d|vr |dsdSd|vr |dsdSd|vr |drdSd|vr |d sdSd |vr |d sdSd |vr|d rd|vrdSd |vr |d sdSd |vr |d sdSd|vr |dsdSd|vr |drdS|tjkr |d sdSd|vr/|d s%|dtjrdSd|vr d|dvrdSd|vr |dsdSd|vr |dsdSd|vr |dsdSdS)a Test if the given word from dictionary is compabilbe with affixes tags. @param stop_tuple: the input word attributes given from dictionary. @type stop_tuple: dict. @param affix_tags: a list of tags given by affixes. @type affix_tags:list. @param procletic: first level prefix vocalized. @type procletic: unicode. @param encletic_nm: first level suffix vocalized. @type encletic_nm: unicode. @return: if the tags are compaatible. @rtype: Boolean. :uعطفhas_conjuctionF تعريفhas_definition is_definedu مجرور is_inflected مرفوعجرhas_prepositionr;مضاف has_pronounu وقايةrAuفعل type_worduاستفهام has_interroguقسم has_qasam تنوينtanwinT) isinstancestrsplitr,r-r1) stop_tuple affix_tags procletic encletic_nms r' validate_tagszstopword_affixer.validate_tagsLs$ j# & & /#))#..J  " ":6F+G "5 J & &z:J/K &5 J & &:l+C &5 J & &z./I &5 J & &z./I &5 j Z%? YcHcHc5 j 4E)F 5 J & &z./I &5 * $ $Z -F $5 * $ $L)A $5 "& N)C 5 J & &J~,F &*4V*<*E*Ebf*M*M '5 J & &8:k;R+R+R5  * *:n3M *5  " ":k+B "5 J & &z(/C &5tr(c<|j||}|S)z ajust vocalization Temporary function @param vocalized: vocalized word. @type vocalized: unicode. @return: ajusted vocalized word. @rtype: unicode. )r%rh)r&rfajusteds r'rjz#stopword_affixer.ajust_vocalizations!&**9i@@r(r:cg}|||rq||||rZ|s|||||}|||||}|r&d|D}|D]}|||S)z generate stopword form c,g|]}t|Sr|)r)rxxs r'r{z-stopword_affixer.get_form..s#B#B#BDGG#B#B#Br()is_valid_cliticscheck_clitic_affixget_tagsrurk) r&rAprocprefsuffencr< newword_list word_tuples r'get_formzstopword_affixer.get_forms  s + + 0&&tS$77 0@==tT3??D#}}T4sCC 0#B#B\#B#B#BL&200 "))$////r(Nc Pg}tj}|s|j|d}|kr!t |dkrfd|D}|D]}|d}t j|j|j |j D]|}|d}|d} |d} | ||| | } ||| || r1|||d | | | }|r||}d |D}|S) z generate all possible affixes We can genearte stopwords based on stop_tuple_list preprared to generate all forms according to a csv file. If stop_tuple_list = None: the current library use arabicstopwords library. T)lemmarIcfg|]-}tj|d+|.S)rf)r, vocalizedlikerh)rxsprAs r'r{z3stopword_affixer.generate_forms..sOUUUb"$"2266+t3L3Ld"S"SUrUUUr(rfrrTFrg r:cg|]}||Sr|r|)rxrs r'r{z3stopword_affixer.generate_forms..s<<<<"<<>N$,T_dmT^\\" <" <qzqzaj }}^T4EE] %%j$cBB<$(==r4QTVZ#[#[L#<&--l;;;E" .s++++1+++r(r)rrhrrrri) r&rArrBrCtaglistproclitic_tags enclitic_tags suffix_tagss r'rzstopword_affixer.get_tagss -11)R@@DDVRPP~&&&+//"==AA&"MM }%%%(,,VR88<.!s'LLLB-ad33LLLr(cg|] }|d Srr|rs r'r{z8stopword_affixer.generate_affix_list..#s999QAaD999r(c<g|]}|dS)rg)replace)rxrrAs r'r{z8stopword_affixer.generate_affix_list..'s'CCC $,,CCCr()rrr+)r&rfr list_affixesrAs @r'generate_affix_listz$stopword_affixer.generate_affix_lists,,T22 :LL^LLLLL99.999LC --.. CCCClCCC r(c||j|idd}|j|idd}|j|idd}||j|iddz }d|vrd|vrdSd|vrd|vrdSd|vrd|vrdSd|vrd |vrdSd |vrd |vrdSd |vrd |vrdS|t jr!|t jrdSd S)a Verify if proaffixes (sytaxic affixes) are compatable with affixes ( conjugation) @param proclitic_nm: first level prefix. @type proclitic_nm: unicode. @param enclitic: first level suffix. @type enclitic: unicode. @param suffix: second level suffix. @type suffix: unicode. @return: compatible. @rtype: True/False. r<r|casesrrFu إضافةru لايضافrru منصوبT) rrhrrrJr,r-r1rZ)r& proclitic_nmrCrBrrrs r'rz#stopword_affixer.check_clitic_affix+s -11,CCGGPRSS+//"==AA&"MM (,,VR88< / / /D"0"0\"0H88\8t===~  \ DD\DL        @@@@D!!!*13&"BBBBBr(r)__doc__resyspprintpathrkrpyarabic.arabyarabyr,$arabicstopwords.stopwords_classifiedarabicstopwords!arabicstopwords.stopwords_lexiconrr aly_stem_stopword_constr alyahmor.basic_affixer alyahmor.aly_stem_stopword_constrr|r(r'rs ++++??????3)))))322222222222222H H H H H }2H H H H H s AA