129 lines
22 KiB
Plaintext
129 lines
22 KiB
Plaintext
|
Ë
|
|||
|
|
>û gˆ= ã óÌ — d dl mZ d dlmZ d dlmZmZmZmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z m Z d dl
|
|||
|
|
Zd dlZd dlmZmZmZmZ d dlmZmZ ddlmZ dd lmZ dd
|
|||
|
|
l m Z dd l m Z dd l m!Z!m"Z"m#Z# dd
|
|||
|
|
l$m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+ ddl,m-Z- dZ.dZ/ e« ja e/« d Z1 e jd ddgg e1dddddd id œd d i¬ « d!e d"e3ded#e
|
|||
|
|
e3 d$e4d%e5d&e4d'e
|
|||
|
|
e fd(„« Z6 G d)„ d*e-« Z7y)+é )ÚCounter)Úislice)ÚAnyÚCallableÚDictÚIterableÚListÚOptionalÚTupleÚcastN)ÚConfigÚModelÚNumpyOpsÚ SequenceCategoricalCrossentropy)ÚFloats2dÚInts2dé )Úutil)ÚErrors)ÚLanguage)ÚDoc)ÚExampleÚvalidate_examplesÚvalidate_get_examples)ÚVocabé )Ú EditTrees)Úvalidate_edit_tree)Úlemmatizer_score)Ú
|
|||
|
|
TrainablePipeé zá
|
|||
|
|
[model]
|
|||
|
|
@architectures = "spacy.Tagger.v2"
|
|||
|
|
|
|||
|
|
[model.tok2vec]
|
|||
|
|
@architectures = "spacy.HashEmbedCNN.v2"
|
|||
|
|
pretrained_vectors = null
|
|||
|
|
width = 96
|
|||
|
|
depth = 4
|
|||
|
|
embed_size = 2000
|
|||
|
|
window_size = 1
|
|||
|
|
maxout_pieces = 3
|
|||
|
|
subword_features = true
|
|||
|
|
ÚmodelÚtrainable_lemmatizerz token.lemmaÚorthé Fz@scorerszspacy.lemmatizer_scorer.v1)r" ÚbackoffÚ
|
|||
|
|
min_tree_freqÚ overwriteÚtop_kÚscorerÚ lemma_accç ð?)ÚassignsÚrequiresÚdefault_configÚdefault_score_weightsÚnlpÚnamer& r' r( r) r* c
|
|||
|
|
ó< — t | j |||||||¬« S )z*Construct an EditTreeLemmatizer component.©r& r' r( r) r* )ÚEditTreeLemmatizerÚvocab)r1 r2 r" r& r' r( r) r* s údC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/pipeline/edit_tree_lemmatizer.pyÚmake_edit_tree_lemmatizerr8 * s- € ô2 Ø � ‰ Ø
|
|||
|
|
Ø ØØ#ØØØô ð ó c óÎ — e Zd ZdZ d,ddddedœdeded ed
|
|||
|
|
ee d e d e
|
|||
|
|
d
|
|||
|
|
e dee fd„Z de
|
|||
|
|
e dee deeee f fd„Zde
|
|||
|
|
e dee fd„Zd„ Zd„ Zd„ Zde
|
|||
|
|
e fd„Zedee df fd„« Z ede
|
|||
|
|
fd „« Z ede fd „« Z d d d œd e g e
|
|||
|
|
e f d!ee d"ee fd#„Z! e"« d$œd%„Z# e"« d$œd&„Z$ e"« fd'„Z% e"« fd(„Z&d"e fd)„Z'd e g e
|
|||
|
|
e f fd*„Z(d-d+„Z)y ).r5 zK
|
|||
|
|
Lemmatizer that lemmatizes each word using a predicted edit tree.
|
|||
|
|
r$ r% Fr r4 r6 r" r2 r& r' r( r) r* c óú — || _ || _ || _ || _ || _ || _ || _ t | j j « | _ i | _
|
|||
|
|
dg i| _ || _ t « | _ y)aí
|
|||
|
|
Construct an edit tree lemmatizer.
|
|||
|
|
|
|||
|
|
backoff (Optional[str]): backoff to use when the predicted edit trees
|
|||
|
|
are not applicable. Must be an attribute of Token or None (leave the
|
|||
|
|
lemma unset).
|
|||
|
|
min_tree_freq (int): prune trees that are applied less than this
|
|||
|
|
frequency in the training data.
|
|||
|
|
overwrite (bool): overwrite existing lemma annotations.
|
|||
|
|
top_k (int): try to apply at most the k most probable edit trees.
|
|||
|
|
ÚlabelsN)r6 r" r2 r& r' r( r) r ÚstringsÚtreesÚ
|
|||
|
|
tree2labelÚcfgr* r Ú numpy_ops) Úselfr6 r" r2 r& r' r( r) r* s r7 Ú__init__zEditTreeLemmatizer.__init__T sn € ð. ˆŒ
|
|||
|
|
؈Œ
|
|||
|
|
؈Œ Ø ˆŒ Ø *ˆÔØ"ˆŒØˆŒ
|
|||
|
|
ä ˜tŸz™z× 1Ñ 1Ó2ˆŒ
|
|||
|
|
Ø*,ˆŒà$,¨b >ˆŒØ ˆŒ Ü!›ˆ�r9 ÚexamplesÚscoresÚreturnc óz — t |d« t dd¬« }g }|D ]¡ }g }t |j |j dd¬« « D ]b \ }}|�|dk( rd} nB| j
|
|||
|
|
j
|
|||
|
|
|j |« }
|
|||
|
|
| j j |
|
|||
|
|
d « } |j | « Œd |j |« Œ£ |||« \ } } | j j j j | « r3t t j" j% | j& ¬
|
|||
|
|
« « ‚t) | « | fS ) NzEditTreeLemmatizer.get_lossFéÿÿÿÿ)Ú normalizeÚ
|
|||
|
|
missing_valueÚLEMMAT)Ú as_stringÚ r ©r2 )r r ÚzipÚ predictedÚ get_alignedr> ÚaddÚtextr? ÚgetÚappendr" ÚopsÚxpÚisnanÚ
|
|||
|
|
ValueErrorr ÚE910Úformatr2 Úfloat)
|
|||
|
|
rB rD rE Ú loss_funcÚtruthsÚegÚ eg_truthsrP Ú
|
|||
|
|
gold_lemmaÚlabelÚtree_idÚd_scoresÚlosss
|
|||
|
|
r7 Úget_losszEditTreeLemmatizer.get_lossz s € ô ˜(Ð$AÔBÜ3¸eÐSUÔVˆ àˆÛˆB؈IÜ+.Ø— ‘ ˜bŸn™n¨WÀ˜nÓ Eö,Ñ'�˜Jð Ð%¨°rÒ)9Ø ‘Eà "Ÿj™jŸn™n¨Y¯^©^¸ZÓ H�GØ ŸO™O× /Ñ /°¸Ó ;�EØ× Ñ Õ'ð,ð
|
|||
|
|
�M‰M˜)Õ $ð ñ # 6¨6Ó2‰ˆ�$Ø �:‰:�>‰>× Ñ × "Ñ " 4Ô (Ü œVŸ[™[× /Ñ /°T·Y±YÐ /Ó ?Ó@Ð @ä�T‹{˜HÐ$Ð$r9 Údocsc ó. — | j dk( r
|
|||
|
|
| j }n,| j t k r
|
|||
|
|
| j }n | j }t t
|
|||
|
|
|« « }t d„ |D « « s]t | j d « }|D �cg c]( }| j j j d|« ‘Œ* }}t |« |k( sJ ‚|S | j j |« }t |« |k( sJ ‚ |||« }t |« |k( sJ ‚|S c c}w )Nr c 3 ó2 K — | ] }t |« –— Œ y w©N)Úlen)Ú.0Údocs r7 ú <genexpr>z-EditTreeLemmatizer.predict.<locals>.<genexpr>£ s è ø€ Ð,¡t ”3�s—8¡tùs ‚r< r )
|
|||
|
|
r) Ú _scores2guesses_top_k_equals_1ÚTOP_K_GUARDRAILÚ _scores2guesses_top_k_greater_1Ú _scores2guesses_top_k_guardrailrk ÚlistÚanyr@ r" rV Úalloc2iÚpredict)rB rg Úscores2guessesÚn_docsÚn_labelsÚ_ÚguessesrE s r7 rv zEditTreeLemmatizer.predict• sû € Ø �:‰:˜Š?Ø !× @Ñ @‰NØ
|
|||
|
|
�Z‰Zœ?Ò
|
|||
|
|
*Ø !× AÑ A‰Nà !× AÑ AˆNô ”T˜$“Z“ ˆÜÑ,¡tÓ,Ô,ä˜4Ÿ8™8 HÑ-Ó.ˆHÙRVÓ$WÑRVÈQ T§Z¡Z§^¡^×%;Ñ%;¸A¸xÕ%HÐRVˆGÐ$WÜ�w“< 6Ò)Ð )Ð)؈NØ—‘×#Ñ# DÓ)ˆÜ�6‹{˜fÒ$Ð$Ð$Ù vÓ.ˆÜ�7‹|˜vÒ%Ð%Ð%؈ùò %Xs  -Dc óº — g }t ||« D ]É \ }}|j d¬« }| j j |« }g }t |« D ]c \ }} | j
|
|||
|
|
d || }
|
|||
|
|
| j j |
|
|||
|
|
| j « �|j |
|
|||
|
|
« ŒS|j d« Œe |j t j |« « ŒË |S )Nr )Úaxisr< rH ) rO ÚargmaxrA ÚasarrayÚ enumerater@ r> ÚapplyrS rU ÚnpÚarray) rB rg rE r{ rm Ú
|
|||
|
|
doc_scoresÚ doc_guessesÚdoc_compat_guessesÚiÚtokenrc s r7 ro z1EditTreeLemmatizer._scores2guesses_top_k_equals_1¯ sÈ € ØˆÜ " 4¨Ö 0‰OˆC�Ø$×+Ñ+°Ð+Ó3ˆKØ Ÿ.™.×0Ñ0°Ó=ˆKà!#Ð Ü % cžN‘��5Ø Ÿ(™( 8Ñ,¨[¸©^Ñ<�Ø—:‘:×#Ñ# G¨U¯Z©ZÓ8ÐDØ&×-Ñ-¨gÕ6à&×-Ñ-¨bÕ1ð +ð
|
|||
|
|
�N‰Nœ2Ÿ8™8Ð$6Ó7Õ 8ð 1ð ˆr9 c ó¤ — g }t | j t | j « « }t ||« D �] \ }}| j
|
|||
|
|
j
|
|||
|
|
|« }g }t |« D ]¿ \ }} t |« D ]› }
|
|||
|
|
t || j « « } | j d | } | j j | | j « �|j | « Œ{t! j" t j$ « j ||| f< Œ� |j d« ŒÁ |j t! j& |« « �Œ |S )Nr< rH )Úminr) rk r< rO rA r r€ ÚrangeÚintr~ r@ r> r� rS rU r‚ ÚfinfoÚfloat32rƒ )
|
|||
|
|
rB rg rE r{ r) rm r„ r† r‡ rˆ rz Ú candidateÚcandidate_tree_ids
|
|||
|
|
r7 rq z2EditTreeLemmatizer._scores2guesses_top_k_greater_1À s € ؈Ü�D—J‘J¤ D§K¡KÓ 0Ó1ˆÜ " 4¨× 0‰OˆC�Ø Ÿ ™ ×/Ñ/°
|
|||
|
|
Ó;ˆJØ!#Ð Ü % cžN‘��5Ü ˜už �AÜ # J¨q¡M×$8Ñ$8Ó$:Ó ;�IØ(,¯©°Ñ(:¸9Ñ(EÐ%Ø—z‘z×'Ñ'Ð(9¸5¿:¹:ÓFÐRØ*×1Ñ1Ð2CÔDÙ Ü/1¯x©x¼¿
|
|||
|
|
¹
|
|||
|
|
Ó/C×/GÑ/G�J˜q )˜|Ò,ð
|
|||
|
|
&ð '×-Ñ-¨bÕ1ð +ð
|
|||
|
|
�N‰Nœ2Ÿ8™8Ð$6Ó7Ö 8ð 1ð ˆr9 c óÞ — g }t ||« D ]Û \ }}t j |« dd | j dz
|
|||
|
|
d…f }| j j |« }g }t ||« D ][ \ }} d}
|
|||
|
|
| D ]> } | j d | } | j j | |j « €Œ<| }
|
|||
|
|
n |j |
|
|||
|
|
« Œ] |j t j |« « ŒÝ |S )N.r rH r< ) rO r‚ Úargsortr) rA r r@ r> r� rS rU rƒ )
|
|||
|
|
rB rg rE r{ rm r„ r… r† rˆ Ú
|
|||
|
|
candidatesrc r� r� s
|
|||
|
|
r7 rr z2EditTreeLemmatizer._scores2guesses_top_k_guardrailÔ sì € ØˆÜ " 4¨Ö 0‰OˆC�Ü Ÿ*™* ZÓ0°Ð6L¸¿¹¸ Àa¹È"Ð6LÐ1LÑMˆKØ Ÿ.™.×0Ñ0°Ó=ˆKà!#Ð Ü%(¨¨kÖ%:Ñ!��zØ �Û!+�IØ(,¯©°Ñ(:¸9Ñ(EÐ%à—z‘z×'Ñ'Ð(9¸5¿:¹:ÓFÑRØ"3˜Ù ð ",ð #×)Ñ)¨'Õ2ð &;ð
|
|||
|
|
�N‰Nœ2Ÿ8™8Ð$6Ó7Õ 8ð 1ð" ˆr9 c óž — t |« D ]¿ \ }}|| }t |d« r|j « }t |« D ]‹ \ }}| j s|| j dk( sŒ%|dk( r/| j
|
|||
|
|
€Œ7t
|
|||
|
|
|| | j
|
|||
|
|
« || _ ŒY| j j ||| j « }||| _
|
|||
|
|
Œ� ŒÁ y )NrT r rH ) r€ ÚhasattrrT r( Úlemmar& Úgetattrr> r� rS Úlemma_) rB rg Úbatch_tree_idsr‡ rm Ú doc_tree_idsÚjrc r– s r7 Úset_annotationsz"EditTreeLemmatizer.set_annotationsé sµ € Ü –o‰FˆAˆsØ)¨!Ñ,ˆLÜ�| UÔ+Ø +× /Ñ /Ó 1� Ü '¨ Ö 5‘
|
|||
|
|
��7Ø—>’> S¨¡V§\¡\°QÓ%6ð "’}Ø Ÿ<™<Ñ3Ü+2°3°q±6¸4¿<¹<Ó+H˜C ™F�Là $§
|
|||
|
|
¡
|
|||
|
|
× 0Ñ 0°¸#¸a¹&¿+¹+Ó F˜Ø(-˜˜A™�
|
|||
|
|
ñ 6ñ &r9 .c ó2 — t | j d « S )z4Returns the labels currently added to the component.r< )Útupler@ ©rB s r7 r< zEditTreeLemmatizer.labelsú s € ô �T—X‘X˜hÑ'Ó(Ð(r9 c ó — y)NT© rŸ s r7 Ú hide_labelsz EditTreeLemmatizer.hide_labelsÿ s € àr9 c ó^ — g }t t | j « « D ]h }| j | }d|v r | j j |d |d< d|v r | j j |d |d< |j |« Œj t
|
|||
|
|
|t | j d « ¬« S )NÚorigÚsubstr< )r> r< ) r‹ rk r> r6 r= rU Údictrž r@ )rB r> rc Útrees r7 Ú
|
|||
|
|
label_dataz EditTreeLemmatizer.label_data s› € àˆÜ œS §¡›_Ö-ˆGØ—:‘:˜gÑ&ˆDؘ‰~Ø #Ÿz™z× 1Ñ 1°$°v±,Ñ ?��V‘ ؘ$‰Ø $§
|
|||
|
|
¡
|
|||
|
|
× 2Ñ 2°4¸±=Ñ A��W‘
|
|||
|
|
Ø �L‰L˜Õ ð
|
|||
|
|
.ô ˜%¬¨d¯h©h°xÑ.@Ó(AÔBÐBr9 N)r1 r< Ú get_examplesr1 r< c ó° — t |d« |€| j |« n| j |« g }g }t |« d« D ]á }|j |j
|
|||
|
|
« g }|j D ]m }|j dk( rd } n&| j |j |j « } |j | j d D �
|
|||
|
|
cg c] }
|
|||
|
|
|
|
|||
|
|
| k( rdnd‘Œ
|
|||
|
|
c}
|
|||
|
|
« Œo t t |« }|j | j j j! |d¬« « Œã | j# « t% |« dkD s/J t& j( j+ | j, ¬ « « ‚t% |« dkD s/J t& j( j+ | j, ¬ « « ‚| j j/ ||¬
|
|||
|
|
« y c c}
|
|||
|
|
w ) Nz EditTreeLemmatizer.initializeé
|
|||
|
|
r r< r, g rŽ )ÚdtyperN )ÚXÚY)r Ú_labels_from_dataÚ _add_labelsr rU ÚxÚ referencer– Ú _pair2labelrS r˜ r@ r r r" rV r Ú_require_labelsrk r ÚE923r[ r2 Ú
|
|||
|
|
initialize) rB r© r1 r< Ú
|
|||
|
|
doc_sampleÚ label_sampleÚexampleÚ gold_labelsrˆ Ú
|
|||
|
|
gold_labelrb s r7 r¶ z EditTreeLemmatizer.initialize sš € ô ˜lÐ,KÔLà ˆ>Ø × "Ñ " <Õ 0à × Ñ ˜VÔ $ð ˆ
|
|||
|
|
؈ Ü ™l›n¨bÖ1ˆGØ × Ñ ˜gŸi™iÔ (Ø-/ˆKØ ×*Ô*�Ø—;‘; !Ò#Ø!%‘Jà!%×!1Ñ!1°%·*±*¸e¿l¹lÓ!K�Jà×"Ñ"ð &*§X¡X¨hÒ%7óá%7˜Eð %¨
|
|||
|
|
Ò 2™¸Ñ;Ø%7ñõð
|
|||
|
|
+ô œx¨Ó5ˆKØ × Ñ §
|
|||
|
|
¡
|
|||
|
|
§¡× 6Ñ 6°{È)Ð 6Ó TÕ Uð# 2ð&
|
|||
|
|
× Ñ Ô Ü�:‹ Ò"ÐF¤F§K¡K×$6Ñ$6¸D¿I¹IÐ$6Ó$FÓFÐ"Ü�<Ó 1Ò$ÐH¤f§k¡k×&8Ñ&8¸d¿i¹iÐ&8Ó&HÓHÐ$à �
|
|||
|
|
‰
|
|||
|
|
× Ñ
|
|||
|
|
¨lÐ Õ;ùòs Ã
|