142 lines
16 KiB
Plaintext
142 lines
16 KiB
Plaintext
|
Ë
|
|||
|
|
>û gâ0 ã ó° — d dl Z d dlmZ d dlmZmZmZmZmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z d dl m
|
|||
|
|
Z
|
|||
|
|
ddlmZ ddlmZmZ ddlmZ dd lmZmZ dd
|
|||
|
|
lmZ dd lmZm Z dd l m Z dd
|
|||
|
|
lm Z m Z m!Z! ddl"m#Z# ddl$m%Z% ejL ddgdddddidœddi¬« ded e e
|
|||
|
|
d e'd e'd e(d e e f d!„« Z)d"ee d#ee'ef fd$„Z* e!jV d« d%„ « Z, G d&„ d'e%« Z-y)(é N)ÚPath)ÚAnyÚCallableÚDictÚIterableÚListÚOptionalÚTupleÚUnion)ÚModelé )Úutil)ÚErrorsÚWarnings)ÚLanguage)ÚLookupsÚ load_lookups)ÚScorer)ÚDocÚToken)ÚExample)ÚSimpleFrozenListÚloggerÚregistry)ÚVocabé )ÚPipeÚ
|
|||
|
|
lemmatizerz token.lemmaÚlookupFz@scorerszspacy.lemmatizer_scorer.v1)ÚmodelÚmodeÚ overwriteÚscorerÚ lemma_accg ð?)ÚassignsÚdefault_configÚdefault_score_weightsÚnlpr Únamer! r" r# c ó8 — t | j |||||¬« S )N©r! r" r# )Ú
|
|||
|
|
LemmatizerÚvocab)r( r r) r! r" r# s úZC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/pipeline/lemmatizer.pyÚmake_lemmatizerr/ s" € ô& Ø � ‰ �5˜$ T°YÀvô ð ó ÚexamplesÚreturnc ó0 — t j | dfi |¤ŽS )NÚlemma)r Úscore_token_attr)r1 Úkwargss r. Úlemmatizer_scorer7 + s € Ü × "Ñ " 8¨WÑ ?¸Ñ ?Ð?r0 c ó — t S ©N)r7 © r0 r. Úmake_lemmatizer_scorerr; / s € ä Ðr0 c óø — e Zd ZdZededeee ee f fd„« Z d%dde dœde
|
|||
|
|
d e e d
|
|||
|
|
eded e
|
|||
|
|
d e e dd
|
|||
|
|
fd„Zed„ « Zdedefd„Z d&d
|
|||
|
|
d
|
|||
|
|
dœde eg ee f de e de e fd„Zej4 fdedd
|
|||
|
|
fd„Zde dee fd„Z de dee fd„Z de de
|
|||
|
|
fd „Z e « d œd e!ee"f d ee fd „Z# e « d œd e!ee"f d ee dd fd!„Z$ e « d œd ee de%fd"„Z& e « d œd#e%d ee dd fd$„Z'y
|
|||
|
|
)'r, z�
|
|||
|
|
The Lemmatizer supports simple part-of-speech-sensitive suffix rules and
|
|||
|
|
lookup tables.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer
|
|||
|
|
r! r2 c ó6 — |dk( rdgg fS |dk( rdgddgfS g g fS )a Returns the lookups configuration settings for a given mode for use
|
|||
|
|
in Lemmatizer.load_lookups.
|
|||
|
|
|
|||
|
|
mode (str): The lemmatizer mode.
|
|||
|
|
RETURNS (Tuple[List[str], List[str]]): The required and optional
|
|||
|
|
lookup tables for this mode.
|
|||
|
|
r Ú lemma_lookupÚruleÚ lemma_rulesÚ lemma_excÚ lemma_indexr: )Úclsr! s r. Úget_lookups_configz Lemmatizer.get_lookups_config< s< € ð �8Ò Ø#Ð$ bÐ)Ð )Ø
|
|||
|
|
�VŠ^Ø"�O k°=Ð%AÐBÐ BØ�Bˆxˆr0 r Fr+ r- r r) r" r# Nc óÀ — || _ || _ || _ || _ t « | _ || _ d| _ | j dk( r| j | _
|
|||
|
|
nv| j dk( r| j | _
|
|||
|
|
nU| j › d�}t | |« s)t t j j! |¬« « ‚t# | |« | _
|
|||
|
|
i | _ || _ y)a& Initialize a Lemmatizer.
|
|||
|
|
|
|||
|
|
vocab (Vocab): The vocab.
|
|||
|
|
model (Model): A model (not yet implemented).
|
|||
|
|
name (str): The component name. Defaults to "lemmatizer".
|
|||
|
|
mode (str): The lemmatizer mode: "lookup", "rule". Defaults to "lookup".
|
|||
|
|
overwrite (bool): Whether to overwrite existing lemmas. Defaults to
|
|||
|
|
`False`.
|
|||
|
|
scorer (Optional[Callable]): The scoring method. Defaults to
|
|||
|
|
Scorer.score_token_attr for the attribute "lemma".
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#init
|
|||
|
|
Fr r? Ú
|
|||
|
|
_lemmatize)r! N)r- r r) Ú_moder Úlookupsr" Ú
|
|||
|
|
_validatedr! Úlookup_lemmatizeÚ lemmatizeÚrule_lemmatizeÚhasattrÚ
|
|||
|
|
ValueErrorr ÚE1003ÚformatÚgetattrÚcacher# )Úselfr- r r) r! r" r# Ú mode_attrs r. Ú__init__zLemmatizer.__init__K s¼ € ð. ˆŒ
|
|||
|
|
؈Œ
|
|||
|
|
؈Œ ؈Œ
|
|||
|
|
Ü “yˆŒ Ø"ˆŒØ ˆŒØ �9‰9˜Ò Ø !× 2Ñ 2ˆD�NØ
|
|||
|
|
�Y‰Y˜&Ò
|
|||
|
|
Ø !× 0Ñ 0ˆD�Nà Ÿ9™9˜+ ZÐ0ˆIܘ4 Ô+Ü ¤§ ¡ ×!4Ñ!4¸$Ð!4Ó!?Ó@Ð@Ü $ T¨9Ó 5ˆDŒN؈Œ
|
|||
|
|
Ø ˆ� r0 c ó — | j S r9 )rG )rS s r. r! zLemmatizer.modeu s € à�z‰zÐr0 Údocc óN — | j s | j t j « | j « } |D ]7 }| j
|
|||
|
|
s|j dk( sŒ | j |« d |_ Œ9 |S # t $ r } || j | |g|« Y d}~yd}~ww xY w)z´Apply the lemmatizer to one document.
|
|||
|
|
|
|||
|
|
doc (Doc): The Doc to process.
|
|||
|
|
RETURNS (Doc): The processed Doc.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#call
|
|||
|
|
r N) rI Ú_validate_tablesr ÚE1004Úget_error_handlerr" r4 rK Úlemma_Ú Exceptionr) )rS rW Ú
|
|||
|
|
error_handlerÚtokenÚes r. Ú__call__zLemmatizer.__call__y s� € ð �ŠØ × !Ñ !¤&§,¡,Ô /Ø ×.Ñ.Ó0ˆ
|
|||
|
|
ð 5Û �Ø—>’> U§[¡[°AÓ%5Ø#'§>¡>°%Ó#8¸Ñ#;�E•Lð ð ˆJøÜò 5Ù ˜$Ÿ)™) T¨C¨5°!× 4Ñ 4ûð 5ús ½ A; Á A; Á; B$ÂB  B$)r( rH Ú get_examplesr( rH c ó¤ — | j | j « \ }}|€Št j d« t | j
|
|||
|
|
j |¬« }t | j
|
|||
|
|
j |d¬« }|j D ]# }|j ||j |« « Œ% || _
|
|||
|
|
| j t j « y)aÑ Initialize the lemmatizer and load in data.
|
|||
|
|
|
|||
|
|
get_examples (Callable[[], Iterable[Example]]): Function that
|
|||
|
|
returns a representative sample of gold-standard Example objects.
|
|||
|
|
nlp (Language): The current nlp object the component is part of.
|
|||
|
|
lookups (Lookups): The lookups object containing the (optional) tables
|
|||
|
|
such as "lemma_rules", "lemma_index", "lemma_exc" and
|
|||
|
|
"lemma_lookup". Defaults to None.
|
|||
|
|
Nz2Lemmatizer: loading tables from spacy-lookups-data)ÚlangÚtablesF)rd re Ústrict)rD r! r Údebugr r- rd re Ú set_tableÚ get_tablerH rY r rZ )rS rb r( rH Úrequired_tablesÚoptional_tablesÚoptional_lookupsÚtables r. Ú
|
|||
|
|
initializezLemmatizer.initializeŒ s¤ € ð ,0×+BÑ+BÀ4Ç9Á9Ó+MÑ(ˆ˜ Ø ˆ?Ü �L‰LÐMÔ NÜ"¨¯
|
|||
|
|
©
|
|||
|
|
¯©ÀÔPˆGÜ +Ø—Z‘Z—_‘_¨_ÀUô Ð ð *×0Ô0�Ø×!Ñ! %Ð)9×)CÑ)CÀEÓ)JÕKð 1à ˆŒ Ø × Ñ œfŸl™lÕ+r0 Ú
|
|||
|
|
error_messagec óî — | j | j « \ }}|D ]K }|| j vsŒt |j | j || j j
|
|||
|
|
¬« « ‚ d| _ y)z8Check that the lookups are correct for the current mode.)r! re ÚfoundTN)rD r! rH rN rP re rI )rS ro rj rk rm s r. rY zLemmatizer._validate_tables¨ sq € à+/×+BÑ+BÀ4Ç9Á9Ó+MÑ(ˆ˜ Û$ˆEؘDŸL™LÒ(Ü Ø!×(Ñ(Ø !ŸY™YØ .Ø "Ÿl™l× 1Ñ 1ð )ó óð ð %ð ˆ�r0 r_ c ó° — | j j di « }|j |j |j « }t |t
|
|||
|
|
« r|g}|S )zÞLemmatize using a lookup-based approach.
|
|||
|
|
|
|||
|
|
token (Token): The token to lemmatize.
|
|||
|
|
RETURNS (list): The available lemmas for the string.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#lookup_lemmatize
|
|||
|
|
r> )rH ri ÚgetÚtextÚ
|
|||
|
|
isinstanceÚstr)rS r_ Ú lookup_tableÚresults r. rJ zLemmatizer.lookup_lemmatize¶ sJ € ð —|‘|×-Ñ-¨n¸bÓAˆ Ø ×!Ñ! %§*¡*¨e¯j©jÓ9ˆÜ �fœcÔ "Ø �XˆF؈
|
|||
|
|
r0 c ót — |j |j |j j f}|| j v r| j | S |j
|
|||
|
|
}|j j « }|dv r9|dk( r#t j t j « |j « gS | j |« r|j « gS | j j di « }| j j di « }| j j di « }t |j! |« |j! |« |j! |« f« s|dk( r|gS |j « gS |j! |i « }|j! |i « } |j! |i « }
|
|||
|
|
|} |j « }g } g }
|
|||
|
|
|
|
|||
|
|
D ]n \ }}|j# |« sŒ|dt% |« t% |« z
|
|||
|
|
|z }|sŒ8||v s|j' « s| j) |« Œ^|
|
|||
|
|
j) |« Œp t+ t, j/ | « « } | j! |g « D ] }|| vsŒ| j1 d|« Œ | s| j3 |
|
|||
|
|
« | s| j) | « | | j |< | S ) zÚLemmatize using a rule-based approach.
|
|||
|
|
|
|||
|
|
token (Token): The token to lemmatize.
|
|||
|
|
RETURNS (list): The available lemmas for the string.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#rule_lemmatize
|
|||
|
|
)Ú ÚeolÚspacerz rB rA r@ ÚpropnNr )ÚorthÚposÚmorphÚkeyrR rt Úpos_ÚlowerÚwarningsÚwarnr ÚW108Ú is_base_formrH ri Úanyrs ÚendswithÚlenÚisalphaÚappendÚlistÚdictÚfromkeysÚinsertÚextend)rS r_ Ú cache_keyÚstringÚuniv_posÚ index_tableÚ exc_tableÚ rules_tableÚindexÚ
|
|||
|
|
exceptionsÚrulesÚorigÚformsÚ oov_formsÚoldÚnewÚforms r. rL zLemmatizer.rule_lemmatizeÄ s\ € ð —Z‘Z §¡¨E¯K©K¯O©OÐ<ˆ Ø ˜Ÿ
|
|||
|
|
™
|
|||
|
|
Ñ "Ø—:‘:˜iÑ(Ð (Ø—‘ˆØ—:‘:×#Ñ#Ó%ˆØ Ð+Ñ +ؘ2Š~Ü—
|
|||
|
|
‘
|
|||
|
|
œhŸm™mÔ,Ø—L‘L“NÐ#Ð #à × Ñ ˜UÔ #Ø—L‘L“NÐ#Ð #Ø—l‘l×,Ñ,¨]¸BÓ?ˆ Ø—L‘L×*Ñ*¨;¸Ó;ˆ Ø—l‘l×,Ñ,¨]¸BÓ?ˆ Üà—‘ Ó)Ø—
|
|||
|
|
‘
|
|||
|
|
˜hÓ'Ø—‘ Ó)ð
|
|||
|
|
ô
|
|||
|
|
ð ˜7Ò"Ø �x�à Ÿ ™ ›Ð'Ð'à—‘ ¨"Ó-ˆØ —]‘] 8¨RÓ0ˆ
|
|||
|
|
Ø—‘ ¨"Ó-ˆØˆØ— ‘ “ ˆØˆØˆ Û ‰HˆC�Ø�‰˜sÕ#Ø Ð 6¤ F£ ¬c°#«hÑ 6Ð7¸#Ñ=�ÙØؘU‘]¨$¯,©,¬.Ø—L‘L Õ&à ×$Ñ$ TÕ*ð ô ”T—]‘] 5Ó)Ó*ˆð
|
|||
|
|
—N‘N 6¨2Ö.ˆDؘ5Ò Ø— ‘ ˜Q Õ%ð /ñ Ø �L‰L˜Ô #ÙØ �L‰L˜Ô Ø %ˆ�
|
|||
|
|
‰
|
|||
|
|
�9Ñ Øˆ r0 c ó — y)a Check whether the token is a base form that does not need further
|
|||
|
|
analysis for lemmatization.
|
|||
|
|
|
|||
|
|
token (Token): The token.
|
|||
|
|
RETURNS (bool): Whether the token is a base form.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#is_base_form
|
|||
|
|
Fr: )rS r_ s r. r‡ zLemmatizer.is_base_form s € ð r0 ©ÚexcludeÚpathr£ c ó\ ‡ ‡— i }ˆˆ fd„|d< ˆ fd„|d< t j ||‰« y)zÞSerialize the pipe to disk.
|
|||
|
|
|
|||
|
|
path (str / Path): Path to a directory.
|
|||
|
|
exclude (Iterable[str]): String names of serialization fields to exclude.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#to_disk
|
|||
|
|
c ó> •— ‰j j | ‰¬« S ©Nr¢ )r- Úto_disk©Úpr£ rS s €€r. ú<lambda>z$Lemmatizer.to_disk.<locals>.<lambda> s ø€ t§z¡z×'9Ñ'9¸!ÀWÐ'9Ô'Mr0 r- c ó: •— ‰j j | « S r9 )rH r¨ ©rª rS s €r. r« z$Lemmatizer.to_disk.<locals>.<lambda> s ø€ ¨¯ © ×)=Ñ)=¸aÔ)@r0 rH N)r r¨ )rS r¤ r£ Ú serializes ` ` r. r¨ zLemmatizer.to_disk s. ù€ ð ˆ Ü Mˆ �'ÑÛ @ˆ �)Ñ Ü � ‰ �T˜9 gÕ.r0 c ó~ ‡ ‡— i }ˆˆ fd„|d< ˆ fd„|d< t j ||‰« ‰ j « ‰ S )aH Load the pipe from disk. Modifies the object in place and returns it.
|
|||
|
|
|
|||
|
|
path (str / Path): Path to a directory.
|
|||
|
|
exclude (Iterable[str]): String names of serialization fields to exclude.
|
|||
|
|
RETURNS (Lemmatizer): The modified Lemmatizer object.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#from_disk
|
|||
|
|
c ó> •— ‰j j | ‰¬« S r§ )r- Ú from_diskr© s €€r. r« z&Lemmatizer.from_disk.<locals>.<lambda>- s ø€ ¨¯©×)=Ñ)=¸aÈÐ)=Ô)Qr0 r- c ó: •— ‰j j | « S r9 )rH r± r s €r. r« z&Lemmatizer.from_disk.<locals>.<lambda>. s ø€ ¨4¯<©<×+AÑ+AÀ!Ô+Dr0 rH )r r± rY )rS r¤ r£ Ú deserializes ` ` r. r± zLemmatizer.from_disk! s? ù€ ð 8:ˆ Ü Qˆ �GÑ Û!Dˆ �IÑ Ü �‰�t˜[¨'Ô2Ø × Ñ Ô Øˆ r0 c óz ‡ ‡— i }ˆˆ fd„|d< ‰ j j |d< t j |‰« S )zçSerialize the pipe to a bytestring.
|
|||
|
|
|
|||
|
|
exclude (Iterable[str]): String names of serialization fields to exclude.
|
|||
|
|
RETURNS (bytes): The serialized object.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/lemmatizer#to_bytes
|
|||
|
|
c ó< •— ‰j j ‰ ¬« S r§ )r- Úto_bytes)r£ rS s €€r. r« z%Lemmatizer.to_bytes.<locals>.<lambda>< s ø€ T§Z¡Z×%8Ñ%8ÀÐ%8Ô%Ir0 r- rH )rH r¶ r )rS r£ r® s `` r. r¶ zLemmatizer.to_bytes3 s9 ù€ ð ˆ Ü Iˆ �'ÑØ #Ÿ|™|× 4Ñ 4ˆ �)Ñ Ü�}‰}˜Y¨Ó0Ð0r0 Ú
|