87 lines
12 KiB
Plaintext
87 lines
12 KiB
Plaintext
|
Ë
|
|||
|
|
>û gû ã
|
|||
|
|
ó — d dl Z d dlmZmZ d dlZddlmZ ddlmZ ddl m
|
|||
|
|
Z
|
|||
|
|
ddl m Z ddl
|
|||
|
|
mZ e
|
|||
|
|
j d g d
|
|||
|
|
¢d ¬ « d
|
|||
|
|
edefd„« Z e
|
|||
|
|
j dg d¢d ¬ « d
|
|||
|
|
efd„« Z e
|
|||
|
|
j ddgd ¬ « d,d
|
|||
|
|
ededefd„« Z e
|
|||
|
|
j( ddddœd ¬« d d dœd e
|
|||
|
|
d ed ed efd „« Z G d!„ d"« Z e
|
|||
|
|
j( d#ddd$œd d%œ¬&« d e
|
|||
|
|
d ed'eeef d(efd)„« Z G d*„ d+« Zy)-é N)ÚAnyÚDicté )Úutil)ÚWarnings)ÚLanguage)ÚMatcher)ÚDocÚmerge_noun_chunks)ú token.depz token.tagz token.posT)ÚrequiresÚ retokenizesÚdocÚreturnc ó — | j d« s| S | j « 5 }| j D ]B }|j j |j j
|
|||
|
|
dœ}|j
|
|||
|
|
||¬« ŒD ddd« | S # 1 sw Y | S xY w)zËMerge noun chunks into a single token.
|
|||
|
|
|
|||
|
|
doc (Doc): The Doc object.
|
|||
|
|
RETURNS (Doc): The Doc object with merged noun chunks.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/pipeline-functions#merge_noun_chunks
|
|||
|
|
ÚDEP)ÚtagÚdep©ÚattrsN)Úhas_annotationÚ
|
|||
|
|
retokenizeÚ noun_chunksÚrootr r Úmerge)r Ú retokenizerÚnpr s úYC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/pipeline/functions.pyr r
|
|||
|
|
sw € ð × Ñ ˜eÔ $؈
|
|||
|
|
Ø � ‰ Ô ˜[Ø—/”/ˆBØ ŸG™GŸK™K°·±· ± Ñ<ˆEØ × Ñ ˜b¨Ð Õ .ñ "÷
|
|||
|
|
ð €J÷
|
|||
|
|
ð €Jús ¤ABÂB Úmerge_entities)zdoc.entsz
|
|||
|
|
token.ent_iobztoken.ent_typec ó — | j « 5 }| j D ]M }|j j |j j |j
|
|||
|
|
dœ}|j
|
|||
|
|
||¬« ŒO ddd« | S # 1 sw Y | S xY w)zÂMerge entities into a single token.
|
|||
|
|
|
|||
|
|
doc (Doc): The Doc object.
|
|||
|
|
RETURNS (Doc): The Doc object with merged entities.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/pipeline-functions#merge_entities
|
|||
|
|
)r r Úent_typer N)r Úentsr r r Úlabelr )r r Úentr s r r r # sj € ð
|
|||
|
|
|
|||
|
|
� ‰ Ô ˜[Ø—8”8ˆCØ ŸH™HŸL™L°·±· ± È3Ï9É9ÑUˆEØ × Ñ ˜c¨Ð Õ /ñ ÷
|
|||
|
|
ð €J÷
|
|||
|
|
ð €Jús ‘A A9Á9BÚmerge_subtokensr r# c ó\ — t | j « }|j d|ddœgg« || « }t j |D ���cg c] \ }}}| ||dz ‘Œ c}}}« }| j « 5 }|D ] } |j
|
|||
|
|
| « Œ ddd« | S c c}}}w # 1 sw Y | S xY w)zõMerge subtokens into a single token.
|
|||
|
|
|
|||
|
|
doc (Doc): The Doc object.
|
|||
|
|
label (str): The subtoken dependency label.
|
|||
|
|
RETURNS (Doc): The Doc object with merged subtokens.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/pipeline-functions#merge_subtokens
|
|||
|
|
ÚSUBTOKú+)r Úopé N)r ÚvocabÚaddr Ú filter_spansr r )
|
|||
|
|
r r# ÚmergerÚmatchesÚ_ÚstartÚendÚspansr Úspans
|
|||
|
|
r r% r% 7 s¥ € ô �S—Y‘YÓ
|
|||
|
|
€FØ
|
|||
|
|
‡J�Jˆx 5°Ñ4Ð5Ð6Ô7Ù�S‹k€GÜ × Ñ ÉÕ Qɹ
|
|||
|
|
¸¸5À#˜s 5¨3°©7Ò 3ÈÓ QÓ R€EØ � ‰ Ô ˜[ÛˆDØ × Ñ ˜dÕ #ñ ÷
|
|||
|
|
ð €Jùô Rß ð €Jús Á
|
|||
|
|
B Á6B!Â!B+Útoken_splitteré é
|
|||
|
|
©Ú
|
|||
|
|
min_lengthÚ split_length)Údefault_configr ÚnlpÚnamer9 r: c ó — t ||¬« S ©Nr8 )Ú
|
|||
|
|
TokenSplitter)r< r= r9 r: s r Úmake_token_splitterrA L s € ô J¸\Ô JÐJó c óz — e Zd Zddedefd„Zdedefd„Zdeee f fd„Z
|
|||
|
|
i fdeee f dd fd
|
|||
|
|
„Z d „ Z d „ Z
|
|||
|
|
d
|
|||
|
|
„ Zd„ Zy )r@ r9 r: c ó — || _ || _ y ©Nr8 )Úselfr9 r: s r Ú__init__zTokenSplitter.__init__X s € Ø$ˆŒØ (ˆÕrB r r c ó — | j dkD ré| j dkD rÚ|j « 5 }|D ]¹ }t |j « | j k\ sŒ&g }g }i }t dt |j « | j « D ]M }|j
|
|||
|
|
|j ||| j z « |j
|
|||
|
|
||| j z f« ŒO |j ||||« Œ» d d d « |S |S # 1 sw Y |S xY w)Nr )r9 r: r ÚlenÚtextÚrangeÚappendÚsplit)rF r r ÚtÚorthsÚheadsr Úis r Ú__call__zTokenSplitter.__call__\ sç € Ø �?‰?˜QÒ 4×#4Ñ#4°qÒ#8Ø— ‘ Ô! [Û �Aܘ1Ÿ6™6“{ d§o¡oÓ5Ø "˜Ø "˜Ø "˜Ü!& q¬#¨a¯f©f«+°t×7HÑ7HÖ!I˜AØ !ŸL™L¨¯©°°A¸×8IÑ8IÑ4IÐ)JÔ KØ !ŸL™L¨!¨Q°×1BÑ1BÑ-BÐ)CÕ Dð "Jð $×)Ñ)¨!¨U°E¸5ÕAñ ÷ "ð ˆ
|
|||
|
|
ˆsˆ
|
|||
|
|
÷ "ð ˆ
|
|||
|
|
ús ¯(C;ÁBC;Ã;Dc ó4 — | j | j dœS r? r8 ©rF s r Ú _get_configzTokenSplitter._get_configj s € à Ÿ/™/Ø × -Ñ -ñ
|
|||
|
|
ð
|
|||
|
|
rB ÚconfigNc ó` — |j dd« | _ |j dd« | _ y )Nr9 r r: )Úgetr9 r: )rF rV s r Ú _set_configzTokenSplitter._set_configp s' € Ø Ÿ*™* \°1Ó5ˆŒØ "ŸJ™J ~°qÓ 9ˆÕrB c ó> ‡ — dˆ fd„i}t j |g « S )NÚcfgc óJ •— t j ‰ j « « S rE )ÚsrslyÚ
|
|||
|
|
json_dumpsrU rT s €r ú<lambda>z(TokenSplitter.to_bytes.<locals>.<lambda>v s ø€ œ5×+Ñ+¨D×,<Ñ,<Ó,>Ô?rB ©r Úto_bytes©rF ÚkwargsÚ serializerss ` r ra zTokenSplitter.to_bytest s# ø€ à Ó?ð
|
|||
|
|
ˆ ô �}‰}˜[¨"Ó-Ð-rB c óD ‡ — dˆ fd„i}t j ||g « ‰ S )Nr[ c óL •— ‰j t j | « « S rE )rY r] Ú
|
|||
|
|
json_loads©ÚbrF s €r r_ z*TokenSplitter.from_bytes.<locals>.<lambda>| s ø€ ˜T× -Ñ -¬e×.>Ñ.>¸qÓ.AÔ BrB ©r Ú
|
|||
|
|
from_bytes©rF Údatarc Ú
|
|||
|
|
deserializerss ` r rk zTokenSplitter.from_bytesz s' ø€ à ÓBð
|
|||
|
|
ˆ
|
|||
|
|
ô
|
|||
|
|
�‰˜˜m¨RÔ0؈ rB c ój ‡ — t j |« }dˆ fd„i}t j ||g « S )Nr[ c óL •— t j | ‰j « « S rE )r] Ú
|
|||
|
|
write_jsonrU ©ÚprF s €r r_ z'TokenSplitter.to_disk.<locals>.<lambda>„ s ø€ œU× -Ñ -¨a°×1AÑ1AÓ1CÔ DrB ©r Ú ensure_pathÚto_disk©rF Úpathrc rd s ` r rv zTokenSplitter.to_disk� s5 ø€ Ü× Ñ Ó%ˆà ÓDð
|
|||
|
|
ˆ ô �|‰|˜D +¨rÓ2Ð2rB c ól ‡ — t j |« }dˆ fd„i}t j ||g « y )Nr[ c óL •— ‰j t j | « « S rE )rY r] Ú read_jsonrr s €r r_ z)TokenSplitter.from_disk.<locals>.<lambda>‹ s ø€ ˜T× -Ñ -¬e¯o©o¸aÓ.@Ô ArB ©r ru Ú from_diskrw s ` r r} zTokenSplitter.from_diskˆ s2 ø€ Ü× Ñ Ó%ˆà ÓAð
|
|||
|
|
ˆ ô
|
|||
|
|
�‰�t˜[¨"Õ-rB )r r )Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__ÚintrG r
|
|||
|
|
rR r Ústrr rU rY ra rk rv r} © rB r r@ r@ W sp „ ñ) 3ð )¸#ó )ð ˜Cð Có ð
|
|||
|
|
˜T # s (™^ó
|