112 lines
14 KiB
Plaintext
112 lines
14 KiB
Plaintext
|
Ë
|
|||
|
|
?û g)5 ã ó0 — d dl Z d dlZd dlmZ d dlmZmZmZmZm Z m
|
|||
|
|
Z
|
|||
|
|
m Z ddl m
|
|||
|
|
Z
|
|||
|
|
ddlmZ ddlmZmZ erdd lmZ e
|
|||
|
|
j* d
|
|||
|
|
« d ed ed
|
|||
|
|
e
|
|||
|
|
eee e f dedede
|
|||
|
|
e e dedegee f fd„« Zdddddddœddded ed ed
|
|||
|
|
e
|
|||
|
|
eee e f dedede
|
|||
|
|
e e dee fd„Z e
|
|||
|
|
j* d« deded
|
|||
|
|
eee e f dedegee f fd „« Z e
|
|||
|
|
j* d « dededegee f fd „« Zdddedee fd „Z dddededee fd „Z dddefd!„Z ddd"œddded
|
|||
|
|
eee e f dededee f d#„Z d$d%œddd&ed'eee e f d
|
|||
|
|
eee eee e f f de de eeee e f f f d(„Z!ddded)ed*e"def
|
|||
|
|
d+„Z#d,„ Z$y)-é N)Úpartial)Ú
|
|||
|
|
TYPE_CHECKINGÚCallableÚDictÚIteratorÚListÚOptionalÚTupleé )Úregistryé )ÚExample)Ú _doc_to_biluo_tags_with_partialÚsplit_bilu_label)ÚLanguagezspacy.combined_augmenter.v1Ú lower_levelÚ
|
|||
|
|
orth_levelÚ
|
|||
|
|
orth_variantsÚwhitespace_levelÚwhitespace_per_tokenÚwhitespace_variantsÚreturnr c ó. — t t | |||||¬« S )a\ Create a data augmentation callback that uses orth-variant replacement.
|
|||
|
|
The callback can be added to a corpus or other data iterator during training.
|
|||
|
|
|
|||
|
|
lower_level (float): The percentage of texts that will be lowercased.
|
|||
|
|
orth_level (float): The percentage of texts that will be augmented.
|
|||
|
|
orth_variants (Optional[Dict[str, List[Dict]]]): A dictionary containing the
|
|||
|
|
single and paired orth variants. Typically loaded from a JSON file.
|
|||
|
|
whitespace_level (float): The percentage of texts that will have whitespace
|
|||
|
|
tokens inserted.
|
|||
|
|
whitespace_per_token (float): The number of whitespace tokens to insert in
|
|||
|
|
the modified doc as a percentage of the doc length.
|
|||
|
|
whitespace_variants (Optional[List[str]]): The whitespace token texts.
|
|||
|
|
RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter.
|
|||
|
|
©r r r r r r )r Úcombined_augmenterr s úWC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/training/augment.pyÚcreate_combined_augmenterr s& € ô. ÜØ Ø Ø#Ø)Ø 1Ø /ô ð ó g r ÚnlpÚexamplec # ó’ K — t j « |k r t | |« }|rŠt j « |k rs|j }|j « } t |j
|
|||
|
|
« | d d< t
|
|||
|
|
| || d |d¬« \ }
|
|||
|
|
} | | d< |j | j |
|
|||
|
|
« | « }|rŽt j « |k rwt t t |j
|
|||
|
|
« |z « « D ]J } t | |t j |« t j dt |j
|
|||
|
|
« « « }ŒL |–— y w)NÚdoc_annotationÚentitiesÚtoken_annotationF©Úlowerr )ÚrandomÚmake_lowercase_variantÚtextÚto_dictr Ú referenceÚmake_orth_variantsÚ from_dictÚmake_docÚrangeÚintÚlenÚmake_whitespace_variantÚchoiceÚ randrange)
|
|||
|
|
r r r r r r r r Úraw_textÚ orig_dictÚ variant_textÚvariant_token_annotÚ_s
|
|||
|
|
r r r 0 s* è ø€ ô ‡}�}ƒ˜Ò$Ü(¨¨gÓ6ˆÙœŸ ™ › ¨:Ò5Ø—<‘<ˆØ—O‘OÓ%ˆ Ü2QØ × Ñ ó3
|
|||
|
|
ˆ Ð"Ñ# JÑ/ô -?Ø Ø Ø Ð(Ñ )Ø Øô -
|
|||
|
|
Ñ)ˆ Ð)ð )<ˆ Ð$Ñ%Ø×#Ñ# C§L¡L° Ó$>À ÓJˆÙœvŸ}™}›Ð1AÒ AÜ”sœ3˜w× 0Ñ 0Ó1Ð4HÑHÓIÖJˆAÜ-ØØÜ—
|
|||
|
|
‘
|
|||
|
|
Ð 1Ó2Ü× Ñ ¤C¨×(9Ñ(9Ó$:Ó;ó ‰Gð Kð ƒMùs ‚EEzspacy.orth_variants.v1Úlevelr& c ó( — t t || |¬« S )a Create a data augmentation callback that uses orth-variant replacement.
|
|||
|
|
The callback can be added to a corpus or other data iterator during training.
|
|||
|
|
|
|||
|
|
level (float): The percentage of texts that will be augmented.
|
|||
|
|
lower (float): The percentage of texts that will be lowercased.
|
|||
|
|
orth_variants (Dict[str, List[Dict]]): A dictionary containing
|
|||
|
|
the single and paired orth variants. Typically loaded from a JSON file.
|
|||
|
|
RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter.
|
|||
|
|
)r r: r& )r Úorth_variants_augmenter)r: r& r s r Ú create_orth_variants_augmenterr= W s € ô Ü ¨}ÀEÐQVô ð r zspacy.lower_case.v1c ó$ — t t | ¬« S )a3 Create a data augmentation callback that converts documents to lowercase.
|
|||
|
|
The callback can be added to a corpus or other data iterator during training.
|
|||
|
|
|
|||
|
|
level (float): The percentage of texts that will be augmented.
|
|||
|
|
RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter.
|
|||
|
|
©r: )r Úlower_casing_augmenterr? s r Ú create_lower_casing_augmenterrA i s € ô Ô)°Ô 7Ð7r c # ó K — |–— y w©N© )r r s r Ú dont_augmentrE v s è ø€ Ø
|
|||
|
|
ƒMùs ‚c # ó` K — t j « |k\ r|–— y t | |« –— y wrC )r' r( )r r r: s r r@ r@ z s( è ø€ ô ‡}�}ƒ˜%Ò Ø‹
|
|||
|
|
ä$ S¨'Ó2Ó2ùs ‚,.c ó, — |j « }t |j « |d d< | j |j j « « }|j D �cg c] }|j ‘Œ c}|d d< |j ||« S c c}w )Nr" r# r$ ÚORTH)r* r r+ r. r) r& Úlower_r- )r r Ú example_dictÚdocÚts r r( r( ƒ s� € Ø—?‘?Ó$€LÜ1PØ×Ñó2€LÐ!Ñ" :Ñ.ð �,‰,�w—|‘|×)Ñ)Ó+Ó
|
|||
|
|
,€CØBI×BSÒBSÓ/TÑBS¸Q°·³ÐBSÑ/T€LÐ#Ñ$ VÑ,Ø × Ñ ˜S ,Ó /Ð/ùò 0Us Á#B)r: r& c # ód K — t j « |k\ r|–— y |j }|j « }t |j « |d d< t | ||d ||d uxr t j « |k ¬« \ }}||d< |j
|
|||
|
|
| j |« |« –— y w)Nr" r# r$ r% )r' r) r* r r+ r, r- r. ) r r r r: r& r5 r6 r7 r8 s r r< r< � s³ è ø€ ô ‡}�}ƒ˜%Ò Ø‹
|
|||
|
|
à—<‘<ˆØ—O‘OÓ%ˆ Ü2QØ × Ñ ó3
|
|||
|
|
ˆ Ð"Ñ# JÑ/ô -?Ø Ø Ø Ð(Ñ )Ø Ø $Ð&ÒB¬6¯=©=«?¸UÑ+Bô -
|
|||
|
|
Ñ)ˆ Ð)ð )<ˆ Ð$Ñ%Ø× Ñ § ¡ ¨\Ó :¸IÓFÓFùs ‚B.B0Fr% ÚrawÚ
|
|||
|
|
token_dictc ób — |j dg « }|j dg « }|s||fS |r-|D �cg c] }|j « ‘Œ }}|j « }|s ||d< ||fS |j dg « }|D � cg c] } t j | d « ‘Œ }
|
|||
|
|
} t t |« « D ]? } t t |« « D ]& } || || d v sŒ|| || d v sŒ |
|
|||
|
|
| || < Œ( ŒA |j dg « }
|
|||
|
|
|
|
|||
|
|
D � cg c] } t j | d « ‘Œ }
|
|||
|
|
} t t |« « D ]Î } t t |
|
|||
|
|
« « D ]µ } || |
|
|||
|
|
| d v sŒ|| t j j |
|
|||
|
|
| d « v sŒ<t j ddg« }t |
|
|||
|
|
| d « d k( r|
|
|||
|
|
| d j || « }n)|
|
|||
|
|
| d D ] }|| |v sŒ |j || « }Œ |
|
|||
|
|
| | || < Œ· ŒÐ ||d< t |« }||fS c c}w c c} w c c} w )
|
|||
|
|
NrH ÚTAGÚsingleÚvariantsÚtagsÚpairedr r
|
|||
|
|
r ) Úgetr& r' r3 r/ r1 Ú itertoolsÚchainÚ
|
|||
|
|
from_iterableÚindexÚconstruct_modified_raw_text)r rN rO r r& ÚwordsrT ÚwÚndsvÚxÚ
|
|||
|
|
punct_choicesÚword_idxÚ punct_idxÚndpvÚpair_idxÚpairs r r, r, ¨ s] € ð
|
|||
|
|
�N‰N˜6 2Ó &€EØ �>‰>˜% Ó $€Dá Ø�JˆÐ Ù Ù$)Ó*¡E˜q�—‘• EˆÐ*Ø�i‰i‹kˆá Ø "ˆ
|
|||
|
|
�6ÑØ�JˆÐ à × Ñ ˜X rÓ *€DÙ;?Ó@¹4°a”V—]‘] 1 Z¡=Õ1¸4€MÐ@Üœ#˜e›*Ö%ˆÜ œs 4›yÖ)ˆIà�X‘ $ y¡/°&Ñ"9Ò9ؘ(‘O t¨I¡°zÑ'BÒBà"/° Ñ":��h’ñ *ð &ð × Ñ ˜X rÓ *€DÙ;?Ó@¹4°a”V—]‘] 1 Z¡=Õ1¸4€MÐ@Üœ#˜e›*Ö%ˆÜ œs 4›yÖ)ˆIØ�H‰~ i¡ °Ñ!8Ò8¸UØñ>ä— ‘ ×.Ñ.¨t°I©¸zÑ/JÓKò>Lô "Ÿ=™=¨!¨Q¨Ó0�ä�t˜I‘ vÑ.Ó/°1Ò4Ø # I™¨vÑ 6× <Ñ <¸TÀ(¹^Ó L‘Hð !% Y¡°
|
|||
|
|
Ô ;˜Ø ™?¨dÒ2Ø'+§z¡z°%¸±/Ó'B™Hð !<ð #0° Ñ":¸8Ñ"D��h’ñ *ð &ð" €JˆvÑÜ
|
|||
|
|
% jÓ
|
|||
|
|
1€CØ �
|
|||
|
|
ˆ?ÐùòM +ùò Aùò As ±H"Á; H'Ä H,Ú
|
|||
|
|
whitespaceÚpositionc ó< — |j « }t |j « |d d< |j di « }|j di « }t |j « dk( s|d|vsxt |j dg « « dkD sZt |j j
|
|||
|
|
« dkD s8|j j
|
|||
|
|
d« r |j j
|
|||
|
|
dd¬ « s|S |j dg « }t |« }d|cxk r|k sJ ‚ J ‚|j j
|
|||
|
|
d
|
|||
|
|
« rkd } |d kD rN||k rI|d |d z
|
|||
|
|
}
|
|||
|
|
|d | } d
|
|||
|
|
|
|
|||
|
|
v r2d
|
|||
|
|
| v r.t |
|
|||
|
|
« \ } }
|
|||
|
|
t | « \ }}| dv r|dv r
|
|||
|
|
|
|
|||
|
|
|k( rd|
|
|||
|
|
› �} |d j || « n|d= |d j ||« |d j |d« |j j
|
|||
|
|
d« r|d j |d« n|d= |j j
|
|||
|
|
d« r|d j ||« n|d= |j j
|
|||
|
|
d« r|d j |d« n|d= |j j
|
|||
|
|
d« r|d j |d« n|d= |j j
|
|||
|
|
dd¬ « r�|dk( r|d j |d« n|d j ||d z
|