100 lines
11 KiB
Plaintext
100 lines
11 KiB
Plaintext
|
Ë
|
|||
|
|
?û gc$ ã óÚ — d dl Z d dlmZmZmZmZmZmZmZ ddl m
|
|||
|
|
Z
|
|||
|
|
m Z ddl m
|
|||
|
|
Z
|
|||
|
|
mZ dee dee fd„Zdee dee fd „Zdee dee fd
|
|||
|
|
„Zdee dee fd „Zdd e
|
|||
|
|
d
|
|||
|
|
efd„Zd e
|
|||
|
|
dee fd„Z dd e
|
|||
|
|
deeeeeeef f d
|
|||
|
|
edee fd„Zd e
|
|||
|
|
dee dee fd„Zd e
|
|||
|
|
dee deeeeeeef f fd„Zdee deeeeef fd„Zdedeeef fd„Zdedefd„Z eZ eZ eZ y)é N)ÚDictÚIterableÚIteratorÚListÚTupleÚUnionÚcasté )ÚErrorsÚWarnings)ÚDocÚSpanÚtagsÚreturnc ó’ — g }t | « } | r7|j t | « « |j t | « « | rŒ7|S )N)ÚlistÚextendÚ _consume_osÚ _consume_ent)r Úouts úYC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/training/iob_utils.pyÚ iob_to_biluor sA € Ø€CÜ �‹:€DÙ
|
|||
|
|
Ø �
|
|||
|
|
‰
|
|||
|
|
”;˜tÓ$Ô%Ø �
|
|||
|
|
‰
|
|||
|
|
”< Ó%Ô&ò ð €Jó c óª — g }| D ]K }|€|j |« Œ|j ddd« j ddd« }|j |« ŒM |S )NúU-úB-é úL-úI-)ÚappendÚreplace)r r Útags r Ú biluo_to_iobr# sT € Ø
|
|||
|
|
€CÛˆØ ˆ;Ø �J‰J�s�Oà—+‘+˜d D¨!Ó,×4Ñ4°T¸4ÀÓCˆCØ �J‰J�s�Oð ð €Jr c # ób K — | r)| d dk( r | j d« –— | r
|
|||
|
|
| d dk( rŒ y y y y w)Nr ÚO)Úpop)r s r r r s6 è ø€ Ù
|
|||
|
|
�4˜‘7˜c’>Ø�h‰h�q‹kÒñ �4˜‘7˜c”>ˆ$�>ˆ$ùs ‚(/«/c ó¨ — | sg S | j d« }d|dd z }d|dd z }d}| r+| d ||hv r"|dz
|
|||
|
|
}| j d« | r
|
|||
|
|
| d ||hv rŒ"|dd }|dk( r=t |« dk( r)t t j j |¬« « ‚d|z gS d|z }d |z }t
|
|||
|
|
d|dz
|
|||
|
|
« D �cg c] }d
|
|||
|
|
|› �‘Œ } }|g| z |gz S c c}w ) Nr ÚIr ÚLr
|
|||
|
|
©r" r r r r )r& ÚlenÚ
|
|||
|
|
ValueErrorr ÚE177ÚformatÚrange)
|
|||
|
|
r r" Ú target_inÚ target_lastÚlengthÚlabelÚstartÚendÚ_Úmiddles
|
|||
|
|
r r r ! s € ٠؈ Ø
|
|||
|
|
�(‰(�1‹+€CØ�c˜!˜"�g‘
|
|||
|
|
€Iؘ˜A˜B˜‘-€KØ
|
|||
|
|
€FÙ
|
|||
|
|
�4˜‘7˜y¨+Ð 6Ñ6Ø�!‰ ˆØ �‰�Œ ñ �4˜‘7˜y¨+Ð 6Ò6ð
|
|||
|
|
��ˆG€EØ
|
|||
|
|
�‚{Ü ˆu‹:˜Š?Ü œVŸ[™[× /Ñ /°CÐ /Ó 8Ó9Ð 9Ø�u‘ ˆ~Ð à�u‘ ˆØ�U‰lˆÜ(-¨a°¸!±Ô(<Ó=Ñ(< 1�B�u�g’,Ð(<ˆÐ=؈w˜Ñ 3 %Ñ'Ð'ùò >s Â7 CÚdocÚmissingc óš — t | | j D �cg c]% }|j |j |j f‘Œ' c}|¬« S c c}w )N©r9 )Úoffsets_to_biluo_tagsÚentsÚ
|
|||
|
|
start_charÚend_charÚlabel_)r8 r9 Úents r Údoc_to_biluo_tagsrB 7 sA € Ü Ø Ø?B¿xºxÓH¹x¸ˆ#�.‰.˜#Ÿ,™,¨¯
|
|||
|
|
©
|
|||
|
|
Ò 3¸xÑHØô ð ùâHs •*A
|
|||
|
|
c óp — t | d¬« }t | « D ] \ }}|j dk( sŒd||< Œ |S )Nú-r; r
|
|||
|
|
r% )rB Ú enumerateÚent_iob)r8 r= ÚiÚtokens r Ú _doc_to_biluo_tags_with_partialrI ? s< € Ü ˜S¨#Ô .€DÜ ˜c–N‰ˆˆ5Ø �=‰=˜AÓ ØˆD�ŠGð #ð €Kr Úentitiesc
|
|||
|
|
ó
|
|||
|
|
— i }| D �ci c] }|j |j “Œ }}| D �ci c]% }|j t |« z |j “Œ' }}| D �cg c] }d‘Œ }}|D ]ó \ } }
|
|||
|
|
} | s |D ] } | | k\ sΠ| |
|
|||
|
|
k sŒd||| < Œ Œ%t | |
|
|||
|
|
« D ]^ }
|
|||
|
|
|
|
|||
|
|
|j « v rBt t j j ||
|
|||
|
|
d ||
|
|||
|
|
d ||
|
|||
|
|
d f| |
|
|||
|
|
| f¬« « ‚| |
|
|||
|
|
| f||
|
|||
|
|
< Œ` |j | « }|j |
|
|||
|
|
« }|€Œ·|€Œº||k( r d| › �||< ŒÈd | › �||< t |dz |« D ]
|
|||
|
|
}d
|
|||
|
|
| › �||< Œ d | › �||< Œõ t « }|D ]( \ } }
|
|||
|
|
} t | |
|
|||
|
|
« D ] }|j |« Œ Œ* | D ]H }t |j |j t |« z « D ] }||v sŒ Œ9 |||j < ŒJ d|v rŽ|dk7 r‰t |« }t j t j j t | j" « d kD r| j" dd d
|
|||
|
|
z n | j" t |« d kD r|dd d
|
|||
|
|
z n|¬« « |S c c}w c c}w c c}w )u¸ Encode labelled spans into per-token tags, using the
|
|||
|
|
Begin/In/Last/Unit/Out scheme (BILUO).
|
|||
|
|
|
|||
|
|
doc (Doc): The document that the entity offsets refer to. The output tags
|
|||
|
|
will refer to the token boundaries within the document.
|
|||
|
|
entities (iterable): A sequence of `(start, end, label)` triples. `start`
|
|||
|
|
and `end` should be character-offset integers denoting the slice into
|
|||
|
|
the original string.
|
|||
|
|
missing (str): The label used for missing values, e.g. if tokenization
|
|||
|
|
doesn’t align with the entity offsets. Defaults to "O".
|
|||
|
|
RETURNS (list): A list of unicode strings, describing the tags. Each tag
|
|||
|
|
string will be of the form either "", "O" or "{action}-{label}", where
|
|||
|
|
action is one of "B", "I", "L", "U". The missing label is used where the
|
|||
|
|
entity offsets don't align with the tokenization in the `Doc` object.
|
|||
|
|
The training algorithm will view these as missing values. "O" denotes a
|
|||
|
|
non-entity token. "B" denotes the beginning of a multi-token entity,
|
|||
|
|
"I" the inside of an entity of three or more tokens, and "L" the end
|
|||
|
|
of an entity of two or more tokens. "U" denotes a single-token entity.
|
|||
|
|
|
|||
|
|
EXAMPLE:
|
|||
|
|
>>> text = 'I like London.'
|
|||
|
|
>>> entities = [(len('I like '), len('I like London'), 'LOC')]
|
|||
|
|
>>> doc = nlp.tokenizer(text)
|
|||
|
|
>>> tags = offsets_to_biluo_tags(doc, entities)
|
|||
|
|
>>> assert tags == ["O", "O", 'U-LOC', "O"]
|
|||
|
|
rD r% r r r
|
|||
|
|
)Úspan1Úspan2Nr r r r é2 z...)ÚtextrJ )ÚidxrG r+ r/ Úkeysr, r ÚE103r. ÚgetÚsetÚaddÚstrÚwarningsÚwarnr ÚW030rO )r8 rJ r9 Útokens_in_entsrH ÚstartsÚendsr6 Úbiluor> r? r3 ÚsÚ token_indexÚ start_tokenÚ end_tokenrG Ú entity_charsÚent_strs r r< r< G sÌ € ð< CE€NÙ.1Ó
|
|||
|
|
2©c Uˆe�i‰i˜Ÿ™Ñ ¨c€FÐ
|
|||
|
|
2Ù9<Ó =¹°ˆE�I‰Iœ˜E›
|
|||
|
|
Ñ " E§G¡GÑ +¸€DÐ =Ù Ó ™#�QŠS˜#€EÐ ã'/Ñ#ˆ
|
|||
|
|
�H˜eÙÛ�ؘ
|
|||
|
|
“? q¨8£|Ø'*�E˜& ™)Ò$ñ ô % Z°Ö :� Ø .×"5Ñ"5Ó"7Ñ7Ü$Ü Ÿ ™ ×*Ñ*à .¨{Ñ ;¸AÑ >Ø .¨{Ñ ;¸AÑ >Ø .¨{Ñ ;¸AÑ >ð# ð
|
|||
|
|
$.¨x¸Ð"?ð
|
|||
|
|
+ó ó ð ð 0:¸8ÀUÐ.K�˜{Ò+ð ;ð !Ÿ*™* ZÓ0ˆKØ Ÿ™ Ó*ˆIàÑ&¨9Ñ+@Ø )Ò+Ø+-¨e¨W¨ �E˜+Ò&à+-¨e¨W¨ �E˜+Ñ&Ü " ;°¡?°IÖ >˜Ø%'¨ w <˜˜ašð ?à)+¨E¨7 |�E˜)Ò$ð; (0ô> “5€LÛ'/Ñ#ˆ
|
|||
|
|
�H˜eÜ�z 8Ö,ˆAØ × Ñ ˜QÕ ñ -ð (0ó ˆÜ�u—y‘y %§)¡)¬c°%«jÑ"8Ö9ˆAØ�LÒ Ùð :ð %ˆE�%—'‘'ŠNð ð ˆe�|˜ 3šÜ�h“-ˆÜ�
|
|||
|
|
‰
|
|||
|
|
Ü �M‰M× Ñ Ü.1°#·(±(«m¸bÒ.@�S—X‘X˜c˜r�] UÒ*ÀcÇhÁhÜ14°W³ ÀÒ1B˜ "˜ ¨Ò-Èð
|
|||
|
|
!ó
|
|||
|
|
ô
|