134 lines
18 KiB
Plaintext
134 lines
18 KiB
Plaintext
|
Ë
|
|||
|
|
?û g
|
|||
|
|
0 ã ó, — d dl Z d dlmZ d dlmZmZmZmZmZm Z m
|
|||
|
|
Z
|
|||
|
|
d dl Z d dl Z d dl m
|
|||
|
|
Z
|
|||
|
|
d dlmZ ddlmZmZmZmZ ddlmZ dd lmZ dd
|
|||
|
|
lmZmZ dd l m Z d d
|
|||
|
|
l m Z d dl m!Z" d dl m#Z# G d„ d« Z$d„ Z%d„ Z&d„ Z' ejP e$e&e'« e$Z)dgZ*y)é N)ÚPath)ÚDictÚIterableÚIteratorÚListÚOptionalÚSetÚUnion)Úndarray)ÚNumpyOpsé )ÚIDSÚORTHÚSPACYÚ intify_attr)Úcopy_reg)ÚErrors)ÚSimpleFrozenListÚ ensure_path)ÚVocabé )Ú
|
|||
|
|
SpanGroups)ÚDOCBIN_ALL_ATTRS)ÚDocc óÐ — e Zd ZdZed e« fdee dedee ddfd„Z
|
|||
|
|
de fd „Z d
|
|||
|
|
e ddfd „Z
|
|||
|
|
d edee fd
|
|||
|
|
„Zdd„Zdefd„Zdedd fd„Zdeeef ddfd„Zdeeef dd fd„Zy)ÚDocBinaÕ Pack Doc objects for binary serialization.
|
|||
|
|
|
|||
|
|
The DocBin class lets you efficiently serialize the information from a
|
|||
|
|
collection of Doc objects. You can control which information is serialized
|
|||
|
|
by passing a list of attribute IDs, and optionally also specify whether the
|
|||
|
|
user data is serialized. The DocBin is faster and produces smaller data
|
|||
|
|
sizes than pickle, and allows you to deserialize without executing arbitrary
|
|||
|
|
Python code.
|
|||
|
|
|
|||
|
|
The serialization format is gzipped msgpack, where the msgpack object has
|
|||
|
|
the following structure:
|
|||
|
|
|
|||
|
|
{
|
|||
|
|
"attrs": List[uint64], # e.g. [TAG, HEAD, ENT_IOB, ENT_TYPE]
|
|||
|
|
"tokens": bytes, # Serialized numpy uint64 array with the token data
|
|||
|
|
"spans": List[Dict[str, bytes]], # SpanGroups data for each doc
|
|||
|
|
"spaces": bytes, # Serialized numpy boolean array with spaces data
|
|||
|
|
"lengths": bytes, # Serialized numpy int32 array with the doc lengths
|
|||
|
|
"strings": List[str] # List of unique strings in the token data
|
|||
|
|
"version": str, # DocBin version number
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
Strings for the words, tags, labels etc are represented by 64-bit hashes in
|
|||
|
|
the token data, and every string that occurs at least once is passed via the
|
|||
|
|
strings object. This means the storage is more efficient if you pack more
|
|||
|
|
documents together, because you have less duplication in the strings.
|
|||
|
|
|
|||
|
|
A notable downside to this format is that you can't easily extract just one
|
|||
|
|
document from the DocBin.
|
|||
|
|
FÚattrsÚstore_user_dataÚdocsÚreturnNc ór — |D �cg c]
|
|||
|
|
}t |« ‘Œ }}d|v rY|D �cg c] }t |« �Œ|‘Œ }}t t j j d|t j « ¬« « d‚t |« }d| _ |D �cg c] }|t k7 sŒ
|
|||
|
|
|t k7 sŒ|‘Œ c}| _ | j j dt « g | _
|
|||
|
|
g | _ g | _ g | _ g | _ g | _ t' « | _ || _ |D ] }| j- |« Œ yc c}w c c}w c c}w )a~ Create a DocBin object to hold serialized annotations.
|
|||
|
|
|
|||
|
|
attrs (Iterable[str]): List of attributes to serialize. 'orth' and
|
|||
|
|
'spacy' are always serialized, so they're not required.
|
|||
|
|
store_user_data (bool): Whether to write the `Doc.user_data` to bytes/file.
|
|||
|
|
docs (Iterable[Doc]): Docs to add.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#init
|
|||
|
|
Nr )ÚdictÚkeyÚkeysz0.1r )r ÚKeyErrorr ÚE983Úformatr r$ ÚsortedÚversionr r r ÚinsertÚtokensÚspacesÚcatsÚ span_groupsÚ user_dataÚflagsÚsetÚstringsr Úadd)Úselfr r r ÚattrÚ int_attrsÚ non_validÚdocs úXC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/tokens/_serialize.pyÚ__init__zDocBin.__init__4 s € ñ 49Ó9±5¨4”[ Õ&°5ˆ Ð9Ø �9Ñ Ù*/ÓM©% $´;¸tÓ3DÑ3Lš¨%ˆIÐMÜÜ— ‘ ×"Ñ"¨°YÄSÇXÁXÃZÐ"ÓPóàð
|
|||
|
|
ô �yÓ!ˆØ ˆŒ Ù',ÓO¡u˜t°¼³ ÀÌà ’d uÑOˆŒ
|
|||
|
|
Ø �
|
|||
|
|
‰
|
|||
|
|
×ј!œTÔ"Ø%'ˆŒ Ø%'ˆŒ Ø "ˆŒ Ø(*ˆÔØ02ˆŒØ!#ˆŒ
|
|||
|
|
Ü!$£ˆŒ Ø .ˆÔ ÛˆCØ �H‰H�S�Mñ ùò% :ùâMùò Ps! …D*¡D/²D/ D4 D4Â%D4c ó, — t | j « S )z7RETURNS: The number of Doc objects added to the DocBin.)Úlenr+ )r4 s r9 Ú__len__zDocBin.__len__X s € ä�4—;‘;Ó Ð ó r8 c ó — |j | j « }t |j « dk( r |j |j d df« }| j
|
|||
|
|
j
|
|||
|
|
|« |j t « }|j d |j d k( sJ ‚|j |j d df« }| j j
|
|||
|
|
t j |t ¬« « | j j
|
|||
|
|
d|j i« |D �]Y }| j j |j « | j j |j" « | j j |j$ « | j j |j& « | j j t) |j* « « | j j |j, « | j j |j. « | j j |j0 « | j j |j2 « �Œ\ | j4 j
|
|||
|
|
|j4 « | j6 r8| j8 j
|
|||
|
|
t; j< |j8 « « | j> j
|
|||
|
|
|j@ jC « « |j@ jE « D ]Ô \ }}|D ]Ê }| j j |jF « |jH |jJ jL j v r%| j j |jN « |jP |jJ jL j v sŒ¦| j j |jR « ŒÌ ŒÖ y)z›Add a Doc's annotations to the DocBin for serialization.
|
|||
|
|
|
|||
|
|
doc (Doc): The Doc object to add.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#add
|
|||
|
|
r r ©ÚdtypeÚhas_unknown_spacesN)*Úto_arrayr r< ÚshapeÚreshaper+ Úappendr r, ÚnumpyÚasarrayÚboolr0 rB r2 r3 ÚtextÚtag_Úlemma_Únorm_ÚstrÚmorphÚdep_Ú ent_type_Ú
|
|||
|
|
ent_kb_id_Úent_id_r- r r/ ÚsrslyÚ
|
|||
|
|
msgpack_dumpsr. ÚspansÚto_bytesÚitemsÚlabel_Úkb_idr8 ÚvocabÚkb_id_ÚidÚid_)r4 r8 Úarrayr, Útokenr# ÚgroupÚspans r9 r3 z
|
|||
|
|
DocBin.add\ s© € ð — ‘ ˜TŸZ™ZÓ(ˆÜ ˆu�{‰{Ó ˜qÒ Ø—M‘M 5§;¡;¨q¡>°1Ð"5Ó6ˆEØ � ‰ ×ј5Ô!Ø— ‘ œeÓ$ˆØ�{‰{˜1‰~ § ¡ ¨a¡ Ò0Ð0Ð0Ø— ‘ § ¡ ¨a¡ °!Ð 4Ó5ˆØ � ‰ ×Ñœ5Ÿ=™=¨´tÔ<Ô=Ø �
|
|||
|
|
‰
|
|||
|
|
×ÑÐ/°×1GÑ1GÐHÔI܈EØ �L‰L× Ñ ˜UŸZ™ZÔ (Ø �L‰L× Ñ ˜UŸZ™ZÔ (Ø �L‰L× Ñ ˜UŸ\™\Ô *Ø �L‰L× Ñ ˜UŸ[™[Ô )Ø �L‰L× Ñ œS §¡Ó -Ô .Ø �L‰L× Ñ ˜UŸZ™ZÔ (Ø �L‰L× Ñ ˜UŸ_™_Ô -Ø �L‰L× Ñ ˜U× -Ñ -Ô .Ø �L‰L× Ñ ˜UŸ]™]Ö +ð ð
|
|||
|
|
� ‰ ×јŸ™Ô"Ø × Ò Ø �N‰N× !Ñ !¤%×"5Ñ"5°c·m±mÓ"DÔ EØ ×Ñ× Ñ § ¡ × 2Ñ 2Ó 4Ô5Ø Ÿ)™)Ÿ/™/Ö+‰JˆC�Û �Ø— ‘ × Ñ §¡Ô-Ø—:‘: §¡§ ¡ ×!7Ñ!7Ñ7Ø—L‘L×$Ñ$ T§[¡[Ô1Ø—7‘7˜dŸh™hŸn™n× 4Ñ 4Ò4Ø—L‘L×$Ñ$ T§X¡XÕ.ñ ñ ,r> r[ c # ó‚ K — | j D ] }|| Œ | j j t « }t t | j « « D �]d }| j | }| j | }| j | }|j d« rd}t ||dd…|f |¬« }|j | j |« }| j | |_ | j | rI| j | t j k7 r)|j j# | j | « n|j j% « |t | j& « k rN| j& | �?t) j* | j& | d¬« } |j& j- | « |–— �Œg yw)aa Recover Doc objects from the annotations, using the given vocab.
|
|||
|
|
Note that the user data of each doc will be read (if available) and returned,
|
|||
|
|
regardless of the setting of 'self.store_user_data'.
|
|||
|
|
|
|||
|
|
vocab (Vocab): The shared vocab.
|
|||
|
|
YIELDS (Doc): The Doc objects.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#get_docs
|
|||
|
|
rB N)Úwordsr, F)Úuse_list)r2 r Úindexr Úranger< r+ r0 r, Úgetr Ú
|
|||
|
|
from_arrayr- r. r Ú _EMPTY_BYTESrV Ú
|
|||
|
|
from_bytesÚclearr/ rT Ú
|
|||
|
|
msgpack_loadsÚupdate)
|
|||
|
|
r4 r[ ÚstringÚorth_colÚir0 r+ r, r8 r/ s
|
|||
|
|
r9 Úget_docszDocBin.get_docs‚ s^ è ø€ ð —l”lˆFØ �&‹Mð #à—:‘:×#Ñ#¤DÓ)ˆÜ”s˜4Ÿ;™;Ó'×(ˆAØ—J‘J˜q‘MˆEØ—[‘[ ‘^ˆFØ(,¯ © °A©ˆFØ�y‰yÐ-Ô.Ø �Ü�e 6ª!¨X¨+Ñ#6¸vÔFˆCØ—.‘. §¡¨VÓ4ˆCØ—y‘y ‘|ˆCŒHà× Ñ Ò" t×'7Ñ'7¸Ñ':¼j×>UÑ>UÒ'UØ— ‘ ×$Ñ$ T×%5Ñ%5°aÑ%8Õ9à— ‘ —‘Ô!Ø”3�t—~‘~Ó&Ò&¨4¯>©>¸!Ñ+<Ð+HÜ !× /Ñ /°·±¸qÑ0AÈEÔ R� Ø—
|
|||
|
|
‘
|
|||
|
|
×$Ñ$ YÔ/ØŒIñ# )ùs ‚F=F?c ój — | j |j k7 r?t t j j d| j |j ¬« « ‚| j
|
|||
|
|
|j
|
|||
|
|
k7 r?t t j j d| j
|
|||
|
|
|j
|
|||
|
|
¬« « ‚| j j |j « | j j |j « | j j |j « | j j |j « | j j |j « | j j |j « | j j |j « y)ab Extend the annotations of this DocBin with the annotations from
|
|||
|
|
another. Will raise an error if the pre-defined attrs of the two
|
|||
|
|
DocBins don't match, or if they differ in whether or not to store
|
|||
|
|
user data.
|
|||
|
|
|
|||
|
|
other (DocBin): The DocBin to merge into the current bin.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#merge
|
|||
|
|
r )ÚparamÚcurrentÚotherr N)r Ú
|
|||
|
|
ValueErrorr ÚE166r' r r+ Úextendr, r2 rn r- r. r0 r/ )r4 rv s r9 Úmergez DocBin.merge¢ s1 € ð �:‰:˜Ÿ™Ò $Ü Ü— ‘ ×"Ñ"¨¸$¿*¹*ÈEÏKÉKÐ"ÓXóð
|
|||
|
|
ð × Ñ 5×#8Ñ#8Ò 8Ü Ü— ‘ ×"Ñ"Ø+Ø × 0Ñ 0Ø ×/Ñ/ð #ó óð
|
|||
|
|
ð
|
|||
|
|
� ‰ ×ј5Ÿ<™<Ô(Ø � ‰ ×ј5Ÿ<™<Ô(Ø � ‰ ×јEŸM™MÔ*Ø � ‰ ×јŸ™Ô$Ø ×Ñ× Ñ × 1Ñ 1Ô2Ø �
|
|||
|
|
‰
|
|||
|
|
×ј%Ÿ+™+Ô&Ø �‰× Ñ ˜eŸo™oÕ.r> c
|
|||
|
|
ó‚ — | j D ]+ }t |j « dk( rŒ J |j « ‚ | j D �cg c]
|
|||
|
|
}t |« ‘Œ }}| j r t j | j « nt j
|
|||
|
|
g « }| j r t j | j « nt j
|
|||
|
|
g « }| j | j |j d« |j d« t j
|
|||
|
|
|d¬« j d« t t | j « « | j | j | j dœ }| j r| j" |d< t% j& t) j* |« « S c c}w )z¡Serialize the DocBin's annotations to a bytestring.
|
|||
|
|
|
|||
|
|
RETURNS (bytes): The serialized DocBin.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#to_bytes
|
|||
|
|
r
|
|||
|
|
ÚCÚint32r@ ) r) r r+ r, Úlengthsr2 r- r0 r. r/ )r+ r< rD rG ÚvstackrH r, r) r ÚtobytesÚlistr( r2 r- r0 r. r r/ ÚzlibÚcompressrT rU )r4 r+ r~ r, Úmsgs r9 rW zDocBin.to_bytesÀ s: € ð —k”kˆFÜ�v—|‘|Ó$¨Ó)Ð 7¨6¯<©<Ó 7Ð)ð "à-1¯[ª[Ó9©[ 6”3�v•;¨[ˆÐ9Ø.2¯kªk”— ‘ ˜dŸk™kÔ*¼u¿}¹}ÈRÓ?PˆØ.2¯kªk”— ‘ ˜dŸk™kÔ*¼u¿}¹}ÈRÓ?Pˆà—|‘|Ø—Z‘ZØ —n‘n SÓ)Ø —n‘n SÓ)Ü —}‘} W°GÔ<×DÑDÀSÓIÜœF 4§<¡<Ó 0Ó1Ø—I‘IØ—Z‘ZØ ×+Ñ+ñ
|
|||
|
|
|
|||
|
|
ˆð × Ò Ø #Ÿ~™~ˆC� Ñ Ü�}‰}œU× 0Ñ 0°Ó 5Ó6Ð6ùò! :s Á F<Ú
|
|||
|
|
bytes_datac ód — t j t j |« « }|d | _ t |d « | _
|
|||
|
|
t j |d d¬« }t j |d t ¬« }t j |d d¬« }|j t | j « z t | j « f}|j! |« }|j! |j d f« }t# « j% ||« | _ t# « j% ||« | _ |d
|
|||
|
|
| _ |j- d |D �cg c] }d ‘Œ c}« | _ |j- d
|
|||
|
|
|D �cg c] }i ‘Œ c}« | _ d|v rt3 |d « | _ ndgt | « z | _ | j& D ]+ }t |j6 « dk( rŒ J |j6 « ‚ | S # t j $ r t t j « ‚w xY wc c}w c c}w )z×Deserialize the DocBin's annotations from a bytestring.
|
|||
|
|
|
|||
|
|
bytes_data (bytes): The data to load from.
|
|||
|
|
RETURNS (DocBin): The loaded DocBin.
|
|||
|
|
|
|||
|
|
DOCS: https://spacy.io/api/docbin#from_bytes
|
|||
|
|
r r2 r~ r} r@ r, r+ Úuint64r r- r. r> r0 r/ Nr
|
|||
|
|
) rT rm r‚ Ú
|