131 lines
30 KiB
Plaintext
131 lines
30 KiB
Plaintext
|
Ë
|
|||
|
|
?û gÉJ ã ó
|
|||
|
|
— d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
|
|||
|
|
Z
|
|||
|
|
d dl m Z d dl
|
|||
|
|
mZ d dlmZmZmZmZ d d lmZ ej, j/ d
|
|||
|
|
« d „ « Zej, j/ d « ej, j3 d
|
|||
|
|
¬« ej, j5 ddg d¢fdg d¢fdg d¢fdg d¢fdg d¢fdg d¢fd g d ¢fg« d „ « « « Zej, j/ d « d „ « Z ej, j/ d!« d"„ « Z ej, j3 d#¬« ej, j/ d$« d%„ « « Z ej, j/ d&« d'„ « Z ej, j/ d(« d)„ « Z ej, j/ d*« d+„ « Z!ej, j/ d,« d-„ « Z"ej, j/ d.« d/„ « Z#ej, j3 d0¬« ej, j/ d1« d2„ « « Z$ej, j/ d3« d4„ « Z%ej, j5 d5d6d7g« ej, j/ d8« d9„ « « Z&ej, j/ d:« d;„ « Z'ej, j/ d<« d=„ « Z(ej, j/ d>« d?„ « Z)ej, j3 d@¬« ej, j/ dA« dB„ « « Z*ej, j5 dCdDg dE¢fdFdFgfg« dG„ « Z+dH„ Z,ej, j5 d5dIg« dJ„ « Z-dK„ Z.dL„ Z/dM„ Z0ej, j5 d5g dN¢« dO„ « Z1ej, j5 d5dPg« dQ„ « Z2ej, j5 d5g dR¢« dS„ « Z3dT„ Z4ej, j5 dUdVg« dW„ « Z5dX„ Z6ej, j5 ddIdYdZidYd[igfg« d\„ « Z7ej, j5 ddIdYdZidYd]igfdIdZd^d_œdYd[igfg« d`„ « Z8ej, j5 ddIdZdadbœdYd[igfg« dc„ « Z9dd„ Z:de„ Z;df„ Z<dg„ Z=dh„ Z>di„ Z?dj„ Z@dk„ ZAdl„ ZBej, j/ dm« dn„ « ZCdo„ ZDy)pé N)ÚGerman)ÚEnglish)ÚORTH)Ú Tokenizer)ÚDoc)ÚExample)Úcompile_infix_regexÚcompile_prefix_regexÚcompile_suffix_regexÚ ensure_path)ÚVocabiç c óz — t t « ddg« } | d }t |g« }t |« }|d |u sJ ‚y )NÚhelloÚworldr )r r
|
|||
|
|
ÚsetÚlist)ÚdocÚtokenÚsÚitemss úeC:\Users\garci\AppData\Roaming\Python\Python312\site-packages\spacy/tests/tokenizer/test_tokenizer.pyÚ
|
|||
|
|
test_issue743r sD € ä
|
|||
|
|
|
|||
|
|
Œe‹g˜ Ð)Ó
|
|||
|
|
*€CØ �‰F€EÜ ˆUˆG‹ €AÜ �‹G€EØ �‰8�uÑ Ð Ñ ó i! zECan not be fixed unless with variable-width lookbehinds, cf. PR #3218)Úreasonz text,tokensz"deserve,"--and)ú"Údeservez,"--Úandzexception;--exclusive)Ú exceptionz;--Ú exclusivezday.--Is)Údayz.--ÚIszrefinement:--just)Ú
|
|||
|
|
refinementz:--Újustz
|
|||
|
|
memories?--To)Úmemoriesz?--ÚTozUseful.=--Therefore)ÚUsefulú.=--Ú Thereforez=Hope.=--Pandora)ú=ÚHoper' ÚPandorac óŽ — | |« }t |« t |« k( sJ ‚|D �cg c] }|j ‘Œ c}|k( sJ ‚yc c}w )z;Test that special characters + hyphens are split correctly.N©ÚlenÚtext)Ú en_tokenizerr/ Útokensr Úts r Ú
|
|||
|
|
test_issue801r3 sH € ñ$ �tÓ
|
|||
|
|
€CÜ ˆs‹8”s˜6“{Ò "Ð"Ð "Ù Ó ™C�qˆA�F‹F˜CÑ FÒ *Ð*Ñ *ùÒ s ¦Ai% c ó� — d} t « j } || « }d|D �cg c] }|j ‘Œ c}v sJ ‚d|D �cg c] }|j ‘Œ c}vsJ ‚|j dt dig« || « }d|D �cg c] }|j ‘Œ c}v sJ ‚d|D �cg c] }|j ‘Œ c}vsJ ‚t « j }|j dt dig« || « }d|D �cg c] }|j ‘Œ c}v sJ ‚d|D �cg c] }|j ‘Œ c}vsJ ‚yc c}w c c}w c c}w c c}w c c}w c c}w )z>Test special-case works after tokenizing. Was caching problem.zTI like _MATH_ even _MATH_ when _MATH_, except when _MATH_ is _MATH_! but not _MATH_.ÚMATHÚ_MATH_N)r Ú tokenizerr/ Úadd_special_caser )r/ r7 r Úws r Útest_issue1061r: 5 sO € ð b€DÜ“ ×#Ñ#€IÙ
|
|||
|
|
�D‹/€CØ ¡cÓ*¡c �a—f“f cÑ*Ñ *Ð*Ð *Ø ©CÓ0©C q˜AŸF›F¨CÑ0Ñ 0Ð0Ð 0à
|
|||
|
|
× Ñ ˜x¬4°Ð*:Ð);Ô<Ù
|
|||
|
|
�D‹/€CØ ©Ó,© 1˜Ÿ›¨Ñ,Ñ ,Ð,Ð ,Ø ©#Ó.©# Q˜!Ÿ&›&¨#Ñ.Ñ .Ð.Ð .ô “ ×#Ñ#€IØ
|
|||
|
|
× Ñ ˜x¬4°Ð*:Ð);Ô<Ù
|
|||
|
|
�D‹/€CØ ©Ó,© 1˜Ÿ›¨Ñ,Ñ ,Ð,Ð ,Ø ©#Ó.©# Q˜!Ÿ&›&¨#Ñ.Ñ .Ð.Ñ .ùò +ùÚ0ùò -ùÚ.ùò -ùÚ.s# ¤D*ÁD/ÂD4 D9Ã2D>ÄEi« c ó* — | d« }t j t |« dfd¬« |_ |j « 5 }|j |dd « ddd« t |« dk( sJ ‚|j j d k( sJ ‚y# 1 sw Y Œ5xY w)
|
|||
|
|
z(Test that doc.merge() resizes doc.tensorza b c dé€ Úf)Údtyper é Né )r@ r< )ÚnumpyÚonesr. ÚtensorÚ
|
|||
|
|
retokenizeÚmergeÚshape)r0 r Ú retokenizers r Útest_issue1963rH K s~ € ñ �yÓ
|
|||
|
|
!€CÜ—‘œS ›X s˜O°3Ô7€C„JØ � ‰ Ô ˜[Ø×ј#˜a ˜(Ô#÷
|
|||
|
|
ä ˆs‹8�qŠ=Ј=Ø �:‰:× Ñ ˜xÒ 'Ð'Ñ '÷
|
|||
|
|
Ð ús Á B Â BzICan not be fixed without variable-width look-behind (which we don't want)iÓ c ó — t « } d} | |« }t |« dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j d
|
|||
|
|
k( sJ ‚|d j dk( sJ ‚y )
|
|||
|
|
z@Test that g is not split of if preceded by a number and a letterz
|
|||
|
|
e2g 2g 52gé r Úe2gé Ú2r? Úgr@ Ú52é N)r r. r/ )ÚnlpÚ testwordsr s r Útest_issue1235rS V s— € ô ‹)€CØ €IÙ
|
|||
|
|
|
|||
|
|
ˆi‹.€CÜ ˆs‹8�qŠ=Ј=Ø ˆq‰6�;‰;˜%Ò Ð Ð Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜$Ò Ð Ð Ø ˆq‰6�;‰;˜#Ò Ð Ñ r iÚ c óÌ — t « } | d« }t |« dk( sJ ‚t | j ddg« « }t |d « dk( sJ ‚t |d « dk( sJ ‚y )NÚ r r rL )r r. r Úpipe)rQ r Údocss r Útest_issue1242rX g se € ä
|
|||
|
|
‹)€CÙ
|
|||
|
|
|
|||
|
|
ˆb‹'€CÜ ˆs‹8�qŠ=Ј=Ü �—‘˜"˜g˜ Ó'Ó (€DÜ ˆt�A‰w‹<˜1Ò Ð Ð Ü ˆt�A‰w‹<˜1Ò Ð Ñ r ié c ó” — t t « g d¢¬« } t t « g d¢¬« }| d |d k7 sJ ‚| d |d k( rJ ‚y)z#Test that tokens compare correctly.©ÚaÚbÚc©Úwords)r[ r] Úer N)r r
|
|||
|
|
)Údoc1Údoc2s r Útest_issue1257rc q sO € ô Œu‹wšoÔ .€DÜ Œu‹wšoÔ .€DØ �‰7�d˜1‘gÒ Ð Ð Ø�A‰w˜$˜q™'Ò!Ð!Ð !Ð!r i_ c óÒ — t t « g d¢¬« } t j t « 5 | d j d« sJ ‚ ddd« | d j d« j dk( sJ ‚t j t « 5 | d j d« sJ ‚ ddd« | d j d« j d k( sJ ‚y# 1 sw Y Œ‰xY w# 1 sw Y Œ9xY w)
|
|||
|
|
zBTest that token.nbor() raises IndexError for out-of-bounds access.)Ú0Ú1rM r^ r éÿÿÿÿNrL re r? rM )r r
|
|||
|
|
ÚpytestÚraisesÚ
|
|||
|
|
IndexErrorÚnborr/ )r s r Útest_issue1375rl z s® € ô Œe‹gš_Ô
|
|||
|
|
-€CÜ � ‰ ”zÕ "Ø�1‰v�{‰{˜2ŒÐ ‰÷
|
|||
|
|
#à ˆq‰6�;‰;�r‹?× Ñ 3Ò &Ð&Ð &Ü � ‰ ”zÕ "Ø�1‰v�{‰{˜1Œ~Ð ‰~÷
|
|||
|
|
#à ˆq‰6�;‰;�q‹>× Ñ #Ò %Ð%Ñ %÷
|
|||
|
|
#Ð "ú÷
|
|||
|
|
#Ð "ús ±CÂ
|
|||
|
|
C ÃCà C&iÐ c ó, ‡‡‡‡— t j d« Št j d« Št j d« Št j d« Šˆˆˆˆfd„} t « } | |« |_ |d« }|D ] }|j rŒJ ‚ y)zBTest that tokenizer can parse DOT inside non-whitespace separatorsz[\[\("']z[\]\)"']z[-~\.]z
|
|||
|
|
^https?://c óŠ •— t | j i ‰j ‰j ‰j ‰j ¬« S )N)Ú
|
|||
|
|
prefix_searchÚ
|
|||
|
|
suffix_searchÚinfix_finditerÚ token_match)r ÚvocabÚsearchÚfinditerÚmatch)rQ Úinfix_reÚ prefix_reÚ
|
|||
|
|
simple_url_reÚ suffix_res €€€€r Ú my_tokenizerz$test_issue1488.<locals>.my_tokenizerŽ s> ø€ ÜØ �I‰IØ Ø#×*Ñ*Ø#×*Ñ*Ø#×,Ñ,Ø%×+Ñ+ô
|
|||
|
|
|
|||
|
|
ð
|
|||
|
|
r zThis is a test.N©ÚreÚcompiler r7 r/ )r{ rQ r r rw rx ry rz s @@@@r Útest_issue1488r † sy û€ ô —
|
|||
|
|
‘
|
|||
|
|
˜?Ó+€IÜ—
|
|||
|
|
‘
|
|||
|
|
˜?Ó+€IÜ�z‰z˜-Ó(€HÜ—J‘JÐ 0Ó1€M÷
|
|||
|
|
ô ‹)€CÙ Ó%€C„MÙ
|
|||
|
|
|
|||
|
|
Ð Ó
|
|||
|
|
€CÛˆØ�z‹zЈzñ r iÖ c óò ‡— t j d« Šdg d¢fdddgfdg d¢fg} ˆfd „}t « } ||« |_ | D ]* \ }} ||« D �cg c] }|j ‘Œ c}|k( rŒ*J ‚ y
|
|||
|
|
c c}w ) z&Test if infix_finditer works correctlyz[^a-z]z
|
|||
|
|
token 123test)r rf rM Ú3Útestz token 1testr Ú1testz hello...test)r ú.r„ r„ r‚ c óH •— t | j i ‰j ¬« S )N)rq )r rs ru )rQ rw s €r Ú
|
|||
|
|
new_tokenizerz%test_issue1494.<locals>.new_tokenizer© s ø€ ܘŸ™ B°x×7HÑ7HÔIÐIr Nr| )Ú
|
|||
|
|
test_casesr† rQ r/ Úexpectedr rw s @r Útest_issue1494r‰ Ÿ sŠ ø€ ô �z‰z˜-Ó(€Hà Ò:Ð;Ø ˜ 'Ð*Ð+Ø Ò9Ð:ð€Jô Jô ‹)€CÙ! #Ó&€C„MÛ$‰ˆˆhÙ(+¨D¬ Ó2© ˜u�—
|
|||
|
|
“
|
|||
|
|
¨ Ñ2°hÓ>Ð>Ð>ñ %ùÚ2s ÁA4zJCan not be fixed without iterative looping between prefix/suffix and infixi c óH — t « } | d« }t |« dk( sJ ‚y)zRTest that checks that a dot followed by a quote is handled
|
|||
|
|
appropriately.
|
|||
|
|
z.First sentence."A quoted sentence" he said ...é N)r r. ©rQ r s r Útest_issue2070r� ² s& € ô ‹)€CÙ
|
|||
|
|
|
|||
|
|
Ð>Ó
|
|||
|
|
?€CÜ ˆs‹8�rŠ>Љ>r in c ót — | d« }t |« dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j d
|
|||
|
|
k( sJ ‚|d j dk( sJ ‚|d j d
|
|||
|
|
k( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚y)zdTest that the tokenizer correctly splits tokens separated by a slash (/)
|
|||
|
|
ending in a digit.
|
|||
|
|
z"Learn html5/css3/javascript/jqueryé r ÚLearnrL Úhtml5r? ú/r@ Úcss3rP rJ Ú
|
|||
|
|
javascripté é ÚjqueryNr- )Ú fr_tokenizerr s r Útest_issue2926r™ Â sÏ € ñ
|
|||
|
|
Ð;Ó
|
|||
|
|
<€CÜ ˆs‹8�qŠ=Ј=Ø ˆq‰6�;‰;˜'Ò !Ð!Ð !Ø ˆq‰6�;‰;˜'Ò !Ð!Ð !Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜&Ò Ð Ð Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜,Ò &Ð&Ð &Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜(Ò "Ð"Ñ "r r/ u ABLEItemColumn IAcceptance Limits of ErrorIn-Service Limits of ErrorColumn IIColumn IIIColumn IVColumn VComputed VolumeUnder Registration of VolumeOver Registration of VolumeUnder Registration of VolumeOver Registration of VolumeCubic FeetCubic FeetCubic FeetCubic FeetCubic Feet1Up to 10.0100.0050.0100.005220.0200.0100.0200.010350.0360.0180.0360.0184100.0500.0250.0500.0255Over 100.5% of computed volume0.25% of computed volume0.5% of computed volume0.25% of computed volume TABLE ItemColumn IAcceptance Limits of ErrorIn-Service Limits of ErrorColumn IIColumn IIIColumn IVColumn VComputed VolumeUnder Registration of VolumeOver Registration of VolumeUnder Registration of VolumeOver Registration of VolumeCubic FeetCubic FeetCubic FeetCubic FeetCubic Feet1Up to 10.0100.0050.0100.005220.0200.0100.0200.010350.0360.0180.0360.0184100.0500.0250.0500.0255Over 100.5% of computed volume0.25% of computed volume0.5% of computed volume0.25% of computed volume ItemColumn IAcceptance Limits of ErrorIn-Service Limits of ErrorColumn IIColumn IIIColumn IVColumn VComputed VolumeUnder Registration of VolumeOver Registration of VolumeUnder Registration of VolumeOver Registration of VolumeCubic FeetCubic FeetCubic FeetCubic FeetCubic Feet1Up to 10.0100.0050.0100.005220.0200.0100.0200.010350.0360.0180.0360.0184100.0500.0250.0500.0255Over 100.5% of computed volume0.25% of computed volume0.5% of computed volume0.25% of computed volumezØoow.jspsearch.eventoracleopenworldsearch.technologyoraclesolarissearch.technologystoragesearch.technologylinuxsearch.technologyserverssearch.technologyvirtualizationsearch.technologyengineeredsystemspcodewwmkmppscem:iB
|
|||
|
|
c ó — | |« }|sJ ‚y)zDCheck that sentence doesn't cause an infinite loop in the tokenizer.N© )r0 r/ r s r Útest_issue2626_2835rœ Ó s € ñ �tÓ
|
|||
|
|
€CÙ €J‰3r i`
|
|||
|
|
c óì — | d« }t |« dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j d
|
|||
|
|
k( sJ ‚|d j d k( sJ ‚|d
|
|||
|
|
j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚y)z`Test that tokenizer correctly splits off punctuation after numbers with
|
|||
|
|
decimal points.
|
|||
|
|
z&I went for 40.3, and got home by 10.0.r‹ r ÚIrL Úwentr? Úforr@ z40.3rP ú,rJ r r• Úgotr– Úhomer� Úbyé z10.0é
|
|||
|
|
r„ Nr- )r0 r s r Útest_issue2656r§ á s € ñ
|
|||
|
|
Ð?Ó
|
|||
|
|
@€CÜ ˆs‹8�rŠ>Ј>Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜&Ò Ð Ð Ø ˆq‰6�;‰;˜%Ò Ð Ð Ø ˆq‰6�;‰;˜&Ò Ð Ð Ø ˆq‰6�;‰;˜#Ò Ð Ð Ø ˆq‰6�;‰;˜%Ò Ð Ð Ø ˆq‰6�;‰;˜%Ò Ð Ð Ø ˆq‰6�;‰;˜&Ò Ð Ð Ø ˆq‰6�;‰;˜$Ò Ð Ð Ø ˆq‰6�;‰;˜&Ò Ð Ð Ø ˆr‰7�<‰<˜3Ò Ð Ñ r iÂ
|
|||
|
|
c ót — | d« }|d j dk( sJ ‚ | d« }|d j dk( sJ ‚y)zFTest that words like 'a' and 'a.m.' don't get exceptional norm values.r[ r ÚamN)Únorm_)r0 r[ r© s r Útest_issue2754r« õ sD € ñ �SÓ€AØ ˆQ‰4�:‰:˜Ò Ð Ð Ù �dÓ €BØ
|
|||
|
|
ˆa‰5�;‰;˜$Ò Ð Ñ r iº c óH — t « } | d« }t |« dk( sJ ‚y)z;Test that the tokenizer doesn't hang on a long list of dotszW880.794.982.218.444.893.023.439.794.626.120.190.780.624.990.275.671 ist eine lange ZahlrJ N)r r. rŒ s r Útest_issue3002r þ s* € ô ‹(€CÙ
|
|||
|
|
|
|||
|
|
Øaó €Cô ˆs‹8�qŠ=Љ=r z;default suffix rules avoid one upper-case letter before dotiy
|
|||
|
|
c óî — t « } | j d« d}d}d} | |« } | |« } | |« }|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚y )NÚ sentencizerz>He gave the ball to I. Do you want to go to the movies with I?z?He gave the ball to I. Do you want to go to the movies with I?z>He gave the ball to I.
|
|||
|
|
Do you want to go to the movies with I?rJ rž )r Úadd_piper/ )rQ Útext1Útext2Útext3Út1Út2Út3s r Útest_issue3449r· sƒ € ô ‹)€C؇L�L� Ô Ø L€EØ M€EØ M€EÙ ˆU‹€BÙ ˆU‹€BÙ ˆU‹€BØ
|
|||
|
|
ˆa‰5�:‰:˜Ò Ð Ð Ø
|
|||
|
|
ˆa‰5�:‰:˜Ò Ð Ð Ø
|
|||
|
|
ˆa‰5�:‰:˜Ò Ð Ñ r z
|
|||
|
|
text,wordszA'B C)ÚAú'ÚBÚCzA-Bc óD — | |« }t j |d|i« y )Nr_ )r Ú from_dict)r0 r/ r_ r s r Útest_gold_misalignedr¾ s# € ñ �tÓ
|
|||
|
|
€CÜ ×Ñ�c˜G UÐ+Õ,r c ó4 — | d« }t |« dk( sJ ‚y )NrU r ©r. )r7 r1 s r Ú test_tokenizer_handles_no_wordrÁ s € Ù
|
|||
|
|
�r‹]€FÜ ˆv‹;˜!Ò ÐÑ r Úloremc ó< — | |« }|d j |k( sJ ‚y )Nr ©r/ ©r7 r/ r1 s r Ú"test_tokenizer_handles_single_wordrÆ % s! € á
|
|||
|
|
�t‹_€FØ �!‰9�>‰>˜TÒ !Ð!Ñ !r c óØ — d} | |« }t |« dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚|d j dk7 sJ ‚y ) Nz
|
|||
|
|
Lorem, ipsum.rP r ÚLoremrL r¡ r? Úipsumr- rÅ s r Ú test_tokenizer_handles_punctrÊ + sz € Ø €DÙ
|
|||
|
|
�t‹_€FÜ ˆv‹;˜!Ò ÐÐ Ø �!‰9�>‰>˜WÒ $Ð$Ð $Ø �!‰9�>‰>˜SÒ Ð Ð Ø �!‰9�>‰>˜WÒ $Ð$Ð $Ø �!‰9�>‰>˜WÒ $Ð$Ñ $r c ó8 — d} | |« }t |« dk( sJ ‚y )NzLorem, (ipsum).r• rÀ rÅ s r Ú#test_tokenizer_handles_punct_bracesrÌ 5 s" € Ø €DÙ
|
|||
|
|
�t‹_€FÜ ˆv‹;˜!Ò ÐÑ r c ó´ — ddg}d} | |« }|d j |vr9t |« dk( sJ ‚|d j dk( sJ ‚|d j dk( sJ ‚y y ) NÚhuÚbnzLorem ipsum: 1984.r rJ rÈ r@ Ú1984)Úlang_r. r/ )r7 Ú
|
|||
|
|
exceptionsr/ r1 s r Ú test_tokenizer_handles_digitsrÓ ; sl € ؘ� €JØ €DÙ
|
|||
|
|
�t‹_€Fà
|
|||
|
|
ˆa�y‡�˜jÑ(Ü�6‹{˜aÒ Ð Ð Ø�a‰y�~‰~ Ò(Ð(Ð(Ø�a‰y�~‰~ Ò'Ð'Ñ'ð )r )z
|
|||
|
|
google.comz
|
|||
|
|
python.orgzspacy.ioz explosion.aizhttp://www.google.comc ó4 — | |« }t |« dk( sJ ‚y ©NrL rÀ rÅ s r Útest_tokenizer_keep_urlsrÖ F s € ñ
|
|||
|
|
�t‹_€FÜ ˆv‹;˜!Ò ÐÑ r z NASDAQ:GOOGc ó4 — | |« }t |« dk( sJ ‚y )Nr@ rÀ rÅ s r Útest_tokenizer_colonsrØ O s € á
|
|||
|
|
�t‹_€FÜ ˆv‹;˜!Ò ÐÑ r )zhello123@example.comzhi+there@gmail.itzmatt@explosion.aic ó4 — | |« }t |« dk( sJ ‚y rÕ rÀ rÅ s r Útest_tokenizer_keeps_emailrÚ U s € ñ �t‹_€FÜ ˆv‹;˜!Ò ÐÑ r c ó8 — d} | |« }t |« dkD sJ ‚y )NaÓ Lorem ipsum dolor sit amet, consectetur adipiscing elit
|
|||
|
|
|
|||
|
|
Cras egestas orci non porttitor maximus.
|
|||
|
|
Maecenas quis odio id dolor rhoncus dignissim. Curabitur sed velit at orci ultrices sagittis. Nulla commodo euismod arcu eget vulputate.
|
|||
|
|
|
|||
|
|
Phasellus tincidunt, augue quis porta finibus, massa sapien consectetur augue, non lacinia enim nibh eget ipsum. Vestibulum in bibendum mauris.
|
|||
|
|
|
|||
|
|
"Nullam porta fringilla enim, a dictum orci consequat in." Mauris nec malesuada justo.rJ rÀ rÅ s r Ú test_tokenizer_handles_long_textrÜ ] s% € ð Z€Dñ �t‹_€FÜ ˆv‹;˜Š?Љ?r Ú file_namezsun.txtc óü — t t « j |z }|j dd¬« 5 }|j « }d d d « t « dk7 sJ ‚ | |« }t |« dkD sJ ‚y # 1 sw Y Œ2xY w)NÚrÚutf8)Úencodingr éd )r Ú__file__ÚparentÚopenÚreadr. )r7 rÝ ÚlocÚinfiler/ r1 s r Ú$test_tokenizer_handle_text_from_fileré k sm € ä
|
|||
|
|
”hÓ
|
|||
|
|
×
|
|||
|
|
&Ñ
|
|||
|
|
&¨Ñ
|
|||
|
|
2€CØ �‰�# ˆÔ '¨6Ø�{‰{‹}ˆ÷
|
|||
|
|
(ä ˆt‹9˜Š>Ј>Ù
|
|||
|
|
�t‹_€FÜ ˆv‹;˜Ò Ð Ñ ÷
|
|||
|
|
(Ð 'ús °A2Á2A;c ó| — d}d} | |« } | |« }|d j dk( sJ ‚|d j dk( sJ ‚y )Nz2Lorem dolor sit amet, consectetur adipiscing elit.z8Lorem ipsum dolor sit amet, consectetur adipiscing elit.r rÈ rÄ )r7 r± r² Útokens1Útokens2s r Ú(test_tokenizer_suspected_freeing_stringsrí u sL € Ø @€EØ F€EÙ˜Ó €GÙ˜Ó €GØ �1‰:�?‰?˜gÒ %Ð%Ð %Ø �1‰:�?‰?˜gÒ %Ð%Ñ %r ÚorthÚloÚremc ó — | j ||« | |« }|d j |d d k( sJ ‚|d j |d d k( sJ ‚y )Nr rî rL ©r8 r/ )r7 r/ r1 r s r Ú test_tokenizer_add_special_caseró ~ sY € à
|
|||
|
|
× Ñ ˜t VÔ,Ù
|
|||
|
|
�D‹/€CØ ˆq‰6�;‰;˜& ™) FÑ+Ò +Ð+Ð +Ø ˆq‰6�;‰;˜& ™) FÑ+Ò +Ð+Ñ +r r} r¸ )rî Útagc ó„ — t j t « 5 | j ||« d d d « y # 1 sw Y y xY w)N)rh ri Ú
|
|||
|
|
ValueErrorr8 rÅ s r Ú$test_tokenizer_validate_special_caser÷ † s, € ô
|
|||
|
|
� ‰ ”zÕ "Ø×"Ñ" 4¨Ô0÷
|
|||
|
|
#× "Ñ "ús š6¶?ÚLO)rî Únormc ó — t « }t |i d d d « }|j | |« || « }|d j |d d k( sJ ‚|d j |d d k( sJ ‚|d j |d d k( sJ ‚y )Nr rî rù rL )r
|