14 lines
2.3 KiB
Plaintext
14 lines
2.3 KiB
Plaintext
|
o
|
|||
|
|
{<7B>g<EFBFBD> <00> @ s d dl Z d dlmZ d dlmZmZmZ ddlmZm Z ddl
|
|||
|
|
m Z m Z m
Z
de jfd d
|
|||
|
|
<EFBFBD>Zdd eeee f d eeee f fd
d<0E>Zdd eeeef d eeee f fdd<10>Zd eeee f d eeee f fdd<12>Zd eeee f d eeee f fdd<14>Zdd efdd<16>ZdS )<18> N)<01> DataCollatorForLanguageModeling)<03>List<73>Dict<63>Any<6E> )<02>train_transform<72>inference_transform<72> )<03>
|
|||
|
|
MIN_HEIGHT<EFBFBD> MIN_WIDTH<54>MAX_TOKEN_SIZE<5A>xc C sX t | j<01>dks J d<02><01>t<02>| <00>}| d d <00>dd <00>f |d d <00>d d<04>f< ||d d <00>df< |S )N<> zx should be 2-dimensionalr <00><><EFBFBD><EFBFBD><EFBFBD>)<04>len<65>shape<70>torch<63> ones_like)r
Zpad_valZlefted_x<5F> r <00>J/Users/Leehy/Code/TexTeller/texteller/models/ocr_model/utils/functional.py<70> left_move s
|
|||
|
|
|
|||
|
|
$r <00>samples<65>returnc C s0 |d usJ d<01><01>|| d dd<04>}| d |d< |S )N<> tokenizer should not be None<6E>
latex_formulaT)<01>return_special_tokens_mask<73>image<67> pixel_valuesr )r <00> tokenizerZtokenized_formular r r <00> tokenize_fn s r c C s<> |d usJ d<01><01>dd<03> | D <00>}t |dd<05>}|| <00>}||d< |<04>d<07>|d< |<04>d <09>|d
|
|||
|
|
< t|d d <0C>|d < tj|d d
d<0E>|d< |S )Nr c S s g | ]}|<01> d <00><01>qS )r )<01>pop)<02>.0<EFBFBD>dicr r r <00>
|
|||
|
|
<listcomp> s z collate_fn.<locals>.<listcomp>F)r <00>mlmr <00> input_ids<64>decoder_input_ids<64>attention_mask<73>decoder_attention_mask<73>labelsi<73><69><EFBFBD><EFBFBD>r )<01>dim)r r r r <00>stack)r r r Z clm_collator<6F>batchr r r <00>
|
|||
|
|
collate_fn s r- c C <00> t | d <00>}|| d< | S <00>Nr )r <00>r Z
processed_imgr r r <00>img_train_transform+ <00> r1 c C r. r/ )r r0 r r r <00>img_inf_transform1 r2 r3 c C s8 | d j tko| d jtkot|| d <00>d <00>td k S )Nr r r% <00>
|
|||
|
|
)<06>heightr
|
|||
|
|
<00>widthr r r )<02>sampler r r r <00> filter_fn7 s <02>r8 )N)r <00> transformersr <00>typingr r r <00>
|
|||
|
|
transformsr r <00>globalsr
|
|||
|
|
r r <00>Tensorr <00>strr r- r1 r3 <00>boolr8 r r r r <00><module> s ,,**
|