-
-
https://github.com/OleehyO/TexTeller/assets/56267907/532d1471-a72e-4960-9677-ec6c19db289f
-TexTeller is an end-to-end formula recognition model based on [TrOCR](https://arxiv.org/abs/2109.10282), capable of converting images into corresponding LaTeX formulas.
+TexTeller is an end-to-end formula recognition model, capable of converting images into corresponding LaTeX formulas.
TexTeller was trained with **80M image-formula pairs** (previous dataset can be obtained [here](https://huggingface.co/datasets/OleehyO/latex-formulas)), compared to [LaTeX-OCR](https://github.com/lukas-blecher/LaTeX-OCR) which used a 100K dataset, TexTeller has **stronger generalization abilities** and **higher accuracy**, covering most use cases.
>[!NOTE]
> If you would like to provide feedback or suggestions for this project, feel free to start a discussion in the [Discussions section](https://github.com/OleehyO/TexTeller/discussions).
->
-> Additionally, if you find this project helpful, please don't forget to give it a star⭐️🙏️
---
@@ -55,15 +31,12 @@ TexTeller was trained with **80M image-formula pairs** (previous dataset can be
@@ -76,18 +49,9 @@ TexTeller was trained with **80M image-formula pairs** (previous dataset can be
-
- Thanks to the
-
- Super Computing Platform of Beijing University of Posts and Telecommunications
-
- for supporting this work😘
-
-
-
@@ -110,153 +74,118 @@ TexTeller was trained with **80M image-formula pairs** (previous dataset can be
## 🚀 Getting Started
-1. Clone the repository:
-
- ```bash
- git clone https://github.com/OleehyO/TexTeller
- ```
-
-2. Install the project's dependencies:
+1. Install the project's dependencies:
```bash
pip install texteller
```
-3. Enter the `src/` directory and run the following command in the terminal to start inference:
+2. If your are using CUDA backend, you may need to install `onnxruntime-gpu`:
```bash
- python inference.py -img "/path/to/image.{jpg,png}"
- # use --inference-mode option to enable GPU(cuda or mps) inference
- #+e.g. python inference.py -img "img.jpg" --inference-mode cuda
+ pip install texteller[onnxruntime-gpu]
```
- > The first time you run it, the required checkpoints will be downloaded from Hugging Face.
-
-### Paragraph Recognition
-
-As demonstrated in the video, TexTeller is also capable of recognizing entire text paragraphs. Although TexTeller has general text OCR capabilities, we still recommend using paragraph recognition for better results:
-
-1. [Download the weights](https://huggingface.co/TonyLee1256/texteller_det/resolve/main/rtdetr_r50vd_6x_coco.onnx?download=true) of the formula detection model to the`src/models/det_model/model/`directory
-
-2. Run `inference.py` in the `src/` directory and add the `-mix` option, the results will be output in markdown format.
+3. Run the following command to start inference:
```bash
- python inference.py -img "/path/to/image.{jpg,png}" -mix
+ texteller inference "/path/to/image.{jpg,png}"
```
-TexTeller uses the lightweight [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) model by default for recognizing both Chinese and English text. You can try using a larger model to achieve better recognition results for both Chinese and English:
-
-| Checkpoints | Model Description | Size |
-|-------------|-------------------| ---- |
-| [ch_PP-OCRv4_det.onnx](https://huggingface.co/OleehyO/paddleocrv4.onnx/resolve/main/ch_PP-OCRv4_det.onnx?download=true) | **Default detection model**, supports Chinese-English text detection | 4.70M |
-| [ch_PP-OCRv4_server_det.onnx](https://huggingface.co/OleehyO/paddleocrv4.onnx/resolve/main/ch_PP-OCRv4_server_det.onnx?download=true) | High accuracy model, supports Chinese-English text detection | 115M |
-| [ch_PP-OCRv4_rec.onnx](https://huggingface.co/OleehyO/paddleocrv4.onnx/resolve/main/ch_PP-OCRv4_rec.onnx?download=true) | **Default recoginition model**, supports Chinese-English text recognition | 10.80M |
-| [ch_PP-OCRv4_server_rec.onnx](https://huggingface.co/OleehyO/paddleocrv4.onnx/resolve/main/ch_PP-OCRv4_server_rec.onnx?download=true) | High accuracy model, supports Chinese-English text recognition | 90.60M |
-
-Place the weights of the recognition/detection model in the `det/` or `rec/` directories within `src/models/third_party/paddleocr/checkpoints/`, and rename them to `default_model.onnx`.
-
-> [!NOTE]
-> Paragraph recognition cannot restore the structure of a document, it can only recognize its content.
+ > See `texteller inference --help` for more details
## 🌐 Web Demo
-Go to the `src/` directory and run the following command:
+Run the following command:
```bash
-./start_web.sh
+texteller web
```
Enter `http://localhost:8501` in a browser to view the web demo.
> [!NOTE]
-> 1. For Windows users, please run the `start_web.bat` file.
-> 2. When using onnxruntime + GPU for inference, you need to install onnxruntime-gpu.
+> Paragraph recognition cannot restore the structure of a document, it can only recognize its content.
-## 🔍 Formula Detection
+## 🖥️ Server
-TexTeller’s formula detection model is trained on 3,415 images of Chinese educational materials (with over 130 layouts) and 8,272 images from the [IBEM dataset](https://zenodo.org/records/4757865), and it supports formula detection across entire images.
-
-
-
-
-
-1. Download the model weights and place them in `src/models/det_model/model/` [[link](https://huggingface.co/TonyLee1256/texteller_det/resolve/main/rtdetr_r50vd_6x_coco.onnx?download=true)].
-
-2. Run the following command in the `src/` directory, and the results will be saved in `src/subimages/`
-
-
-Advanced: batch formula recognition
-
-After **formula detection**, run the following command in the `src/` directory:
-
-```shell
-python rec_infer_from_crop_imgs.py
-```
-
-This will use the results of the previous formula detection to perform batch recognition on all cropped formulas, saving the recognition results as txt files in `src/results/`.
-
-
-
-## 📡 API Usage
-
-We use [ray serve](https://github.com/ray-project/ray) to provide an API interface for TexTeller, allowing you to integrate TexTeller into your own projects. To start the server, you first need to enter the `src/` directory and then run the following command:
+We use [ray serve](https://github.com/ray-project/ray) to provide an API server for TexTeller. To start the server, run the following command:
```bash
-python server.py
+texteller launch
```
| Parameter | Description |
| --------- | -------- |
| `-ckpt` | The path to the weights file,*default is TexTeller's pretrained weights*. |
| `-tknz` | The path to the tokenizer,*default is TexTeller's tokenizer*. |
-| `-port` | The server's service port,*default is 8000*. |
-| `--inference-mode` | Whether to use "cuda" or "mps" for inference,*default is "cpu"*. |
-| `--num_beams` | The number of beams for beam search,*default is 1*. |
-| `--num_replicas` | The number of service replicas to run on the server,*default is 1 replica*. You can use more replicas to achieve greater throughput.|
-| `--ncpu_per_replica` | The number of CPU cores used per service replica,*default is 1*.|
-| `--ngpu_per_replica` | The number of GPUs used per service replica,*default is 1*. You can set this value between 0 and 1 to run multiple service replicas on one GPU to share the GPU, thereby improving GPU utilization. (Note, if --num_replicas is 2, --ngpu_per_replica is 0.7, then 2 GPUs must be available) |
-| `-onnx` | Perform inference using Onnx Runtime, *disabled by default* |
+| `-p` | The server's service port,*default is 8000*. |
+| `--num-replicas` | The number of service replicas to run on the server,*default is 1 replica*. You can use more replicas to achieve greater throughput.|
+| `--ncpu-per-replica` | The number of CPU cores used per service replica,*default is 1*.|
+| `--ngpu-per-replica` | The number of GPUs used per service replica,*default is 1*. You can set this value between 0 and 1 to run multiple service replicas on one GPU to share the GPU, thereby improving GPU utilization. (Note, if --num_replicas is 2, --ngpu_per_replica is 0.7, then 2 GPUs must be available) |
+| `--num-beams` | The number of beams for beam search,*default is 1*. |
+| `--use-onnx` | Perform inference using Onnx Runtime, *disabled by default* |
-> [!NOTE]
-> A client demo can be found at `src/client/demo.py`, you can refer to `demo.py` to send requests to the server
+To send requests to the server:
+
+```python
+# client_demo.py
+
+import requests
+
+server_url = "http://127.0.0.1:8000/predict"
+
+img_path = "/path/to/your/image"
+with open(img_path, 'rb') as img:
+ files = {'img': img}
+ response = requests.post(server_url, files=files)
+
+print(response.text)
+```
+
+## 🐍 Python API
+
+We provide several easy-to-use Python APIs for formula OCR scenarios. Please refer to our [documentation](https://oleehyo.github.io/TexTeller/) to learn about the corresponding API interfaces and usage.
+
+## 🔍 Formula Detection
+
+TexTeller's formula detection model is trained on 3,415 images of Chinese materials and 8,272 images from the [IBEM dataset](https://zenodo.org/records/4757865).
+
+
+
+
+
+We provide a formula detection interface in the Python API. Please refer to our [API documentation](https://oleehyo.github.io/TexTeller/) for more details.
## 🏋️♂️ Training
-### Dataset
+Please setup your environment before training:
-We provide an example dataset in the `src/models/ocr_model/train/dataset/` directory, you can place your own images in the `images/` directory and annotate each image with its corresponding formula in `formulas.jsonl`.
-
-After preparing your dataset, you need to **change the `DIR_URL` variable to your own dataset's path** in `**/train/dataset/loader.py`
-
-### Retraining the Tokenizer
-
-If you are using a different dataset, you might need to retrain the tokenizer to obtain a different vocabulary. After configuring your dataset, you can train your own tokenizer with the following command:
-
-1. In `src/models/tokenizer/train.py`, change `new_tokenizer.save_pretrained('./your_dir_name')` to your custom output directory
-
- > If you want to use a different vocabulary size (default 15K), you need to change the `VOCAB_SIZE` variable in `src/models/globals.py`
- >
-2. **In the `src/` directory**, run the following command:
+1. Install the dependencies for training:
```bash
- python -m models.tokenizer.train
+ pip install texteller[train]
```
+2. Clone the repository:
+
+ ```bash
+ git clone https://github.com/OleehyO/TexTeller.git
+ ```
+
+### Dataset
+
+We provide an example dataset in the `examples/train_texteller/dataset/train` directory, you can place your own training data according to the format of the example dataset.
+
### Training the Model
-1. Modify `num_processes` in `src/train_config.yaml` to match the number of GPUs available for training (default is 1).
-2. In the `src/` directory, run the following command:
+In the `examples/train_texteller/` directory, run the following command:
```bash
- accelerate launch --config_file ./train_config.yaml -m models.ocr_model.train.train
+ accelerate launch train.py
```
-You can set your own tokenizer and checkpoint paths in `src/models/ocr_model/train/train.py` (refer to `train.py` for more information). If you are using the same architecture and vocabulary as TexTeller, you can also fine-tune TexTeller's default weights with your own dataset.
-
-In `src/globals.py` and `src/models/ocr_model/train/train_args.py`, you can change the model's architecture and training hyperparameters.
-
-> [!NOTE]
-> Our training scripts use the [Hugging Face Transformers](https://github.com/huggingface/transformers) library, so you can refer to their [documentation](https://huggingface.co/docs/transformers/v4.32.1/main_classes/trainer#transformers.TrainingArguments) for more details and configurations on training parameters.
+Training arguments can be adjusted in [`train_config.yaml`](./examples/train_texteller/train_config.yaml).
## 📅 Plans
@@ -266,13 +195,11 @@ In `src/globals.py` and `src/models/ocr_model/train/train_args.py`, you can chan
- [X] ~~Handwritten formulas support~~
- [ ] PDF document recognition
- [ ] Inference acceleration
-- [ ] ...
## ⭐️ Stargazers over time
[](https://starchart.cc/OleehyO/TexTeller)
-
## 👥 Contributors
diff --git a/assets/README_zh.md b/assets/README_zh.md
index 4dee0f7..d696480 100644
--- a/assets/README_zh.md
+++ b/assets/README_zh.md
@@ -1,52 +1,28 @@
-📄 English | 中文
+📄 中文 | [English](./README.md)
+ 
-
+ 
+