lol

.gitattributes

@@ -1,31 +0,0 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,12 +1,136 @@
----
-title: Pixelplanetocr
-emoji: 🦀
-colorFrom: gray
-colorTo: purple
-sdk: gradio
-sdk_version: 3.3.1
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# IterVM: Iterative Vision Modeling Module for Scene Text Recognition
+
+The official code of [IterNet](https://arxiv.org/abs/2204.02630).
+
+We propose IterVM, an iterative approach for visual feature extraction which can significantly improve scene text recognition accuracy.
+IterVM repeatedly uses the high-level visual feature extracted at the previous iteration to enhance the multi-level features extracted at the subsequent iteration.
+
+
+![framework](./figures/framework.png)
+
+
+## Runtime Environment
+```
+pip install -r requirements.txt
+```
+Note: `fastai==1.0.60` is required.
+
+## Datasets
+<details>
+  <summary>Training datasets (Click to expand) </summary>
+    1. [MJSynth](http://www.robots.ox.ac.uk/~vgg/data/text/) (MJ): 
+        - Use `tools/create_lmdb_dataset.py` to convert images into LMDB dataset
+        - [LMDB dataset BaiduNetdisk(passwd:n23k)](https://pan.baidu.com/s/1mgnTiyoR8f6Cm655rFI4HQ)
+    2. [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) (ST):
+        - Use `tools/crop_by_word_bb.py` to crop images from original [SynthText](http://www.robots.ox.ac.uk/~vgg/data/scenetext/) dataset, and convert images into LMDB dataset by `tools/create_lmdb_dataset.py`
+        - [LMDB dataset BaiduNetdisk(passwd:n23k)](https://pan.baidu.com/s/1mgnTiyoR8f6Cm655rFI4HQ)
+    3. [WikiText103](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-v1.zip), which is only used for pre-trainig language models:
+        - Use `notebooks/prepare_wikitext103.ipynb` to convert text into CSV format.
+        - [CSV dataset BaiduNetdisk(passwd:dk01)](https://pan.baidu.com/s/1yabtnPYDKqhBb_Ie9PGFXA)
+</details>
+
+<details>
+  <summary>Evaluation datasets (Click to expand) </summary>
+- Evaluation datasets, LMDB datasets can be downloaded from [BaiduNetdisk(passwd:1dbv)](https://pan.baidu.com/s/1RUg3Akwp7n8kZYJ55rU5LQ), [GoogleDrive](https://drive.google.com/file/d/1dTI0ipu14Q1uuK4s4z32DqbqF3dJPdkk/view?usp=sharing).
+    1. ICDAR 2013 (IC13)
+    2. ICDAR 2015 (IC15)
+    3. IIIT5K Words (IIIT)
+    4. Street View Text (SVT)
+    5. Street View Text-Perspective (SVTP)
+    6. CUTE80 (CUTE)
+</details>
+
+<details>
+  <summary>The structure of `data` directory (Click to expand) </summary>
+- The structure of `data` directory is
+    ```
+    data
+    ├── charset_36.txt
+    ├── evaluation
+    │   ├── CUTE80
+    │   ├── IC13_857
+    │   ├── IC15_1811
+    │   ├── IIIT5k_3000
+    │   ├── SVT
+    │   └── SVTP
+    ├── training
+    │   ├── MJ
+    │   │   ├── MJ_test
+    │   │   ├── MJ_train
+    │   │   └── MJ_valid
+    │   └── ST
+    ├── WikiText-103.csv
+    └── WikiText-103_eval_d1.csv
+    ```
+</details>
+
+## Pretrained Models
+
+Get the pretrained models from [GoogleDrive](https://drive.google.com/drive/folders/1C8NMI8Od8mQUMlsnkHNLkYj73kbAQ7Bl?usp=sharing). Performances of the pretrained models are summaried as follows:
+
+|Model|IC13|SVT|IIIT|IC15|SVTP|CUTE|AVG|
+|-|-|-|-|-|-|-|-|
+|IterNet|97.9|95.1|96.9|87.7|90.9|91.3|93.8|
+
+## Training
+
+1. Pre-train vision model
+    ```
+    CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python main.py --config=configs/pretrain_vm.yaml
+    ```
+2. Pre-train language model
+    ```
+    CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --config=configs/pretrain_language_model.yaml
+    ```
+3. Train IterNet
+    ```
+    CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python main.py --config=configs/train_iternet.yaml
+    ```
+Note:
+- You can set the `checkpoint` path for vision model (vm) and language model separately for specific pretrained model, or set to `None` to train from scratch
+
+
+## Evaluation
+
+```
+CUDA_VISIBLE_DEVICES=0 python main.py --config=configs/train_iternet.yaml --phase test --image_only
+```
+Additional flags:
+- `--checkpoint /path/to/checkpoint` set the path of evaluation model 
+- `--test_root /path/to/dataset` set the path of evaluation dataset
+- `--model_eval [alignment|vision]` which sub-model to evaluate
+- `--image_only` disable dumping visualization of attention masks
+
+## Run Demo
+[<a href="https://colab.research.google.com/drive/1XmZGJzFF95uafmARtJMudPLLKBO2eXLv?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>](https://colab.research.google.com/drive/1XmZGJzFF95uafmARtJMudPLLKBO2eXLv?usp=sharing)
+
+```
+python demo.py --config=configs/train_iternet.yaml --input=figures/demo
+```
+Additional flags:
+- `--config /path/to/config` set the path of configuration file 
+- `--input /path/to/image-directory` set the path of image directory or wildcard path, e.g, `--input='figs/test/*.png'`
+- `--checkpoint /path/to/checkpoint` set the path of trained model
+- `--cuda [-1|0|1|2|3...]` set the cuda id, by default -1 is set and stands for cpu
+- `--model_eval [alignment|vision]` which sub-model to use
+- `--image_only` disable dumping visualization of attention masks
+
+
+## Citation
+If you find our method useful for your reserach, please cite
+```bash 
+@article{chu2022itervm,
+  title={IterVM: Iterative Vision Modeling Module for Scene Text Recognition},
+  author={Chu, Xiaojie and Wang, Yongtao},
+  journal={arXiv preprint arXiv:2204.02630},
+  year={2022}
+}
+ ```
+
+## License
+The project is only free for academic research purposes, but needs authorization for commerce. For commerce permission, please contact [email protected].
+
+## Acknowledgements
+This project is based on [ABINet](https://github.com/FangShancheng/ABINet.git).
+Thanks for their great works.
+

app.py

@@ -0,0 +1,30 @@
+import glob
+import gradio as gr
+from demo import get_model, preprocess, postprocess, load
+from utils import Config, Logger, CharsetMapper
+
+config = Config('configs/train_iternet.yaml')
+config.model_vision_checkpoint = None
+model = get_model(config)
+model = load(model, 'workdir/train-iternet/best-train-iternet.pth')
+charset = CharsetMapper(filename=config.dataset_charset_path, max_length=config.dataset_max_length + 1)
+
+def process_image(image):
+  img = image.convert('RGB')
+  img = preprocess(img, config.dataset_image_width, config.dataset_image_height)
+  res = model(img)
+  return postprocess(res, charset, 'alignment')[0][0]
+
+title = "Interactive demo: ABINet"
+description = "Demo for ABINet, ABINet uses a vision model and an explicit language model to recognize text in the wild, which are trained in end-to-end way. The language model (BCN) achieves bidirectional language representation in simulating cloze test, additionally utilizing iterative correction strategy. To use it, simply upload a (single-text line) image or use one of the example images below and click 'submit'. Results will show up in a few seconds."
+article = "<p style='text-align: center'><a href='https://arxiv.org/pdf/2103.06495.pdf'>Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition</a> | <a href='https://github.com/FangShancheng/ABINet'>Github Repo</a></p>"
+
+iface = gr.Interface(fn=process_image, 
+                     inputs=gr.inputs.Image(type="pil"), 
+                     outputs=gr.outputs.Textbox(),
+                     title=title,
+                     description=description,
+                     article=article,
+                     examples=glob.glob('figs/test/*.png'))
+
+iface.launch(debug=True, share=True,enable_queue=True)

callbacks.py

@@ -0,0 +1,360 @@
+import logging
+import shutil
+import time
+
+import editdistance as ed
+import torchvision.utils as vutils
+from fastai.callbacks.tensorboard import (LearnerTensorboardWriter,
+                                          SummaryWriter, TBWriteRequest,
+                                          asyncTBWriter)
+from fastai.vision import *
+from torch.nn.parallel import DistributedDataParallel
+from torchvision import transforms
+
+import dataset
+from utils import CharsetMapper, Timer, blend_mask
+
+
+class IterationCallback(LearnerTensorboardWriter):
+    "A `TrackerCallback` that monitor in each iteration."
+    def __init__(self, learn:Learner, name:str='model', checpoint_keep_num=5,
+                 show_iters:int=50, eval_iters:int=1000, save_iters:int=20000,
+                 start_iters:int=0, stats_iters=20000):
+        #if self.learn.rank is not None: time.sleep(self.learn.rank)  # keep all event files
+        super().__init__(learn, base_dir='.', name=learn.path, loss_iters=show_iters, 
+                        stats_iters=stats_iters, hist_iters=stats_iters)
+        self.name, self.bestname = Path(name).name, f'best-{Path(name).name}'
+        self.show_iters = show_iters
+        self.eval_iters = eval_iters
+        self.save_iters = save_iters
+        self.start_iters = start_iters
+        self.checpoint_keep_num = checpoint_keep_num
+        self.metrics_root = 'metrics/'  # rewrite
+        self.timer = Timer()
+        self.host = self.learn.rank is None or self.learn.rank == 0
+
+    def _write_metrics(self, iteration:int, names:List[str], last_metrics:MetricsList)->None:
+        "Writes training metrics to Tensorboard."
+        for i, name in enumerate(names):
+            if last_metrics is None or len(last_metrics) < i+1: return
+            scalar_value = last_metrics[i]
+            self._write_scalar(name=name, scalar_value=scalar_value, iteration=iteration)
+
+    def _write_sub_loss(self, iteration:int, last_losses:dict)->None:
+        "Writes sub loss to Tensorboard."
+        for name, loss in last_losses.items():
+            scalar_value = to_np(loss)
+            tag = self.metrics_root + name
+            self.tbwriter.add_scalar(tag=tag, scalar_value=scalar_value, global_step=iteration)
+
+    def _save(self, name):
+        if isinstance(self.learn.model, DistributedDataParallel):
+            tmp = self.learn.model
+            self.learn.model = self.learn.model.module
+            self.learn.save(name)
+            self.learn.model = tmp
+        else: self.learn.save(name)
+
+    def _validate(self, dl=None, callbacks=None, metrics=None, keeped_items=False):
+        "Validate on `dl` with potential `callbacks` and `metrics`."
+        dl = ifnone(dl, self.learn.data.valid_dl)
+        metrics = ifnone(metrics, self.learn.metrics)
+        cb_handler = CallbackHandler(ifnone(callbacks, []), metrics)
+        cb_handler.on_train_begin(1, None, metrics); cb_handler.on_epoch_begin()
+        if keeped_items: cb_handler.state_dict.update(dict(keeped_items=[]))
+        val_metrics = validate(self.learn.model, dl, self.loss_func, cb_handler)
+        cb_handler.on_epoch_end(val_metrics)
+        if keeped_items: return cb_handler.state_dict['keeped_items']
+        else: return cb_handler.state_dict['last_metrics']
+
+    def jump_to_epoch_iter(self, epoch:int, iteration:int)->None:
+        try:
+            self.learn.load(f'{self.name}_{epoch}_{iteration}', purge=False)
+            logging.info(f'Loaded {self.name}_{epoch}_{iteration}')
+        except: logging.info(f'Model {self.name}_{epoch}_{iteration} not found.')
+
+    def on_train_begin(self, n_epochs, **kwargs):
+        # TODO: can not write graph here
+        # super().on_train_begin(**kwargs)
+        self.best = -float('inf')
+        self.timer.tic()
+        if self.host:
+            checkpoint_path = self.learn.path/'checkpoint.yaml'
+            if checkpoint_path.exists():
+                os.remove(checkpoint_path)
+            open(checkpoint_path, 'w').close()
+        return {'skip_validate': True, 'iteration':self.start_iters}  # disable default validate
+
+    def on_batch_begin(self, **kwargs:Any)->None:
+        self.timer.toc_data()
+        super().on_batch_begin(**kwargs)
+
+    def on_batch_end(self, iteration, epoch, last_loss, smooth_loss, train, **kwargs):
+        super().on_batch_end(last_loss, iteration, train, **kwargs)
+        if iteration == 0: return
+
+        if iteration % self.loss_iters == 0:
+            last_losses = self.learn.loss_func.last_losses
+            self._write_sub_loss(iteration=iteration, last_losses=last_losses)
+            self.tbwriter.add_scalar(tag=self.metrics_root + 'lr',
+                scalar_value=self.opt.lr, global_step=iteration)
+
+        if iteration % self.show_iters == 0:
+            log_str = f'epoch {epoch} iter {iteration}: loss = {last_loss:6.4f},  ' \
+                      f'smooth loss = {smooth_loss:6.4f}'
+            logging.info(log_str)
+            # log_str = f'data time = {self.timer.data_diff:.4f}s, runing time = {self.timer.running_diff:.4f}s'
+            # logging.info(log_str)
+
+        if iteration % self.eval_iters == 0:
+            # TODO: or remove time to on_epoch_end
+            # 1. Record time 
+            log_str = f'average data time = {self.timer.average_data_time():.4f}s, ' \
+                      f'average running time = {self.timer.average_running_time():.4f}s'
+            logging.info(log_str)
+
+            # 2. Call validate
+            last_metrics = self._validate()
+            self.learn.model.train()
+            log_str = f'epoch {epoch} iter {iteration}: eval loss = {last_metrics[0]:6.4f},  ' \
+                      f'ccr = {last_metrics[1]:6.4f},  cwr = {last_metrics[2]:6.4f},  ' \
+                      f'ted = {last_metrics[3]:6.4f},  ned = {last_metrics[4]:6.4f},  ' \
+                      f'ted/w = {last_metrics[5]:6.4f}, '
+            logging.info(log_str)
+            names = ['eval_loss', 'ccr', 'cwr', 'ted', 'ned', 'ted/w']
+            self._write_metrics(iteration, names, last_metrics)
+
+            # 3. Save best model
+            current = last_metrics[2]
+            if current is not None and current > self.best:
+                logging.info(f'Better model found at epoch {epoch}, '\
+                             f'iter {iteration} with accuracy value: {current:6.4f}.')
+                self.best = current
+                self._save(f'{self.bestname}')
+
+        if iteration % self.save_iters == 0 and self.host:
+            logging.info(f'Save model {self.name}_{epoch}_{iteration}')
+            filename = f'{self.name}_{epoch}_{iteration}'
+            self._save(filename)
+
+            checkpoint_path = self.learn.path/'checkpoint.yaml'
+            if not checkpoint_path.exists():
+                open(checkpoint_path, 'w').close()
+            with open(checkpoint_path, 'r') as file:
+                checkpoints = yaml.load(file, Loader=yaml.FullLoader) or dict()
+            checkpoints['all_checkpoints'] = (
+                checkpoints.get('all_checkpoints') or list())
+            checkpoints['all_checkpoints'].insert(0, filename)
+            if len(checkpoints['all_checkpoints']) > self.checpoint_keep_num:
+                removed_checkpoint = checkpoints['all_checkpoints'].pop()
+                removed_checkpoint =  self.learn.path/self.learn.model_dir/f'{removed_checkpoint}.pth'
+                os.remove(removed_checkpoint)
+            checkpoints['current_checkpoint'] = filename
+            with open(checkpoint_path, 'w') as file:
+                yaml.dump(checkpoints, file)
+
+
+        self.timer.toc_running()
+
+    def on_train_end(self, **kwargs):
+        #self.learn.load(f'{self.bestname}', purge=False)
+        pass
+    
+    def on_epoch_end(self, last_metrics:MetricsList, iteration:int, **kwargs)->None:
+        self._write_embedding(iteration=iteration)
+
+
+class TextAccuracy(Callback):
+    _names = ['ccr', 'cwr', 'ted', 'ned', 'ted/w']
+    def __init__(self, charset_path, max_length, case_sensitive, model_eval):
+        self.charset_path = charset_path
+        self.max_length = max_length
+        self.case_sensitive = case_sensitive
+        self.charset = CharsetMapper(charset_path, self.max_length)
+        self.names = self._names
+
+        self.model_eval = model_eval or 'alignment'
+        assert self.model_eval in ['vision', 'language', 'alignment']
+ 
+    def on_epoch_begin(self, **kwargs):
+        self.total_num_char = 0.
+        self.total_num_word = 0.
+        self.correct_num_char = 0.
+        self.correct_num_word = 0.
+        self.total_ed = 0.
+        self.total_ned = 0.
+
+    def _get_output(self, last_output):
+        if isinstance(last_output, (tuple, list)): 
+            for res in last_output:
+                if res['name'] == self.model_eval: output = res
+        else: output = last_output
+        return output
+    
+    def _update_output(self, last_output, items):
+        if isinstance(last_output, (tuple, list)): 
+            for res in last_output:
+                if res['name'] == self.model_eval: res.update(items)
+        else: last_output.update(items)
+        return last_output
+
+    def on_batch_end(self, last_output, last_target, **kwargs):
+        output = self._get_output(last_output)
+        logits, pt_lengths = output['logits'], output['pt_lengths']
+        pt_text, pt_scores, pt_lengths_ = self.decode(logits)
+        assert (pt_lengths == pt_lengths_).all(), f'{pt_lengths} != {pt_lengths_} for {pt_text}'
+        last_output = self._update_output(last_output, {'pt_text':pt_text, 'pt_scores':pt_scores})
+
+        pt_text = [self.charset.trim(t) for t in pt_text]
+        label = last_target[0]
+        if label.dim() == 3: label = label.argmax(dim=-1)  # one-hot label
+        gt_text = [self.charset.get_text(l, trim=True) for l in label]
+        
+        for i in range(len(gt_text)):
+            if not self.case_sensitive:
+                gt_text[i], pt_text[i] = gt_text[i].lower(), pt_text[i].lower()
+            distance = ed.eval(gt_text[i], pt_text[i])
+            self.total_ed += distance
+            self.total_ned += float(distance) / max(len(gt_text[i]), 1)
+
+            if gt_text[i] == pt_text[i]:
+                self.correct_num_word += 1
+            self.total_num_word += 1
+            
+            for j in range(min(len(gt_text[i]), len(pt_text[i]))):
+                if gt_text[i][j] == pt_text[i][j]:
+                    self.correct_num_char += 1
+            self.total_num_char += len(gt_text[i])
+
+        return {'last_output': last_output}
+
+    def on_epoch_end(self, last_metrics, **kwargs):
+        mets = [self.correct_num_char / self.total_num_char,
+                self.correct_num_word / self.total_num_word,
+                self.total_ed,
+                self.total_ned,
+                self.total_ed / self.total_num_word]
+        return add_metrics(last_metrics, mets)
+
+    def decode(self, logit):
+        """ Greed decode """
+        # TODO: test running time and decode on GPU
+        out = F.softmax(logit, dim=2)
+        pt_text, pt_scores, pt_lengths = [], [], []
+        for o in out:
+            text = self.charset.get_text(o.argmax(dim=1), padding=False, trim=False)
+            text = text.split(self.charset.null_char)[0]  # end at end-token
+            pt_text.append(text)
+            pt_scores.append(o.max(dim=1)[0])
+            pt_lengths.append(min(len(text) + 1, self.max_length))  # one for end-token
+        pt_scores = torch.stack(pt_scores)
+        pt_lengths = pt_scores.new_tensor(pt_lengths, dtype=torch.long)
+        return pt_text, pt_scores, pt_lengths
+
+
+class TopKTextAccuracy(TextAccuracy):
+    _names = ['ccr', 'cwr']
+    def __init__(self, k, charset_path, max_length, case_sensitive, model_eval):
+        self.k = k
+        self.charset_path = charset_path
+        self.max_length = max_length
+        self.case_sensitive = case_sensitive
+        self.charset = CharsetMapper(charset_path, self.max_length)
+        self.names = self._names
+ 
+    def on_epoch_begin(self, **kwargs):
+        self.total_num_char = 0.
+        self.total_num_word = 0.
+        self.correct_num_char = 0.
+        self.correct_num_word = 0.
+
+    def on_batch_end(self, last_output, last_target, **kwargs):
+        logits, pt_lengths = last_output['logits'], last_output['pt_lengths']
+        gt_labels, gt_lengths = last_target[:]
+
+        for logit, pt_length, label, length in zip(logits, pt_lengths, gt_labels, gt_lengths):
+            word_flag = True
+            for i in range(length):
+                char_logit = logit[i].topk(self.k)[1]
+                char_label = label[i].argmax(-1)
+                if char_label in char_logit: self.correct_num_char += 1
+                else: word_flag = False
+                self.total_num_char += 1
+            if pt_length == length and word_flag:
+                self.correct_num_word += 1
+            self.total_num_word += 1
+
+    def on_epoch_end(self, last_metrics, **kwargs):
+        mets = [self.correct_num_char / self.total_num_char,
+                self.correct_num_word / self.total_num_word,
+                0., 0., 0.]
+        return add_metrics(last_metrics, mets)
+
+
+class DumpPrediction(LearnerCallback):
+
+    def __init__(self, learn, dataset, charset_path, model_eval, image_only=False, debug=False):
+        super().__init__(learn=learn)
+        self.debug = debug
+        self.model_eval = model_eval or 'alignment'
+        self.image_only = image_only
+        assert self.model_eval in ['vision', 'language', 'alignment']
+
+        self.dataset, self.root = dataset, Path(self.learn.path)/f'{dataset}-{self.model_eval}'
+        self.attn_root = self.root/'attn'
+        self.charset = CharsetMapper(charset_path)
+        if self.root.exists(): shutil.rmtree(self.root)
+        self.root.mkdir(), self.attn_root.mkdir()
+
+        self.pil = transforms.ToPILImage()
+        self.tensor = transforms.ToTensor()
+        size = self.learn.data.img_h, self.learn.data.img_w
+        self.resize = transforms.Resize(size=size, interpolation=0)
+        self.c = 0
+
+    def on_batch_end(self, last_input, last_output, last_target, **kwargs):
+        if isinstance(last_output, (tuple, list)):
+            for res in last_output:
+                if res['name'] == self.model_eval: pt_text = res['pt_text']
+                if res['name'] == 'vision': attn_scores = res['attn_scores'].detach().cpu()
+                if res['name'] == self.model_eval: logits = res['logits']
+        else:
+            pt_text = last_output['pt_text']
+            attn_scores = last_output['attn_scores'].detach().cpu()
+            logits = last_output['logits']
+
+        images = last_input[0] if isinstance(last_input, (tuple, list)) else last_input
+        images = images.detach().cpu()
+        pt_text = [self.charset.trim(t) for t in pt_text]
+        gt_label = last_target[0]
+        if gt_label.dim() == 3: gt_label = gt_label.argmax(dim=-1)  # one-hot label
+        gt_text = [self.charset.get_text(l, trim=True) for l in gt_label]
+
+        prediction, false_prediction = [], []
+        for gt, pt, image, attn, logit in zip(gt_text, pt_text, images, attn_scores, logits):
+            prediction.append(f'{gt}\t{pt}\n')
+            if gt != pt:
+                if self.debug:
+                    scores = torch.softmax(logit, dim=-1)[:max(len(pt), len(gt)) + 1]
+                    logging.info(f'{self.c} gt {gt}, pt {pt}, logit {logit.shape}, scores {scores.topk(5, dim=-1)}')
+                false_prediction.append(f'{gt}\t{pt}\n')
+
+            image = self.learn.data.denorm(image)
+            if not self.image_only:
+                image_np = np.array(self.pil(image))
+                attn_pil = [self.pil(a) for a in attn[:, None, :, :]]
+                attn = [self.tensor(self.resize(a)).repeat(3, 1, 1) for a in attn_pil]
+                attn_sum = np.array([np.array(a) for a in attn_pil[:len(pt)]]).sum(axis=0)
+                blended_sum = self.tensor(blend_mask(image_np, attn_sum))
+                blended = [self.tensor(blend_mask(image_np, np.array(a))) for a in attn_pil]
+                save_image = torch.stack([image] + attn + [blended_sum] + blended)
+                save_image = save_image.view(2, -1, *save_image.shape[1:])
+                save_image = save_image.permute(1, 0, 2, 3, 4).flatten(0, 1)
+                vutils.save_image(save_image, self.attn_root/f'{self.c}_{gt}_{pt}.jpg', 
+                                nrow=2, normalize=True, scale_each=True)
+            else:
+                self.pil(image).save(self.attn_root/f'{self.c}_{gt}_{pt}.jpg')
+            self.c += 1
+
+        with open(self.root/f'{self.model_eval}.txt', 'a') as f: f.writelines(prediction)
+        with open(self.root/f'{self.model_eval}-false.txt', 'a') as f: f.writelines(false_prediction)  

configs/pretrain_itervm.yaml

@@ -0,0 +1,60 @@
+global:
+  name: pretrain-itervm
+  phase: train
+  stage: pretrain-vision
+  workdir: workdir
+  seed: ~
+ 
+dataset:
+  train: {
+    roots: ['data/training/MJ/MJ_train/', 
+            'data/training/MJ/MJ_test/', 
+            'data/training/MJ/MJ_valid/', 
+            'data/training/ST'],
+    batch_size: 384
+  }
+  test: {
+    roots: ['data/evaluation/IIIT5k_3000', 
+            'data/evaluation/SVT', 
+            'data/evaluation/SVTP',
+            'data/evaluation/IC13_857',
+            'data/evaluation/IC15_1811',
+            'data/evaluation/CUTE80'],
+    batch_size: 384
+  }
+  data_aug: True
+  multiscales: False
+  num_workers: 14
+
+training:
+  epochs: 8
+  show_iters: 50
+  eval_iters: 3000
+  save_iters: 3000
+
+optimizer:
+  type: Adam
+  true_wd: False
+  wd: 0.0
+  bn_wd: False
+  clip_grad: 20
+  lr: 0.0001
+  args: {
+    betas: !!python/tuple [0.9, 0.999], # for default Adam 
+  }
+  scheduler: {
+    periods: [6, 2],
+    gamma: 0.1,
+  }
+
+model:
+  name: 'modules.model_vision.BaseIterVision'
+  checkpoint: ~
+  vision: {
+    loss_weight: 1.,
+    attention: 'position',
+    backbone: 'transformer',
+    backbone_ln: 3,
+    iter_size: 3,
+    backbone_alpha_d: 0.5,
+  }

configs/pretrain_language_model.yaml

@@ -0,0 +1,45 @@
+global:
+  name: pretrain-language-model
+  phase: train
+  stage: pretrain-language
+  workdir: workdir
+  seed: ~
+ 
+dataset:
+  train: {
+    roots: ['data/WikiText-103.csv'],
+    batch_size: 4096
+  }
+  test: {
+    roots: ['data/WikiText-103_eval_d1.csv'],
+    batch_size: 4096
+  }
+
+training:
+  epochs: 80
+  show_iters: 50
+  eval_iters: 6000
+  save_iters: 3000
+
+optimizer:
+  type: Adam
+  true_wd: False
+  wd: 0.0
+  bn_wd: False
+  clip_grad: 20
+  lr: 0.0001
+  args: {
+    betas: !!python/tuple [0.9, 0.999], # for default Adam 
+  }
+  scheduler: {
+    periods: [70, 10],
+    gamma: 0.1,
+  }
+
+model:
+  name: 'modules.model_language.BCNLanguage'
+  language: {
+    num_layers: 4,
+    loss_weight: 1.,
+    use_self_attn: False
+  }

configs/pretrain_vm.yaml

@@ -0,0 +1,51 @@
+global:
+  name: pretrain-vm
+  phase: train
+  stage: pretrain-vision
+  workdir: workdir
+  seed: ~
+ 
+dataset:
+  train: {
+    roots: ['output_tbell_dataset/'],
+    batch_size: 20
+  }
+  test: {
+    roots: ['output_tbell_dataset/'],
+    batch_size: 20
+  }
+  data_aug: True
+  multiscales: False
+  num_workers: 1
+
+training:
+  epochs: 8
+  show_iters: 50
+  eval_iters: 50
+  # save_iters: 3000
+
+optimizer:
+  type: Adam
+  true_wd: False
+  wd: 0.0
+  bn_wd: False
+  clip_grad: 20
+  lr: 0.0001
+  args: {
+    betas: !!python/tuple [0.9, 0.999], # for default Adam 
+  }
+  scheduler: {
+    periods: [6, 2],
+    gamma: 0.1,
+  }
+
+model:
+  name: 'modules.model_vision.BaseVision'
+  checkpoint: ~
+  vision: {
+    loss_weight: 1.,
+    attention: 'position',
+    backbone: 'transformer',
+    backbone_ln: 3,
+    backbone_alpha_d: 0.5,
+  }

configs/template.yaml

@@ -0,0 +1,67 @@
+global:
+  name: exp
+  phase: train
+  stage: pretrain-vision
+  workdir: /tmp/workdir
+  seed: ~
+ 
+dataset:
+  train: {
+    roots: ['data/training/MJ/MJ_train/', 
+            'data/training/MJ/MJ_test/', 
+            'data/training/MJ/MJ_valid/', 
+            'data/training/ST'],
+    batch_size: 128
+  }
+  test: {
+    roots: ['data/evaluation/IIIT5k_3000', 
+            'data/evaluation/SVT', 
+            'data/evaluation/SVTP',
+            'data/evaluation/IC13_857',
+            'data/evaluation/IC15_1811',
+            'data/evaluation/CUTE80'],
+    batch_size: 128
+  }
+  charset_path: data/charset_36.txt
+  num_workers: 4
+  max_length: 25  # 30
+  image_height: 32
+  image_width: 128
+  case_sensitive: False
+  eval_case_sensitive: False
+  data_aug: True
+  multiscales: False
+  pin_memory: True
+  smooth_label: False
+  smooth_factor: 0.1
+  one_hot_y: True
+  use_sm: False
+
+training:
+  epochs: 6
+  show_iters: 50
+  eval_iters: 3000
+  save_iters: 20000
+  start_iters: 0
+  stats_iters: 100000
+
+optimizer:
+  type: Adadelta # Adadelta, Adam
+  true_wd: False
+  wd: 0. # 0.001
+  bn_wd: False
+  args: {
+    # betas: !!python/tuple [0.9, 0.99], # betas=(0.9,0.99) for AdamW
+    # betas: !!python/tuple [0.9, 0.999], # for default Adam 
+  }
+  clip_grad: 20
+  lr: [1.0, 1.0, 1.0]  # lr: [0.005, 0.005, 0.005]   
+  scheduler: {
+    periods: [3, 2, 1],
+    gamma: 0.1,
+  }
+
+model:
+  name: 'modules.model_iternet.IterNet'
+  checkpoint: ~
+  strict: True

configs/train_iternet.yaml

@@ -0,0 +1,65 @@
+global:
+  name: train-iternet
+  phase: train
+  stage: train-super
+  workdir: workdir
+  seed: ~
+ 
+dataset:
+  train: {
+    roots: ['output_pixelplanet_dataset/'],
+    batch_size: 20
+  }
+  test: {
+    roots: ['output_pixelplanet_dataset/'],
+    batch_size: 20
+  }
+  data_aug: True
+  multiscales: False
+  num_workers: 8
+
+training:
+  epochs: 1000
+  show_iters: 500
+  eval_iters: 500
+  # save_iters: 1
+
+optimizer:
+  type: Adam
+  true_wd: False
+  wd: 0.0
+  bn_wd: False
+  clip_grad: 20
+  lr: 0.0001
+  args: {
+    betas: !!python/tuple [0.9, 0.999], # for default Adam 
+  }
+  scheduler: {
+    periods: [6, 4],
+    gamma: 0.1,
+  }
+
+model:
+  name: 'modules.model_iternet.IterNet'
+  iter_size: 3
+  ensemble: ''
+  use_vision: False
+  vision: {
+    checkpoint: workdir/train-iternet/best-train-iternet.pth,
+    loss_weight: 1.,
+    attention: 'position',
+    backbone: 'transformer',
+    backbone_ln: 3,
+    iter_size: 3,
+    backbone_alpha_d: 0.5,
+  }
+  # language: {
+  #   checkpoint:  workdir/pretrain-language-model/pretrain-language-model.pth,
+  #   num_layers: 4,
+  #   loss_weight: 1.,
+  #   detach: True,
+  #   use_self_attn: False
+  # }
+  alignment: {
+    loss_weight: 1.,
+  }

dataset.py

@@ -0,0 +1,278 @@
+import logging
+import re
+
+import cv2
+import lmdb
+import six
+from fastai.vision import *
+from torchvision import transforms
+
+from transforms import CVColorJitter, CVDeterioration, CVGeometry
+from utils import CharsetMapper, onehot
+
+
+class ImageDataset(Dataset):
+    "`ImageDataset` read data from LMDB database."
+
+    def __init__(self,
+                 path:PathOrStr,
+                 is_training:bool=True,
+                 img_h:int=32,
+                 img_w:int=100,
+                 max_length:int=25,
+                 check_length:bool=True,
+                 case_sensitive:bool=False,
+                 charset_path:str='data/charset_36.txt',
+                 convert_mode:str='RGB',
+                 data_aug:bool=True,
+                 deteriorate_ratio:float=0.,
+                 multiscales:bool=True,
+                 one_hot_y:bool=True,
+                 return_idx:bool=False,
+                 return_raw:bool=False,
+                 **kwargs):
+        self.path, self.name = Path(path), Path(path).name
+        assert self.path.is_dir() and self.path.exists(), f"{path} is not a valid directory."
+        self.convert_mode, self.check_length = convert_mode, check_length
+        self.img_h, self.img_w = img_h, img_w
+        self.max_length, self.one_hot_y = max_length, one_hot_y
+        self.return_idx, self.return_raw = return_idx, return_raw
+        self.case_sensitive, self.is_training = case_sensitive, is_training
+        self.data_aug, self.multiscales = data_aug, multiscales
+        self.charset = CharsetMapper(charset_path, max_length=max_length+1)
+        self.c = self.charset.num_classes
+
+        self.env = lmdb.open(str(path), readonly=True, lock=False, readahead=False, meminit=False)
+        assert self.env, f'Cannot open LMDB dataset from {path}.'
+        with self.env.begin(write=False) as txn:
+            self.length = int(txn.get('num-samples'.encode()))
+
+        if self.is_training and self.data_aug:
+            self.augment_tfs = transforms.Compose([
+                CVGeometry(degrees=45, translate=(0.0, 0.0), scale=(0.5, 2.), shear=(45, 15), distortion=0.5, p=0.5),
+                CVDeterioration(var=20, degrees=6, factor=4, p=0.25),
+                CVColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.1, p=0.25)
+            ])
+        self.totensor = transforms.ToTensor()
+    
+    def __len__(self): return self.length
+
+    def _next_image(self, index):
+        next_index = random.randint(0, len(self) - 1)
+        return self.get(next_index)
+
+    def _check_image(self, x, pixels=6):
+        if x.size[0] <= pixels or x.size[1] <= pixels: return False
+        else: return True
+
+    def resize_multiscales(self, img, borderType=cv2.BORDER_CONSTANT): 
+        def _resize_ratio(img, ratio, fix_h=True):
+            if ratio * self.img_w < self.img_h:
+                if fix_h: trg_h = self.img_h
+                else: trg_h = int(ratio * self.img_w)
+                trg_w = self.img_w
+            else: trg_h, trg_w = self.img_h, int(self.img_h / ratio)
+            img = cv2.resize(img, (trg_w, trg_h))
+            pad_h, pad_w = (self.img_h - trg_h) / 2, (self.img_w - trg_w) / 2
+            top, bottom = math.ceil(pad_h), math.floor(pad_h)
+            left, right = math.ceil(pad_w), math.floor(pad_w)
+            img = cv2.copyMakeBorder(img, top, bottom, left, right, borderType)
+            return img
+        
+        if self.is_training: 
+            if random.random() < 0.5:
+                base, maxh, maxw = self.img_h, self.img_h, self.img_w
+                h, w = random.randint(base, maxh), random.randint(base, maxw)
+                return _resize_ratio(img, h/w)
+            else: return _resize_ratio(img, img.shape[0] / img.shape[1])  # keep aspect ratio
+        else:  return _resize_ratio(img, img.shape[0] / img.shape[1])  # keep aspect ratio
+
+    def resize(self, img):
+        if self.multiscales: return self.resize_multiscales(img, cv2.BORDER_REPLICATE)
+        else: return cv2.resize(img, (self.img_w, self.img_h))
+         
+    def get(self, idx):
+        with self.env.begin(write=False) as txn:
+            image_key, label_key = f'image-{idx+1:09d}', f'label-{idx+1:09d}'
+            try:
+                label = str(txn.get(label_key.encode()), 'utf-8')  # label
+                label = re.sub('[^0-9a-zA-Z]+', '', label)
+                if self.check_length and self.max_length > 0:
+                    if len(label) > self.max_length or len(label) <= 0:
+                        #logging.info(f'Long or short text image is found: {self.name}, {idx}, {label}, {len(label)}')
+                        return self._next_image(idx)
+                label = label[:self.max_length]
+
+                imgbuf = txn.get(image_key.encode())  # image
+                buf = six.BytesIO()
+                buf.write(imgbuf)
+                buf.seek(0)
+                with warnings.catch_warnings():
+                    warnings.simplefilter("ignore", UserWarning) # EXIF warning from TiffPlugin
+                    image = PIL.Image.open(buf).convert(self.convert_mode)
+                if self.is_training and not self._check_image(image):
+                    #logging.info(f'Invalid image is found: {self.name}, {idx}, {label}, {len(label)}')
+                    return self._next_image(idx)
+            except:
+                import traceback
+                traceback.print_exc()
+                logging.info(f'Corrupted image is found: {self.name}, {idx}, {label}, {len(label)}')
+                return self._next_image(idx)
+            return image, label, idx
+
+    def _process_training(self, image):
+        if self.data_aug: image = self.augment_tfs(image)
+        image = self.resize(np.array(image))
+        return image
+
+    def _process_test(self, image):
+        return self.resize(np.array(image)) # TODO:move is_training to here
+
+    def __getitem__(self, idx):
+        image, text, idx_new = self.get(idx)
+        if not self.is_training: assert idx == idx_new, f'idx {idx} != idx_new {idx_new} during testing.'
+
+        if self.is_training: image = self._process_training(image)
+        else: image = self._process_test(image)
+        if self.return_raw: return image, text
+        image = self.totensor(image)
+
+        length = tensor(len(text) + 1).to(dtype=torch.long)  # one for end token
+        label = self.charset.get_labels(text, case_sensitive=self.case_sensitive)
+        label = tensor(label).to(dtype=torch.long)
+        if self.one_hot_y: label = onehot(label, self.charset.num_classes)
+
+        if self.return_idx: y = [label, length, idx_new]
+        else: y = [label, length]
+        return image, y
+
+
+class TextDataset(Dataset):
+    def __init__(self,
+                 path:PathOrStr, 
+                 delimiter:str='\t',
+                 max_length:int=25, 
+                 charset_path:str='data/charset_36.txt', 
+                 case_sensitive=False, 
+                 one_hot_x=True,
+                 one_hot_y=True,
+                 is_training=True,
+                 smooth_label=False,
+                 smooth_factor=0.2,
+                 use_sm=False,
+                 **kwargs):
+        self.path = Path(path)
+        self.case_sensitive, self.use_sm = case_sensitive, use_sm
+        self.smooth_factor, self.smooth_label = smooth_factor, smooth_label
+        self.charset = CharsetMapper(charset_path, max_length=max_length+1)
+        self.one_hot_x, self.one_hot_y, self.is_training = one_hot_x, one_hot_y, is_training
+        if self.is_training and self.use_sm: self.sm = SpellingMutation(charset=self.charset)
+
+        dtype = {'inp': str, 'gt': str}
+        self.df = pd.read_csv(self.path, dtype=dtype, delimiter=delimiter, na_filter=False)
+        self.inp_col, self.gt_col = 0, 1
+
+    def __len__(self): return len(self.df)
+
+    def __getitem__(self, idx):
+        text_x = self.df.iloc[idx, self.inp_col]
+        text_x = re.sub('[^0-9a-zA-Z]+', '', text_x)
+        if not self.case_sensitive: text_x = text_x.lower()
+        if self.is_training and self.use_sm: text_x = self.sm(text_x)
+
+        length_x = tensor(len(text_x) + 1).to(dtype=torch.long)  # one for end token
+        label_x = self.charset.get_labels(text_x, case_sensitive=self.case_sensitive)
+        label_x = tensor(label_x)
+        if self.one_hot_x:
+            label_x = onehot(label_x, self.charset.num_classes)
+            if self.is_training and self.smooth_label: 
+                label_x = torch.stack([self.prob_smooth_label(l) for l in label_x])
+        x =  [label_x, length_x]
+    
+        text_y = self.df.iloc[idx, self.gt_col]
+        text_y = re.sub('[^0-9a-zA-Z]+', '', text_y)
+        if not self.case_sensitive: text_y = text_y.lower()
+        length_y = tensor(len(text_y) + 1).to(dtype=torch.long)  # one for end token
+        label_y = self.charset.get_labels(text_y, case_sensitive=self.case_sensitive)
+        label_y = tensor(label_y)
+        if self.one_hot_y: label_y = onehot(label_y, self.charset.num_classes)
+        y = [label_y, length_y]
+
+        return x, y
+
+    def prob_smooth_label(self, one_hot):
+        one_hot = one_hot.float()
+        delta = torch.rand([]) * self.smooth_factor
+        num_classes = len(one_hot)
+        noise = torch.rand(num_classes)
+        noise = noise / noise.sum() * delta
+        one_hot = one_hot * (1 - delta) + noise
+        return one_hot
+
+
+class SpellingMutation(object):
+    def __init__(self, pn0=0.7, pn1=0.85, pn2=0.95, pt0=0.7, pt1=0.85, charset=None):
+        """ 
+        Args:
+            pn0: the prob of not modifying characters is (pn0)
+            pn1: the prob of modifying one characters is (pn1 - pn0)
+            pn2: the prob of modifying two characters is (pn2 - pn1), 
+                 and three (1 - pn2)
+            pt0: the prob of replacing operation is pt0.
+            pt1: the prob of inserting operation is (pt1 - pt0),
+                 and deleting operation is (1 - pt1)
+        """
+        super().__init__()
+        self.pn0, self.pn1, self.pn2 = pn0, pn1, pn2
+        self.pt0, self.pt1 = pt0, pt1
+        self.charset = charset
+        logging.info(f'the probs: pn0={self.pn0}, pn1={self.pn1} ' + 
+                     f'pn2={self.pn2}, pt0={self.pt0}, pt1={self.pt1}')
+
+    def is_digit(self, text, ratio=0.5):
+        length = max(len(text), 1)
+        digit_num = sum([t in self.charset.digits for t in text])
+        if digit_num / length < ratio: return False
+        return True
+
+    def is_unk_char(self, char):
+        # return char == self.charset.unk_char
+        return (char not in self.charset.digits) and (char not in self.charset.alphabets)
+
+    def get_num_to_modify(self, length):
+        prob = random.random()
+        if prob < self.pn0: num_to_modify = 0
+        elif prob < self.pn1: num_to_modify = 1
+        elif prob < self.pn2: num_to_modify = 2
+        else: num_to_modify = 3
+        
+        if length <= 1: num_to_modify = 0
+        elif length >= 2 and length <= 4: num_to_modify = min(num_to_modify, 1)
+        else: num_to_modify = min(num_to_modify, length // 2)  # smaller than length // 2
+        return num_to_modify
+
+    def __call__(self, text, debug=False):
+        if self.is_digit(text): return text
+        length = len(text)
+        num_to_modify = self.get_num_to_modify(length)
+        if num_to_modify <= 0: return text
+
+        chars = []
+        index = np.arange(0, length)
+        random.shuffle(index)
+        index = index[: num_to_modify]
+        if debug: self.index = index
+        for i, t in enumerate(text):
+            if i not in index: chars.append(t)
+            elif self.is_unk_char(t): chars.append(t)
+            else:
+                prob = random.random()
+                if prob < self.pt0: # replace
+                    chars.append(random.choice(self.charset.alphabets))
+                elif prob < self.pt1: # insert
+                    chars.append(random.choice(self.charset.alphabets))
+                    chars.append(t)
+                else: # delete
+                    continue
+        new_text = ''.join(chars[: self.charset.max_length-1])
+        return new_text if len(new_text) >= 1 else text

demo.py

@@ -0,0 +1,109 @@
+import argparse
+import logging
+import os
+import glob
+import tqdm
+import torch
+import PIL
+import cv2
+import numpy as np
+import torch.nn.functional as F
+from torchvision import transforms
+from utils import Config, Logger, CharsetMapper
+
+def get_model(config):
+    import importlib
+    names = config.model_name.split('.')
+    module_name, class_name = '.'.join(names[:-1]), names[-1]
+    cls = getattr(importlib.import_module(module_name), class_name)
+    model = cls(config)
+    logging.info(model)
+    model = model.eval()
+    return model
+
+def preprocess(img, width, height):
+    img = cv2.resize(np.array(img), (width, height))
+    img = transforms.ToTensor()(img).unsqueeze(0)
+    mean = torch.tensor([0.485, 0.456, 0.406])
+    std  = torch.tensor([0.229, 0.224, 0.225])
+    return (img-mean[...,None,None]) / std[...,None,None]
+
+def postprocess(output, charset, model_eval):
+    def _get_output(last_output, model_eval):
+        if isinstance(last_output, (tuple, list)): 
+            for res in last_output:
+                if res['name'] == model_eval: output = res
+        else: output = last_output
+        return output
+
+    def _decode(logit):
+        """ Greed decode """
+        out = F.softmax(logit, dim=2)
+        pt_text, pt_scores, pt_lengths = [], [], []
+        for o in out:
+            text = charset.get_text(o.argmax(dim=1), padding=False, trim=False)
+            text = text.split(charset.null_char)[0]  # end at end-token
+            pt_text.append(text)
+            pt_scores.append(o.max(dim=1)[0])
+            pt_lengths.append(min(len(text) + 1, charset.max_length))  # one for end-token
+        return pt_text, pt_scores, pt_lengths
+
+    output = _get_output(output, model_eval)
+    logits, pt_lengths = output['logits'], output['pt_lengths']
+    pt_text, pt_scores, pt_lengths_ = _decode(logits)
+    
+    return pt_text, pt_scores, pt_lengths_
+
+def load(model, file, device=None, strict=True):
+    if device is None: device = 'cpu'
+    elif isinstance(device, int): device = torch.device('cuda', device)
+    assert os.path.isfile(file)
+    state = torch.load(file, map_location=device)
+    if set(state.keys()) == {'model', 'opt'}:
+        state = state['model']
+    model.load_state_dict(state, strict=strict)
+    return model
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', type=str, default='configs/train_iternet.yaml',
+                        help='path to config file')
+    parser.add_argument('--input', type=str, default='figures/demo')
+    parser.add_argument('--cuda', type=int, default=-1)
+    parser.add_argument('--checkpoint', type=str, default='workdir/train-iternet/best-train-iternet.pth')
+    parser.add_argument('--model_eval', type=str, default='alignment', 
+                        choices=['alignment', 'vision', 'language'])
+    args = parser.parse_args()
+    config = Config(args.config)
+    if args.checkpoint is not None: config.model_checkpoint = args.checkpoint
+    if args.model_eval is not None: config.model_eval = args.model_eval
+    config.global_phase = 'test'
+    config.model_vision_checkpoint, config.model_language_checkpoint = None, None
+    device = 'cpu' if args.cuda < 0 else f'cuda:{args.cuda}'
+
+    Logger.init(config.global_workdir, config.global_name, config.global_phase)
+    Logger.enable_file()
+    logging.info(config)
+
+    logging.info('Construct model.')
+    model = get_model(config).to(device)
+    model = load(model, config.model_checkpoint, device=device)
+    charset = CharsetMapper(filename=config.dataset_charset_path,
+                            max_length=config.dataset_max_length + 1)
+
+    if os.path.isdir(args.input):
+        paths = [os.path.join(args.input, fname) for fname in os.listdir(args.input)]
+    else:
+        paths = glob.glob(os.path.expanduser(args.input))
+        assert paths, "The input path(s) was not found"
+    paths = sorted(paths)
+    for path in tqdm.tqdm(paths):
+        img = PIL.Image.open(path).convert('RGB')
+        img = preprocess(img, config.dataset_image_width, config.dataset_image_height)
+        img = img.to(device)
+        res = model(img)
+        pt_text, _, __ = postprocess(res, charset, config.model_eval)
+        logging.info(f'{path}: {pt_text[0]}')
+
+if __name__ == '__main__':
+    main()