Study 03 — Self-Supervised Visual Representation Learning for Khmer OCR

Serey OCR Research Journal · October 21, 2026 · Journal version 0.3
Status: Critical literature and architecture review. No Khmer self-supervised model was pretrained, fine-tuned, or evaluated in this study.

Abstract

Self-supervised visual learning can exploit text images without transcriptions, potentially reducing dependence on costly labeled Khmer recognition data. Yet the most useful learned representations must preserve distinctions between visually similar consonants, dependent vowels, subscript forms, marks, and numerals. This study reviews masked autoencoding (MAE), contrastive-plus-generative text learning (DiG), masked visual/language pretraining (MaskOCR), and linguistics-aware masked image modeling (LMIM). It examines the mechanisms, published evidence, compute assumptions, and transfer limitations. The main conclusion is that the invariances induced by pretraining are as important as the quantity of unlabeled data. A model that becomes robust to fonts and scan noise at the cost of ignoring fine Khmer marks could perform worse on exact transcription. No published results reviewed here establish a Khmer OCR improvement.

1. Why self-supervised learning matters

Typical text recognition trains on an image paired with its transcription. Producing accurate labels is expensive in languages with complex shaping and may be particularly difficult on damaged or administrative documents. Self-supervised learning (SSL) uses a target derived from the input image itself—such as reconstructing hidden patches or making two views agree—so the pretraining stage need not have human text labels [1]. A later recognition stage still generally requires labeled examples and defensible evaluation.

Three terms must not be conflated:

A general image encoder may learn useful stroke-like features but text recognition additionally needs composition and sequence alignment. The extent to which SSL encodes the latter is central to the literature.

2. Four architectural families

2.1 Masked autoencoders: learn by reconstructing hidden pixels

Masked Autoencoders (MAE) randomly remove many image patches and train an encoder/decoder to reconstruct them. The original CVPR 2022 work uses an asymmetric design: the encoder sees only unmasked patches, and a lightweight decoder predicts the missing image content. Its widely cited 75% masking ratio is a result from general visual representation learning, not an optimal Khmer OCR setting [1].

For text, a reconstruction objective may learn local structure and font texture while allowing shortcuts from nearby pixels. A system might reconstruct a plausible mark from local texture without developing the exact discriminative representation needed to tell visually neighboring characters apart.

Khmer research hypothesis: masking could help learn unfamiliar fonts, printed strokes, and scan distortions from unlabeled text-line images. It could also obscure an entire distinguishing mark in a small number of patches, making the pretext task counterproductive unless masking geometry is considered.

2.2 DiG: discriminative and generative learning together

Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition (ACM Multimedia 2022) combines contrastive learning—which encourages useful agreement between augmented views—with masked image modeling—which reconstructs hidden features [2]. The authors report improvements on irregular scene-text benchmarks compared with earlier self-supervised methods. Those results pertain to the paper's scene-text data and evaluation settings, not to Khmer document lines.

The design highlights a tension: strong invariance can help with illumination, blur and fonts, but two samples that differ only in a small Unicode-significant Khmer mark must not become indistinguishable. A good Khmer representation would be invariant to irrelevant appearance while remaining sensitive to orthographic content.

2.3 MaskOCR: pretrain both the encoder and the decoder

MaskOCR: Scene Text Recognition with Masked Vision–Language Pre-training (TMLR 2024) pretrains a visual encoder on unlabeled images through masked image modeling, then separately develops the recognition decoder using masked image–language modeling over text-derived synthetic images; it reports English and Chinese recognition improvements [3]. The recognition architecture remains an encoder–decoder system.

Important distinction: the encoder's self-supervised phase and the decoder's synthetic text-based language phase are not the same operation. MaskOCR is evidence that unlabeled visual representations and language-conditioned recognition can complement each other, not proof that fully label-free Khmer transcription is solved.

This family is conceptually relevant to Serey OCR's existing autoregressive recognizer even if CTC is never adopted.

2.4 LMIM: combine visual reconstruction with linguistics-aware consistency

Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025) introduces two branches using the same source text image [4]. One branch sees heavily masked patches. The other sees an augmented view retaining the same text content but a different appearance. The branches share an encoder. A global representation-alignment loss and a reconstruction loss encourage features that combine visual evidence with global consistency.

Conceptually:

1. Unlabeled text image → masked reconstruction view.
2. The same image → appearance-changed guidance view (same underlying text).
3. Shared encoder produces both feature streams.
4. A decoder reconstructs teacher-derived masked features; a global alignment term makes representations consistent.
5. The pretrained encoder is subsequently fine-tuned with labeled text recognition data.

In the paper, reconstruction and alignment are optimized jointly, with an illustrative objective L = L_reconstruction + L_alignment [4, Sec. 3]. Its guidance mechanism does not depend on knowing the text string for each unlabeled input. However, it does depend on the augmented view preserving the original content. That condition is particularly consequential for Khmer marks.

The released recipe is also not simply a small, from-scratch, CPU pretraining procedure. The official repository starts from an MAERec teacher checkpoint and illustrates eight-GPU distributed pretraining. Its default setup uses 32×128 inputs, 4×4 patches, 80% random masking and a ViT-Small encoder; the downstream recognition configuration includes a six-layer Transformer decoder [4, code]. These are reported settings, not recommendations to copy to Khmer.

3. What the published experiments actually show

The clearest controlled evidence here comes from LMIM's own ablations on the English Union14M seven-subset benchmark [4, Table 1]. These are paper-reported recognition accuracies after the specified training setup, not unsupervised transcription accuracy:

The +0.6-point difference between the second and third row isolates the additional alignment term within this reported ablation. It does not establish which component would improve Khmer recognition.

Other paper-reported findings [4]:

• Mask ratio: random masking of 80% outperformed the paper's sampled 60% and 90% ratios under its tested setup (81.2% versus 79.1% and 80.4% in Table 5). This cannot determine a Khmer-optimal ratio.
• Reconstruction target: teacher features from MAERec yielded 81.2%, while pixel reconstruction yielded 76.4% in Table 4. It suggests the target representation, not merely masking, matters.
• Augmentation strength: medium augmentation yielded 81.2%, compared with 79.2% weak and 79.3% strong in Table 2. Stronger transformations are not always better.
• English Union14M: with longer pretraining, the authors report an average of 86.3% on its seven subsets (Table 7), under a different training-duration comparison.
• Chinese: the authors report 83.6% scene and 82.0% web subset accuracy (Table 6); this is not Khmer evidence.
• Six common English benchmarks: the authors report a 97.0% arithmetic mean in a corrected-label setting (Table 8). Its evaluation definitions—including English word accuracy ignoring case and symbols elsewhere in the paper—are not interchangeable with codepoint-exact Khmer CER.

The official repository reports downstream ViT-Small + Transformer decoder variants with about 35.8M parameters for English and 43.8M for Chinese. That is not a direct speed, accuracy or memory comparison with Serey OCR's ~19.2M model [4, code].

Evidence caution: published averages mix different subsets, data sizes, pretraining schedules and recognition metrics. They cannot be interpreted as a predicted percentage-point improvement in Serey OCR.

4. Khmer-specific opportunities and risks

4.1 Learn appearance invariance without erasing orthography

Khmer recognition needs robustness to fonts, layout rendering, scanner artifacts, lighting, antialiasing and printing quality. But small dependent vowels and combining marks are content-bearing visual structures, not disposable noise. An augmentation that removes them can create a false positive pair: visually different labels incorrectly treated as the same text.

These are failure hypotheses and evaluation considerations, not measured Khmer augmentation outcomes. A semantics-aware objective must not learn to ignore the very details a legal or financial document requires.

4.2 Visual context is not proof of a transcription

The same context that helps with ambiguous printed words may bias a recognizer toward a common spelling. That is risky for Khmer personal names, rare place names, document identifiers, years, dates and numerical amounts. SSL may improve the visual encoder yet later decoder losses can still favor linguistically plausible substitutions. Character, cluster, diacritic and exact-field preservation are therefore distinct scientific questions.

4.3 The source domain matters

LMIM and DiG are largely motivated by scene-text imagery. Clean document lines, degraded photocopies, handwritten annotations and complex official forms differ in background, text length, resolution, and content distribution. Unlabeled image quantity alone cannot establish target-domain benefit, and the original published training images are not proof of adequate representation for Khmer.

4.4 Pretraining is not detection

A stronger recognizer encoder cannot recover text that an upstream detector never crops. Page-level region discovery and document structure require separate study. Thus detection omissions, recognizer omissions, and serialization issues should not be collapsed into a single visual-pretraining hypothesis.

5. Comparison for the Serey OCR research baseline

The preceding studies address output representation (Study 01) and supervised recognition architecture (Study 02). Study 03 concerns how to initialize visual representations before ordinary text recognition training; it does not choose the eventual CTC versus autoregressive decoder.

6. Research questions that remain open

1. Granularity: Does the pretraining objective encourage representations of individual strokes, complete orthographic clusters, or only broad visual texture?
2. Sensitivity: Which visual transformations preserve Khmer transcription exactly, and which create false-invariance between distinct text labels?
3. Masking geometry: Would spatial masking need to account for the density and vertical arrangement of Khmer marks rather than copy English scene-text patch ratios?
4. Teacher dependence: How much of LMIM's reported benefit comes from its external MAERec teacher feature targets versus its new alignment mechanism?
5. Target domain: Would visual pretraining transfer from general scene text to low-resolution government scans, colored headings and unusual fonts?
6. Context risk: Could linguistic guidance reduce general word errors while increasing rare-name, date or number substitutions?
7. Efficiency: Can pretraining improve an existing compact encoder, or does the demonstrated result depend on larger ViT models and substantial GPU resources?
8. Evaluation: Which separately grounded evidence would distinguish improved visual discrimination from mere linguistic prior correction?

These are research hypotheses and evaluation design questions. This journal entry is not a model-building or data-acquisition task.

7. Provisional synthesis

Three lessons are strongest:

• Pretraining objective matters: reconstructing raw pixels, teacher features, and consistency across visual views do not teach the same thing.
• Data transformations matter as much as data volume: for Khmer, the safe notion of “same text despite altered appearance” must exclude changes that erase distinct code points or visual clusters.
• Architecture and training are separable decisions: the concept of self-supervised pretraining could be scientifically relevant without adopting LMIM's entire ViT–Transformer recognition pipeline or replacing the existing Serey OCR architecture.

Research conclusion: LMIM is a particularly worthwhile paper for Khmer OCR representation-learning research, but its English/Chinese ablations do not justify a Khmer performance claim. The next literature study should compare modern document-focused vision-language OCR architectures and examine what they add beyond specialist recognition, including text grounding and hallucination risks.

8. Sources

1. He, K. et al. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR. https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html
2. Yang, M. et al. (2022). Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition. ACM Multimedia. https://arxiv.org/abs/2207.00193
3. Lyu, P. et al. (2024). MaskOCR: Scene Text Recognition with Masked Vision–Language Pre-training. TMLR. https://openreview.net/forum?id=KNAWoKKpi3
4. Zhang, Y. et al. (2025). Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition. CVPR, 9318–9328. https://arxiv.org/html/2503.18746 ; https://github.com/zhangyifei01/LMIM
5. Unicode Consortium. Unicode Technical Note #61: Khmer Encoding Structure. https://www.unicode.org/notes/tn61/utn61-Khmer_Encoding_Structure_V1.pdf

Suggested citation: Serey OCR Research Journal, Study 03, v0.3 (October 21, 2026). This is an interpretive literature study, not a peer-reviewed paper or original Khmer model result. Authorship and publication URL to be completed by the author.