{"action":"created","issue":{"url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/318","repository_url":"https://api.github.com/repos/tesseract-ocr/tesseract","labels_url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/318/labels{/name}","comments_url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/318/comments","events_url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/318/events","html_url":"https://github.com/tesseract-ocr/tesseract/issues/318","id":153677637,"number":318,"title":"Unicharset — Incomplete properties","user":{"login":"ne0zer0","id":10687250,"avatar_url":"https://avatars.githubusercontent.com/u/10687250?v=3","gravatar_id":"","url":"https://api.github.com/users/ne0zer0","html_url":"https://github.com/ne0zer0","followers_url":"https://api.github.com/users/ne0zer0/followers","following_url":"https://api.github.com/users/ne0zer0/following{/other_user}","gists_url":"https://api.github.com/users/ne0zer0/gists{/gist_id}","starred_url":"https://api.github.com/users/ne0zer0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ne0zer0/subscriptions","organizations_url":"https://api.github.com/users/ne0zer0/orgs","repos_url":"https://api.github.com/users/ne0zer0/repos","events_url":"https://api.github.com/users/ne0zer0/events{/privacy}","received_events_url":"https://api.github.com/users/ne0zer0/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-05-08T21:05:15Z","updated_at":"2016-05-09T17:09:32Z","closed_at":null,"body":"Hi,\r\n\r\nI get some strange result when I try to train Tesseract.\r\n\r\nSome part are very improved comparing to the default eng.tessdata, when some part are strangely added or modified, while the image quality is very good (24 become eat ???; uppercase letter become lowercase; some words are cut in two words; etc)\r\n\r\nI think it may be cause by unicharset.\r\n\r\nIndeed, when I try to generate a unicharset file with the following command :\r\n`unicharset_extractor eng.palladio-regular.exp8.box`\r\nI get an incomplete file. Here the result :\r\n\r\n        115\r\n        NULL 0 NULL 0\r\n        Joined 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# Joined [4a 6f 69 6e 65 64 ]\r\n        |Broken|0|1 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# Broken\r\n        d 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# d [64 ]\r\n        i 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# i [69 ]\r\n        f 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# f [66 ]\r\n        e 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# e [65 ]\r\n        r 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# r [72 ]\r\n        n 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# n [6e ]\r\n        t 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# t [74 ]\r\n        N 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# N [4e ]\r\n        w 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# w [77 ]\r\n        A 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# A [41 ]\r\n        c 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# c [63 ]\r\n        l 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# l [6c ]\r\n        s 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# s [73 ]\r\n        p 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# p [70 ]\r\n        a 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# a [61 ]\r\n        g 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# g [67 ]\r\n        2 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 2 [32 ]\r\n        3 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 3 [33 ]\r\n        T 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# T [54 ]\r\n        o 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# o [6f ]\r\n        S 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# S [53 ]\r\n        v 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# v [76 ]\r\n        ~ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ~ [7e ]\r\n        D 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# D [44 ]\r\n        C 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# C [43 ]\r\n        h 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# h [68 ]\r\n        ' 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ' [27 ]\r\n        7 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 7 [37 ]\r\n        « 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# « [ab ]\r\n        : 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# : [3a ]\r\n        # 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# # [23 ]\r\n        1 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 1 [31 ]\r\n        Z 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# Z [5a ]\r\n        _ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# _ [5f ]\r\n        M 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# M [4d ]\r\n        u 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# u [75 ]\r\n        m 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# m [6d ]\r\n        P 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# P [50 ]\r\n        H 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# H [48 ]\r\n        O 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# O [4f ]\r\n        ( 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ( [28 ]\r\n        ) 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ) [29 ]\r\n        q 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# q [71 ]\r\n        y 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# y [79 ]\r\n        | 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# | [7c ]\r\n        U 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# U [55 ]\r\n        0 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 0 [30 ]\r\n        % 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# % [25 ]\r\n        x 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# x [78 ]\r\n        F 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# F [46 ]\r\n        R 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# R [52 ]\r\n        I 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# I [49 ]\r\n        , 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# , [2c ]\r\n        ! 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ! [21 ]\r\n        E 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# E [45 ]\r\n        b 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# b [62 ]\r\n        \\ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# \\ [5c ]\r\n        8 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 8 [38 ]\r\n        ? 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ? [3f ]\r\n        & 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# & [26 ]\r\n        ; 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ; [3b ]\r\n        B 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# B [42 ]\r\n        k 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# k [6b ]\r\n        - 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# - [2d ]\r\n        > 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# > [3e ]\r\n        L 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# L [4c ]\r\n        . 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# . [2e ]\r\n        — 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# — [2014 ]\r\n        4 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 4 [34 ]\r\n        » 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# » [bb ]\r\n        € 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# € [20ac ]\r\n        W 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# W [57 ]\r\n        J 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# J [4a ]\r\n        é 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# é [e9 ]\r\n        9 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 9 [39 ]\r\n        ® 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ® [ae ]\r\n        $ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# $ [24 ]\r\n        5 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 5 [35 ]\r\n        } 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# } [7d ]\r\n        [ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# [ [5b ]\r\n        Y 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# Y [59 ]\r\n        § 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# § [a7 ]\r\n        \" 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# \" [22 ]\r\n        { 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# { [7b ]\r\n        ¢ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ¢ [a2 ]\r\n        / 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# / [2f ]\r\n        Q 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# Q [51 ]\r\n        6 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# 6 [36 ]\r\n        G 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# G [47 ]\r\n        ” 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ” [201d ]\r\n        ° 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ° [b0 ]\r\n        K 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# K [4b ]\r\n        ¥ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ¥ [a5 ]\r\n        V 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# V [56 ]\r\n        © 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# © [a9 ]\r\n        z 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# z [7a ]\r\n        + 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# + [2b ]\r\n        = 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# = [3d ]\r\n        £ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# £ [a3 ]\r\n        < 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# < [3c ]\r\n        ’ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ’ [2019 ]\r\n        ‘ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ‘ [2018 ]\r\n        j 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# j [6a ]\r\n        X 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# X [58 ]\r\n        ] 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ] [5d ]\r\n        * 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# * [2a ]\r\n        “ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# “ [201c ]\r\n        @ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# @ [40 ]\r\n        • 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# • [2022 ]\r\n        – 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# – [2013 ]\r\n        … 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# … [2026 ]\r\n        ^ 0 0,255,0,255,0,0,0,0,0,0 NULL 0 0 0 \t# ^ [5e ]\r\n\r\nWhen I try to fix it with with set_unicharset_properties:\r\n`set_unicharset_properties --F font_properties -U unicharset -O output_unicharset --script_dir=/ `\r\n\r\nI get these warnings :\r\n\r\n        Loaded unicharset of size 115 from file unicharset\r\n        Setting unichar properties\r\n        Other case É of é is not in unicharset\r\n        Warning: properties incomplete for index 3 = d\r\n        Warning: properties incomplete for index 4 = i\r\n        Warning: properties incomplete for index 5 = f\r\n        Warning: properties incomplete for index 6 = e\r\n        Warning: properties incomplete for index 7 = r\r\n        Warning: properties incomplete for index 8 = n\r\n        Warning: properties incomplete for index 9 = t\r\n        Warning: properties incomplete for index 10 = N\r\n        Warning: properties incomplete for index 11 = w\r\n        Warning: properties incomplete for index 12 = A\r\n        Warning: properties incomplete for index 13 = c\r\n        Warning: properties incomplete for index 14 = l\r\n        Warning: properties incomplete for index 15 = s\r\n        Warning: properties incomplete for index 16 = p\r\n        Warning: properties incomplete for index 17 = a\r\n        Warning: properties incomplete for index 18 = g\r\n        Warning: properties incomplete for index 19 = 2\r\n        Warning: properties incomplete for index 20 = 3\r\n        Warning: properties incomplete for index 21 = T\r\n        Warning: properties incomplete for index 22 = o\r\n        Warning: properties incomplete for index 23 = S\r\n        Warning: properties incomplete for index 24 = v\r\n        Warning: properties incomplete for index 25 = ~\r\n        Warning: properties incomplete for index 26 = D\r\n        Warning: properties incomplete for index 27 = C\r\n        Warning: properties incomplete for index 28 = h\r\n        Warning: properties incomplete for index 29 = '\r\n        Warning: properties incomplete for index 30 = 7\r\n        Warning: properties incomplete for index 31 = «\r\n        Warning: properties incomplete for index 32 = :\r\n        Warning: properties incomplete for index 33 = #\r\n        Warning: properties incomplete for index 34 = 1\r\n        Warning: properties incomplete for index 35 = Z\r\n        Warning: properties incomplete for index 36 = _\r\n        Warning: properties incomplete for index 37 = M\r\n        Warning: properties incomplete for index 38 = u\r\n        Warning: properties incomplete for index 39 = m\r\n        Warning: properties incomplete for index 40 = P\r\n        Warning: properties incomplete for index 41 = H\r\n        Warning: properties incomplete for index 42 = O\r\n        Warning: properties incomplete for index 43 = (\r\n        Warning: properties incomplete for index 44 = )\r\n        Warning: properties incomplete for index 45 = q\r\n        Warning: properties incomplete for index 46 = y\r\n        Warning: properties incomplete for index 47 = |\r\n        Warning: properties incomplete for index 48 = U\r\n        Warning: properties incomplete for index 49 = 0\r\n        Warning: properties incomplete for index 50 = %\r\n        Warning: properties incomplete for index 51 = x\r\n        Warning: properties incomplete for index 52 = F\r\n        Warning: properties incomplete for index 53 = R\r\n        Warning: properties incomplete for index 54 = I\r\n        Warning: properties incomplete for index 55 = ,\r\n        Warning: properties incomplete for index 56 = !\r\n        Warning: properties incomplete for index 57 = E\r\n        Warning: properties incomplete for index 58 = b\r\n        Warning: properties incomplete for index 59 = \\\r\n        Warning: properties incomplete for index 60 = 8\r\n        Warning: properties incomplete for index 61 = ?\r\n        Warning: properties incomplete for index 62 = &\r\n        Warning: properties incomplete for index 63 = ;\r\n        Warning: properties incomplete for index 64 = B\r\n        Warning: properties incomplete for index 65 = k\r\n        Warning: properties incomplete for index 66 = -\r\n        Warning: properties incomplete for index 67 = >\r\n        Warning: properties incomplete for index 68 = L\r\n        Warning: properties incomplete for index 69 = .\r\n        Warning: properties incomplete for index 70 = —\r\n        Warning: properties incomplete for index 71 = 4\r\n        Warning: properties incomplete for index 72 = »\r\n        Warning: properties incomplete for index 73 = €\r\n        Warning: properties incomplete for index 74 = W\r\n        Warning: properties incomplete for index 75 = J\r\n        Warning: properties incomplete for index 76 = é\r\n        Warning: properties incomplete for index 77 = 9\r\n        Warning: properties incomplete for index 78 = ®\r\n        Warning: properties incomplete for index 79 = $\r\n        Warning: properties incomplete for index 80 = 5\r\n        Warning: properties incomplete for index 81 = }\r\n        Warning: properties incomplete for index 82 = [\r\n        Warning: properties incomplete for index 83 = Y\r\n        Warning: properties incomplete for index 84 = §\r\n        Warning: properties incomplete for index 85 = \"\r\n        Warning: properties incomplete for index 86 = {\r\n        Warning: properties incomplete for index 87 = ¢\r\n        Warning: properties incomplete for index 88 = /\r\n        Warning: properties incomplete for index 89 = Q\r\n        Warning: properties incomplete for index 90 = 6\r\n        Warning: properties incomplete for index 91 = G\r\n        Warning: properties incomplete for index 92 = ”\r\n        Warning: properties incomplete for index 93 = °\r\n        Warning: properties incomplete for index 94 = K\r\n        Warning: properties incomplete for index 95 = ¥\r\n        Warning: properties incomplete for index 96 = V\r\n        Warning: properties incomplete for index 97 = ©\r\n        Warning: properties incomplete for index 98 = z\r\n        Warning: properties incomplete for index 99 = +\r\n        Warning: properties incomplete for index 100 = =\r\n        Warning: properties incomplete for index 101 = £\r\n        Warning: properties incomplete for index 102 = <\r\n        Warning: properties incomplete for index 103 = ’\r\n        Warning: properties incomplete for index 104 = ‘\r\n        Warning: properties incomplete for index 105 = j\r\n        Warning: properties incomplete for index 106 = X\r\n        Warning: properties incomplete for index 107 = ]\r\n        Warning: properties incomplete for index 108 = *\r\n        Warning: properties incomplete for index 109 = “\r\n        Warning: properties incomplete for index 110 = @\r\n        Warning: properties incomplete for index 111 = •\r\n        Warning: properties incomplete for index 112 = –\r\n        Warning: properties incomplete for index 113 = …\r\n        Warning: properties incomplete for index 114 = ^\r\n\r\nAnd this incomplete file :\r\n\r\n        115\r\n        NULL 0 Common 0\r\n        Joined 7 0,255,0,255,0,0,0,0,0,0 Latin 1 0 1 Joined\t# Joined [4a 6f 69 6e 65 64 ]a\r\n        |Broken|0|1 f 0,255,0,255,0,0,0,0,0,0 Common 2 10 2 |Broken|0|1\t# Broken\r\n        d 3 0,255,0,255,0,0,0,0,0,0 Latin 26 0 3 d\t# d [64 ]a\r\n        i 3 0,255,0,255,0,0,0,0,0,0 Latin 54 0 4 i\t# i [69 ]a\r\n        f 3 0,255,0,255,0,0,0,0,0,0 Latin 52 0 5 f\t# f [66 ]a\r\n        e 3 0,255,0,255,0,0,0,0,0,0 Latin 57 0 6 e\t# e [65 ]a\r\n        r 3 0,255,0,255,0,0,0,0,0,0 Latin 53 0 7 r\t# r [72 ]a\r\n        n 3 0,255,0,255,0,0,0,0,0,0 Latin 10 0 8 n\t# n [6e ]a\r\n        t 3 0,255,0,255,0,0,0,0,0,0 Latin 21 0 9 t\t# t [74 ]a\r\n        N 5 0,255,0,255,0,0,0,0,0,0 Latin 8 0 10 N\t# N [4e ]A\r\n        w 3 0,255,0,255,0,0,0,0,0,0 Latin 74 0 11 w\t# w [77 ]a\r\n        A 5 0,255,0,255,0,0,0,0,0,0 Latin 17 0 12 A\t# A [41 ]A\r\n        c 3 0,255,0,255,0,0,0,0,0,0 Latin 27 0 13 c\t# c [63 ]a\r\n        l 3 0,255,0,255,0,0,0,0,0,0 Latin 68 0 14 l\t# l [6c ]a\r\n        s 3 0,255,0,255,0,0,0,0,0,0 Latin 23 0 15 s\t# s [73 ]a\r\n        p 3 0,255,0,255,0,0,0,0,0,0 Latin 40 0 16 p\t# p [70 ]a\r\n        a 3 0,255,0,255,0,0,0,0,0,0 Latin 12 0 17 a\t# a [61 ]a\r\n        g 3 0,255,0,255,0,0,0,0,0,0 Latin 91 0 18 g\t# g [67 ]a\r\n        2 8 0,255,0,255,0,0,0,0,0,0 Common 19 2 19 2\t# 2 [32 ]0\r\n        3 8 0,255,0,255,0,0,0,0,0,0 Common 20 2 20 3\t# 3 [33 ]0\r\n        T 5 0,255,0,255,0,0,0,0,0,0 Latin 9 0 21 T\t# T [54 ]A\r\n        o 3 0,255,0,255,0,0,0,0,0,0 Latin 42 0 22 o\t# o [6f ]a\r\n        S 5 0,255,0,255,0,0,0,0,0,0 Latin 15 0 23 S\t# S [53 ]A\r\n        v 3 0,255,0,255,0,0,0,0,0,0 Latin 96 0 24 v\t# v [76 ]a\r\n        ~ 0 0,255,0,255,0,0,0,0,0,0 Common 25 10 25 ~\t# ~ [7e ]\r\n        D 5 0,255,0,255,0,0,0,0,0,0 Latin 3 0 26 D\t# D [44 ]A\r\n        C 5 0,255,0,255,0,0,0,0,0,0 Latin 13 0 27 C\t# C [43 ]A\r\n        h 3 0,255,0,255,0,0,0,0,0,0 Latin 41 0 28 h\t# h [68 ]a\r\n        ' 10 0,255,0,255,0,0,0,0,0,0 Common 29 10 29 '\t# ' [27 ]p\r\n        7 8 0,255,0,255,0,0,0,0,0,0 Common 30 2 30 7\t# 7 [37 ]0\r\n        « 10 0,255,0,255,0,0,0,0,0,0 Common 31 10 72 «\t# « [ab ]p\r\n        : 10 0,255,0,255,0,0,0,0,0,0 Common 32 6 32 :\t# : [3a ]p\r\n        # 10 0,255,0,255,0,0,0,0,0,0 Common 33 4 33 #\t# # [23 ]p\r\n        1 8 0,255,0,255,0,0,0,0,0,0 Common 34 2 34 1\t# 1 [31 ]0\r\n        Z 5 0,255,0,255,0,0,0,0,0,0 Latin 98 0 35 Z\t# Z [5a ]A\r\n        _ 10 0,255,0,255,0,0,0,0,0,0 Common 36 10 36 _\t# _ [5f ]p\r\n        M 5 0,255,0,255,0,0,0,0,0,0 Latin 39 0 37 M\t# M [4d ]A\r\n        u 3 0,255,0,255,0,0,0,0,0,0 Latin 48 0 38 u\t# u [75 ]a\r\n        m 3 0,255,0,255,0,0,0,0,0,0 Latin 37 0 39 m\t# m [6d ]a\r\n        P 5 0,255,0,255,0,0,0,0,0,0 Latin 16 0 40 P\t# P [50 ]A\r\n        H 5 0,255,0,255,0,0,0,0,0,0 Latin 28 0 41 H\t# H [48 ]A\r\n        O 5 0,255,0,255,0,0,0,0,0,0 Latin 22 0 42 O\t# O [4f ]A\r\n        ( 10 0,255,0,255,0,0,0,0,0,0 Common 43 10 44 (\t# ( [28 ]p\r\n        ) 10 0,255,0,255,0,0,0,0,0,0 Common 44 10 43 )\t# ) [29 ]p\r\n        q 3 0,255,0,255,0,0,0,0,0,0 Latin 89 0 45 q\t# q [71 ]a\r\n        y 3 0,255,0,255,0,0,0,0,0,0 Latin 83 0 46 y\t# y [79 ]a\r\n        | 0 0,255,0,255,0,0,0,0,0,0 Common 47 10 47 |\t# | [7c ]\r\n        U 5 0,255,0,255,0,0,0,0,0,0 Latin 38 0 48 U\t# U [55 ]A\r\n        0 8 0,255,0,255,0,0,0,0,0,0 Common 49 2 49 0\t# 0 [30 ]0\r\n        % 10 0,255,0,255,0,0,0,0,0,0 Common 50 4 50 %\t# % [25 ]p\r\n        x 3 0,255,0,255,0,0,0,0,0,0 Latin 106 0 51 x\t# x [78 ]a\r\n        F 5 0,255,0,255,0,0,0,0,0,0 Latin 5 0 52 F\t# F [46 ]A\r\n        R 5 0,255,0,255,0,0,0,0,0,0 Latin 7 0 53 R\t# R [52 ]A\r\n        I 5 0,255,0,255,0,0,0,0,0,0 Latin 4 0 54 I\t# I [49 ]A\r\n        , 10 0,255,0,255,0,0,0,0,0,0 Common 55 6 55 ,\t# , [2c ]p\r\n        ! 10 0,255,0,255,0,0,0,0,0,0 Common 56 10 56 !\t# ! [21 ]p\r\n        E 5 0,255,0,255,0,0,0,0,0,0 Latin 6 0 57 E\t# E [45 ]A\r\n        b 3 0,255,0,255,0,0,0,0,0,0 Latin 64 0 58 b\t# b [62 ]a\r\n        \\ 10 0,255,0,255,0,0,0,0,0,0 Common 59 10 59 \\\t# \\ [5c ]p\r\n        8 8 0,255,0,255,0,0,0,0,0,0 Common 60 2 60 8\t# 8 [38 ]0\r\n        ? 10 0,255,0,255,0,0,0,0,0,0 Common 61 10 61 ?\t# ? [3f ]p\r\n        & 10 0,255,0,255,0,0,0,0,0,0 Common 62 10 62 &\t# & [26 ]p\r\n        ; 10 0,255,0,255,0,0,0,0,0,0 Common 63 10 63 ;\t# ; [3b ]p\r\n        B 5 0,255,0,255,0,0,0,0,0,0 Latin 58 0 64 B\t# B [42 ]A\r\n        k 3 0,255,0,255,0,0,0,0,0,0 Latin 94 0 65 k\t# k [6b ]a\r\n        - 10 0,255,0,255,0,0,0,0,0,0 Common 66 3 66 -\t# - [2d ]p\r\n        > 0 0,255,0,255,0,0,0,0,0,0 Common 67 10 102 >\t# > [3e ]\r\n        L 5 0,255,0,255,0,0,0,0,0,0 Latin 14 0 68 L\t# L [4c ]A\r\n        . 10 0,255,0,255,0,0,0,0,0,0 Common 69 6 69 .\t# . [2e ]p\r\n        — 10 0,255,0,255,0,0,0,0,0,0 Common 70 10 70 -\t# — [2014 ]p\r\n        4 8 0,255,0,255,0,0,0,0,0,0 Common 71 2 71 4\t# 4 [34 ]0\r\n        » 10 0,255,0,255,0,0,0,0,0,0 Common 72 10 31 »\t# » [bb ]p\r\n        € 0 0,255,0,255,0,0,0,0,0,0 Common 73 4 73 €\t# € [20ac ]\r\n        W 5 0,255,0,255,0,0,0,0,0,0 Latin 11 0 74 W\t# W [57 ]A\r\n        J 5 0,255,0,255,0,0,0,0,0,0 Latin 105 0 75 J\t# J [4a ]A\r\n        é 3 0,255,0,255,0,0,0,0,0,0 Latin 76 0 76 é\t# é [e9 ]a\r\n        9 8 0,255,0,255,0,0,0,0,0,0 Common 77 2 77 9\t# 9 [39 ]0\r\n        ® 0 0,255,0,255,0,0,0,0,0,0 Common 78 10 78 ®\t# ® [ae ]\r\n        $ 0 0,255,0,255,0,0,0,0,0,0 Common 79 4 79 $\t# $ [24 ]\r\n        5 8 0,255,0,255,0,0,0,0,0,0 Common 80 2 80 5\t# 5 [35 ]0\r\n        } 10 0,255,0,255,0,0,0,0,0,0 Common 81 10 86 }\t# } [7d ]p\r\n        [ 10 0,255,0,255,0,0,0,0,0,0 Common 82 10 107 [\t# [ [5b ]p\r\n        Y 5 0,255,0,255,0,0,0,0,0,0 Latin 46 0 83 Y\t# Y [59 ]A\r\n        § 10 0,255,0,255,0,0,0,0,0,0 Common 84 10 84 §\t# § [a7 ]p\r\n        \" 10 0,255,0,255,0,0,0,0,0,0 Common 85 10 85 \"\t# \" [22 ]p\r\n        { 10 0,255,0,255,0,0,0,0,0,0 Common 86 10 81 {\t# { [7b ]p\r\n        ¢ 0 0,255,0,255,0,0,0,0,0,0 Common 87 4 87 ¢\t# ¢ [a2 ]\r\n        / 10 0,255,0,255,0,0,0,0,0,0 Common 88 6 88 /\t# / [2f ]p\r\n        Q 5 0,255,0,255,0,0,0,0,0,0 Latin 45 0 89 Q\t# Q [51 ]A\r\n        6 8 0,255,0,255,0,0,0,0,0,0 Common 90 2 90 6\t# 6 [36 ]0\r\n        G 5 0,255,0,255,0,0,0,0,0,0 Latin 18 0 91 G\t# G [47 ]A\r\n        ” 10 0,255,0,255,0,0,0,0,0,0 Common 92 10 92 \"\t# ” [201d ]p\r\n        ° 0 0,255,0,255,0,0,0,0,0,0 Common 93 4 93 °\t# ° [b0 ]\r\n        K 5 0,255,0,255,0,0,0,0,0,0 Latin 65 0 94 K\t# K [4b ]A\r\n        ¥ 0 0,255,0,255,0,0,0,0,0,0 Common 95 4 95 ¥\t# ¥ [a5 ]\r\n        V 5 0,255,0,255,0,0,0,0,0,0 Latin 24 0 96 V\t# V [56 ]A\r\n        © 0 0,255,0,255,0,0,0,0,0,0 Common 97 10 97 ©\t# © [a9 ]\r\n        z 3 0,255,0,255,0,0,0,0,0,0 Latin 35 0 98 z\t# z [7a ]a\r\n        + 0 0,255,0,255,0,0,0,0,0,0 Common 99 3 99 +\t# + [2b ]\r\n        = 0 0,255,0,255,0,0,0,0,0,0 Common 100 10 100 =\t# = [3d ]\r\n        £ 0 0,255,0,255,0,0,0,0,0,0 Common 101 4 101 £\t# £ [a3 ]\r\n        < 0 0,255,0,255,0,0,0,0,0,0 Common 102 10 67 <\t# < [3c ]\r\n        ’ 10 0,255,0,255,0,0,0,0,0,0 Common 103 10 103 '\t# ’ [2019 ]p\r\n        ‘ 10 0,255,0,255,0,0,0,0,0,0 Common 104 10 104 '\t# ‘ [2018 ]p\r\n        j 3 0,255,0,255,0,0,0,0,0,0 Latin 75 0 105 j\t# j [6a ]a\r\n        X 5 0,255,0,255,0,0,0,0,0,0 Latin 51 0 106 X\t# X [58 ]A\r\n        ] 10 0,255,0,255,0,0,0,0,0,0 Common 107 10 82 ]\t# ] [5d ]p\r\n        * 10 0,255,0,255,0,0,0,0,0,0 Common 108 10 108 *\t# * [2a ]p\r\n        “ 10 0,255,0,255,0,0,0,0,0,0 Common 109 10 109 \"\t# “ [201c ]p\r\n        @ 10 0,255,0,255,0,0,0,0,0,0 Common 110 10 110 @\t# @ [40 ]p\r\n        • 10 0,255,0,255,0,0,0,0,0,0 Common 111 10 111 •\t# • [2022 ]p\r\n        – 10 0,255,0,255,0,0,0,0,0,0 Common 112 10 112 -\t# – [2013 ]p\r\n        … 10 0,255,0,255,0,0,0,0,0,0 Common 113 10 113 ...\t# … [2026 ]p\r\n        ^ 0 0,255,0,255,0,0,0,0,0,0 Common 114 10 114 ^\t# ^ [5e ]\r\n\r\nWe can see that some parts are missing.\r\n\r\nHere from the documentation to see the difference, so the \"missing\" part :\r\nhttps://github.com/tesseract-ocr/tesseract/blob/a3ba11b030345d32829b1e8355afea5419978d82/doc/unicharset.5.asc\r\n\r\n        EXAMPLE (v3.02)\r\n\r\n        110\r\n        NULL 0 NULL 0\r\n        N 5 59,68,216,255,87,236,0,27,104,227 Latin 11 0 1 N\r\n        Y 5 59,68,216,255,91,205,0,47,91,223 Latin 33 0 2 Y\r\n        1 8 59,69,203,255,45,128,0,66,74,173 Common 3 2 3 1\r\n        9 8 18,66,203,255,89,156,0,39,104,173 Common 4 2 4 9\r\n        a 3 58,65,186,198,85,164,0,26,97,185 Latin 56 0 5 a\r\n        . . .\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/comments/217926438","html_url":"https://github.com/tesseract-ocr/tesseract/issues/318#issuecomment-217926438","issue_url":"https://api.github.com/repos/tesseract-ocr/tesseract/issues/318","id":217926438,"user":{"login":"ne0zer0","id":10687250,"avatar_url":"https://avatars.githubusercontent.com/u/10687250?v=3","gravatar_id":"","url":"https://api.github.com/users/ne0zer0","html_url":"https://github.com/ne0zer0","followers_url":"https://api.github.com/users/ne0zer0/followers","following_url":"https://api.github.com/users/ne0zer0/following{/other_user}","gists_url":"https://api.github.com/users/ne0zer0/gists{/gist_id}","starred_url":"https://api.github.com/users/ne0zer0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ne0zer0/subscriptions","organizations_url":"https://api.github.com/users/ne0zer0/orgs","repos_url":"https://api.github.com/users/ne0zer0/repos","events_url":"https://api.github.com/users/ne0zer0/events{/privacy}","received_events_url":"https://api.github.com/users/ne0zer0/received_events","type":"User","site_admin":false},"created_at":"2016-05-09T17:09:32Z","updated_at":"2016-05-09T17:09:32Z","body":"Thank you for your reply,\r\n\r\nBut what about theses questions:\r\n\r\n> But if I put Latin.unicharset, what I get is not what is adapted for my fonts,\r\n> xheights can change per fonts:\r\n> https://raw.githubusercontent.com/tesseract-ocr/langdata/master/Latin.xheights\r\n> So how can I be sure that this default value is correct for my all fonts???\r\n> \r\n> As says ggdhines, according to the documentation, we have to compute the actual size of our fonts:\r\n> \r\n> https://github.com/tesseract-ocr/tesseract/blob/a3ba11b030345d32829b1e8355afea5419978d82/doc/unicharset.5.asc\r\n> \r\n>     CAVEATS\r\n> \r\n>     Although the unicharset reader maintains the ability to read unicharsets of older formats and will assign default values to missing fields, the accuracy will be degraded.\r\n> \r\n> What you suggest is likely to produce such degraded result. (what I seem to experiment)\r\n> \r\n> Always for the above link:\r\n> \r\n>     Further, most other data files are indexed by the unicharset file, so changing it without re-generating the others is likely to have dire consequences\r\n> \r\n> So, as it is stated that \"assign default values to missing fields, the accuracy will be degraded\", and \"is likely to have dire consequences\", your proposition can not be accepted, because it lead to what I experiment if I do like you say: strange results.\r\n> \r\n> Unless I did not understand anything, in which case, as I am not the only one, you have to review the documentation.\r\n\r\n"}}