{"push_id":1030980974,"size":149,"distinct_size":149,"ref":"refs/heads/master","head":"97543daa5b1fb47efec82dc55ab3c26ddf9fe8eb","before":"e902bc3906ac0ef55ff189dae6ba69a191c3117d","commits":[{"sha":"c47908e663ea7b441bbac0445e584bd364811b5d","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Bump version to development version 1.3.1.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/c47908e663ea7b441bbac0445e584bd364811b5d"},{"sha":"fb8c5b6d4c218a2d6e9194a2d0e8944482d4b245","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Document the behaviour stability of tokenizers.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/fb8c5b6d4c218a2d6e9194a2d0e8944482d4b245"},{"sha":"6979f4b844151cba1e376e65314d6a91444c9c7a","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Move eos_exception test to utf8_tokenizer.\n\nThis allows customizing the eos_word_exceptions, which will be needed in\nczech_tokenizer soon to implement czech/slovak eow_word_exceptions.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/6979f4b844151cba1e376e65314d6a91444c9c7a"},{"sha":"f5dd1d2166f09cc2ff275d5dd9a8454f37405d01","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Support different modes in czech_tokenizer.\n\nWe want to support also Slovak mode.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/f5dd1d2166f09cc2ff275d5dd9a8454f37405d01"},{"sha":"0f81a2885f573cafcd0c7e7f0dc8ec4309efc71a","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Implement hyphenated_sequences_index.\n\nThe hyphenated_sequences_index allows joining hyphenated words,\nwhen these are split by the tokenizer, as in czech_tokenizer.\n\nPreviously, the Ragel tokenizer contained the hyphenated words.\nSuch tokenizer was quite large (260kB C++ source vs 115kB source)\nand the list of hyphenated words was not configurable.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/0f81a2885f573cafcd0c7e7f0dc8ec4309efc71a"},{"sha":"c560bd96512aae76145bd5a29508d5cee6819dd9","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Add SLOVAK_GENERIC mode to czech_tokenizer.\n\nThe SLOVAK_GENERIC mode has the same core tokenizer, and Slovak-specific\neos_word_exceptions.\n\nCurrently there are no Slovak-specific hyphenated_sequences, as there\nare no hyphenated words in Slovak morphological dictionary.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/c560bd96512aae76145bd5a29508d5cee6819dd9"},{"sha":"d38407be1b70041f9fefe9cd67a2f321e9b07465","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Rename hyphenated_sequences_index to hyphenated_sequences_map.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/d38407be1b70041f9fefe9cd67a2f321e9b07465"},{"sha":"e2d7720f927e59dab9157c590e9e34fef552dcbd","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Ignore .swp files.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/e2d7720f927e59dab9157c590e9e34fef552dcbd"},{"sha":"2023f5311bdc4a4e1e4e08540c09d6a6fd643072","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Make emergency_sentence_split proper method instead of inline.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/2023f5311bdc4a4e1e4e08540c09d6a6fd643072"},{"sha":"a72617d98216c13a7a7d534c49b6d4ebabc47aa3","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Change the way how token splitting is implemented.\n\nPreviously, token splitting was done in the tokenizer itself. That has\nseveral problems:\n- it is complicated for the splitting rules to be customizable\n- the resulting source and binary is quite large, as many rules fire\n  on suffixes\n- marking the place where to split is burdensome\n\nNow we have a separate method split_token, which is given a token\nand it tries to split it. This allows:\n- choosing the split method (with correct signature) dynamically\n- smaller source/binary, as the matching in the split_token is performed\n  backwards\n- marking the split point \"visually\"","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/a72617d98216c13a7a7d534c49b6d4ebabc47aa3"},{"sha":"7ff32b0f2f8ad141aaa3794e92b584f1f46f49a7","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Make english_tokenizer versionable and modify latest version.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/7ff32b0f2f8ad141aaa3794e92b584f1f46f49a7"},{"sha":"4d9ce76491a427a5726123158700885267909f78","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Make split_token also perform the split, not only return its length.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/4d9ce76491a427a5726123158700885267909f78"},{"sha":"d1d842e2633f1f33126dc8836ca59ec3d434e23d","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Remove two inactive rules from split_token.\n\nThe rules removed were 't|is and 't|was. Nevertheless, such inputs are\ntokenized as ' tis and ' twas. The same applied for English tokenizer\nfrom Treex.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/d1d842e2633f1f33126dc8836ca59ec3d434e23d"},{"sha":"db798227079b5e12e9431657661ff68f80baade1","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Add ENGLISH_V3 morpho_id.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/db798227079b5e12e9431657661ff68f80baade1"},{"sha":"826016a96557c480d9ed599295ae4042af125a58","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Make czech_tokenizer and czech_morpho versionable.\n\nUse language,version pair, because Slovak can also be used.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/826016a96557c480d9ed599295ae4042af125a58"},{"sha":"ed12e4d6d200470c3a93e2c64cd2b133f7146c45","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Make lists using morpho_ids ascending.\n\nNamely morpho::parse, morpho::load and the switch in encode_dictionary.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/ed12e4d6d200470c3a93e2c64cd2b133f7146c45"},{"sha":"876af72cc358d928508436dedc1a469d9a392936","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Add morpho_id SLOVAK_PDT.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/876af72cc358d928508436dedc1a469d9a392936"},{"sha":"42ab768bcae09c51fa07c677b366b650c9bb97bb","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Fix signed/unsigned warning by reordering operations.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/42ab768bcae09c51fa07c677b366b650c9bb97bb"},{"sha":"d2fd9fdd592e5b5ecb9c9e0afa4fe54bb022c9db","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Use lstlisting package for automatic line breaking...\n\nin verbatim environment.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/d2fd9fdd592e5b5ecb9c9e0afa4fe54bb022c9db"},{"sha":"b64e5ce56275bb3fe66a89bb2f49f5f639166939","author":{"email":"ff0f0a8b656f0b44c26933acd2e367b6c1211290@ucw.cz","name":"Milan Straka"},"message":"Remove the first and last empty line from txt target...\n\ninside txt2tags and not in the Makefile.","distinct":true,"url":"https://api.github.com/repos/uetoyo/morphodita/commits/b64e5ce56275bb3fe66a89bb2f49f5f639166939"}]}