{"action":"created","issue":{"url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/58","repository_url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator","labels_url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/58/comments","events_url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/58/events","html_url":"https://github.com/MicrosoftTranslator/DocumentTranslator/issues/58","id":323340702,"node_id":"MDU6SXNzdWUzMjMzNDA3MDI=","number":58,"title":"Infinite loop in SplitList","user":{"login":"jsypkens","id":37302381,"node_id":"MDQ6VXNlcjM3MzAyMzgx","avatar_url":"https://avatars1.githubusercontent.com/u/37302381?v=4","gravatar_id":"","url":"https://api.github.com/users/jsypkens","html_url":"https://github.com/jsypkens","followers_url":"https://api.github.com/users/jsypkens/followers","following_url":"https://api.github.com/users/jsypkens/following{/other_user}","gists_url":"https://api.github.com/users/jsypkens/gists{/gist_id}","starred_url":"https://api.github.com/users/jsypkens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsypkens/subscriptions","organizations_url":"https://api.github.com/users/jsypkens/orgs","repos_url":"https://api.github.com/users/jsypkens/repos","events_url":"https://api.github.com/users/jsypkens/events{/privacy}","received_events_url":"https://api.github.com/users/jsypkens/received_events","type":"User","site_admin":false},"labels":[{"id":184349144,"node_id":"MDU6TGFiZWwxODQzNDkxNDQ=","url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-15T18:53:35Z","updated_at":"2018-09-11T15:53:38Z","closed_at":null,"author_association":"NONE","body":"In DocumentTranslationManager, the method \"SplitList\" has an infinite loop possibility. When an individual element exceeds the maxSize, it will do the following:\r\n\r\n1. Attempt to add UP TO groupSize elements to the list\r\n2. Remove them, one by one, as they all exceed the maxSize\r\n3. Once it gets to the last item in the list, it also exceeds maxSize, so it gets removed\r\n4. An empty list is added to the \"result\" list.\r\n5. Loop - start again at 1 above.  Continually adding to the result list.\r\n\r\nThis is more apparent with Text Translator v3 API, as the element and list sizes are smaller. One contiguous paragraph doesn't need to be very long to cause this behavior."},"comment":{"url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/comments/420323680","html_url":"https://github.com/MicrosoftTranslator/DocumentTranslator/issues/58#issuecomment-420323680","issue_url":"https://api.github.com/repos/MicrosoftTranslator/DocumentTranslator/issues/58","id":420323680,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDMyMzY4MA==","user":{"login":"jsypkens","id":37302381,"node_id":"MDQ6VXNlcjM3MzAyMzgx","avatar_url":"https://avatars1.githubusercontent.com/u/37302381?v=4","gravatar_id":"","url":"https://api.github.com/users/jsypkens","html_url":"https://github.com/jsypkens","followers_url":"https://api.github.com/users/jsypkens/followers","following_url":"https://api.github.com/users/jsypkens/following{/other_user}","gists_url":"https://api.github.com/users/jsypkens/gists{/gist_id}","starred_url":"https://api.github.com/users/jsypkens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsypkens/subscriptions","organizations_url":"https://api.github.com/users/jsypkens/orgs","repos_url":"https://api.github.com/users/jsypkens/repos","events_url":"https://api.github.com/users/jsypkens/events{/privacy}","received_events_url":"https://api.github.com/users/jsypkens/received_events","type":"User","site_admin":false},"created_at":"2018-09-11T15:53:38Z","updated_at":"2018-09-11T15:53:38Z","author_association":"NONE","body":"I made a few other changes to the method as well, and the implementation changed a little bit, so I'm not sure if my fix would be what you're looking for.\r\n\r\nI separated out the \"maxSize\" into two parameters - maxListSize and maxElementSize, then looped through each element in values, checking if it would exceed the group or the element size. I also changed the return type to List<List<<Tuple<int,string>>, for two main reasons:\r\n1. It will track the source line number of each string element, so I can translate all the elements from multiple batches in parallel and then re-assemble the translated elements in the right order\r\n2. The v3 API calls fail when translating empty strings or whitespace (which is common when extracting text out of documents), so I'm removing those when translating, but re-inserting the white space in the correct place after translation returns\r\n\r\n`     public static List<List<Tuple<int, string>>> SplitList2(IEnumerable<string> values, int groupSize, int maxListSize, int maxElementSize)\r\n        {\r\n            List<List<Tuple<int, string>>> result = new List<List<Tuple<int, string>>>();\r\n            int currentLineNumber = 0;\r\n            int currentListSize = 0;\r\n\r\n            List<Tuple<int, string>> newList = new List<Tuple<int, string>>();\r\n\r\n            foreach (var value in values)\r\n            {\r\n                if (newList.Count > groupSize)\r\n                {\r\n                    result.Add(newList);\r\n                    newList = new List<Tuple<int, string>>();\r\n                    currentListSize = 0;\r\n                }\r\n\r\n                if (value.Length > maxElementSize) //current item is longer than item max, split it on sentence/paragraph\r\n                {\r\n                    if (value.Length > maxListSize)\r\n                    {\r\n                        //Split this item up! by itself, it's too long for the list. Use BreakSentence API to split on sentences.                        \r\n                        BreakSentenceResponse[] BRR = TranslationServiceFacade.BreakSentences(new string[] { value });\r\n                        List<string> sentences = new List<string>();\r\n                        int currentPosition = 0;\r\n                        foreach (var sentSplitLocation in BRR[0].sentLen) //BRR[0]: we're only passing in one value, we should only get one out\r\n                        {\r\n                            string currentSentence = value.Substring(currentPosition, sentSplitLocation);\r\n                            if (sentences.Count > 0)\r\n                            {\r\n                                //Check to see if we will exceed the maxElementSize if we added this sentence\r\n                                if (sentences.Last().Length + currentSentence.Length + 1 > maxElementSize)\r\n                                    sentences.Add(currentSentence); //would exceed the max, so just add it to the list separately\r\n                                else\r\n                                    sentences[sentences.Count() - 1] += \" \" + currentSentence; //wouldn't exceed the max, append to the current list\r\n                            }\r\n                            else //first sentence, just add it\r\n                                sentences.Add(currentSentence);\r\n                            currentPosition += sentSplitLocation;\r\n                        }\r\n\r\n                        //loop through sentences, and add them to the list to be translated.\r\n                        foreach (var sent in sentences) //make sure extractedSentence is under maxElementSize\r\n                        {\r\n                            if (currentListSize + sent.Length >= maxListSize || newList.Count >= groupSize)\r\n                            {\r\n                                result.Add(newList);\r\n                                newList = new List<Tuple<int, string>>();\r\n                                currentListSize = 0;\r\n                            }\r\n                            newList.Add(new Tuple<int, string>(currentLineNumber, sent));\r\n                        }\r\n\r\n                    }\r\n                    else  //the element isn't too big for the list by itself...\r\n                    {\r\n                        if (currentListSize + value.Length >= maxListSize) //next item will put the list over the max, start a new list\r\n                        {\r\n                            result.Add(newList);\r\n                            newList = new List<Tuple<int, string>>();\r\n                            currentListSize = 0;\r\n                        }\r\n\r\n                        BreakSentenceResponse[] BRR = TranslationServiceFacade.BreakSentences(new string[] { value });\r\n                        List<string> sentences = new List<string>();\r\n                        int currentPosition = 0;\r\n                        foreach (var sentSplitLocation in BRR[0].sentLen) //we're only passing in one value, we should only get one out\r\n                        {\r\n                            string currentSentence = value.Substring(currentPosition, sentSplitLocation);\r\n                            if (sentences.Count > 0)\r\n                            {\r\n                                if (sentences.Last().Length + currentSentence.Length + 1 > maxElementSize)\r\n                                    sentences.Add(currentSentence);\r\n                                else\r\n                                    sentences[sentences.Count() - 1] += \" \" + currentSentence;\r\n                            }\r\n                            else\r\n                                sentences.Add(currentSentence);\r\n                            currentPosition += sentSplitLocation;\r\n                        }\r\n\r\n\r\n                        foreach (var sent in sentences) //make sure extractedSentence is under maxElementSize\r\n                        {\r\n                            if (currentListSize + sent.Length >= maxListSize || newList.Count >= groupSize)\r\n                            {\r\n                                result.Add(newList);\r\n                                newList = new List<Tuple<int, string>>();\r\n                                currentListSize = 0;\r\n                            }\r\n                            newList.Add(new Tuple<int, string>(currentLineNumber, sent));\r\n                            currentListSize += sent.Length;\r\n                        }\r\n                    }\r\n\r\n                }\r\n                else // element does not exceed the max element size\r\n                {\r\n                    if (currentListSize + value.Length >= maxListSize) //make sure it won't exceed the current list size\r\n                    {\r\n                        result.Add(newList);\r\n                        newList = new List<Tuple<int, string>>();\r\n                        currentListSize = 0;\r\n                    }\r\n\r\n                    newList.Add(new Tuple<int, string>(currentLineNumber, value));\r\n                    currentListSize += value.Length;\r\n                }\r\n                currentLineNumber++;\r\n            }\r\n            if (newList.Count > 0)\r\n                result.Add(newList);\r\n\r\n            foreach (var list in result)\r\n            {\r\n\r\n\r\n                int totalLength = 0;\r\n                foreach (var item in list)\r\n                {\r\n                    totalLength += item.Item2.Length;\r\n                }\r\n                if (totalLength > maxListSize)\r\n                {\r\n                    logger.Error(\"list exceeds max. why?\");\r\n                }\r\n            }\r\n\r\n            return result;\r\n        }\r\n`"}}
{"actor":{"display_login":"jsypkens"}}