{"action":"closed","issue":{"url":"https://api.github.com/repos/shangjingbo1226/SegPhrase/issues/3","repository_url":"https://api.github.com/repos/shangjingbo1226/SegPhrase","labels_url":"https://api.github.com/repos/shangjingbo1226/SegPhrase/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/shangjingbo1226/SegPhrase/issues/3/comments","events_url":"https://api.github.com/repos/shangjingbo1226/SegPhrase/issues/3/events","html_url":"https://github.com/shangjingbo1226/SegPhrase/issues/3","id":136453213,"number":3,"title":"Using SegPhrase on other text corpus ","user":{"login":"palansuya","id":17479365,"avatar_url":"https://avatars.githubusercontent.com/u/17479365?v=3","gravatar_id":"","url":"https://api.github.com/users/palansuya","html_url":"https://github.com/palansuya","followers_url":"https://api.github.com/users/palansuya/followers","following_url":"https://api.github.com/users/palansuya/following{/other_user}","gists_url":"https://api.github.com/users/palansuya/gists{/gist_id}","starred_url":"https://api.github.com/users/palansuya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palansuya/subscriptions","organizations_url":"https://api.github.com/users/palansuya/orgs","repos_url":"https://api.github.com/users/palansuya/repos","events_url":"https://api.github.com/users/palansuya/events{/privacy}","received_events_url":"https://api.github.com/users/palansuya/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-02-25T17:21:44Z","updated_at":"2016-07-12T02:31:38Z","closed_at":"2016-07-12T02:31:38Z","body":"I am trying to run SegPhrase on a dataset of Academic papers on bioengineering. Each line of the of the data is one paper. I have changed to the train_toy.sh file's RAW_TEXT to be my new dataset. The rest it kept the same. \r\n\r\nWhen, however, I run the training, it fail's with the following messages. \r\n\r\n> \r\n# Sentences = 79178\r\n# tokens =  532593\r\n# of distinct tokens =  12633\r\n# of frequent pattern =  11862\r\nfeature extraction done.\r\n===Auto Label Enable===\r\nTraceback (most recent call last):\r\n  File \"src/classification/auto_label_generation.py\", line 112, in <module>\r\n    kmeans.fit(matrixOther)\r\n  File \"/home/apps/python/python-2.7.9/lib/python2.7/site-packages/sklearn/cluster/k_means_.py\", line 1235, in fit\r\n    X = check_array(X, accept_sparse=\"csr\", order='C', dtype=np.float64)\r\n  File \"/home/apps/python/python-2.7.9/lib/python2.7/site-packages/sklearn/utils/validation.py\", line 398, in check_array\r\n    _assert_all_finite(array)\r\n  File \"/home/apps/python/python-2.7.9/lib/python2.7/site-packages/sklearn/utils/validation.py\", line 54, in _assert_all_finite\r\n    \" or a value too large for %r.\" % X.dtype)\r\nValueError: Input contains NaN, infinity or a value too large for dtype('float64').\r\n[Warning] failed to open data/wiki.label.auto under parameters = r\r\n./train_ChipSeq.sh: line 52: 46775 Segmentation fault      ./bin/predict_quality results/feature_table_0.csv ${DATA_LABEL} results/ranking.csv outsideSentence,log_occur_feature,constant,frequency 0 TRAIN results/random_forest_0.model\r\n# Sentences = 88897\r\n# Unigrams = 15865\r\n[Warning] failed to open results/ranking.csv under parameters = r\r\n./train_ChipSeq.sh: line 58: 46778 Segmentation fault      ./bin/adjust_probability tmp/sentences.buf ${OMP_NUM_THREADS} results/ranking.csv results/patterns.csv ${DISCARD_RATIO} ${MAX_ITERATION} ./results/ ${DATA_LABEL} ./results/penalty.1\r\n[Warning] failed to open ./results/penalty.1 under parameters = r\r\n./train_ChipSeq.sh: line 61: 46782 Segmentation fault      ./bin/recompute_features results/iter${MAX_ITERATION_1}_discard${DISCARD_RATIO}/length results/feature_table_0.csv results/patterns.csv tmp/sentencesWithPunc.buf results/feature_table_1.csv ./results/penalty.1 1\r\n[Warning] failed to open data/wiki.label.auto under parameters = r\r\n./train_ChipSeq.sh: line 62: 46783 Segmentation fault      ./bin/predict_quality results/feature_table_1.csv ${DATA_LABEL} results/ranking_1.csv outsideSentence,log_occur_feature,constant,frequency 0 TRAIN results/random_forest_1.model\r\n[Warning] failed to open results/ranking_1.csv under parameters = r\r\n./train_ChipSeq.sh: line 63: 46784 Segmentation fault      ./bin/adjust_probability tmp/sentences.buf ${OMP_NUM_THREADS} results/ranking_1.csv results/patterns.csv ${DISCARD_RATIO} ${MAX_ITERATION} ./results/1. ${DATA_LABEL} ./results/penalty.2\r\n[Warning] failed to open ./results/penalty.2 under parameters = r\r\n./train_ChipSeq.sh: line 66: 46788 Segmentation fault      ./bin/build_model results/1.iter${MAX_ITERATION_1}_discard${DISCARD_RATIO}/ 6 ./results/penalty.2 results/segmentation.model\r\n===Unigram Disable===\r\n[Warning] failed to open results/1.iter6_discard0.00//length1.csv under parameters = r\r\n[Warning] failed to open results/1.iter6_discard0.00//length2.csv under parameters = r\r\n[Warning] failed to open results/1.iter6_discard0.00//length3.csv under parameters = r\r\n[Warning] failed to open results/1.iter6_discard0.00//length4.csv under parameters = r\r\n[Warning] failed to open results/1.iter6_discard0.00//length5.csv under parameters = r\r\n[Warning] failed to open results/1.iter6_discard0.00//length6.csv under parameters = r\r\nTraceback (most recent call last):\r\n  File \"src/postprocessing/filter_by_support.py\", line 37, in <module>\r\n    main(sys.argv[1:])\r\n  File \"src/postprocessing/filter_by_support.py\", line 17, in main\r\n    for line in open(segmented_corpus_filename):\r\nIOError: [Errno 2] No such file or directory: 'results/1.iter5_discard0.00/segmented.txt'\r\n\r\nI speculate maybe the each data,i.e. paper, is too long to parse? \r\n\r\nWhat do you suggest I do in order to use SegPhrase on this dataset. \r\n\r\nAlso may I ask, for the train_dblp.sh, how did you collect or obtain the DBLP.label? \r\n\r\nI figure it will provide a higher quality phrases when labels are provide rather than enabling Auto Label feature. \r\n\r\n"}}
{"actor":{"display_login":"shangjingbo1226"}}