{"action":"created","issue":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/168","repository_url":"https://api.github.com/repos/kbenoit/quanteda","labels_url":"https://api.github.com/repos/kbenoit/quanteda/issues/168/labels{/name}","comments_url":"https://api.github.com/repos/kbenoit/quanteda/issues/168/comments","events_url":"https://api.github.com/repos/kbenoit/quanteda/issues/168/events","html_url":"https://github.com/kbenoit/quanteda/issues/168","id":155361575,"number":168,"title":"Error in validObject(r) :    invalid class “dgTMatrix” object: length(Dimnames[2]) differs from Dim[2] which is 33411","user":{"login":"7804j","id":1547347,"avatar_url":"https://avatars.githubusercontent.com/u/1547347?v=3","gravatar_id":"","url":"https://api.github.com/users/7804j","html_url":"https://github.com/7804j","followers_url":"https://api.github.com/users/7804j/followers","following_url":"https://api.github.com/users/7804j/following{/other_user}","gists_url":"https://api.github.com/users/7804j/gists{/gist_id}","starred_url":"https://api.github.com/users/7804j/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/7804j/subscriptions","organizations_url":"https://api.github.com/users/7804j/orgs","repos_url":"https://api.github.com/users/7804j/repos","events_url":"https://api.github.com/users/7804j/events{/privacy}","received_events_url":"https://api.github.com/users/7804j/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-17T21:04:49Z","updated_at":"2016-05-18T09:53:30Z","closed_at":"2016-05-18T08:19:33Z","body":"Hi again,\r\n\r\nWhen I create a dfm from my emails corpus, I get the error message:\r\n\r\n> Creating a dfm from a corpus ...\r\n>   ... lowercasing\r\n>   ... tokenizing\r\n>   ... indexing documents: 1,882 documents\r\n>   ... indexing features: 4,596 feature types\r\n>Error in validObject(r) : \r\n>  invalid class “dgTMatrix” object: length(Dimnames[2]) differs from Dim[2] which is 33411\r\n\r\n\r\nBelow is my code. The data comes from https://www.kaggle.com/kaggle/hillary-clinton-emails/ again.\r\n\r\nAlso, interestingly, when I write LIMIT 100 at the end of the SQL query, with exactly the same code for everything else, I don't have any problem. So it might be linked with the size of the corpus.\r\n\r\n\r\n\r\n-----------------------------\r\n\r\n\r\n#Connect to db\r\ndb <- dbConnect(dbDriver(\"SQLite\"), \"output/database.sqlite\")\r\n\r\n#get all emails\r\nemails <- dbGetQuery(db, \"\r\n                     SELECT ExtractedBodyText body,MetadataSubject subject, MetadataDateSent date \r\n                     FROM Emails e \r\n                     INNER JOIN Persons p \r\n                     ON e.SenderPersonId=P.Id \r\n                     WHERE p.Name='Hillary Clinton'  \r\n                     AND e.ExtractedBodyText != '' \r\n                     ORDER BY RANDOM()\")\r\n\r\n#Create new column with weekdays, and column weekend\r\nemails = emails %>% separate(date,\"date\", sep = \"T\") %>% mutate(\r\n  weekday = weekdays(as.Date(emails$date)),\r\n  weekend = ifelse(weekday %in% c('Saturday','Sunday'),1,0)\r\n  )\r\n\r\n\r\n#Clean some of the email bodies that still contain part of the header (manual inspection of emails)\r\nemails = emails %>% mutate(body = sub(\"H <.*Re:.*\\\\n\", \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*Re:\", \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*RELEASE IN.*B6\", \"\", body)) %>%\r\n                    mutate(body = sub(\"RELEASE\\\\nIN PART B6\\\\n\", \"\", body)) %>%\r\n                    mutate(body = sub(\"RELEASE\\\\nIN PART B6\", \"\", body)) %>%\r\n                    mutate(body = sub(\"RELEASE IN PART\\\\nB6\", \"\", body)) %>%\r\n                    mutate(body = sub(\"RELEASE IN PART.*\\\\B1\", \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*Fw:\", \"\", body)) %>%\r\n                    mutate(body = sub('Declassify on: 04/23/2035', \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*B6\\\\nB6\\\\n\", \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*PM\\\\n\", \"\", body)) %>%\r\n                    mutate(body = sub(\"H <.*AM\\\\n\", \"\", body))\r\n\r\n#Create corpus\r\nemail_corpus = corpus(emails$body)\r\n\r\n#Create dfm, after stemming and removing stopwords\r\nemail_dfm <- dfm(email_corpus,ignoredFeatures = stopwords(\"english\"), stem = TRUE)\r\n"},"comment":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/comments/219979523","html_url":"https://github.com/kbenoit/quanteda/issues/168#issuecomment-219979523","issue_url":"https://api.github.com/repos/kbenoit/quanteda/issues/168","id":219979523,"user":{"login":"7804j","id":1547347,"avatar_url":"https://avatars.githubusercontent.com/u/1547347?v=3","gravatar_id":"","url":"https://api.github.com/users/7804j","html_url":"https://github.com/7804j","followers_url":"https://api.github.com/users/7804j/followers","following_url":"https://api.github.com/users/7804j/following{/other_user}","gists_url":"https://api.github.com/users/7804j/gists{/gist_id}","starred_url":"https://api.github.com/users/7804j/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/7804j/subscriptions","organizations_url":"https://api.github.com/users/7804j/orgs","repos_url":"https://api.github.com/users/7804j/repos","events_url":"https://api.github.com/users/7804j/events{/privacy}","received_events_url":"https://api.github.com/users/7804j/received_events","type":"User","site_admin":false},"created_at":"2016-05-18T09:53:30Z","updated_at":"2016-05-18T09:53:30Z","body":"This still doesn't work for me, even after updating, as long as I have this code before creating the dfm:\r\n\r\n```\r\nemails = emails %>% mutate(body = sub(\"H <.Re:.\\n\", \"\", body)) %>%\r\nmutate(body = sub(\"H <.Re:\", \"\", body))\r\n```\r\n\r\nSo it looks like the sub() created the bug (maybe by creating some empty documents, as you mentioned above).\r\n\r\nHere is my sessionInfo():\r\n```\r\nR version 3.3.0 (2016-05-03)\r\nPlatform: x86_64-w64-mingw32/x64 (64-bit)\r\nRunning under: Windows >= 8 x64 (build 9200)\r\n\r\nlocale:\r\n[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252\r\n[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    \r\n\r\nattached base packages:\r\n[1] stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n[1] gmodels_2.16.2   quanteda_0.9.6-1 tidyr_0.4.1      ggplot2_2.1.0    dplyr_0.4.3      RSQLite_1.0.0    DBI_0.4-1       \r\n\r\nloaded via a namespace (and not attached):\r\n [1] Rcpp_0.12.5      lattice_0.20-33  gtools_3.5.0     assertthat_0.1   MASS_7.3-45      chron_2.3-47     grid_3.3.0       R6_2.1.2        \r\n [9] plyr_1.8.3       gtable_0.2.0     magrittr_1.5     scales_0.4.0     stringi_1.0-1    lazyeval_0.1.10  gdata_2.17.0     data.table_1.9.6\r\n[17] ca_0.64          Matrix_1.2-6     tools_3.3.0      munsell_0.4.3    parallel_3.3.0   colorspace_1.2-6\r\n```"}}