{"action":"created","issue":{"url":"https://api.github.com/repos/ropensci/onboarding/issues/191","repository_url":"https://api.github.com/repos/ropensci/onboarding","labels_url":"https://api.github.com/repos/ropensci/onboarding/issues/191/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/onboarding/issues/191/comments","events_url":"https://api.github.com/repos/ropensci/onboarding/issues/191/events","html_url":"https://github.com/ropensci/onboarding/issues/191","id":293003503,"number":191,"title":"Submission: rtika","user":{"login":"predict-r","id":4193801,"avatar_url":"https://avatars0.githubusercontent.com/u/4193801?v=4","gravatar_id":"","url":"https://api.github.com/users/predict-r","html_url":"https://github.com/predict-r","followers_url":"https://api.github.com/users/predict-r/followers","following_url":"https://api.github.com/users/predict-r/following{/other_user}","gists_url":"https://api.github.com/users/predict-r/gists{/gist_id}","starred_url":"https://api.github.com/users/predict-r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/predict-r/subscriptions","organizations_url":"https://api.github.com/users/predict-r/orgs","repos_url":"https://api.github.com/users/predict-r/repos","events_url":"https://api.github.com/users/predict-r/events{/privacy}","received_events_url":"https://api.github.com/users/predict-r/received_events","type":"User","site_admin":false},"labels":[{"id":405979113,"url":"https://api.github.com/repos/ropensci/onboarding/labels/5/awaiting-reviewer(s)-response","name":"5/awaiting-reviewer(s)-response","color":"fbca04","default":false},{"id":185501572,"url":"https://api.github.com/repos/ropensci/onboarding/labels/package","name":"package","color":"fbca04","default":false},{"id":381071918,"url":"https://api.github.com/repos/ropensci/onboarding/labels/topic:data-extraction","name":"topic:data-extraction","color":"0e8a16","default":false},{"id":369897644,"url":"https://api.github.com/repos/ropensci/onboarding/labels/topic:text-mining","name":"topic:text-mining","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"noamross","id":571752,"avatar_url":"https://avatars1.githubusercontent.com/u/571752?v=4","gravatar_id":"","url":"https://api.github.com/users/noamross","html_url":"https://github.com/noamross","followers_url":"https://api.github.com/users/noamross/followers","following_url":"https://api.github.com/users/noamross/following{/other_user}","gists_url":"https://api.github.com/users/noamross/gists{/gist_id}","starred_url":"https://api.github.com/users/noamross/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noamross/subscriptions","organizations_url":"https://api.github.com/users/noamross/orgs","repos_url":"https://api.github.com/users/noamross/repos","events_url":"https://api.github.com/users/noamross/events{/privacy}","received_events_url":"https://api.github.com/users/noamross/received_events","type":"User","site_admin":false},"assignees":[{"login":"noamross","id":571752,"avatar_url":"https://avatars1.githubusercontent.com/u/571752?v=4","gravatar_id":"","url":"https://api.github.com/users/noamross","html_url":"https://github.com/noamross","followers_url":"https://api.github.com/users/noamross/followers","following_url":"https://api.github.com/users/noamross/following{/other_user}","gists_url":"https://api.github.com/users/noamross/gists{/gist_id}","starred_url":"https://api.github.com/users/noamross/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noamross/subscriptions","organizations_url":"https://api.github.com/users/noamross/orgs","repos_url":"https://api.github.com/users/noamross/repos","events_url":"https://api.github.com/users/noamross/events{/privacy}","received_events_url":"https://api.github.com/users/noamross/received_events","type":"User","site_admin":false}],"milestone":null,"comments":20,"created_at":"2018-01-31T01:49:34Z","updated_at":"2018-03-06T00:47:48Z","closed_at":null,"author_association":"NONE","body":"### Summary\r\n\r\n-   What does this package do? (explain in 50 words or less):\r\n\r\nExtract text or metadata from over a thousand file types. This is an R interface to Apache Tika, a content and metadata extraction tool. Tika is written in Java and is maintained by the Apache Software Foundation since 2007. \r\n\r\n-   Paste the full DESCRIPTION file inside a code block below:\r\n\r\n```\r\nPackage: rtika\r\nType: Package\r\nTitle: R Interface to 'Apache Tika' \r\nVersion: 0.1.2\r\nAuthor: Sasha Goodman <goodmansasha@gmail.com>[aut], The Apache Software Foundation [aut,cph]\r\nMaintainer: Sasha Goodman <goodmansasha@gmail.com>\r\nSuggests: \r\n    sys,\r\n    jsonlite,\r\n    xml2,\r\n    data.table,\r\n    curl,\r\n    testthat,\r\n    knitr,\r\n    rmarkdown,\r\n    covr,\r\n    Matrix\r\nLicense: Apache License 2.0 | file LICENSE\r\nSystemRequirements: Java (>=7) | openjdk-7-jre (via apt) | java-1.7.0-openjdk (via yum) | openjdk-8-jre (via apt) | java-1.8.0-openjdk (via yum) \r\nDescription: Extract text or metadata from over a thousand file types. Get either plain text or structured XHTML. This R interface includes the Tika software. Its source is available at https://github.com/apache/tika. \r\nDepends: R (>= 3.1.0)\r\nEncoding: UTF-8\r\nLazyData: true\r\nRoxygenNote: 6.0.1\r\nURL: http://github.com/predict-r/rtika\r\nBugReports: http://github.com/predict-r/rtika/issues\r\nVignetteBuilder: knitr\r\n\r\n```\r\n\r\n-   URL for the package (the development repository, not a stylized html page):\r\n\r\nhttps://github.com/predict-r/rtika\r\n\r\n- Please indicate which category or categories from our [package fit policies](https://github.com/ropensci/onboarding/blob/master/policies.md#package-fit) this package falls under ***and why**(? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):\r\n\r\nIt fits into data extraction, especially text extraction. This package helps extract either a plain text or XHTML rendition of most common file types, without leaving R. It includes parsers for Microsoft Office (both the old and new formats), OpenOffice, iWorks, WordPerfect, PDF, ePub, rtf, HTML, XML, and Mbox email. \r\n\r\nThe hours of labor spent on these parsers is unbelievable. I recall Tika was intended to prevent duplicate work at Apache, and has since became a foundational library for Apache search engine projects.\r\n\r\nTika also detects metadata from over a thousand file formats. These include basic features like Content-Encoding and mime type sniffing. Beyond that, there are specialized extractors for scientific formats (ISA-Tab, Matlab), geospatial formats (GDAL, ISO-19139), image Exif data from tiff or jpeg, and many more.\r\n\r\nThe long list of formats is here: https://tika.apache.org/1.17/formats.html\r\n\r\n-   Who is the target audience and what are scientific applications of this package?  \r\n\r\nComputational social scientists, digital humanities scholars, historians, or any person that works with large digital archives filled with possibly unpredictable file formats. \r\n\r\nFor example, consider a social scientist studying the California state legislature, as I am. The legislature publishes a database that includes, among other things, Microsoft Word documents created over the past decades. These are at http://leginfo.legislature.ca.gov/.\r\n\r\nI've used Tika to automatically extract text from decades of Word documents, even if I don't know the particular version of Word beforehand. Further, with the XHTML output option, I can preserve tables inside each Word document.\r\n\r\n-   Are there other R packages that accomplish the same thing? If so, how does\r\nyours differ or meet [our criteria for best-in-category](https://github.com/ropensci/onboarding/blob/master/policies.md#overlap)?\r\n\r\nThere are specialized parsers for several of the file types Tika processes, such as antiword for older Microsoft Word documents. Overlap is going to be inevitable. I have used antiword and it worked well, except that some of my files did not have a file type affix (e.g. .docx or .doc), and could be in either format, and so it took some trial and error to get it working. It turned out antiword does not process the modern XML based Word formats. It also does not render the Word document into XHTML, which could preserve tables from the Word document. \r\n\r\nListing all the document parsers that Tika overlaps with would take an enormous amount of space, since it processes over a thousand file types. In general, however, Tika's aims to streamline text extraction within large archives. Archives often contain heterogeneous and unpredictable file formats. Tika handles most of what I throw at it.\r\n\r\nThe other package I found that interfaces with Apache Tika is tikaR, released in September of 2017 (See: https://github.com/kyusque/tikaR). It provides only XHMTL renditions of documents, and uses the rJava package to interface with Tika. \r\n\r\nI was a Java developer and implemented a rJava interface with Tika in 2012, when I was considering my first package for R (the project is empty, but I set up a repository on r-forge back then:  https://r-forge.r-project.org/projects/r-tika/). My rJava code worked, but I concluded at the time that the Apache Tika command line interface (CLI) was sufficient for myself and most people. I ended up just crafting some specialized system calls to the command line for my own work. The r-forge repository remains empty because I didn't want to maintain a complex Java codebase.\r\n\r\nRecently, the Tika project came out with a very nice batch processor in their command line that I found to be the most efficient method to handle tens of thousands of documents. It was too good to keep to myself. \r\n\r\nUnlike the rJava functions used by tikaR, I essentially send Tika a file list to process, and then craft a specialized call to the command line to invoke the batch processor. Retrieving the converted results is pretty straightforward. The code is simple and easy to maintain. \r\n\r\nUnlike tikaR, I've included a variety of tests and keep track of test coverage, and anticipate that maintaining this package will be fairly simple, even without Java knowledge. There are also more features implemented in my package than tikaR, such as the ability to use multiple threads.\r\n\r\n-   If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.\r\n\r\n### Requirements\r\n\r\nConfirm each of the following by checking the box.  This package:\r\n\r\n- [x] does not violate the Terms of Service of any service it interacts with. \r\n- [x] has a CRAN and OSI accepted license.\r\n- [x] contains a README with instructions for installing the development version. \r\n- [x] includes documentation with examples for all functions.\r\n- [x] contains a vignette with examples of its essential functions and uses.\r\n- [x] has a test suite.\r\n- [x] has continuous integration, including reporting of test coverage, using services such as Travis CI, Coveralls and/or CodeCov.\r\n- [x] I agree to abide by [ROpenSci's Code of Conduct](https://github.com/ropensci/onboarding/blob/master/policies.md#code-of-conduct) during the review process and in maintaining my package should it be accepted.\r\n\r\n#### Publication options\r\n\r\n- [x] Do you intend for this package to go on CRAN?  \r\n- [ ] Do you wish to automatically submit to the [Journal of Open Source Software](http://joss.theoj.org/)? If so:\r\n    - [ ] The package has an **obvious research application** according to [JOSS's definition](http://joss.theoj.org/about#submission_requirements).\r\n    - [ ] The package contains a `paper.md` matching [JOSS's requirements](http://joss.theoj.org/about#paper_structure) with a high-level description in the package root or in `inst/`.\r\n    - [ ] The package is deposited in a long-term repository with the DOI: \r\n    - (*Do not submit your package separately to JOSS*)\r\n- [ ] Do you wish to submit an Applications Article about your package to [Methods in Ecology and Evolution](http://besjournals.onlinelibrary.wiley.com/hub/journal/10.1111/(ISSN)2041-210X/)? If so:\r\n\t- [ ] The package is novel and will be of interest to the broad readership of the journal.\r\n\t- [ ] The manuscript describing the package is no longer than 3000 words.\r\n\t- [ ] You intend to archive the code for the package in a long-term repository which meets the requirements of the journal.\r\n\t- (*Please do not submit your package separately to Methods in Ecology and Evolution*)\r\n\r\n### Detail\r\n\r\n- [x] Does `R CMD check` (or `devtools::check()`) succeed?  Paste and describe any errors or warnings:\r\n\r\n- [x] Does the package conform to [rOpenSci packaging guidelines](https://github.com/ropensci/onboarding/blob/master/packaging_guide.md)? Please describe any exceptions:\r\n\r\n- If this is a resubmission following rejection, please explain the change in circumstances:\r\n\r\n- If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:\r\n\r\nJeroen Ooms has worked on text file parsers and I respect his work immensely. \r\n"},"comment":{"url":"https://api.github.com/repos/ropensci/onboarding/issues/comments/370620626","html_url":"https://github.com/ropensci/onboarding/issues/191#issuecomment-370620626","issue_url":"https://api.github.com/repos/ropensci/onboarding/issues/191","id":370620626,"user":{"login":"juliasilge","id":12505835,"avatar_url":"https://avatars3.githubusercontent.com/u/12505835?v=4","gravatar_id":"","url":"https://api.github.com/users/juliasilge","html_url":"https://github.com/juliasilge","followers_url":"https://api.github.com/users/juliasilge/followers","following_url":"https://api.github.com/users/juliasilge/following{/other_user}","gists_url":"https://api.github.com/users/juliasilge/gists{/gist_id}","starred_url":"https://api.github.com/users/juliasilge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juliasilge/subscriptions","organizations_url":"https://api.github.com/users/juliasilge/orgs","repos_url":"https://api.github.com/users/juliasilge/repos","events_url":"https://api.github.com/users/juliasilge/events{/privacy}","received_events_url":"https://api.github.com/users/juliasilge/received_events","type":"User","site_admin":false},"created_at":"2018-03-06T00:47:48Z","updated_at":"2018-03-06T00:47:48Z","author_association":"NONE","body":"Ah, I love the new vignette!!! 💕 It is full of so much useful, detailed information. This is going to be a great resource for users. I also am happy to see the other issues in the review addressed; I reviewed the package once more, did R CMD check and `goodpractice::gp()` again, and I have checked off the box in my review.\r\n\r\nThe only thing significant remaining in the checks is the NOTE about the size of the package."}}
{"actor":{"display_login":"juliasilge"}}