{"action":"created","issue":{"url":"https://api.github.com/repos/IQSS/dataverse/issues/4923","repository_url":"https://api.github.com/repos/IQSS/dataverse","labels_url":"https://api.github.com/repos/IQSS/dataverse/issues/4923/labels{/name}","comments_url":"https://api.github.com/repos/IQSS/dataverse/issues/4923/comments","events_url":"https://api.github.com/repos/IQSS/dataverse/issues/4923/events","html_url":"https://github.com/IQSS/dataverse/issues/4923","id":348209655,"node_id":"MDU6SXNzdWUzNDgyMDk2NTU=","number":4923,"title":"An extra '/' at the beginning of the ID in one record blocks whole dataverse","user":{"login":"pkiraly","id":218218,"node_id":"MDQ6VXNlcjIxODIxOA==","avatar_url":"https://avatars1.githubusercontent.com/u/218218?v=4","gravatar_id":"","url":"https://api.github.com/users/pkiraly","html_url":"https://github.com/pkiraly","followers_url":"https://api.github.com/users/pkiraly/followers","following_url":"https://api.github.com/users/pkiraly/following{/other_user}","gists_url":"https://api.github.com/users/pkiraly/gists{/gist_id}","starred_url":"https://api.github.com/users/pkiraly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pkiraly/subscriptions","organizations_url":"https://api.github.com/users/pkiraly/orgs","repos_url":"https://api.github.com/users/pkiraly/repos","events_url":"https://api.github.com/users/pkiraly/events{/privacy}","received_events_url":"https://api.github.com/users/pkiraly/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-07T07:59:41Z","updated_at":"2018-08-07T13:35:11Z","closed_at":null,"author_association":"NONE","body":"We have a strange Dataverse issue.\r\nA Java exception is blocking the rendering of the Dataverse front\r\npage. The error is recorded in the server.log as:\r\n\r\n```\r\n2018-08-06T16:32:06.773+0200] [glassfish 4.1] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=56 _ThreadName=jk-connector(5)] [timeMillis: 1533565926773] [levelValue: 900] [[\r\n  StandardWrapperValve[HomepageServlet]: Servlet.service() for servlet HomepageServlet threw exception\r\njava.lang.IllegalArgumentException: Failed to parse identifier: doi:/10.25625/VUHOIJ\r\n  at edu.harvard.iq.dataverse.GlobalId.<init>(GlobalId.java:38)\r\n  at edu.harvard.iq.dataverse.DatasetVersionServiceBean.populateDatasetSearchCard(DatasetVersionServiceBean.java:851)\r\n  at sun.reflect.GeneratedMethodAccessor322.invoke(Unknown Source)\r\n  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n  at java.lang.reflect.Method.invoke(Method.java:498)\r\n  at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)\r\n  at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)\r\n  at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)\r\n  at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)\r\n```\r\n\r\nI found that the problem is caused by a Solr record:\r\n\r\n```\r\n{\r\n        \"id\":\"dataset_5707_draft\",\r\n        \"entityId\":5707,\r\n        \"dataverseVersionIndexedBy_s\":\"4.9.1\",\r\n        \"identifier\":\"doi:/10.25625/VUHOIJ\",\r\n        \"dsPersistentId\":\"doi:/10.25625/VUHOIJ\",\r\n        \"persistentUrl\":\"https://doi.org//10.25625/VUHOIJ\",\r\n        \"dvObjectType\":\"datasets\",\r\n        \"publicationStatus\":[\"Unpublished\",\r\n          \"Draft\"],\r\n        \"dateSort\":\"2018-08-06T10:54:29.896Z\",\r\n        \"dateFriendly\":\"Aug 6, 2018\",\r\n        \"isHarvested\":false,\r\n        \"metadataSource\":\"eRA\",\r\n        \"datasetVersionId\":127,\r\n        \"citation\":\"Harald Kusch, 2018, \\\"Test Export\\\", https://doi.org//10.25625/VUHOIJ, eRA, DRAFT VERSION\",\r\n        \"citationHtml\":\"Harald Kusch, 2018, \\\"Test Export\\\", <a href=\\\"https://doi.org//10.25625/VUHOIJ\\\" target=\\\"_blank\\\">https://doi.org//10.25625/VUHOIJ</a>, eRA, DRAFT VERSION\",\r\n        \"nameSort\":\"Test Export\",\r\n        \"title\":\"Test Export\",\r\n        \"language\":[\"English\"],\r\n        \"language_ss\":[\"English\"],\r\n        \"subject\":[\"Medicine, Health and Life Sciences\"],\r\n        \"subject_ss\":[\"Medicine, Health and Life Sciences\"],\r\n        \"dsDescriptionValue\":[\"Test Export\"],\r\n        \"authorName\":[\"Harald Kusch\"],\r\n        \"authorName_ss\":[\"Harald Kusch\"],\r\n        \"datasetContactName\":[\"Harald Kusch\"],\r\n        \"subtreePaths\":[\"/3\"],\r\n        \"parentId\":\"3\",\r\n        \"parentName\":\"electroniclabnotebook\",\r\n        \"_version_\":1608064864314458112}]\r\n}\r\n```\r\n\r\nI identified several place in Postgres which might cause this result in Solr.\r\n\r\nIn `doidataciteregistercache` table:\r\n\r\n* doi: `doi:/10.25625/VUHOIJ`\r\n* url: `https://data.gro.uni-goettingen.de/citation?persistentId=doi:/10.25625/VUHOIJ`\r\n* xml:\r\n```\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<resource xsi:schemaLocation=\"http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd\"\r\n  xmlns=\"http://datacite.org/schema/kernel-3\"\r\n  xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\r\n  <identifier identifierType=\"DOI\">/10.25625/VUHOIJ</identifier>\r\n  <creators>\r\n    <creator>\r\n      <creatorName>Harald Kusch</creatorName>\r\n    </creator>\r\n  </creators>\r\n  <titles>\r\n    <title>Test Export</title>\r\n  </titles>\r\n  <publisher>eRA</publisher>\r\n  <publicationYear>2018</publicationYear>\r\n  <resourceType resourceTypeGeneral=\"Dataset\"/>\r\n  <descriptions>\r\n    <description descriptionType=\"Abstract\">Test Export</description>\r\n  </descriptions>\r\n  <contributors>\r\n    <contributor contributorType=\"ContactPerson\">\r\n      <contributorName>Harald Kusch</contributorName>\r\n    </contributor>\r\n  </contributors>\r\n</resource>\r\n```\r\n\r\nIn the dvobject table:\r\n\r\n* storageidentifier: `file:///10.25625/VUHOIJ`\r\n* identifier: `10.25625/VUHOIJ`\r\n\r\nAs you can see there are several extra '/' as part of the DOI identifier. At this point it is not clear what made that extra character, it is Dataverse or the client which use the Dataverse API?\r\n\r\nThe important thing I would like to solve is the original problem, i.e. the blocking DOI parsing issue.\r\nSo I cleared the database, and removed these extra '/'-s from everywhere, removed the Solr index, and reindexed the whole database again. The problem still existed: the DOI in Solr is what I showed you.\r\n\r\nI continue investigation, and check how the dsPersistentId is created in Solr. In `DvObject` class (in edu.harvard.iq.dataverse package) I found this method:\r\n\r\n```\r\n    public GlobalId getGlobalId() {\r\n        // FIXME should return NULL when the fields are null. Currenntly, \r\n        //       a lot of code depends call this method, so this fix can't be \r\n        //       a part of the current PR.\r\n        return new GlobalId(getProtocol(), getAuthority(), getIdentifier());\r\n    }\r\n```\r\n\r\nI also found that in the same class there is a named query:\r\n\r\n```\r\n    @NamedQuery(name = \"DvObject.findByGlobalId\",\r\n            query = \"SELECT o FROM DvObject o WHERE o.identifier=:identifier and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype\"),\r\n```\r\nso `authority` also comes from the `dvobject` table. I checked it, and found that for this record its content is an empty string.\r\n\r\nI modified the database manually, so\r\n\r\n| identifier | authority |\r\n| --- | --- |\r\n| 10.25625/VUHOIJ | |\r\n\r\nbecame\r\n\r\n| identifier | authority |\r\n| --- | --- |\r\n| VUHOIJ | 10.25625 |\r\n\r\nAfter clearing the index, and reindexing it again the original issue were solved, and now our front page is working again. There are some points however where Dataverse code base could be improved:\r\n\r\n1) catch the DOI parsing exception, and instead of blocking the whole page the error should be reported to the admin with some more information, and the search result list should skip the particular record but should contain the rest.\r\n\r\n2) Since the DOI is coming from Dataverse, it should check it after creation whether they are parsable. \r\n\r\n3) When Dataverse stores the new identifiers is should add a proper check against the content of dvobject's table columns (identifier, authority) whether they are syntactically correct or not.\r\n\r\n4) The database stores the DOI in doidataciteregistercache.doi table, so it doesn't seem logical to not use that in indexing, and recreating it again from parts.\r\n\r\n5) Tt would be also nice if these tables would have documentation\r\n\r\nIn the next days we will try to reproduce the issue, and figure it out if it was dependent or independent from the client application (this records and two others were imported from RSpace lab notebook). I will add a comment about the result as we have something to report.\r\n\r\nBest,\r\nPéter"},"comment":{"url":"https://api.github.com/repos/IQSS/dataverse/issues/comments/411057988","html_url":"https://github.com/IQSS/dataverse/issues/4923#issuecomment-411057988","issue_url":"https://api.github.com/repos/IQSS/dataverse/issues/4923","id":411057988,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTA1Nzk4OA==","user":{"login":"pdurbin","id":21006,"node_id":"MDQ6VXNlcjIxMDA2","avatar_url":"https://avatars2.githubusercontent.com/u/21006?v=4","gravatar_id":"","url":"https://api.github.com/users/pdurbin","html_url":"https://github.com/pdurbin","followers_url":"https://api.github.com/users/pdurbin/followers","following_url":"https://api.github.com/users/pdurbin/following{/other_user}","gists_url":"https://api.github.com/users/pdurbin/gists{/gist_id}","starred_url":"https://api.github.com/users/pdurbin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdurbin/subscriptions","organizations_url":"https://api.github.com/users/pdurbin/orgs","repos_url":"https://api.github.com/users/pdurbin/repos","events_url":"https://api.github.com/users/pdurbin/events{/privacy}","received_events_url":"https://api.github.com/users/pdurbin/received_events","type":"User","site_admin":false},"created_at":"2018-08-07T13:35:11Z","updated_at":"2018-08-07T13:35:11Z","author_association":"MEMBER","body":"@pkiraly when you run\r\n\r\ncurl http://localhost:8080/api/admin/settings/:Authority\r\n\r\nDoes the value start with a slash (\"/\")? It shouldn't.  Here's how it looks on a server I use for testing:\r\n\r\n    {\"status\":\"OK\",\"data\":{\"message\":\"10.5072\"}}\r\n\r\nYou can get a dump of all database settings with this:\r\n\r\n    curl http://localhost:8080/api/admin/settings | jq .\r\n\r\nI believe you can get a dump of all Dataverse-related JVM options with this:\r\n\r\n    /usr/local/glassfish4/glassfish/bin/asadmin list-jvm-options | egrep 'dataverse|doi'\r\n\r\nThat's what we document at http://guides.dataverse.org/en/4.9.1/installation/config.html#jvm-options at least. 😄 \r\n\r\n"}}
{"actor":{"display_login":"pdurbin"}}