Linking the Data: Building effective Authority and Identity Lookup
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Khan, Huda Jaliluddin
Rayle, E. Lynette
Eichmann, David
Warner, Simeon
Krafft, Dean
The Mellon Foundation-funded Linked Data for Libraries Labs (LD4L-Labs) and Linked Data for Libraries Production (LD4P) projects are exploring the library community’s transition to Linked Open Data. Authority and identity lookups are integral to cataloging workflows and provide excellent opportunities for exploring how to leverage the power of Linked Data for reconciliation and more effective lookups. Central questions in this work include the implementation, performance and reliability of lookup services; what multiple authority lookups mean with respect to reconciliation; and user interface design. We are currently experimenting with caching mechanisms that address the issues of access, reliability, and speed, including an approach that uses a Linked Data Fragments server for caching and serving up authority results. We are also exploring the implementation of a containerized and potentially mirrored search index populated using Jena and Fuseki to execute SPARQL queries against Linked Data sources. We are extending the Questioning Authority (QA) work being developed for the Hydra repository to enable more configurable Linked Data lookups that can be used across different technology stacks. VitroLib, an experimental cataloging tool being developed using the Vitro platform, integrates QA lookups to support catalogers searching for and linking to authorities. We will demonstrate the lookup configuration and integration into both the Hydra repository and VitroLib. All the software is open source.
 So, thank you for being here. I don't want to say it's been an honor and a privilege because every time a character says that in a movie, they don't live for the next 10 minutes. They're dead. And I want to stay till the end of this conference. So this is probably not my most creative title, possibly because Lynette and I are working on this together and she keeps me real. Otherwise, it would have had 3,000 more words. But it does do a good job of talking about what we'll talk about. Of course, I'm already on the next slide. And so really, there's two main strands to this talk. One of them is thinking about how we are trying to really integrate authorities and lookup services in user-facing systems. So, Viewtrelib being a cataloging editor that we're working on and Hierax being a repository front end. And then the other strand of this work that Lynette will really get into is, well, what does it actually take to build something that allows for authorities to be accessed in a reliable and hopefully performant way? So those are the two main themes that we'll run through this talk. And we'll begin with why. So the background and motivation, we'll go over that a little bit because it's usually a good idea to talk about why you're doing something, especially when you're already doing it and having a presentation about it. And then hopefully we'll have time to talk a little bit about some future steps because this is still a work in progress. And then I'll run back on stage because I never knew when to end a conversation. So the background, you've already heard about the LD4 universe of projects probably three or four times by this point. So I won't spend a whole lot of time here. But in brief, this work that we're talking about is part of the LD4 grants. And the grants are looking at how to transition library systems to link data. So in brief, I'm going to just show this extremely oversimplified rendition of a mark record. How many cataloging-ish people in the room? A couple ontology designers. Okay. So my objective is to make all of you angry because this is a very simplified view of a mark which shows that you have the audio CD Made in America which came out in 1980 and was performed by the Blues Brothers. And you can see there's a link to the name authority file which relies on making sure the string is exactly what you're matching to in your work. And so the information about work instance item have all been merged together, sort of flattened out in a mark record. And what we're doing when we try to transition to link data is take those pieces and pry them apart. So you've got the work which is Made in America which has an instance which thankfully has the same title, otherwise this would be a confusing slide. And that instance was published in 1980. And now the Made in America work is linking to the Blues Brothers entity. So there's a URI out there that uniquely identifies Blues Brothers and you can have variant labels and other things off of it but we're no longer relying on a string or a flat representation. We're pointing to an object. And it was interesting, I think George Oates mentioned or had a list with Mark Twain and Mark Twain Spirit was one of them and Mark Twain, the spirit, has its own real world object. So try to think about that if you will. And so why is any of this important in this context? Well, we're transitioning to link data and so who's actually responsible for putting this stuff together and for cataloging it? Well, it's catalogers. So I did what every good citizen should do and went to Wikipedia to look it up. And in addition to catalogers being the foundation of all library service, so the three quarters of this room that raise your hands, thank you. They're the tasks that are responsible for it that I thought were really interesting. So it's not just data entry that catalogers do. There's a lot of cognitive load that you have to carry. Which includes things like describing what something is really about, what are the subjects and the authorities. Who is this really by and is this the person that is really the person, is the authority record I have, is this the one I want to pick? So I've already mentioned the authority file. That's usually what people use and it uses the string as a unique identifier. And again, going back to what it is that catalogers are doing, they're trying to think about, well, is this work about forest fires or something else? What is the genre here of this particular recording? Disambiguation is a big part of this. So you need context and information for this. So it's Mark Twain the author and not Mark Twain my best friend from down the street. I don't know why your best friend would be in a catalog, but it could happen. And so you need context and information and access to the authority information to do that work. So going trying this back to the work we're doing in this project, what we're looking at is designing these systems. And one of the questions is how do we integrate information about authorities and how do we architect all of this? So the two examples that we'll touch on briefly before Lynette launches into the technical details are VTROLib, which is a cataloging editor as I said, it's a prototype that we're looking at to see how we can support cataloging using link data and producing link data. And part of that is enabling lookup and use of authorities. And it's called VTROLib because we are great at naming things in this project, as I'm sure all of you already know. It's based on VTRO, which is an ontology agnostic editor, which actually underlines Vivo and Tatiana will mention that briefly. So we took that and we added a lip to it because we're trying to make it into a cataloging specific editor. And the second example will be Hierarchs, which Lynette will touch on and which is the repository interface. So without further ado, and I have to say, I think I talked to Alicia right before our presentation and I was like, you're doing a live demo and I thought that's gutsy. I too remember once when I was full of hope and young and I now have screenshots. So here are the screenshots. So imagine if you will that you're a cataloger who comes in, say, let's say you're Cecilia and it's your job to try to catalog this particular audio record, the Blues Brothers, Made in America. I won't go into all of the details here, but let's just say you're putting in the information for the title and the type of work this is. And when you get to where you're trying to put in information about the performer, this is just an autocomplete where you typed in Blues Brothers and through the magic that we will discuss further in this presentation, it's going off to the name authority file from the Library of Congress, retrieving Blues Brothers and all of the things that match that term along with the URIs. So when you select it, you are actually tying that to the URI. Similarly for subject headings, this is going off to the Library of Congress subject headings and bringing back soul music and then you can put that in. And as you go through the rest of this form, you're filling in information at the instance level as well. And so you can put in information about the publisher, which again is going to the name authority file. So there's three autocomplete examples going to the name authority file. And so you have this information and when you hit submit, what you're looking at here is a page that describes the work with a link to the instance, but you can see that Blues Brothers and soul music are in fact in there. And so this leads to some interesting design questions. These are fairly simple autocomplete lookups from a front end point of view, but like I said, catalogers probably could use some more context. Now we do know that they definitely still want to see the authority record, but one of our design questions, and this is again a work in progress, is trying to understand what pieces of information can we give them that can help them in their disambiguation and analysis. So taking John Form, which is the last property on the slide, is an example. If you click on that, let's say in this particular example, you've already added one, so you have video recordings, but you would like to search some more. And this is going to the Library of Congress John Forms again. So you type in animation and hit search. And what this does is it gets back information, not just of the title, but also the alternate labels and some links to broader and narrower. And if I scroll down, you'll see one example at the bottom where you have notes, so some additional notes that are coming back. If you click on the links in the results, like for Clay Animation Television, I realize I keep pointing at my screen and that helps no one, but just follow along please. If you click on that, that will open up the actual authority record because this is something that we've seen catalogers would like even with all the contextual information. So what just happened? This is a quick run through. Again, the technical details will not get into more, but from the point of view of what you just saw, we had the query animation. That goes off to code that's sitting within Vitrolib that takes that and translates that to the questioning authority lookup, which again, Lynette will talk about, but that's what it's actually talking to. It's talking to a questioning authority endpoint, I suppose you could call it, with that query. And then magic happens. Again, Lynette will explain the magic. And it goes off and searches the Library of Congress, John Forms, comes back with some data. So this is sort of standardized data or standard format data coming back from questioning authority. You've got URI label and context, gives that back to Vitrolib, which has its own internal standard. And then it takes that, massages it into what the front end expects, and then you see the label and the alternate labels show up. So that kind of explains what just happened on a high level. And now Lynette, who is our Sam Vera guru. And is there anyone from Iceland who can tell us how to actually say this word? No, okay. So we'll just keep calling it Sam Vera. So Lynette will talk about Hierax. Yes, so the main point of showing the Vitrolib and Hierax is to point out that this same process was used for two different technology stacks. So Vitrolib is a Java application and Hierax is a Ruby application. And the same process works in both. So this is just showing in autocomplete field as well. And I'm not going to go into detail. Everybody knows how autocompletes work. So you select from the autocomplete Ezra, we come back and we get the pref label. We also get the URI. So the URI is considered the value of this field. But we also cache the label just for display purposes later. So the context part hasn't been implemented in Hierax yet, but there are some mockups associated with it. And what I wanted to show in this is that there are various fields here. And I know they're kind of small. So there's authoritative label, variable label, and it's small for me too. Field of activity, occupation, and then a multitude of dates that are associated with it. And all of that is part of the configuration that drives questioning authority. So for example, this is what works well for LOC, named authority data. And those particular fields were requested by our metadata folks. But this is another authority that also works in QA, which is AgriVoc, which is more hierarchically oriented. So it's got broader and narrower. And the only thing we have to do different is change the configuration file to say that what we want to show in our context is broader, narrower, and same as. The processing code that sits underneath it is exactly the same in both cases. So I also wanted to point out that it's not just for entering data. You can use it to enhance your data as well. So this is an example on a show page. And in this case, the actual value for this field is the Hive products with that URI. And then at the time of showing it, we reach out and say, hey, what are some of the other data that we can show with this that the user might find useful? So we put in broader, narrower, and same as. And you can also, so that particular example was from the same authority. This one is from a different authority. So there wasn't a lot of data in OC-LC-FAST for Cornell-Esra. So we go out to DBpedia and grab some additional data. So you can get it from multiple authorities. And again, DBpedia is just going through questioning authority. Well, we're going through questioning authority to get the data from DBpedia. So it's the same process. So what I really want to talk about a little bit is to give you some introduction to the architecture. First, technical motivations. I'm not going to cover the link data part because I think everyone here already drank that Kool-Aid. So I think we're OK with that. But some of the goals of the implementation are to provide a single process for accessing any number of authorities. That's one of the things that link data tells us we can do, that we put it all in these standard formats that's easy for machines to talk about. So I wanted to test that and see can we really do that. The other is that we really want to provide an efficient and reliable access. And we want a means to provide the user who's making a selection with enough detail that they can make as accurate a selection as possible. So we approached the problem by identifying some of the challenges. We identified a couple of authorities we wanted to try and do this with. And the first challenge by far was finding documentation that told me how to access everything. Once I located that, it became clear that the API that you use to access things, the various authorities are different. Some don't have any support for link data at all. And some have partial support where you can get data about a single term, but you can't do a query and get link data back. Some require logins, some have Sparklin points versus like a REST style type API. The third challenge was varying results formats, but I would say by far that was the easiest challenge to solve because we just used a standard library that took whatever format it was and threw it into a graph. And then I processed the graph. So that was pretty easy to deal with. And varying ontologies also did not present a great challenge in the configuration itself. I identified what predicate plays what role. So again, that one wasn't too challenging to deal with. So the approach we took is questioning authority was a gem that already existed, but it did non-link data and it had a one-off code for every authority you wanted to access. So if you wanted to access a new authority, you would have to go out and write a new module in order to do that. So what I did was I wrote a module to access link data. And I only wrote it once and all the authorities that I accessed through here that are link data use this same process. So this sits in the middle as a normalization layer. Up at the top is the application that wants to access it. And at the bottom is whatever authority that you want to access. So much like who just showed earlier, you send a URL request to questioning authority service and it will translate that into a request to the actual authority that you're accessing, which will return to you link data. Question authority goes through a process of normalizing that link data and only returns what you requested. So who to show does slightly more extended version because it included additional context. This is sort of like the minimal record. It has a URI, an ID, and a label, which is all you need to do autocomplete. So I just want to show this is what the URLs look like if you go directly against the authorities. They each are different. They have different parameters they pass, different structures. Some require a login. And when you go against questioning authority, these are the URLs you call questioning authority with. They are exactly the same except that you identify what authority you're going against and what query you want. And there is some leeway to define the parameters the way you want if you're writing your own configuration. So if an authority provides additional parameters, you're not limited. You can use those. And I'm not showing the results because if I showed the results from the ontologies, it's quite extensive. But this is the normalized results. And the thing I want you to take away from this is that's three different authorities, O, CLC, FAST, GeoNames, and AgriVoc, and the structure looks exactly the same. So from the application's perspective, they can just use the same process regardless of what field they're trying to fill in, regardless of what authority is actually pushing that forward. So after we met those challenges, we went on to a few more. So our second set of challenges were, well, first I'm going to give a hint as to how I'm going to solve this because I want to talk about it as I talk about these. We're going to do a cache server where we actually put the triples from the external authorities into a local cache. And so one is reliability and efficiency. If it's an external server, we have no control over load factors, server uptime. So if we find that our server is getting bogged down, we can throw more resources at it if it's our server. Accuracy is kind of a mixed bag. We can write our own algorithms that determine how we search for things and find search results. One of the things we lose when it's our own is sometimes the authorities have specialized knowledge that they have in their data set that we don't have. And so we won't have that as part of our algorithm. But the last one is not available from any authority that I worked with. None of them have order rankings. So if I get back 50 results, because this is a graph situation, I pop those 50 results into a graph, there's no order. So I can't distinguish between the best match or the worst match. So by putting it into our own, I'll show you how we solve that problem. So here's our cache server. We basically have a Genevuse Qsaki triple store that we dump all of our triples into and we have an index that sits over top of that. And we can tweak that index as much as we want to improve the efficiency of what we get back, the reliability of what we get back. So the process basically is that when our query comes in, you do a search through the Lucene index and it gets back and extracts out the rank order from the search and also the URI. It does a second search of the Genevuse Qsaki triple store to get all the additional context that it wants to include in the results. And then it has a process where it combines those together. So it stuffs that rank order into a predicate in the search results. And so then when I get that back, I now have a predicate that I can sort things on before it gets presented to the user at the end. And basically in questioning authority itself, when I do that normalized last layer where it's setting things up with the URI, the label, and the ID, I've already ordered them. So that when the end application gets it, they can just put it in, use the order as it comes from QA. So the other thing I want to point out here, this is just a slightly extended version of the diagram I showed a minute ago. But now across the bottom, you can see there's different kinds of data sources that we can get it from, direct from the authority or from our own cache version, or you can get it from other caches that you might choose to set up. So we had another set of challenges that we are currently working on, and that is the disambiguation. We are part way through that as Huda showed where we actually can do a little more context in what we present back to the application so that they can give that to users to make a better choice. The other is that we're working on reconciliation across multiple sources. So what we want to be able to do is actually do the search. By having the triples ourselves, we can do a search across multiple authorities. And there's all sorts of questions that come out in that about how we match up URIs and things like that. So I'm torn between saying, do you want to do questions or what's next? I just will real quickly say what's next and then we'll do some questions. For the architectural side of things, what we want to do right now is generalize that process that we're doing to access the additional context because right now I've basically done it for LOC genre and I want to be able to apply that same process to others. And the multi authority search I mentioned and also one of the things that happens when you start collecting the link data yourself, you have to deal with caching and cache refresh. And that's a whole other topic so I'm not going to go into more detail than that. We're also looking at the potential to mirror cache servers to also improve fallback and availability of the data. And who is going to talk just one second about the user experience and then we'll take a few seconds. I just wanted to quickly say as part of the work that we're doing with the authority lookup stuff in VTRAlib, in general with VTRAlib we've tried to use a user-centered design process which means we really actually want to understand how catalogers do things and what we can do to support them. So there's a lot more work in that area for trying to understand how to actually get the lookup context to work with them, what pieces of information work and how can we support those tasks. As part of the experience there's also the matter of search results and how they're ranked. We already have a situation where we showed them the search results from our cache server and they said we want something a little more exact because I don't want to see these other 20 things that I don't want in my results set. So how to tweak that, how to make that work for them. And then a quick word for the fact that we talked about two platforms but there's potential for applying this in additional platforms such as Folio perhaps. So that's it. So I'm just going to say that the tiny URL there is for the slides themselves. There's actually additional information in the slides and sort of like an appendix. Great. Thank you very much. All right. Do we have questions for Huda and Lynette on this excellent work? Yep. Yeah. Thank you for an excellent talk. I might have been responsible for one of the services, namely the one that's about AgriVoc. How was you? Sorry. Well, yeah. In a way. But yeah, I think you're talking about linked data access but actually the search part where you actually send a query somewhere, it's not part of linked data really. I mean, the response might be linked data in those cases but the original, I mean, there's something like an API there that you're sending a search to. And so that's something that there is really no standard for except maybe it's something like Open Search which is really broad. You can apply it in many settings. And then for the responses, that's also something where linked data doesn't actually shine. Even if the server had a sort of a notion of order, it's difficult to represent that in the response. So that, I mean, that was just a comment. But I think you're sort of hitting the limits of linked data both ways, both in sending a query and getting the result. All right. Thank you. Was there comments, other questions? I would say I'm happy to talk about that more if you want to. Sorry. I didn't mean to interrupt. I was wondering if you were just like processing the brilliance of Osmo right now. Well, I have several opinions about it but I don't know. I do think that linked data, sometimes when we talk about it, we look at it as everything has to be linked data. And I'm very pragmatic in the way I approach things. And to me, the part that linked data promises us is easy interchange. And so when I can use the same process to access linked data from lots of different places, I feel like I'm getting the benefit of linked data. And then I'm trying to smooth the path and make it easier for consumers on the other end. So I don't feel bad that I'm not passing everything as linked data. Great. All right. Well, thank you again, Huda and Lynette. Thank you. This is the last round. And I wish everyone had a wonderful evening and thank you again. And you have our
 So, thank you for being here.  I don't want to say it's been an honor and a privilege because every time a character  says that in a movie, they don't live for the next 10 minutes.  They're dead.  And I want to stay till the end of this conference.  So this is probably not my most creative title, possibly because Lynette and I are working  on this together and she keeps me real.  Otherwise, it would have had 3,000 more words.  But it does do a good job of talking about what we'll talk about.  Of course, I'm already on the next slide.  And so really, there's two main strands to this talk.  One of them is thinking about how we are trying to really integrate authorities and lookup  services in user-facing systems.  So, Viewtrelib being a cataloging editor that we're working on and Hierax being a repository  front end.  And then the other strand of this work that Lynette will really get into is, well, what  does it actually take to build something that allows for authorities to be accessed in a  reliable and hopefully performant way?  So those are the two main themes that we'll run through this talk.  And we'll begin with why.  So the background and motivation, we'll go over that a little bit because it's usually  a good idea to talk about why you're doing something, especially when you're already  doing it and having a presentation about it.  And then hopefully we'll have time to talk a little bit about some future steps because  this is still a work in progress.  And then I'll run back on stage because I never knew when to end a conversation.  So the background, you've already heard about the LD4 universe of projects probably three  or four times by this point.  So I won't spend a whole lot of time here.  But in brief, this work that we're talking about is part of the LD4 grants.  And the grants are looking at how to transition library systems to link data.  So in brief, I'm going to just show this extremely oversimplified rendition of a mark record.  How many cataloging-ish people in the room?  A couple ontology designers.  Okay.  So my objective is to make all of you angry because this is a very simplified view of  a mark which shows that you have the audio CD Made in America which came out in 1980 and  was performed by the Blues Brothers.  And you can see there's a link to the name authority file which relies on making sure  the string is exactly what you're matching to in your work.  And so the information about work instance item have all been merged together, sort of  flattened out in a mark record.  And what we're doing when we try to transition to link data is take those pieces and pry  them apart.  So you've got the work which is Made in America which has an instance which thankfully has  the same title, otherwise this would be a confusing slide.  And that instance was published in 1980.  And now the Made in America work is linking to the Blues Brothers entity.  So there's a URI out there that uniquely identifies Blues Brothers and you can have variant labels  and other things off of it but we're no longer relying on a string or a flat representation.  We're pointing to an object.  And it was interesting, I think George Oates mentioned or had a list with Mark Twain and  Mark Twain Spirit was one of them and Mark Twain, the spirit, has its own real world  object.  So try to think about that if you will.  And so why is any of this important in this context?  Well, we're transitioning to link data and so who's actually responsible for putting  this stuff together and for cataloging it?  Well, it's catalogers.  So I did what every good citizen should do and went to Wikipedia to look it up.  And in addition to catalogers being the foundation of all library service, so the three quarters  of this room that raise your hands, thank you.  They're the tasks that are responsible for it that I thought were really interesting.  So it's not just data entry that catalogers do.  There's a lot of cognitive load that you have to carry.  Which includes things like describing what something is really about, what are the subjects  and the authorities.  Who is this really by and is this the person that is really the person, is the authority  record I have, is this the one I want to pick?  So I've already mentioned the authority file.  That's usually what people use and it uses the string as a unique identifier.  And again, going back to what it is that catalogers are doing, they're trying to think about,  well, is this work about forest fires or something else?  What is the genre here of this particular recording?  Disambiguation is a big part of this.  So you need context and information for this.  So it's Mark Twain the author and not Mark Twain my best friend from down the street.  I don't know why your best friend would be in a catalog, but it could happen.  And so you need context and information and access to the authority information to do that  work.  So going trying this back to the work we're doing in this project, what we're looking  at is designing these systems.  And one of the questions is how do we integrate information about authorities and how do we  architect all of this?  So the two examples that we'll touch on briefly before Lynette launches into the technical  details are VTROLib, which is a cataloging editor as I said, it's a prototype that we're  looking at to see how we can support cataloging using link data and producing link data.  And part of that is enabling lookup and use of authorities.  And it's called VTROLib because we are great at naming things in this project, as I'm  sure all of you already know.  It's based on VTRO, which is an ontology agnostic editor, which actually underlines Vivo and  Tatiana will mention that briefly.  So we took that and we added a lip to it because we're trying to make it into a cataloging  specific editor.  And the second example will be Hierarchs, which Lynette will touch on and which is the  repository interface.  So without further ado, and I have to say, I think I talked to Alicia right before our  presentation and I was like, you're doing a live demo and I thought that's gutsy.  I too remember once when I was full of hope and young and I now have screenshots.  So here are the screenshots.  So imagine if you will that you're a cataloger who comes in, say, let's say you're Cecilia  and it's your job to try to catalog this particular audio record, the Blues Brothers, Made in  America.  I won't go into all of the details here, but let's just say you're putting in the information  for the title and the type of work this is.  And when you get to where you're trying to put in information about the performer, this  is just an autocomplete where you typed in Blues Brothers and through the magic that we  will discuss further in this presentation, it's going off to the name authority file  from the Library of Congress, retrieving Blues Brothers and all of the things that match that  term along with the URIs.  So when you select it, you are actually tying that to the URI.  Similarly for subject headings, this is going off to the Library of Congress subject headings  and bringing back soul music and then you can put that in.  And as you go through the rest of this form, you're filling in information at the instance  level as well.  And so you can put in information about the publisher, which again is going to the name  authority file.  So there's three autocomplete examples going to the name authority file.  And so you have this information and when you hit submit, what you're looking at here  is a page that describes the work with a link to the instance, but you can see that Blues  Brothers and soul music are in fact in there.  And so this leads to some interesting design questions.  These are fairly simple autocomplete lookups from a front end point of view, but like I  said, catalogers probably could use some more context.  Now we do know that they definitely still want to see the authority record, but one  of our design questions, and this is again a work in progress, is trying to understand  what pieces of information can we give them that can help them in their disambiguation  and analysis.  So taking John Form, which is the last property on the slide, is an example.  If you click on that, let's say in this particular example, you've already added one, so you  have video recordings, but you would like to search some more.  And this is going to the Library of Congress John Forms again.  So you type in animation and hit search.  And what this does is it gets back information, not just of the title, but also the alternate  labels and some links to broader and narrower.  And if I scroll down, you'll see one example at the bottom where you have notes, so some  additional notes that are coming back.  If you click on the links in the results, like for Clay Animation Television, I realize  I keep pointing at my screen and that helps no one, but just follow along please.  If you click on that, that will open up the actual authority record because this is something  that we've seen catalogers would like even with all the contextual information.  So what just happened?  This is a quick run through.  Again, the technical details will not get into more, but from the point of view of what  you just saw, we had the query animation.  That goes off to code that's sitting within Vitrolib that takes that and translates that  to the questioning authority lookup, which again, Lynette will talk about, but that's  what it's actually talking to.  It's talking to a questioning authority endpoint, I suppose you could call it, with that query.  And then magic happens.  Again, Lynette will explain the magic.  And it goes off and searches the Library of Congress, John Forms, comes back with some  data.  So this is sort of standardized data or standard format data coming back from questioning authority.  You've got URI label and context, gives that back to Vitrolib, which has its own internal  standard.  And then it takes that, massages it into what the front end expects, and then you see the  label and the alternate labels show up.  So that kind of explains what just happened on a high level.  And now Lynette, who is our Sam Vera guru.  And is there anyone from Iceland who can tell us how to actually say this word?  No, okay.  So we'll just keep calling it Sam Vera.  So Lynette will talk about Hierax.  Yes, so the main point of showing the Vitrolib and Hierax is to point out that this same  process was used for two different technology stacks.  So Vitrolib is a Java application and Hierax is a Ruby application.  And the same process works in both.  So this is just showing in autocomplete field as well.  And I'm not going to go into detail.  Everybody knows how autocompletes work.  So you select from the autocomplete Ezra, we come back and we get the pref label.  We also get the URI.  So the URI is considered the value of this field.  But we also cache the label just for display purposes later.  So the context part hasn't been implemented in Hierax yet, but there are some mockups  associated with it.  And what I wanted to show in this is that there are various fields here.  And I know they're kind of small.  So there's authoritative label, variable label, and it's small for me too.  Field of activity, occupation, and then a multitude of dates that are associated with  it.  And all of that is part of the configuration that drives questioning authority.  So for example, this is what works well for LOC, named authority data.  And those particular fields were requested by our metadata folks.  But this is another authority that also works in QA, which is AgriVoc, which is more hierarchically  oriented.  So it's got broader and narrower.  And the only thing we have to do different is change the configuration file to say that  what we want to show in our context is broader, narrower, and same as.  The processing code that sits underneath it is exactly the same in both cases.  So I also wanted to point out that it's not just for entering data.  You can use it to enhance your data as well.  So this is an example on a show page.  And in this case, the actual value for this field is the Hive products with that URI.  And then at the time of showing it, we reach out and say, hey, what are some of the other  data that we can show with this that the user might find useful?  So we put in broader, narrower, and same as.  And you can also, so that particular example was from the same authority.  This one is from a different authority.  So there wasn't a lot of data in OC-LC-FAST for Cornell-Esra.  So we go out to DBpedia and grab some additional data.  So you can get it from multiple authorities.  And again, DBpedia is just going through questioning authority.  Well, we're going through questioning authority to get the data from DBpedia.  So it's the same process.  So what I really want to talk about a little bit is to give you some introduction to the  architecture.  First, technical motivations.  I'm not going to cover the link data part because I think everyone here already drank  that Kool-Aid.  So I think we're OK with that.  But some of the goals of the implementation are to provide a single process for accessing  any number of authorities.  That's one of the things that link data tells us we can do, that we put it all in these  standard formats that's easy for machines to talk about.  So I wanted to test that and see can we really do that.  The other is that we really want to provide an efficient and reliable access.  And we want a means to provide the user who's making a selection with enough detail that  they can make as accurate a selection as possible.  So we approached the problem by identifying some of the challenges.  We identified a couple of authorities we wanted to try and do this with.  And the first challenge by far was finding documentation that told me how to access everything.  Once I located that, it became clear that the API that you use to access things, the  various authorities are different.  Some don't have any support for link data at all.  And some have partial support where you can get data about a single term, but you can't  do a query and get link data back.  Some require logins, some have Sparklin points versus like a REST style type API.  The third challenge was varying results formats, but I would say by far that was the easiest  challenge to solve because we just used a standard library that took whatever format  it was and threw it into a graph.  And then I processed the graph.  So that was pretty easy to deal with.  And varying ontologies also did not present a great challenge in the configuration itself.  I identified what predicate plays what role.  So again, that one wasn't too challenging to deal with.  So the approach we took is questioning authority was a gem that already existed, but it did  non-link data and it had a one-off code for every authority you wanted to access.  So if you wanted to access a new authority, you would have to go out and write a new module  in order to do that.  So what I did was I wrote a module to access link data.  And I only wrote it once and all the authorities that I accessed through here that are link  data use this same process.  So this sits in the middle as a normalization layer.  Up at the top is the application that wants to access it.  And at the bottom is whatever authority that you want to access.  So much like who just showed earlier, you send a URL request to questioning authority  service and it will translate that into a request to the actual authority that you're  accessing, which will return to you link data.  Question authority goes through a process of normalizing that link data and only returns  what you requested.  So who to show does slightly more extended version because it included additional context.  This is sort of like the minimal record.  It has a URI, an ID, and a label, which is all you need to do autocomplete.  So I just want to show this is what the URLs look like if you go directly against the authorities.  They each are different.  They have different parameters they pass, different structures.  Some require a login.  And when you go against questioning authority, these are the URLs you call questioning authority  with.  They are exactly the same except that you identify what authority you're going against  and what query you want.  And there is some leeway to define the parameters the way you want if you're writing your own  configuration.  So if an authority provides additional parameters, you're not limited.  You can use those.  And I'm not showing the results because if I showed the results from the ontologies,  it's quite extensive.  But this is the normalized results.  And the thing I want you to take away from this is that's three different authorities,  O, CLC, FAST, GeoNames, and AgriVoc, and the structure looks exactly the same.  So from the application's perspective, they can just use the same process regardless of  what field they're trying to fill in, regardless of what authority is actually pushing that  forward.  So after we met those challenges, we went on to a few more.  So our second set of challenges were, well, first I'm going to give a hint as to how I'm  going to solve this because I want to talk about it as I talk about these.  We're going to do a cache server where we actually put the triples from the external  authorities into a local cache.  And so one is reliability and efficiency.  If it's an external server, we have no control over load factors, server uptime.  So if we find that our server is getting bogged down, we can throw more resources at it if  it's our server.  Accuracy is kind of a mixed bag.  We can write our own algorithms that determine how we search for things and find search results.  One of the things we lose when it's our own is sometimes the authorities have specialized  knowledge that they have in their data set that we don't have.  And so we won't have that as part of our algorithm.  But the last one is not available from any authority that I worked with.  None of them have order rankings.  So if I get back 50 results, because this is a graph situation, I pop those 50 results  into a graph, there's no order.  So I can't distinguish between the best match or the worst match.  So by putting it into our own, I'll show you how we solve that problem.  So here's our cache server.  We basically have a Genevuse Qsaki triple store that we dump all of our triples into  and we have an index that sits over top of that.  And we can tweak that index as much as we want to improve the efficiency of what we  get back, the reliability of what we get back.  So the process basically is that when our query comes in, you do a search through the  Lucene index and it gets back and extracts out the rank order from the search and also  the URI.  It does a second search of the Genevuse Qsaki triple store to get all the additional context  that it wants to include in the results.  And then it has a process where it combines those together.  So it stuffs that rank order into a predicate in the search results.  And so then when I get that back, I now have a predicate that I can sort things on before  it gets presented to the user at the end.  And basically in questioning authority itself, when I do that normalized last layer where  it's setting things up with the URI, the label, and the ID, I've already ordered them.  So that when the end application gets it, they can just put it in, use the order as  it comes from QA.  So the other thing I want to point out here, this is just a slightly extended version of  the diagram I showed a minute ago.  But now across the bottom, you can see there's different kinds of data sources that we can  get it from, direct from the authority or from our own cache version, or you can get  it from other caches that you might choose to set up.  So we had another set of challenges that we are currently working on, and that is the  disambiguation.  We are part way through that as Huda showed where we actually can do a little more context  in what we present back to the application so that they can give that to users to make  a better choice.  The other is that we're working on reconciliation across multiple sources.  So what we want to be able to do is actually do the search.  By having the triples ourselves, we can do a search across multiple authorities.  And there's all sorts of questions that come out in that about how we match up URIs and  things like that.  So I'm torn between saying, do you want to do questions or what's next?  I just will real quickly say what's next and then we'll do some questions.  For the architectural side of things, what we want to do right now is generalize that  process that we're doing to access the additional context because right now I've basically done  it for LOC genre and I want to be able to apply that same process to others.  And the multi authority search I mentioned and also one of the things that happens when  you start collecting the link data yourself, you have to deal with caching and cache refresh.  And that's a whole other topic so I'm not going to go into more detail than that.  We're also looking at the potential to mirror cache servers to also improve fallback and  availability of the data.  And who is going to talk just one second about the user experience and then we'll take  a few seconds.  I just wanted to quickly say as part of the work that we're doing with the authority  lookup stuff in VTRAlib, in general with VTRAlib we've tried to use a user-centered design  process which means we really actually want to understand how catalogers do things and  what we can do to support them.  So there's a lot more work in that area for trying to understand how to actually get the  lookup context to work with them, what pieces of information work and how can we support  those tasks.  As part of the experience there's also the matter of search results and how they're ranked.  We already have a situation where we showed them the search results from our cache server  and they said we want something a little more exact because I don't want to see these  other 20 things that I don't want in my results set.  So how to tweak that, how to make that work for them.  And then a quick word for the fact that we talked about two platforms but there's potential  for applying this in additional platforms such as Folio perhaps.  So that's it.  So I'm just going to say that the tiny URL there is for the slides themselves.  There's actually additional information in the slides and sort of like an appendix.  Great.  Thank you very much.  All right.  Do we have questions for Huda and Lynette on this excellent work?  Yep.  Yeah.  Thank you for an excellent talk.  I might have been responsible for one of the services, namely the one that's about AgriVoc.  How was you?  Sorry.  Well, yeah.  In a way.  But yeah, I think you're talking about linked data access but actually the search part where  you actually send a query somewhere, it's not part of linked data really.  I mean, the response might be linked data in those cases but the original, I mean, there's  something like an API there that you're sending a search to.  And so that's something that there is really no standard for except maybe it's something  like Open Search which is really broad.  You can apply it in many settings.  And then for the responses, that's also something where linked data doesn't actually shine.  Even if the server had a sort of a notion of order, it's difficult to represent that  in the response.  So that, I mean, that was just a comment.  But I think you're sort of hitting the limits of linked data both ways, both in sending  a query and getting the result.  All right.  Thank you.  Was there comments, other questions?  I would say I'm happy to talk about that more if you want to.  Sorry.  I didn't mean to interrupt.  I was wondering if you were just like processing the brilliance of Osmo right now.  Well, I have several opinions about it but I don't know.  I do think that linked data, sometimes when we talk about it, we look at it as everything  has to be linked data.  And I'm very pragmatic in the way I approach things.  And to me, the part that linked data promises us is easy interchange.  And so when I can use the same process to access linked data from lots of different places,  I feel like I'm getting the benefit of linked data.  And then I'm trying to smooth the path and make it easier for consumers on the other  end.  So I don't feel bad that I'm not passing everything as linked data.  Great.  All right.  Well, thank you again, Huda and Lynette.  Thank you.  This is the last round.  And I wish everyone had a wonderful evening and thank you again.  And you have our
Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_0_71.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_72_683.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_684_2843.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_2844_5135.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_5136_5435.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_5436_6431.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_6432_9491.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_9492_12035.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_12036_12179.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_12180_12911.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_12912_13187.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_13188_13319.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_13320_15455.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_15456_15575.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_15576_16175.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_16176_17459.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_17460_18143.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_18144_19535.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_19536_24575.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_24576_30347.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_30348_31259.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_31260_34943.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_34944_35015.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_35016_35087.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_35088_36887.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_36888_37283.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_37284_38183.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_38184_38327.png Linking_the_Data__Building_effective_Authority_and_SM3W6bUFO2c_38328_38477.png