so is it correct or no oops you never know whether you have got these things on right or not you never know whether they are on right or not yeah yeah well so we hope feels wrong okay so i do not need that okay okay so david came in to me and to say i something we had been talking about for a while was some stuff i had been working on about using these topic models with with networks with n things with links between them documents that are linked n it is actually it is related to rapple the rapple also no but it is not actually that th this is something i have not actually been working on for a couple of months now just because i came up with something more interesting that i have been working on okay but th at the time basically i had been trying to use p l s a and those types of models on web pages and that came up with a big problem in that most web pages have very very little text on them but you can tell what they are about based on the other texts on the site the texts on the things they are linked to so what i wanted to know was can you combine the topic models that the pages that have lots of text on them with something else that that tells you what a page that has no or very little text on it based on the links yeah with the link here in so quite simply if you have a page one which is like a home page and that will maybe have 5 links and a picture on it and then because it links to page 2 and page 3 can you tell what this is about yeah yeah so i started looking at google page rank which kinda does this by saying well the links go the other way it says if this page is about fish or in google they say if this page is good in some way and it links to this other page and we do not know what this is then that page is more likely to be good and it is just page rank is just a simple algorithm for doing this over the whole big network so i wanted to combine the topic models that tell you beforehand what you think this page and this page are about with this to maybe get a better idea so the idea is say you have got some text on a page that is only part of the picture about what of what the page is really about and the links pointing to that page are also part of the picture so i basically came up with an algorithm that is just a i it plugs p l s a plugs page rank into p l s a right so page rank you basically just d say you basically take the how good this page is which is x and how good this page is which is y and you then say this page is based on how do you do this based on the the links coming into it it is just a like a sum over all the the links of well x link i guess of the value yeah 0 of of the value of the previous r page rank or of the the but it is not it is yeah if if so if this is page rank of x page rank of y okay i think this is roughly it off hand page rank of z so you say page rank of z is equal to one over a constant to normalize it sum over all the links into z of page rank of the link yeah tha it is like that yeah so similar idea if you got a vector that says what the topics are say you have learned from p l s a a d a distribution of topics for every page so if you say a probability of topic one for for page z is equal to and you can use the same sort of thing right so nicking a whole bunch of maths from how you work out page rank it comes up with just an iterative algorithm which is just the current probability of topic one given z so that is probability old plus okay i yeah plus that and then you normalize between them so you say alpha and one minus alpha yeah okay so w basically what you are saying here one minus alpha say link contribution you can call this i guess so what you are saying is that a page is partly from its text that you have got on the page and it is partly from what the links say it is about so that alpha thing is how much do you trust the text on the page and how much do you trust the links huh so that alpha could be different for every page so what i did then was try this on a big database of web pages okay and i had some results from that but they were all on temp 2 so that is not important right now but as it is not really important i can regenerate that the problem was that this gives you a new vector of topics for every page so you have got 2 vectors you have got the one that you have just got from p l s a or whatever model and you have got the one that you get from this how do you tell which one is better i had no way to evaluate this well how do you tell that this page because you do not know where the topics are cause they are learned automatically so huh yeah so wait i do not get it you you have 2 representation one which comes from this link stuff and one f this one is not based on p l s a too it it is yeah because b yeah you initialize it with actually thi this is kinda wrong yeah it is the initialisation of okay y this is not old but start this is how you this is how you you start it off a t yeah okay yeah but then it is iterative so this this is what p l s a gives you to start with and then you iterate this and it becomes something else yeah yeah okay okay okay okay so you you should 1st train the p l s a stuff then you initialize this link spreading thing and then you iterate over okay yeah that is it exactly yeah and it converge and so you have the boost initial p l s a and p l s a plus the link thing then exactly yeah so you have now got you have got p l s a gives you 2 probabilities right so it gives you a probability of a topic given a document which is z given d and a probability of a word given a topic yeah so this is the same for both models yeah yeah 0 this is is frozen frozen okay well yeah what you do is you are taking this and iterating over it the probability of the topic given the document ah okay so have i made that clear i am i am not sure if i am explaining this well yeah yeah it is okay yeah okay right so y i you m so th yeah you you you were saying that you would like to evaluate that now yeah so we were struggling at how do you evaluate this yeah right because i mean if you look at the way that p l s a and all those kind of models have been evaluated it is th they never quantitative quantitatively do anything they always say well look at the clustering there is here we have got some documents that all appear to be about the same thing perplexity and here is where we get some other documents that all appear to be about the same thing and you can not really tell exactly how good that is you can just get an idea that it is yeah or they use the perplexity on the the likelihood to say 0 good this likelihood is is it is smaller than the other yeah yeah so tha that is the 1st thing th that thing when i spoke to you about this you suggested d use the likelihood and pedro came up with a really good example why that could be bad so you have you have the 0 let me thing of what this was it was to do with cars basically if you have you have a topic that is representing cars and you can tell that that is representing cars by looking at this distribution and it is the same for both now you want to tell which i i this page pedro is example of what it was is that is says on the page lotus elise okay and that no f i can not i just remember it was a really can you ig remember what it was 0 no re no i do not even remember the the you said i would tell you that evaluation should be done based on likelihood or perplexity no okay yeah i did not really consider doing it on perplexity actually that to be honorable but okay well it is bu yeah it is basically the same but it is the same but but maybe you you could have some some other labelli li or the task like if if you imagine you have some labeling on of of the web pages like like like for example you you look at at at at at a set of pages i dunno yeah another task or and you have the m the directory entries of these pages so so f like yahoo or entries okay so you have categories and and entries so for example you have i dunno cars s u v etcetera so what you would like would be that 2 pages on in the same category should be closer than than 2 pages being spread across different categories is that fair though because i think one of the things with these topic models is that you do not know you can not make them cluster but you you would not use this you will not use the yahoo directory or n the directory in the training step of i of the model okay yeah okay this is something you have just for evaluation and then for example you you your task would be to say i have some page and and i know where it is in the in the in the directory and i have another page i do not know where it is and i would like to to know whether it is in the same category as the 1st one just by comparing their aspects i it is making the assumption that this human categorisation is related to the one that is discovered yeah if they have the same yeah so basically what you would like to check if is is is there some similarity in the in the in the aspect distribution tells me something about the sim the the fact that they could be on the same category in the the this end end label categories yeah okay and that is is that really fair though can you say that if a set of documents if the clustering of these documents is closer to the effecti what you what you have got there is a clustering of documents a as your devaluation yeah 0 yeah yeah yeah thi this these labels the yahoo categories that is that is a clustering yeah yeah yeah so you are by saying that this clustering is closer to that clustering is is does that necessarily mean that it is better no but no no no i no no no no it is does not have to be that if if you use the latent representations or distribution over aspects no no no no no how would you could use this to save 2 documents in the same directory or not and this you should be able to evaluate then so so so w what what you would have have basically would be to have a directory which give you labels for some part of the documents okay okay so this is a directory and this is some some some some web page okay so the labeled one okay and you have some page which are unlabelled but you have links between those 2 so what what you would do is that you would try and p l s a and and and use a model over that then then you would like to for example what f what could be very simple would be just to then to match each pages like that so let us say you have you have a a page which is labeled and you say okay the the label page 2 should get should be the one w which it is the most similar to or any kind of classification rules you can imagine and so and so you say okay what what is if i from from this page which have the similar so so the the criterion would be to compare those 2 distribution okay and then and then you would like to assign to this document too where you do not know the labels the same one as this one and and here that you use only for test so this is your then you can evaluate how wrong you are or cu how close you are from the real assignment the right directory category kay but i have the feeling i in in this way i think it is fair if you kn if you know only if you know only that and that and what you would like to discover is this this is a r like a real task like like if if if this directory company would like to extend this directory to new pages without the hassle of having human labeling but in this case you you have to use the label one yeah yeah you have t you use this one your los you you only have the labels for some small part of the but how does it enter in that we i think your point was that you you train this without labeling 1st yeah so i i you have to train your so you you would train it you this is this is t t all all those links are known your training data the links between some labeled document and unlabelled document yeah but the point is that you should change in a way that to take the label into account or not i d i think you can do it as a 2 step process okay 1st you you train well maybe s it wou could be better if if if you take it into account but but maybe you can you can do just a 2 step process 1st one you have this unsupervised task over both p l and p u okay okay and then you from these unsupervised starts you can know which document of p l is close from the document of p u or the reverse yeah and you can infer some label to a document of p l clearly wha we we had done these you can train after that an s v m saying this is category one or 2 on the label document and then we will have an unlabelled document and you just say this is directory one or 2 and if it should improve it if you say this feature extraction process is better than you can have a better cleaner separation between those 2 well th yeah the the thing that worries me about this approach is that say the really really really simple example of this where your directory had 2 categories in it good and bad yeah yeah yeah and and and you run p l s a on it now that does not do it good and bad it does i dunno english and french pages so they do not match but the clustering is still very good in in one respect but but you you can extend that up even if you have a 100 categories yeah well but p l s a can find a 100 clusters that are a good clustering but are completely unrelated to the directory yeah well but i mean if you have to judge it is with respect to a given task i mean you can not say of course it is always good because it is maximizing the likelihood of whatever no but w but but what i want to know i is that which of them has the better representation of a given page yeah but so so given a random page and these 2 models and the wo the words that these 2 models give which is more likely to yeah yeah yeah but but here you can you you say given a random page of p u which of those 2 model is more likely to to to to to say for example that thi the page of p l which is the most similar so you so so from this so you take any distribution comparison things so so hoffmann in his paper was doing just cosine similarity but you can do anything so so you just compare those 2 distributions so of d one and d 2 okay and so with that you can you can determine for example for a document 0 of the set 2 to the most similar so you you you you select d one so in this set which is the most similar to d 2 according to the distribution from one or the other model and then and then for this document you automatically assign to it the label of the document one and you check with this which is your your your if you are right or not this is what what this is very related to what frank did with the with the keyword image annotation okay yeah it is the same so you would have a set of keywords the set of image label with keywords and he has a set of unlabelled image with some and it would check whether the the label he assigns so in this case annotation word h are correct and i think it is fair it is fair setup yeah okay fus so maybe this classification rule is not the best there there is some might be some work to do but the general idea i think it is it is fair evaluations it is a task s okay i am i am just n concerned that that is that that is it is answering a question that evaluates in some way but maybe not in the way that that can be dep well that maybe not is the same way i was thinking but maybe it is a better way so what i mean by that is that sorry i am trying to process all this yeah yeah i it is the same thing like it it does tell you whether this which clustering is better for this task yeah but it does not tell you which clustering better represents the pages necessarily no but according to this directory yeah but i but it it is it you you could say that it possibly doe it probably does yeah cause i do not think there is any way to say generally this this is better or worse no yeah okay the the there is not v perfect clustering clus wh what is the how do you judge what is a good clustering i i i think it is w yeah yeah it has to be re it has to be really to with respect to this this clustering makes sense or i dunno yeah yeah what is good with the directory is that you have a good coverage of various topic humans humans can label but then if you want to detect different languages this is another task i think the task of this latent model is to discover some some semantic like some topic proximity between documents and here this la this directory is doing the same thing like associating some topics with some documents manually so comparing them is is is is good setup with respect to the goal yeah i and you are not exactly compar you are comparing them i compare those 2 like the p l s a and the one over this same task of assigning good directory ah okay okay yeah yeah yeah i do not know if if it if it will provide good result result or not but it is clearly a benchmark for those 2 goals yeah you can e even keep the bag of words and do the cosine comparison or also like a baseline yeah on on the infra yeah yeah just showing just showing how just 1st how better this is with respect to bag of words and hopefully then how you new regularized p l z given d is yeah like bag 0 yeah yeah you can have tree setups yeah i dunno just yeah it is basic okay yeah basic cosine over the p of t given d so over all this empirical distribution and the 2 aspect models but i think you m you still not combi we have to repeat over and over yeah no i th i am i am coming round to it actually no i am so i i i think i g i i get it this is like advertisement no no but it is another no no it is joke i i th i think it probably is fair i am just i i have gotten uncomfortable with it but i think that will go if i think about it enough no it is jus but but yeah the it it but it just seems a little suspect yeah but i think that is just from the whole point that you can not tell which clustering is b better yeah yeah yeah just know where but what i would thought before was could you get people to evaluate this by by hand basically this is really yeah yeah could you give people a set of pages and like say you did it with 5 topics for example yeah now you give people 5 sheets you give a t a a user 5 sheets of paper each with words in varying sizes representing the probability of that topic for example yeah try give the user these pieces of paper to get an idea of what these topics are that is this p of w given z yeah and then you give them some pages and say which one would you which piece of paper would you put that with yeah and then you get like an intuitive ide like how would a person cluster pages given the what these topics supposedly are yeah cause i you have got something that is fixed the the p of w given z with respect to both models yeah so i was tr i was hoping you could use that to but the point is that the z the the the aspect like the z are are are come unsupervisely it is it is more like if you were giving the people you you said to them okay divide me this bunch of documents into 5 topics and you do not give them the topics a priori you know so the 2 person could come up with very different one is very interesting i in in fact i i have seen a paper in i c m l where the guy was trying to learn which word he aspects of clustering so i give you i give the real users the person a bunch of marbles and they begin to say okay and they have to divide it in 5 group and in the end it is not al it is it is always different to the clustering each people has because it is completely subjective yeah okay but in but there is some characteristic that are for transparency color which are features that made the clustering that can be extracted even if the clustering it is always different yeah right yeah okay so i i think i think it is yeah so so that is only to say that it is it can be really infinitive clustering see the the thing about web pages i thought as well in that getting humans to evaluate it in some way is that you have most of the web pages that are well a lot of the web pages that are important in some way are are are home pages so google dot com does not actually say the words search on it i think i maybe it says go but you only know it is about search through the thing but someone looking at it can tell that it is a search engine because of the way it is laid out because of the pictures on it and because of all of that yeah there is a lot of information that is completely lost in this bag of words representation and i was hoping that in some way you could use that to evaluate this yeah i have no idea how but yeah say another another example apart from this search i do not get it exactly what you mean then do not do not use a google page because it is too s root but another page but eve ev even someone would would provide very different labels like like someone would say it is a page in english someone would say it is a it is a it is a it is a it is a web page wh which has very few information in it or anything so i think there is no hard clustering and what is good with those model is that they they can they they are intended to provide you some some kind of way to compare 2 documents rather than to give you pre defined sets of documents yeah this is not hard clustering the 0 i i do not think 111 aspect or one latent topic has any sense by his own yeah yeah it is when you have the word set and the word distribution that you c you can tell something if you just take one of them alone so you take all the p of z given d and all of the p of w given giv given z you you you can do anything with that it is it is the whole set i would say an an and this is 0 it it should be evaluated meaning that like for example the the example about marbles is that the point what you can learn from all these sets made by people is that is that those 2 those 2 marble would be consider as more likely to be in the same sets because they they share some common properties which are not seen as the same for every one but they share some common properties that people will notice and if you look at the 2 the d all possible pairs of marble yeah you would have some mu much more agreement between people than than if you look at the predefined sets people would would just cut at some precise point you know what is okay i think i know it but it yeah thi this is effectively what we have here though and th that directory is one clustering of the marbles yeah and you are hoping that it it so what you are saying is that i have 2 clusterings of marbles for my 2 models yeah and i have a test clustering of marbles it is saying that one of these is more close to the test t t to the the one that we have been given well you do not you do not really have 2 clustering no no no no you do not you do not really have 2 clustering no it does is that it is saying that what does that actually say 0 it is not that that does not it says it is g better for this task but it does not really say anything else yeah no no no no you do not have 2 clustering of of of marbles it does not have to match what you have is y is you have is you have one clustering of marble which is this one and then you have lots of people telling you i consider those marbles as very have to have very salient feature that they share in common i i c i would like them to be in the same cluster and you c and and and those 2 so and one of them is labeled and the other is not labeled okay so you look at that you say 0 someone told this one was very similar and this one is said to be blue for example i dunno if it is the label so i put it in the blue bag okay but but you do not you never use the blue bag you do not ask the people to look at the blue bag you just look at the people you you just have the people to compare those 2 marbles and then it happens that some of them have been clustered and and people being your algorithm that is the point yeah yeah yeah yeah this is i reverse both role it is getting but yeah it just re okay the this is the players and so and so if i get that right what what the algorithms are still coming up with is in this way you are looking at a clustering of marbles yeah if each of the algorithms comes up with a defin i i no no no no no it comes up with not with the clustering but with some way to compare 2 marbles okay some would say would say i consider those 21s are completely dissimilar and the other ones say they are very similar ah okay right that makes sense yeah and if you find that in your unl unlabelled clustering those 2 balls marbles happen to be in the same cluster you said the user algorithm might be wrong but you average that over lots of documents okay because the the latent aspects really does not have to match the final directory category exactly it can be anything but it is just a way of saying yeah those documen documents are similar in some sense so we are not trying to to fit yeah okay is it just it just sometime dangerous to say th these latent aspects means this okay yeah i think yeah i am convinced i am convinced alright good good good so it is good right alright well thanks i am trying to think of anything i have not said about the stuff but that from looking at the results i got it looked quite good just like hand reading them in that one of the things i did was i started a i a web crawl at the idiap homepage it did not very many pages only like 2000 or something like that but enough to get the pages that were close to idiap to have a lot of links between them so that there was enough to be able to run this link clustering algorithm so what you find is that the pages around idiap there is a lot of english pa speaking pages and a lot of french speaking pages so of course this p l s a completely separated those out into into categories so when i looked at the the word likelihoods there were 2 topics one was very clearly english words that most likely word was the and one was very clearly french words most likely words was de i think and you had onl 2 topics no i had many topics or no yeah the others looked like garbage could not tell what they might have been but anyway tha that is often the case yeah when you look at it you can tell very clearly that one or 2 topics are a b not the rest yeah yeah yeah yeah yeah yeah and the others usually do not mean anything yeah yeah yeah it is they make sense only so global optimisation the likelihood is optimisable globally over the whole aspects etcetera so it is it is you never ask the model to do hard clustering of documents you just ask it to find a distribution over aspects such that a document is more likely according to the restriction that the number aspe aspect is limited and and so it the it just makes sense on the looking at the whole set so y i think all evaluation should take that into account and look at the whole aspect set ok like here you look at the aspect set because you look at the similarity of 2 distribution an any saint any kind of thing you where you un unfold the model and look at it manually looks weird like like in those hoffmann papers where show you columns with words or things that w anyway that when i run it on that what i found was looking through some of the sites i looked for the sites that the models differed on that where one said it was probably an english phrase and the other said it was probably a french phrase and i found a good few examples where you had a site that looked well one that i remember off hand was it was a page and it was a big picture and links down the site but the links were all pictures so you could not the the words were in french and the links but there were no words picked up by the when i did the crawl for the model and there was a small bit down here and in this there was meant to be picture in there but it did not load or b it was meant to be something else and it said in english it said 404 this page cannot be found so all the words on this page were english but it was very very clearly a french page so the 1st model said it was an english page cause all the words were in english 2nd model said very definitely a french page because all of these pointed to french sites and it was only pointed to it was something dot f r as well it was linki okay okay it is okay so it was a a good point for you yeah for your model good so i i found a couple of examples that were kinda like that like this yeah okay but the the problem that i found when i ran it the th the thing that was a bad point was that if you have say tryin i was trying to say if you could evaluate this with search given some keywords like which model would more likely find you a p good document that that sort of approach what i found was that say you had th tha that is not too relevant necessarily but if you have say a topic distribution for one page and it very very highly scores on the p l s a for one of the the topics and that is right okay this model because it is averaging out over all the pages nearby all of the topic distributions become more flat it it becomes more uncertain about everything okay so i do not know if that is necessarily a good thing or not yeah probably not yeah yeah in in some cases it is going to be a bad thing b because ev every page is linked by the way the algorithm works yeah every page is linked eventually to everything else so it every page has information incorporated in it or i the distribution learned for every page has information incorporated in it from every other page is distribution as well so it averages out it flattens all the distributions depending on what this alpha is yeah yeah yeah but also it depends on that could be a good effect and because if you if you can be if you can if you are highly reliable about one page but all the page linking to it are actually about very different topics so you could say maybe i was wrong from deciding only due to the text that this page was about this aspect distribution because because all the other pages are voting in different direction well i i think that the thing is while there is a balance yeah that that there is a balance in the for example take a page like slashdot and it is like a news site for geeks and you have nerds yeah nerd nerd news for nerds stuff that matters yeah all of the like 100s of 1000s of pages point to that but they are about completely different things so you in effect are losing information about what that page is about because there is so many things pointing to it and they all average out as yeah this alpha i can it be dependent of the of the document yeah so that and how many text is in it or something yeah th that is my 1st idea w was how i set this i remember for the lotus thing was not there so i thought you want to probably set this alpha per page there is probably some heuristics for example this page there is no text in it will set it very low in user information for the links so problem is this is what pedro immediately said what if you have a page that looks like this it has got a big picture of a car pedro is always speaks about cars yeah cars and web anyway and it says lotus elise so you know that it is definitely about car but it is only got 2 words so you can not just do it by words because those 2 words completely specify into something or other su you suggested doing likelihood but at that time and i can not remember what my problem was with that it might not have been a good thing but you have got 2 likelihoods for it as well but i think for any kind of way you have to select those alpha you you have to look at the effect on on the re on the t evaluation task no maybe not and then maybe you can find some patterns that are then then once you have the once you have the task yeah maybe do it by cross validation something you could look at at which pages are completely missed by one model which others are are successful with one model and and you can maybe infer some some rules afterwards but if you have no task you can blindly select s good alphas okay there is like like if if you look at if you look at your task and you see that i dunno 0 of course yeah the the long pages for example have good p l s a model without fusing links so then you can infer some heuristics like okay the alpha should be proportional to the lengths or anything like that or to the number of image in the pages or anything but you should have some some task where you can individually look at the performance on every on every page yeah yeah w w i think we have agree on that yeah yeah that that is fair and and and and at as a 2nd step you could infer some kind of rule about how to select alphas with respect to the task yeah okay maybe it is v it is very simple rules but select them blindly s might be odd it yeah d no yeah i so i i think that that is too easy you can do it with some heuristics and you can learn it from the task results in or combine the 2 so tha that is kinda straightforward i think yeah probably heuristic it will be easier to compute because yeah the the the thing about doing it through cross validation or something like that with the task is the problem with doing a page rank style algorithm is that is that it gets better it it converges better and it gives you better results yeah the closer a network is to being a small world network so small world network mean basically some far links but generally very clustered locally yeah yeah now the thing about doing a web crawl is the bigger your web crawl the closer it will be to the overall structure of the net n the internet which is a small world network the smaller your web crawl the less small world network like it is going to be yeah so the more pages you get the better your results and people running this style of algorithm the page rank style stuff generally run it on about 2000000 or 20000000 pages or whatever the the stanford web base data set that i was using for this for for a bit of this yeah they have a a d i can not remember off hand but they have m maybe a bit over a theta byte worth of web pages in this a bunch of servers but but but still li like the the the work i did on on hyperlinks like i used wikipedia yeah maybe wikipedia is a good thing to do i i think it is g because it has fairly high level of inside links between was i was concerned about wikipedia though was that it is while it has a high level of inside links that has a lot of directory style links as well that might mess up it actually being a see wh while you have yeah well you can categorize them because they are th it is there is a structure on the on the web page so you you can select only the links which are within the article or within the categories or it is it is it is well structured it is it is easy to manipulate and ah okay that is good that plus i guess it is actually straightforward to say to start with is this a small world network and then run it on that yeah you can even isolate the directory from the categories of wikipedia you were running that on what we what was it like half a 1000000 pages something like that well yeah actually i i i divided in 3 equal size parts which have 150 documents is it and that was for half of it if i remember okay okay the whole thing was there are a 1000 documents and well but did you manage to to divide the link properly or well what i did with the link well but this is not th exactly the same setup you would have okay so in this setup the whole link structure could be known and the only thing you wou do not you w do not know would be the assignation to to a category in my setup it was different because i i sh i could not use the links between different sets but this is not i do not want to mess you up with that it is not the same kind of experiment but you could just use the whole corpus like this and and for some of them you assume you do not know the the category assignment and then you can run this set of experiments over wikipedia okay i mean i think off hand thinking about this model it would have to work better because 2 things in the same category are going to be linked yeah so it it yeah but this is the yeah this is the hypothesis between this in this p l s a with link stuff okay 0 yeah i thi yeah that is i think tha it is going to be more true in some networks and in some tasks than others and i think in the wikipedia it is probably going to be it is a good one to test on yeah but you can you can think also at at other task like that like for example you can take proceedings and assume the categories are the keywords of the of the d of the documents and then if you have the archive of a journal you can you can use the l the links between the citation links so if you want to have a bigger stuff so wikipedia would be kind of very small with lots of engli in links okay and then if you take a journal you will have less links and and more documents and then you would be likely to say that p l s a might be better over that because you do not have enough links so you you you can find examples which are not the web i think it is good to work in closed set okay wikipedia is kind of closed set ma even if there is lots of outside links but there is lots of inside links you have to think at at at at at at at database where there is lot of in links wikipedia might be one citation for example if if you take the nips arch archive which is available nips author always other nips author which are often themselves etcetera well it is reported i i will not say so but but but but yeah i think th th this kind of setup maybe web pages it is hard to find sets of web pages which have good links stuff if you start from one page or something okay yeah it i will maybe i will have a look at the wikipedia stuff anyway i am stuck doing something else for the next month or 2 anyway so this is always the case okay the there is another thing this was new but i have not thought about this too much but it is on the same lines in that have have you guys seen this correlated topic models work which ones correlated topic models it is david blei the guy that came up with with l d a no no l d a he has got a new model that is it is published in nips this year but his co author put it on their web site a month or 2 ago so you know l d a d you all know l d a or yeah okay so a l d a you basically draw a a dirichlet which says like what type of document this is effectively and then you draw a topic given that dirichlet distribution and then you draw words given that alright well that is for for every word and that is once for the document and that is you have got a parameter coming at this globally so that that is roughly what the model is plus other little bits but anyway so what they did was they they reworked this because one of the problems with l d a and i think p l s a has this problem as well in is that the more topics you have the less expressive it can be because the topics it cluster it it pushes a document if it is pulling a document into one topic it pushes another topic away from that does not that make sense so the topics are necessarily trying to be different from each other so the more topics you have the more th they are sort of squeezing into the space of what things can be about and things are kinda pushed out so once you m go over you try it one corpus and once you go over about a 100 maybe 200 topics the the new you add more topics and it does not get better in any way the model does not the n new topics are effectively going to just be garbage alright so what they did was they they they stopped using the dirichlet and they are starting to use a gaussian there so it it in to spare the details basically they have they have got a variable here i think they call it eta and they from this generate a distribution of topics in some way but it is generated from a gaussian space from a from a a space so it is given a a mean and covariance and but it is discrete no the yeah so what it does the this is the the bit that took me forever to understand actually same model but this is given a gaussian space it maps that down into the simplex of what a topic can be about the so the simplex basically if you have got a say you have got 2 topics now say i i say you get 3 topics that makes more easier to draw for this is theta one theta 2 theta 3 these are the probabilities that that a document is about this theta one is the probability that this document is about topic one theta 2 is so these guys all have to add up to one yeah so that basically means that what a document is about lies on this space between them yeah pyramid yeah so this is the simplex so basically a distribution generated by this guy is basically a point on this simplex so what they do is they have got a distribution that maps a gaussian and some space into a distribution on this thing so it maps it into gaussian in that okay the cool thing about that is that if you have got 2 gaussians you can tell how close they are so you can now tell how close 2 topic distributions are right ah okay so you can have as many as you like and you can tell that they are close to each other so there is load of ideas that i had about this in that if you can tell that 2 topic distributions are close to each other so for example if you split a document into 2 parts and said that the 2 parts have to be close in the distribution you can see how the doc t the document changes for exa stu stuff based on that with this idea what if rather than working with this p of z given d thing that you had from p l s a but what if you worked from these gaussians that you get for every document and you said that in some some function that said that the gaussian representing this should be similar to the ones linking to it now i have not thought about any of the maths for this but but it should be straightforward i guess to to formulate a basic in terms of the mean should be similar and the covariance maybe should be similar as well so it might actually be a better model again yeah i dunno okay yeah i will send you guys the link for that correlated topic model stuff if you want but i do not know if it is too interesting to you yeah kay yeah thanks okay kay alright well than thanks for hearing me out and the idea especially okay yeah maybe i can do more with this well in in 3 months in 2 months yeah alright well there are lots of good ideas that are recorded and someone will yeah 0 yeah it is too late 3 months we should wait yeah yeah b because the data like will be released very soon bastien you can not release this data for 4 months
i do not know i am not really i am sorry i am sorry back a bit easy yeah you never know whether you have got these things on right or not you never know whether they are on right or not yeah well so we hope it feels wrong okay so i do not need that okay so david kim and to mean to say something we had been talking about for a while was some stuff i had been working on about using these topic models with with with networks things with links between them documents that are linked they are re proleting it is related to ripple but it is not actually that this is something i have not actually been working on for a couple of months now just because i came up with something more interesting that i have been working on but at the time basically i had been trying to use plsa and those types of models on web pages and i would came up with a big problem in that most web pages have very very little text on them but you can tell what they are about based on the other texts on the site the text and the things they linked to it so what i wanted to know was can you combine the topic models that describe the pages that have lots of text on them with something else that tells you what a page that has no or very little text on it based on the links yeah so quite simply if you have a page one which is like a home page and it will maybe have 5 links and a picture on it and then because it links to page 2 and page 3 can you tell what this is about yeah yeah so i started looking at google pagerank which kind of does this by saying where the links go the other way it says if this page is about fish or in google they say if this page is good in some way and it links to this other page and we do not know what this is then that is page is more likely to be good and it is just page rank is just a simple algorithm for doing this over a whole big network so i wanted to combine the topic models that tell you beforehand what you think this page and this page are about with this to maybe get a better idea so the idea is say you have got some text on a page that is only part of the picture about what the page is really about and the links pointing to that page are also part of the picture so i basically came up with an algorithm that is just a it plugs plsa or plugs page rank into plsa so page rank you basically just say you basically take the how good this page is which is x and how good this page is which is y and you then say this page is based on how do you do this based on the links come in to it it is just a sum over all the links of x i guess of the value of the value but it is not of the previous page rank yeah so if this is page rank of x page rank of y i think this is roughly offhand page rank of z so you say page rank of z is equal to one over a constant to normalize it sum over all the links into z of page rank of the link yeah it is like that yeah so similar idea if you have got a vector that says what the topics are so you have learned from plsa a distribution of topics for every page so if you say probability of topic one for page z is equal to it and you can use the same sort of thing right so knocking a whole bunch of maths from how you work out page rank it comes up with just an iterative algorithm which is just the current probability of topic one given z so that is probably old plus yeah plus that and then you normalize between them so you say alpha and one minus alpha so basically what you are saying here one minus alpha say link contribution you can call this i guess so what you are saying is that a page is partly from its text that you have got on the page and it is partly from what the link say it is about so the alpha thing is how much do you trust the text on the page and how much do you trust the links so that alpha could be different for every page so what i did then was try this on a big database of web pages i had some results from that but they were all intent to so that is not important right now but it is not really important i can regenerate that the problem was that this gives you a new vector of topics for every page so you have got 2 vectors you have got the one that you just got from plsa or whatever model and you have got the one that you get from this how do you tell which one is better i had no way to evaluate this well how do you tell that this paid because you do not know what the topics are because they are learned automatically so wait i do not get it you have 2 representations one which comes from this link stuff and one this one is not based on plsa too it is yeah because initialization of it yeah you initialize it with actually this is kind of wrong this is not old but start this is how you this is how you start it off so this is what plsa gives you to start with and then you iterate this and it becomes something else so you should 1st train the whole psa stuff yeah then you initialize this link spreading thing and then you iterate that is it exactly yeah and it converge and so you have the both the initial plsa and the plsa plus the link thing exactly yeah so you have now got plsa gives you 2 probabilities right so it gives you a probability of a topic given a document which is a given d and a probability of a word given a topic so this is the same for both models yeah 0 this is a 1000 what you do is you are taking this and iterating over the probability of the topic given the document so i have made that clear i am not sure if i am explaining this no no okay right so you yeah you were saying that you would like to evaluate that now yeah so we were struggling how do you evaluate this because and if you look at the way that plsa and all those kind of models have been evaluated it is they never quantitatively do anything they always say well look at the clustering there is here we have got some documents that all appear to be about the same thing and here is we have got some other documents that all appear about the same thing and you can not really tell exactly how good that is you can just get an idea that it is or they use the perplexity or the likelihood they say 0 good this likelihood is smaller than the other yeah so that is the 1st thing that i think when i spoke to you about this you suggested use the likelihood and pedro came up were a really good example of why that could be bad so you have the you have the let me think of what this was it was to do with cars basically if you have a topic that is representing cars and you can tell that that is representing cars and you can tell that that is representing cars by looking at this distribution and it is the same for both now you want to tell which this page pedro is example of what it was is that it says on the page lotus elise okay and that i can not i just remember it was it really can you remember what it was 0 no really no i do not even remember but you said i would tell you that evaluation should be done based on likelihood or perplexity yeah i did not really consider doing it on perplexity actually to be on yeah it is basically the same but maybe you could have some some you could have some other labellee or the tasks like if you imagine you have some labeling on off of the web pages like like like for example you you look at at at at a set of pages and you have the the directory entries of those pages okay so so like yahoo are demos entries so you have categories and and entries so for example you have i do not know cars suv etc so what you would like would be that 2 pages in the same category should be closer than 2 pages being spread across different categories is that fair though because i think one of the things with these topic models is that you do not know you can not make them close you would not use the yahoo directory or the directory in the training step of the model okay okay this is something you have just for evaluation and then for example your task would be to say i have some page and i know where it is in the directory and i have another page i do not know where it is and i would like to know whether it is in the same category as the 1st one just by comparing there aspect it is making the assumption that this human categorization is related to the one that is discovered if they have the same yeah so basically what you would like to check is is some similarity in the in the aspect distribution tells me something about the the the fact that they could be on the same category in the the this and label categories okay is that really fair though can you say that if a set of documents if the clustering of these documents is closer to the effect of what you have got there is a clustering of documents as your devaluation yeah these labels the yahoo categories that is a clustering so you are by saying that this clustering is closer to that clustering does that necessarily mean that it is better no no no it does not have to be if you use if you use the latent representation so distribution over aspects you could use this to save 2 documents are in the same directory or not and this you should be able to evaluate then so what you would have basically would be to have a directory which give you labels for some part of the documents ok so this is a directory and this is some web page ok so the labeled one ok and you have some page which are unlabeled but you have links between those 2 so what you would do is that you would train both plsa and the other model over that then you would like to for example what could be very simple would be just to then to match each pages like that so let us say you have a page which is labeled and you say okay the label the label the the label page 2 should get should be the one which it is the most similar to or any kind of classification rules you can imagine and so you say okay what if i iterate from this page which have the similar so the criterion would be to compare those 2 distribution okay and then you would like to assign to this document to where you do not know the label the same one as this one and here that you use only for test so this is your grand truth then you can evaluate how wrong you are or how close you are from the real assignment to the right directory category in this way i think it is fair if you know only if you know only that and that and what you would like to discover is this cover is this this is a like a real task like like if this directory company would like to extend his directory to new pages without the hassle of having human labeling but in this case you have to use the label one yeah you have to use this one you only have the labels for some small part of the but how does it enter in that well i think your point was you train this without labeling 1st yeah so you would train it you are training data all those links are known the links between some label document and unlabeled document yeah the point is that you should change in a way to take the labels into account i think you can do it as a 2 step process 1st you train well maybe it could be better better if you take it into account but maybe you can do just a 2 step process 1st one you have this unsupervised task over both pl and pu and then you from this unsupervised task you can know which document of pl is closed from document of pu or the reverse and you can infer some label clearly what we have done is you can train after that an svm saying this this is category one or 2 on the label document and then you have an unlabel document and you just say this is directory one or 2 if you should improve if you say this feature extraction process is better then you are going to have a better cleaner separation between those 2 yeah the thing that worries me about this approach is that say the really really really simple example of this where your directory had 2 categories in it good and bad and you run plsa on it now that does not do good and bad it does i do not know english and french pages so they do not match but the clustering was still very good in one respect but you can extend that up even if you have 100 categories plsa can find 100 clusters that are a good clustering but are completely unrelated to the directory but i mean if you have to judge bad and good it is respect to a given task i mean you can not say of course it is always good because it is maximizing the likelihood know is that which of them has the better representation of a given page so given a random page and these 2 models and the words that these 2 models give which is more likely to yeah but yeah you can you you say given a random page of p u which of those 2 models is more likely to say for example that the page of pl which is the most similar so you so from this so you take any distribution comparison thing so so offman in his paper was doing just cosine similarity but you can do kl anything so so you just compare those 2 distributions of d one and d 2 okay and so with that you can you can determine for example for a document of the set 2 to the most similar so you you you you select the one so in this set which is the most similar to the 2 according to the distribution from one or the other model and then and then for this document you automatically assign to it the label of the document one and you check with this which is your your your ground truth if you are right or not this is what is very related to what is florenton did with the keyword image annotation so you would have a set of keywords the set of image labeled with keywords and he has a set of unlabeled image with some ground truths and you would check whether the label he assigned so in this case annotation word are correct and i think it is fair it is fair setup okay so maybe this classification rule is not the best there there might be some work to do but the general idea i think it is fair evaluation it is a task okay i am just concerned that that is it is answering a question that evaluates in some way but maybe not in the way that well maybe not in the same way i was thinking but maybe it is a better way okay so what i mean by that is that so i am trying to process all this yeah it is the same thing it does tell you whether this which clustering is better for this task but it does not tell you which clustering better represents the pages necessarily no but you could say that it possibly does it probably does i do not think there is any way to say generally this is better or worse worse there is not a perfect clustering what is the how do you judge it has to be related to with respect to this this clustering makes sense what is good with the directory is that you have a good coverage of various topics humans can label but then if you want to detect different languages this is another task i think the task of this latent model is to discover some semantic like some topic proximity between documents and and here this directory is doing the same thing like associating some topics with some documents manually so comparing them is good setup and you are not exactly compared you are comparing them i compare those 2 like the plsa and the one over this same task of assigning good directory i do not know if it will provide good result or not but it is clearly a benchmark for those 2 goals you can even keep the bag of words and do the cosine comparison or on the just showing just showing how just 1st how better this is with respect to bag of words and yeah hopefully then how you like regularized p of z given d is yeah you can have okay just it is basic yeah basic cosine over the p of t given d so road is empirical distribution and the 2 hidden aspects models i think you still not comply no i am coming around to it we have to repeat over and over this is like advertisement i think i get it i think it probably is fair i am just i have got uncomfortable with it but i think that will go if i think about it enough but yeah the it just seems a lot of suspect but i think that is just from the whole point that you can not tell which clustering is better what i thought before was could you get people to evaluate this by hand basically could you give people a set of pages say you did it with 5 topics for example no you give people 5 sheets you give a user 5 sheets of paper each with words in varying sizes representing the probability of that topic for example give the user these pieces of paper to get an idea what these topics are that is this p w given z and then you give them some pages and say which one would you which piece of paper would you put that with and then you get an intuitive idea like how would a person cluster pages given what these topics supposedly are because you have got something that is fixed at the p of w given z with respect to both models so i was hoping you could use that the hidden aspect like the z are unsupervisely it is more like if you were giving the people you said to them okay divide me this bunch of documents into 5 topics and you do not give them the topics a priori so the 2 persons could come up with very different one is very interesting in fact i have seen a paper in icml where the guy was trying to learn which were the aspects of clustering so i give several users several persons a bunch of marbles and they begin to say ok and they have to divide it in 5 group and in the end it is always different the clustering each people has because it is completely subjective yeah ok but there is some characteristic that are transparency color which are features that made the clustering that can be extracted even if the clustering is always different ok so i think i am saying that it can be really infinite see the thing about web pages i thought as well and that getting humans to evaluate it in some way is that you have most of the web pages that are well a lot of the web pages that are important in some way are homepages so google com does not actually say the word search on it i think it maybe says go but you only know it is about search through the thing but someone looking at it can tell that it is a search engine because of the way it is laid out because of the pictures on it and because of all of that there is a lot of information that is completely lost in this bag of words representation and i was hoping that in some way you could use that to evaluate this i have no idea how but say another example apart from this search i do not get exactly what you mean do not use the google page because it is too rude even someone would provide very different labels like someone would say it is a page in english someone would say it is a it is a it is a it is a it is a it is a web page which has very few information or anything so i think there is no hard clustering and what is good with those models is that they they can they are intended to provide you some some kind of way to compare to documents rather than to give you predefined sets of documents it is not hard clustering i do not think one aspect or one latin topic has any sense by his own it is when you have the whole set and the whole distribution that you can tell something if you just take one of them alone so you take all the p of z given d and all the p of w given z you can not do anything with that it is the whole set i would say and this is how it should be evaluated meaning that like for example the mikhail example about marble is that the point what you can learn from all these sets made by people is that is that those 2 marble would be considered as more likely to be in the same set because they share some common properties which are not seen as the same for everyone but they share some common properties that people will notice and if you look at the 2 the old possible pairs of marble you would have much more agreement between people than if you look at the predefined sets people would just cut at some precise point you know what is okay i think you know but yeah this is effectively what we have here though and that directory is one clustering of the marbles and you are hoping that so what you are saying is that i have 2 clustering of marbles from my 2 models and i have a 10 test clustering of marbles is saying that one of these is more close to the test to the one that we have been given is that it saying that what does that actually say it does not really say anything else no no no you do not have 2 clustering of marbles what you have is you have is you have one clustering of marble which is this one and then you have lots of people telling you i consider those marbles as very to have very salient feature that they should share in common i would like them to be in the same cluster and those 2 and one of them is labeled and the other is not labeled so you look at that you say 0 someone told this one was very similar and this one is said to be blue for example i do not know if it is the label so i put it in the blue bag but you never use the blue bag you do not ask the people to look at the blue bag you just look at the people you just ask the people to compare those 2 marbles and then it happens that some of them have been clustered people being your algorithm that is the point yeah yeah yeah this is i reverse most wrong yeah it just it is just this is really is right what the algorithms are still coming up with is in this way looking at a clustering of marbles yeah each of the algorithms comes up with a different it comes up with not with a clustering but with some way to compare to marbles okay so my algorithm would say okay right that makes sense would say i consider those 21s are completely dissimilar and the other one say they are very similar and if you find that in your un label clustering those 2 balls barbells happen to be in the same cluster you said the other algorithm might be wrong but you average that over lots of documents okay because the latent aspects really does not have too much the final directory it can be anything but it is just a way of saying yeah those documents are similar in some sense so we are not going to it is just sometimes dangerous to say this latent aspect means this okay i am convinced i am convinced all right so it is good good good right all right well thanks i am trying to think of anything i have not said about the stuff i did from looking at the results i got it looked quite good just like hand reading them one of the things i did was i started a web crawl at the edap homepage it did not very many pages only like 2000 or something like that but enough to get the pages that were close to edap to have a lot of links between them so that there was enough to be able to run this link clustering algorithm so what you find is that the pages around edaap there is a lot of english speaking pages and a lot of french speaking pages of course this plsa completely separated those out into categories so when i looked at the word likelihoods there were 2 topics one was very clearly english words the most likely word was there and one was very clearly french words most likely word was there i think you had on 2 topics no i had many topics the others looked like garbage could not tell what they might be but anyway that is often the case when you look at it you can tell very clearly what one or 2 topics are not the rest the others usually do not mean anything it makes sense only it is a global optimization the likelihood is optimized globally over the whole aspects etc so you never ask the model to do hard clustering of documents you just ask it to find a distribution over aspects such that a document is more likely according to the restriction that the number aspect is limited and so it just makes sense on looking at the whole set so i think all evaluation should take that to account and look at the whole aspect set like here you look at the whole aspect set because you look at the similarity of 2 distribution any kind of thing where you unfold the model and look at it manually looks weird like in those offman papers where show you columns with words or things anyway when i ran it on that what i found was looking through some of the sites i looked for the sites that the models differed on where one said it was probably an english page and the other said it was probably a french page and i found a good few examples where you had a site that looked well one that i remember offhand was it was a page and it was a big picture in links down the side but the links were all pictures so you could not the words were in french in the links but there were no words picked up by the when i did the crawl for the model and there was a small bit down here and in this there was meant to be a picture in there but it did not load or it was meant to be a picture in there but it was meant to be something else and it said in english it said 404 this page cannot be found so all the words in this page were english but it was very very clearly a french page so the 1st model said it was an english page because all the words were in english the 2nd model said it very definitely a french page because all of these pointed to french sites and it was only pointed to it was something fr as well okay so it was a good point for you yeah for your model so i found a couple of examples that were kind of like that but the problem that i found when i ran it the thing that was a bad point was that if you have say i was trying to say if you could evaluate this with search given some keywords which model would more likely find you a good document that sort of approach what i found was that say you had that that is not too relevant necessarily but if you have say a topic distribution for one page and it very very highly scores on the plsa for one of the topics and that is right this model because it is averaging out over all the pages nearby all of the topic distributions become more flat it becomes more uncertain about everything so i do not know if that is necessarily a good thing or not in some cases it is going to be a bad thing because every page is linked by the way the algorithm works every page is linked eventually to everything else so every page has information incorporated in it the distribution learned for every page has information incorporated in it from every other pages of distribution as well so it averages out it flattens all the distributions depending on what this alpha is but also it depends on that could be a good effect because if you can be if you are highly reliable about one page but all the page looking to it are actually about very different topics so you could say maybe i was wrong from deciding only due to the text that this page was about this aspect distribution because all the other pages are voting in different direction well i think that the thing is while there is a balance yeah there is a balance in that for example take a page like slash dot it is like a news site for for geeks and you have nerds yeah news for nerds stuff that matter all of the like 100s of 1000s of pages point to that but they are about completely different things so you in effect are losing information about what that page is about because there is so many things pointing to it in the all average out this alpha can it be dependent of the document yeah so and how many texts is in need that is my 1st idea was how to set this i remember what the lotus thing was now so i thought you want to probably set this alpha per page there is probably some heuristics for example this page there is no text in it we will set it very low and use the information for the links problem is this is what pedro immediately said what if you have a page that looks like this it has got a big picture of a car pedro always speaks about cars yeah cars anyway and it says lotus elise so you know that it is definitely about car but it is only got 2 words only get 2 words so you can not just do it by words because those 2 words completely specify it into something or other you suggested doing likelihood but at that time and i can not remember what my problem was with that it might not have been a good thing but you have got 2 likelihoods for it as well but i think for any kind of way you have to select those alpha you have to look at the effect on an evaluation task and then maybe you can find some patterns that are then once you have the task you could look at which pages are completely missed by one model which others are successful with one model and you can maybe infer some rules afterwards but if you have no task you can blindly select good alphas like if you look at your task and you see that task and you see that i do not know the long pages for example have good plsm model without using links so then you can infer some heuristics like okay the alpha should be proportional to the lengths or anything like that or to the number of images in the pages or anything but you should have some some tasks where you can individually look at the performance on every page yeah and and and as a 2nd step could infer some kind of rule about how to select alphas with respect to the task yeah okay maybe it is very simple rules but select them blindly it might be hard yeah so i think that there is 2 ways you can do it with some heuristics and you can learn it from the task results or combine the 2 so that is kind of straightforward i think probably heuristic will be easier to compute yeah about doing it through cross validation or something like that with the task is the problem with doing a page rank style algorithm is that it gets better it converges better and it gives you better results the closer a network is to being a small world network so small world network you know basically some far links but generally very clustered locally now the thing about doing a web crawl is the bigger your web crawl the closer it will be to the overall structure of the internet which is a small world network the smaller your web crawl the less more world network like it is going to be so the more pages yet the better your results and people running this style of algorithm the page rank style stuff generally run it on about 2000000 or 20000000 pages or whatever the stanford webbase data set that i was using for this for a bit of this they have i can not remember offhand but they have maybe over a petabyte worth of web pages in this the bunch of servers but still like the work i did on hyperlinks like i used the wikipedia yeah maybe wikipedia is a good thing to do it because it has fairly high level of inside links what i was concerned about wikipedia though was that it is while it has a high level of inside links it has a lot of directory style links as well that might mess up actually being a yeah well you can categorize them because they are there is a structure on the the web page so you can select only the links which are within the article or within the categories okay that is good it is well structured it is so easy to manipulate plus i guess it is actually straightforward to say to start with is this a small world network and then run on that you can even isolate the directory from the categories of wikipedia you were running that on what was it like half a 1000000 pages something like that well yeah and that was for half of it i divided in 3 equal size parts which of 150 documents okay that was 1000 documents and well but did you manage to divide the link properly well what i did with the link well but this is not exactly the same setup you would have so in this setup the whole link structure could be known and the only thing you would do not know would be the assignation to a category in my setup it was different because i could not use the link use the links between different sets but this is not i do not want to miss you up with that it is not the same kind of experiments but you could just use the whole corpus like this and and for some of them you assume you do not know the the category assignment and then you can run the set of experiments over wikipedia i mean i think offhand thinking about this model it would have to work better because 2 things in the same category are going to be linked yeah so but this is the yeah yeah this is the hypothesis between this in this plsa with link stuff okay yeah i think that it is going to be more true in some networks and in some tasks than others but you can think also at other tasks like that like for example you can take proceedings and assume the categories are the keywords of the documents and then if you have the archive of a journal you can use the links between the site limitations links so if you want to have a bigger stuff so wikipedia would be kind of very small with lots of in links and then if you take a journal you will have less links and more documents and then you would be likely to say that plsa might be better over that because you do not have enough links so you can find examples which are not the web i think it is good to work in closed set wikipedia is kind of close set even if there is lots of outside links but there is a lot of inside links you have to think at database where there is a lot of in links wikipedia might be one citation for example if you take the nips archive which is available nips author always sits on nips author which are often themselves etc well it is recorded i will not say it so but but but but yeah i think this kind of setup maybe web pages it is hard to find sets of web pages which have good links stuff if you start from one page or something okay yeah maybe have a look at the wikipedia stuff i am stuck doing something else for the next month or 2 anyways okay there is another thing this was new i have not thought about this too much but it is on the same lines have you guys seen this correlated topic models work which ones topic models david bly the guy that came up with lda he has got a new model that is published to neps this year but he is co author put it on their website a month or 2 ago so you know lda you all know lda you all know lda yeah okay so lda you basically draw a dreasla which says like what type of document this is effectively and then you draw a topic given the derile distribution and then you draw a word given that all right well that is for for every word and that is once for the document and that is you have got a parameter coming at this globally so that is roughly what the model is just a little bit but anyway so what they did was they rework this because one of the problems with lda and i think plsa has this problem as well is that the more topics you have the less expressive it can be because the topics it clusters it pushes a document if it is pulling a document into one topic it pushes another topic away from that does not that make sense so the topics are necessarily trying to be different from each other so the more topics you have the more they are sort of squeezing into the space of what things can be about and things are kind of pushed out so once you go over they tried it on one corpus and once you go over about 100 maybe 200 topics the new you add more topics and it does not get better in any way the model does not the new topics are effect they are going to just be garbage all right so what they did was they stopped using adirishly and they are starting to use a gaussian there so to spare the details basically they have got a variable here i think they call it eta and from this generate a distribution of topics in some way but it is generated from a gaussian space from a space so it is given a mean and covariance but it is disquid no yeah so what it does this is the bit that took me forever to understand actually same model but this is given a gaussian space it maps that down into the simplex of what a topic can be about the simplex basically if you have got say you have got 2 topics say you have got 3 topics that may you get 3 topics that makes more easier to draw for this is theta one theta 2 theta 3 these are the probabilities that a document is about theta one is the probability that this document is about topic one theta 2 so these guys all have to add up to one so that basically means that what a document is about lies on this space between them so this is the simplex so basically a distribution generated by this guy is basically a point on this simplex so what they do is they have got a distribution that maps a gaussian in some space into a distribution on this thing so it maps it into a gaussian in that the cool thing about that is that if you have got 2 gaussian you can tell how close they are so you can now tell how close to topic distributions are right so you can have as many as you like and you can tell that they are close to each other so there is loads of ideas that i had about this in the in that if you can tell that 2 topic distributions are close to each other so for example if you split a document in the 2 parts and said that the 2 parts have to be close in the distribution you can see how the document changes for you say it is stuff based on that with this idea what if rather than working with this p of z given d thing that you get from plsa but what if you work from these gaussian that you get for every document and you said that in some function that you get from some function that said the gaussian representing this should be similar to the ones linking to it and i have not thought about any of the maths for this but it should be straightforward i guess to formulate it basically in terms of the mean should be similar in the covariance maybe it should be similar as well so it might actually be a better model again i do not know i will send you guys the link for that correlated topic model stuff if you want but i do not know if it is doing i do not know if it is too interesting to you okay yeah thanks for hearing me out in the idea especially okay well maybe i can do more with us well in 3 months yeah in 3 months well there are lots of good ideas that are recorded and someone will 0 yeah it is too late 3 months yeah because the data will be released very soon bastien you can not release this data for 4 months that is cool thanks guys i know if it is useful to you yes that makes me at some point i am going to have to go to david and convince him to let me spend a month to working on this again you do not have to convince you have to say i am going to work on this one have you met david