kay okay gosh kay does anyone want to see steve is feedback from the specification is there much more in it than he d i i dry read it the last time right is there much more in it than he said yesterday not really just what he is talking about like duplication of effort and like duplication of effort and stuff and yeah he was saying that we should maybe think about having a prototype for week 6 which is next week next week yeah so we should probably prioritize our packages yeah now i would say if for the prototype if we just like wherever possible p chunk in the stuff that we have pre annotated and stuff and for the stuff that we do not have pre annotated write like a stupid baseline then we should probably be able to basically that means we focus on on the interface 1st sort of so that we we take the the ready made parts and just see how we get them work together in the interface the way we want and and then we have a working prototype and then we can go back and replace pieces either by our own components or by more sophisticated compo po components of our own so it is probably feasible yeah the thing is i am away this weekend yeah so that is for me yeah i mean if we just want to have some data for the user face could even be random data 0 yeah yeah yeah no but also i might like the the similarity thing like my just my matrix itself for my stuff i c i i think i can do that fairly quickly because i have the algorithms yeah i am yeah i think today is meeting is really the one where we where we sort of settle down the data structure and as soon as we have that yeah probably like after today is meeting we then actually need to well go back 1st of all and look at nite x m l to see in how far that that which we want is compatible with that which nite x m l offers us and then just sort of everyone make sure everyone understand the interface yeah so i think if today we decide on what data we want to have now and and later maybe even today we go and look at nite x m l or some of us look at nite x m l in a bit more detail just trying to make some sense of that code and see how does the representation work in their system and then sort of with that knowledge we should be able to then say okay that type of nite x m l data we want to load into it and this is how everyone can access it and then we should be able to go from there has has anyone actually looked at the java code for the huh no no i have looked looked at the documentation and n like seen enough to make me think that we want to use the nite x m l framework because they have a good a event model that synchronizes sort of the data and and every display element so that takes a lot of work away from us sort of that would be a reason for staying within their framework and using their general classes but beyond that i have not looked at it at all which is something we should really do yeah i think so who actually like for this whole discussion i mean who of us is doing stuff that is happening on line and who of us is doing stuff that is happening off line like my data is coming c the basic word importance is off line as well yeah the combined measure might not be if we want to wait what the user has typed in into the search okay okay mine is going to be mostly using the off line but the actual stuff it is doing will be on line but it will not be very processor intensive or memory intensive i do not think kay so basically apart from the display module the i the display itself we do not have an extremely high degree of interaction between sort of our modules that create the stuff and and the interface so the interface is mainly while it is running just working on data that is just loaded from a file i guess there is not yeah i i do not know about the search functionality that might be online depends how it is going to work yeah i know th yeah the search is i guess the search is sort of a strange beast anyway because for the search we are leaving the nite x m l framework yeah but that is still sort of that is good that means that at least like we do not have the type of situation where somebody has to do like a 1000000000 calculations on on data on line cause that would make it a lot more like that would mean that our interface for the data would have to be a lot more careful about how it performs and and everything and nobody is modifying that data at at on line time at all it seems nobody is making any changes to the actual data on line do not think so so that is actually making it a lot easier that basically means our browser really is a viewer mostly yeah which is not doing much with the data except for sort of selecting a piece piece of it and and displaying it are we still going to go for dumping it into a database are we still going to dump it into a database well some parts relevant for the search yes i would say so cause if we are i reckon we should all read our classes out of the database it will be so much easier well if we are going to dump the part of it into a database anyway we might as well dump all the fields we want into the database calculate everything from there then we do not even have to worry that much about the underlying x m l representation we can just query it yeah but nobody of us is doing much of searching from the data in the on line stage and for all together like the display itself i think we are easier if we if it is sitting on the x m l than if it is sitting on the s q l stuff because if it is sitting on the x m l we have the the nite x m l framework with all its functionality for synchronizing through all the different levels whenever there is a change whenever something is moving forward and stuff and we can just more or less look at their code like how their player moves forward and how that moving forward is represented in different windows and stuff so i think in the actual browser itself i do not want to sit on the s q l if we can sit on the x m l because sitting on the x m l we have all we have so much help and for y for like the p the calculations that we are doing apart from the search it seems that everyone needs some special representations anyway well if we are going to do that we should try and store everything in in an x m l format as well you mean our results yeah yeah in in the nite x m l x m l format so with their time stamps and stuff so that it is easy to to tie together saint things yes yeah what i am like what we have to think about is if we go with this multi level idea like this idea that sort of if you start with a whole meeting series as one entity as one thing that you display as one whole sort of that then the individual chunks of the individual meetings whereas and then you can click on a meeting and then sort of the meeting is the whole thing and the chunks are the individual segments that means sort of we have multiple levels of of representation which we probably if we if we do it this way like we f we have to discuss that if we do it this way then we should probably find some abstraction model so that the interface in the sense like deals with it as if it is same so that the interface does not really have to worry whether it is a meeting in the whole meeting series or a segment within a meeting you know what i mean yes and that is probably stuff that we have to sort of like process twice then like for example that like the summary of a meeting within the whole meeting corpus or meeting series y is meeting series a good word for that i do not really know what how to call it you know what i mean like not not the whole corpus but every meeting that has to do with one topic yeah that makes sense so in in the meeting se series so that a summary for a meeting within the meeting series are sort of compiled off line by a summary module and that is separate from a summary of a segment within a meeting cause i do not think we can i am not so sure well we do not even need to do that cause if we got our information density calculated off line so all we do is treat the whole lot as one massive document i mean they will it is not going to be so big that we can not load in a information density for every utterance and we can just summarize based on that i i thought we would just have like one big summary with all the different importance levels displayed on it and depending on what our zoom level is we just display a part of it so are we doing that at all levels are we and we would have one very big thing off line and from that we would just select what we are displaying and just have different like fine grainedness levels sort of yes so for example you would give a high value to those sequences you want to display in the meeting series summary kay so the only thing that yeah so the only thing that would happen basically if i double click let us say from the whole meeting series on a single meeting and you just cut off is that the zoom level changes like the th the start and the end position changes and the zoom level changes that was what i sh i thought yeah i think you can do it on line i i thought we could not do that i thought like i was under the impression that we could not do that because we could not load the data for all that but i do not know i mean that i do not think there is really much point in doing like that when it is just going to feed off in the end the information density measure basically and that is all calculated off line so what you are really doing is sorting a list is the p computationally hard part of it so i am s not sure if i got it i was well like the ideas we are calculating are information density all off line 1st for every utterance in the whole corpus right so what you do is you say if you are looking at a series of meetings you just say well our whole document comprises of all these stuck together and then all you have to do is sort them by j information density like maybe weighted with the search terms and then extract them okay so i do not think it is too slow to do on line to be honest okay i wa i was just worried about the total memory complexity of it but i i completely admit i mean i just sort of like th took that from some thing that jonathan once said about not loading everything but maybe i was just wrong about it how many utterances w is that but i think the m difference might be that we want just want to have the words and that is not so much what he meant with not possibly loading everything was that you m load all the annotation stuff yeah and i w yeah yeah yeah all the sound files all yeah yeah so what we have is we would have a word like we would have words with some priority levels and they would basically be because even the selection would would the summaries automatically feed from just how prioritized an individual word or how indiv prioritized an individual utterance is or i are the summaries sort of refined from it and made by a machine to make sentences and stuff or are they just sort of taking out the words with the highest priority and then the words of the 2nd highest priority well on the utterance level i was thinking so the utterances with the highest like mean information density and the u okay are we doing it on th the whole thing on the utterance level in or are we doing it on word level like the information density calculation well the trouble with doing it on the word level is if you want the audio to synch up you have got no way of getting in and extracting just that word i mean it is impossible we i think we have start and end times for words actually but it is yeah for every single word yeah but it m it might s but it might sound crazy in the player 0 okay we should really maybe we can do that together at some point today that we check out how the player works yeah i do not think that will do it but there is maybe some merit in altogether doing it on an utterance level in the end we will have to buffer it i r i i am getting quite lost at the moment because w what is our difference between the se the importance measure and the skimming so i mean do we do both or is it the same thing well the skimming is going to use the importance yeah okay but like at 1st it is just going to be i d f so but when when we talk about summaries you talk about this abo about skimming and not about well mostly skimming yeah well but also about the displays i mean the displays in the in the text body in the in the latest draft that we had sort of we came up with the idea that it is not displaying utterance for utterance but it is also displaying a summarized version in you know like below the below the graph the part yeah maybe yeah r yeah right is not that the skimming is not that the skimming 0 yeah f it is just like there there is like audio skimming and there is displayed skimming yeah but it use the same data yeah yeah ma maybe there is some merit of going altogether for utterance level and not even bother to calculate i mean if you have to do it internally then you can do it but maybe like not even store the importance levels for individual words and just sort of rank utterances as a whole well the nice thing about that is it will automatically be in sentences well more or less yeah so it will make more sense and if you get just extract words cause it it might be better skimming and less memory required at the same time and i mean if you if you know how to do it for individual words then you can just in the worst case if you can not find anything else just sort of make the mean of the words over the utterance you know what i mean i am not quite so what it did you want to do it i you just wanted to assign w it is it is well what is the smallest chunk at the moment you are thinking of of assigning an importance measure to is it a word or is it an utterance i thought about words so we are thinking of like maybe just storing it on a per utterance level because it is it is less stuff to store probably for dave in the in the audio playing and for in the display it is probably better if you have whole utterances than i do not know like what it is like if you just take single words out of utterances yeah that probably does not make any sense at all whereas if you just show important utterances but the utterance as a whole it makes more sense a yeah so it does not actually make a difference for your algorithm cause it just means that if you are working on a word level then we just mean it over the utterance and yeah i think we also thought about combining that measure with the measures i get from s hot spots and so on okay they are on so that would also be on utterance level i think 0 so that is good anyway then yeah i think i see it because that makes it a lot easier than to t put it on utterance level but it will need to be calculated at word level though because otherwise there will not be enough occurrences of the terms to make any meaningful sense 0 yeah no but i mean like how how jasmine does it internally i do not know but it is probably yeah you probably have to work on word levels for importance yeah but there should be ways of easily going from a word level to an utterance level yeah i reckon you can just mean it over the sentence okay yeah but how about those words which do not carry any meaning at all the and uhs and something like that yeah prob because if we if we average average over over a whole utterance all the words i think we should filter them and there are quite unimportant words in there but quite important words as well i think we should just disregard the the maybe we should have like a cut off so it a w word only gets a value if it is above a certain threshold so anything that has less than say nought 5 importance gets assigned to 0 okay alright well we do a pre filtering of sort of the whole thing yeah that is the other th sort of like but that like the problem with that is it is easy to do in the text level but that would mean it would still play the in your audio unless we sort of also store what pieces we cut out for the audio yeah yeah i think before we can like answer that specific question how we c deal with that it is probably good for us to look at what the audio player is capable of doing i think we will have to buffer the audio but i do not think it will be very hard yes i think it would be like an hour or 2 is work so what do you mean by buffering like you think directly feeding like just build an another f wave file essentially yeah you just concatenate them together but yeah but not but not stored on the hard disk and then loaded in but loaded in directly from memory yeah i mean i bet there would be packages in memory yeah so just like unp there is bound to be like a media wave object or something like that and just build one in memory but it is probably a stream if it exists in java it would be probably some binary stream going in of some type i do not know i have no idea okay yeah but it must have like classes for dealing with files and if it has classes for concatenating files you can do it in memory okay okay so i mean so that means that there is probably even if you go on an per utterance level so there is still some merit on within utterances cutting out stuff which clearly is not relevant at all and that maybe also for the audio we would have to do so let us say we play the whole au phrase but then in addition to that we have some information that says minus that part of something that is okay that we can do well what i think i might try and build is basically a class that you just feed it a linked list of different wave forms and it will just string them all together with maybe i do not know 10th of a 2nd silence in between each one or something like that yeah maybe even i mean that is sort of that depends on how how advanced we get if maybe if we realize that there is massive differences in in gain or in something you can probably just make some simple simple normalization but that really depends on how much time we have and and how much is necessary normalize it yeah yeah if like i d i do not know anything about audio and i have never seen the player so if you find that the player accepts some n input from memory and if it is easy to do then i guess that is that is fairly doable so but that means in the general structure we are actually quite lucky so we we have we load into memory for the whole series of meetings just the utterances and rankings for the utterances and some information probably that says well the i guess that goes with the utterance who is speaking 0 yeah yeah we will need that yes sure because then we can also do the display about who is speaking we also really want to be able to search by who is speaking as well yeah but i am i am still confused cause i thought like that is just what jonathan said we do c that we can not do like load a massive document of that size it does not matter cause all the calculation is done off line on the other hand the other hand i mean it should not be like should be like 50 mega byte in ram or something it should not be massive should it actually 5000 megabyte is quite big in ram just thinking what is the simp so we do get an error message with the project if we load everything into the project with all the data they load so we know that does not work so our hope is essentially that we load less into it yes yeah what is this lazy loading thing somebody explain lazy loading to me it just means it loads on demand it only loads when it needs a particular type of file like when it is being accessed ah okay so that is that only by type of file like if if if the same thing is in different files would it then maybe like you know if if utterances are split over 3 or 10 or w 100 different files is then a chance maybe that it does not try to load them all into memory at the same time but just yeah i think that is the idea it just loads the particular ones it needs so why does it fail then in the 1st place then it should not ever fail because then it should never but if you were doing a search over the whole corpus you would have to load them all yeah but yeah but it it it failed right when you load it right the nite x m l kit so that is interesting yes right oops it does so i define baseline and what it loads for example it loads all the utterances and so on but it does not load the discourse acts and for example not the and what is what else there not the summaries it only loads those on demand let us check that out i will p i will probably ask jonathan about it so alternatively if we realize we can not do the whole thing in one go we can probably just process some sort of meta data you know what i mean like sort of sort of for the whole series chunks representing the individual meetings or some like something that represents the whole series in in a v in a structure very similar to the structure in which we represent individual meetings but with data sort of always combined from the whole series so instead of having an single utterance that we display it would probably be like that would be representing a whole topic a segment in a meeting and sort of so that wi using the same data saint y you mean that you basically split up th the big thing into different summaries for example that you have a very top level summary and a separate file for for each level well in a sense i am i am thinking of in a sense of like creating a virtual a virtual meeting out of the whole meeting series sort of that is easy you just like create a new x m l document in memory yeah sort of like off line create a virtual meeting which which basically treats the meeting series as if it was a meeting and treats the individual meetings within the series as if they were segments and treats the individual segments within meetings as if they were utterances you know so we just sort of we shift it one level up and in that way we could probably use the same algorithm and just like make vir like one or 2 ifs that say okay if you are on a whole document a whole series level and that was a double click then do not just go into that segment but load a new file or something like it but in general use the same algorithm that would be an alternative if we can not actually load the whole thing and cause also like even if we maybe this whole like maybe i am worrying too much about the whole series in one thing display because actually i mean probably users would not view that one too often i do not think it is really that much of a problem because if it is too big what we can do is just well all the off line stuff does not really matter and all we can do is just process a bit at a time like for summarisation say we wanted a 100 utterances in the summary just look at the meeting take the top 100 utterances in each other meeting if it scores higher than the ones already in the summary so far just replace them and then you only have to process one meeting at a time yeah but i am i am still worried like for example for the display if you actually if you want a display like for the whole series the information density levels based on and and the f and the only granularity you have is individual utterances that means you have to through every single utterance in a series of 70 hours of meetings okay so maybe we should build a b store a mean measure for the segments and meetings as well yeah yeah and if you make that structurally very similar to the the le like one level down like the way how we store individual utterances and stuff then maybe we can more or less use the same code and just make a few ifs and stuff yeah so so but still so in in general we are having we are having utterances and they have a score and that is as much as we really need and of cou and they also have a time a time information of course and speaker and a and a s and a speaker information yeah speaker and topic segmenting we will need as well yeah so an information which topic they are in yeah yeah and and probably separate to that an information about the different topics like that yeah so so the skimming can work on that because the skimming just sort of sorts the utterances and puts as many in as it needs well yeah and then it will f preserve the order when it is displayed the yeah yeah yeah it will it will play them in some order in which they were set because otherwise it is going to be more entertaining yeah but that that is enough data for the skimming and the the searching so what the searching does is the searching leaves the whole framework yeah i think so goes to the s q l database and gets like basically in the end gets just a time marker for where that is like that utterance that we are concerned with and then we have to find i am sure there is some way in in nite x m l to just say set position to that time mark and then it shifts the whole frame and it alerts every single element of the display and the display updates yeah we do not want it in to develop a little tree display as well for multiple results yeah yeah yeah but that would be quite easy to do that we can ju yeah but so so if if somethi so yeah so if in that tree display somebody clicks on something you just need to find the time stamp yeah and then you sort of feed the time stamp to and the nite x m l central manager and that central manager alerts everything that is there like alerts the skim like the the audio display yeah alerts the text display alerts the visual display and says we have a new time frame and then they all sort of do their update routines with respect to the current level of zoom so how much do they display and starting position at where the or maybe the mid position of it i do not know like w if start where the thing was found or if that thing wa was found it is in the middle of the part that we display that i do not know but that we can decide about but a general sort of it is the same thing if like whether you play and it moves forward or whether you jump to a position through search it is essentially for all the window handling it is the same event yeah it is only that the event gets triggered by the search routine which sort of push that into nite x m l and says please go there now so we should basically make our own x m l document in memory that everyone is module changes that rather than the underlying data and then have that x m l nite x m l document tied to the interface why do we have to do it in memory but that stuff is so well you can make it in a file if you want i mean like the information is coming from off line so we probably we do not even have to change the utterance document right because the whole way like the whole beauty of the nite x m l is that it ties together lots of different files so we can just create an additional x m l file which for every utterance like the utterances have i d s i presume some references so we just we tie p just a very short x m l file yeah yes which it is the only information it has that has whatever a number for for the weight for the information density and we just tie that to the existing utterances and tie them to the existing speaker changes but there is no i d for an utterance i think it is just for individual words so how do we do that then n no no it is f for we for utterances as well no you are right yeah i think it is just for one it is for word yeah i think i think those segments for each utterance are split up so we have to yeah think so well otherwise we probably have to go over it and like add some integer that we just increment from top to bottom sort of to every utterance as an as an i d some type or un or try to understand how nite x m l i d s work and maybe there is some special route we have to follow when we use these i d s yeah i am pretty sure it is already there it is alm pretty sure that is already there the the utterances are numbered yeah the the girl said the utterances themselves are not numbered at the moment i am not quite sure i have only seen that the the individual words have got an i d okay okay okay yeah so i guess that would be solvable if not yeah you always could have a look at the time stamps and then take the ones that belong together to form an utterance no i i think we would just take the segments that are already that were yeah there is this segments file sorry yeah if they are already there is it is easy but it would be possible okay you know the x m l segments okay yeah okay is that a board marker pen actually 0 that i do not know 0 yeah i think so that is just so like to make a list of all this stuff or we probably can somebody can do it on paper all these fancy pens so what so the stuff we have we have utterances and speakers and weights for utterances so for for every utterance sort of like the utterance has a speaker and a weight which is coming from outside or we just tie it to it and there is segments which they are utterances are not they the segments are utterances are not they ye that is the impression i get yeah 0 so sorry yeah yeah that is topic s topic segments i meant like they are they are a super unit alright okay so so the utterances are tied to topic segments 0 and if the time stamps are on a word level then we b somehow have to extract time stamps for utterances where they start there there are time stamps for well well that is easy segments and for th segments is for example when when you look at the data what is displayed in one line w what segments now okay what when when you look at it in the is the is that the same as utterances that is that the same as utterances that ye well it is close enough is not it i think so is not yeah it may not be exact every time but it is a so sort of size we are looking for for ex i i compared it with what i did for the pause duration extraction and basically it is words that are uttered in a sequence without pauses but sometimes however there are short pauses in it and they are indicated by square brackets pause or something in the data what so that is 0 right but that is 101 segment or is that 2 segments then someti but the annotators decided what was one segment and what was not yeah okay okay so but but generally utterances is that which we just called sorry segments is that which we just called utterances now like it is it is the sa it is sort of like one person is contribution at a time sort of thingy dingy i think so yeah yeah okay okay so yeah so we have those and and then we have some f field somewhere else which has topics topics yeah yeah and and a topic is basically they are just on the i d probably with a start time or something and and the utterances referenced to those topics i guess yeah i think that is the right one so the topics do not contain any redundant thing of like showing the whole topic again but they just sort of say a number and where they start and where they finish and the utterances then say which topic they belong to yeah but i think for some annotations an uttera ca utterance can have several types for example for the dialog acts and so on yeah no but i was thinking of the topic segmentation now and and f for that there would only be one okay yeah that should be for right because it is sort of like it is just a time window yeah should be yeah yeah so if this lazy loading works then this should definitely fit into i mean not memory then because it would not all be in memory at the same time so if we just have those sort of that information like a long list of all the utterances slash segments and like short or smaller lists which give weight to them and even though probably if there is a lot of over head in having 2 different files we can probably merge the weights into it off line you know what i mean like if if there is a lot of bureaucracy involved with having 2 different trees and whether one ties to the other because the one has the weight for the other yeah then it is probably quicker to just but why do not we just write it as a new x m l file can nite handle just loading arbitrary new like attributes and stuff i mean i would have thought they would make it able to yeah i thought that was the whole beauty that like you can just make a new x m l file and sort of tie that to the other and and it tre yeah so why do we need to have 2 x m l trees in memory at once 0 yeah so no i did not i did not mean tree no no i meant just like handling 2 different files internally sort of c i was just thinking you know like if if the overhead for having the same amount of data coming from 2 d files instead of from one file is massive then it would probably be for us easy to just like off line put the the weight into into the file that has the segments yeah segments slash utterances already but that we can figure out i mean if it is going horrendously wrong the other thing is that would mean we would be using their parser as well which means we would not have to parse anything which be quite nice yeah cause their parser is probably much faster than anything we have come up with anyway yeah yeah no we would we would be completely using like the whole infrastructure and basically just i mean the main difference really between our project and theirs really is that we load a different part of the data but otherwise we are doing it the same way that they are doing it so we just we are sort of running different types of queries on it we in a sense we i think we are running queries it is not just about what we load and what we do not load but we are l running queries in the sense that we dynamically select by by weights do not we that we have to check how fast that is like to say give us all the ones that whether that works with their query language whether that is too many results and whether we shou you know if cause if it i let us say i mean if if their query language is strange and if it would return b 10000000 results and it can not handle it then we can just write our individual components in the way that they know which what the threshold is so they still get all the data and just they internally say 0 no this is less than 3 and i am not going to display it or something yeah i mean we can process it in chunks if it gets too big basically we can just process it all in chunks if it gets too big to load it into memory yeah no i am just thinking for this whole thing of like a different level sort of cutting out different different pieces whether we do that through a query where we say give us everything that is ab above this and this weight or whether we skip the same infrastructure but every individual module like the player and the display say like they still get sort of all the different utterances all the different pieces but they say 0 this piece i leave out because it is below the current threshold level i think we probably want to store sorry i think we probably want to store a hierarchical information density as well so like an informan mation density score for each meeting and each topic segment cause otherwise we would be recalculating the same thing over and over and over again yeah that would be much more efficient to do that yeah yeah and that will obviously make it much easier to display when do we need the one for the meet well it may not for the whole meeting but like okay yeah i guess for the so when we have the display will we display the whole series then if we have for the individual topic segments within the meetings if we have ready calculated disp measures then we do not have to sort of extract that data from the individual utterances yeah exactly yeah and that is also fairly easy to store along with our segments is not it yeah for the segments are we extracting some type of title for them that we craft with some fancy algorithm or manually or we are just taking the single most highly valued key word utterance for the segment heading well we can start off like that well i was going to start off i have v got sort of half way through implementing one that does just i d f and then just i can change that to work on whatever it is probably like in in the end probably it would not be the best thing if it is just the high most highly ranked phrase or key word because like for example for an introduction that would most definitely not be anything that has any title anywhere similar to introduction or something yeah and it should be weighted by stuff like the hot spots and the key words in the search and stuff like that yeah also like for this part maybe if we go over it with named entity in the end if i mean w if one of the people doing dil has some named entity code to spare and just like at least for the for sort of for finding topics titles for for segments just take a named entity which has a really high what is it called d f i d f whatever cause you would probably be quite likely if they are talking about a conference or a person that that would be a named entity which is very highly fr frequented in that part did he not say something about named entities so i thought he said there was not very many yeah he said they are quite sparse so that basically was do not bother basing too much of your general calculation on it yeah but like especially if they are sparse probably individual named entities which describe what a what a segment is about would probably be quite good like if there is some name of some conference they would could probably say that name of the conference quite often even though he is right that they make indirect references to it anyway sorry you s you said you are currently in implementing the idea what exactly are you computing yeah it is not t f i d f it is just inverse document frequency okay cause it is really easy to do basically okay there is just like for a baseline really yeah you are able to do that in java yeah well i am half way through it is not working yet but it will do yeah so you are doing that on a on a per word level yeah okay and then averaging it over the utterances okay but it is not like related to the corpus at all it is just working on an arbitrary text file at the moment okay cool i was just wondering where you had the corpus from at the moment no huh so it it seems that the data structure is not a big problem and that basically we do not have to have all these massive discussions of how we exactly interact with the data structure because most of our work is not done with that data structure in memory in the browser but it is just done off line and everyone can ha represent it anyway they want as long as they sort of store it in a useful x m l representation in the end so like it would be useful to know how everyone is going to store their things though yeah that would mean understanding the nite x m l x m l sort of format in a lot more detail yeah we should i think we should just have a long session in the computer room together and like now that we know a bit more what we want take a closer look at nite x m l yeah well i have got like a few hours free yeah i have had a b i have had a look at the the topic segments how it is stored like after this and then yeah th those are few per meeting and it well it gives a time stamp and inside each one there is the actual like utterance segments and the list of them that occurred and they are all numbered so that is where that is stored good yeah i have not looked at this stuff much at all yeah so i guess if i am going to be segmenting it with a l c seg then that is like same format i would want to put it back out in so it would be equivalent yeah yeah who is who is sort of doing the the the central coordination of of of the browser application now like well like the integration what do you mean integration yeah or but also like all these elements like like the loading and yeah integration and and like handling the data loading and stuff i do not know i do not think anyone has been allocated to do that yet nah i am sort of like i think i will take over the display just because i have started with a bit and found it found it doable yeah yeah yeah definitely so somebody should sort of be the one person who is who understands most about what is t centrally going on with with the with the project like with the with the browser as a whole and where the data comes in and yeah any volunteers it is the most boring task it is also a complicated one yeah it c could be difficult yeah yeah yeah i know but b i guess we can do it like several people together it is probably just those people have to work together a lot and very closely and just make sure that they are always f understand what the other one is doing yeah well i guess the important thing is to get the crucial m modules built or at least simple versions of them ye yeah yeah or or ready made versions of them for that matter and yeah and then yeah and then we will maybe have to prioritize somebody into just integrating it yeah but i think actually like at the moment the integration comes 1st i mean it is sort of at the moment the building the browser comes 1st and then only comes the creating new sophisticated data chunks because that is sort of the whole thing about having a prototype system which is more or less working on on chunk data but it at least we have the framework in which we can then test everything and and look at everything cause before we have that it is going to be very difficult for anyone to really see how much the work that they are doing is making sense because you just well i guess you can see something from the data that you have in your individual x m l s files files that you create but it would be nice to have some basic system which just displays some stuff so maybe we should try doing something really simple like just displaying a whole meeting and like just being able to scroll through it or something like that or just adapt like their like just sort of go from their system and and adapt that piece for piece and see how we could yeah how we could arran like adapt it to our system does anyone want to like just sit with me and like play for 3 hours with nite x m l at some point are you free after this i would not like to be cause i would like to go to the gym i am theoretically free but if there is any time t how about friday then cause i am off all friday you have nothing no free time on wednesday wednesday i have got a 9 til 12 9 til 12 and then nothi you have or you yeah nothing in the afternoon i have got nothing in the afternoon so anytime wednesday afternoon i would be cool i think okay so you ha yeah where about just in appleton tower yo forrest hill whatever one is easier to discuss stuff i do not know i will be in the appleton tower anyway i am not biased okay what time do you want to do well i will be there from 12 i have got some other stuff that needs done on matlab so if you are not there at 12 i can just work on that okay so i will just meet you in in 18 a in the afternoon so yeah i guess at the moment nobody critically depends on like the nite x m l stuff working right now right like at the moment you can all do your stuff and i can do my l s a stuff and i can even do the display to a vast degree without actually having their supplying framework working so it is not that crucial yeah i think so yeah i w i w i would need the raw text pretty soon because i have to find out how i have to put the segments into bins yeah actually i need the raw text as well and then yeah yeah but i was i was i was more thinking of the sort of the the whole browser framework as a running program now yeah no that is not necessary yeah i think we all need the raw text in different in different flavors do not we yeah yeah why w jasmine i thought you just said that you would looked at extracting the text yes i did but i have only just got the notes i have to still have to order everything by the time and yeah so you you said you did it in python yeah yeah i think it is quite easy after the yeah did you use b the x l x m l parser in python yeah yeah right yeah sounds pretty good so so cause yeah i was having a look in it a look at it as well and i noticed the the speakers are all in that separate file so did did you have to combine them all and and then re order them yeah b i w that is what i was thought that you just combine them and then order the time stamps accordingly yeah ye yeah c right yeah so that is approach well i was going to do so yeah we may as well collaborate okay what i found out was that there are quite a lot of things without without s time stamps in the beginning in the word files yeah and x m l files yes but that is yeah that is just an i d or something i do not know yeah everything that is a word has a sti time stamp just numbers yes but what are the other things that is some kind of number f maybe the file number or something that is in the beginning what is that i am not sure i what you mean do you know i think there are quite a lot of numbers in the beginning where n there is no time stamp for the numbers it is think they say quite a lot of numbers and before that there is this number was it but number within the x m l context yeah there i are numbers in the the w tag but there are no time stamps are they spoken numbers like do they look like they are utterances numbers yeah there is the number task is not there that is part of the whole thing that is at the end that is at the end i think her time okay yeah in the beginning as well sometimes i think 0 right at least i saw some yeah maybe did not have a look at our meetings yeah we have to probably cut that out anyway for our project i do not know yeah it is probably going to screw up a lot of our data otherwise yeah if not sure if it what it does to document it would probably make the yeah if if you have segments for that probably the i i think it would not as it occurs i mean it would be it occurs in every meeting so and i think it even has its own annotation like digits or something so that should be really easy to cut out okay i am just thinking like it pro it pro probably like the l s a would perform quite well on it it would probably find another number task quite easily seeing that it is a constrained vocabulary with a high co occurrence of the same 9 words yeah i am sure so that wou 10 word but what it is it actually that numbers ah it is just to test the system i think yeah okay so but there are no time stamps annotated to that so i think it is also something that they they said the numbers in order right it is it is quite strange they have to read numbers from yeah i think it is it the it sounded like they wanted to check out how well they were doing with overlapping and stuff because basically it is like they are reading them at different speeds but you know in which order they are said i did not have a look at that anyway so icsi has some reasons for doing it they must have been pissed off saying like numbers at the end of every meeting and also there are different combinations of letters b r e and something like that is it everything ordered are the time stamps global or are they local at any point i thought they were local to th a particular meeting they okay dave if you would or actually for well if you are doing i d f s or you whatever you call your your frequencies i always mix up the name you need some dictionary for that at some point though like you need to have some representation of a word as not not that specific occurrence of that word token but of of of a given word form because you are making counts for word forms right yeah i am just building a dictionary yeah so we should work together on that because i need a dictionary as well 0 mine is just going to use the hash map one in java cause i am only going to do it on small documents okay it is just like bef until the information density is up and running kay just something to get give me something to work with okay did not you say that the 0 the ord so it is only going to use quite small documents you see to start with yeah but for i am just wondering for the whole thing does somebody wo who was it of you 2 who said that there is some program which spits out a dictionary probably with frequencies yeah it is rainbow it is i think it is just the dictionary in the 1st place but okay is anyone of you for the for the document frequency over total frequency you going to have total frequencies of words then with that right like over the whole corpus sort of no i have to bin it up and so i will only have counts for each each bin or something or why does it need to be classified into like different segments it is because rainbow is a text classification system and i think it is not possible to have just one class that is the problem can we just fill a 2nd class with junk that we do not care about maybe we could like i do not know copies of shakespeare or something yeah sure you sure we could do that but i do not that makes sense cause if what we are looking for is the frequency statistics i do not see how that would be changed by the classification if we need just frequencies maybe we should just calculate them by using perl or something i the i do not know well there maybe another tool available yeah it is quite easy to just count and s or sort them by frequency yeah w using which tool are you talking about just using a perl script be careful with that like my experience with the british national corpus was that there is far more word types than you ever think because anything that is sort of unusual generally is a new word type is it too big yeah like any typo or any strange thing where they put 2 words together and also any number as a word type of its own so you can easily end up with 100000s of words when you did not expect them so generally dictionaries can grow bigger then you think they do i do not know how you how many terms you can handle in perl well you can probably also you can probably pre filter like with regular expressions even just say if it consists of only dig digits then skip it or even if it consists any special characters yeah then skip it because it is probably something with a dot in between which is usually not something you want to have and i can not remember who has got it might be wordnet but one of these big corpuses has a list of stop words that you can download and they are just basically lists of really uninteresting boring words that we could filter out before we do that it is like that is one the papers i read that is one things they did right at the beginning is they have got this big s stop list and they just ignore all of those throughout the experiment what i did for my project i just ignored the 100 most frequent words because they actually end up all being articles and and everything and stuff so we need like several of us need a dictionary am i the only one who needs it with frequencies yeah i it would be useful for me as well am i the only one who needs it with frequencies or it i think that would be useful for me as well frequencies yeah yeah well i guess as soon as we have the raw text we can probably just start with the java hash map and like just hash map over it and see how far we get yeah i mean we can probably on a machine with a few 100 megabyte ram you can go quite far you can write it on beefy so even if it goes wrong and even if it has a 1000000 words be well all you really want to do is look into getting some sub set of the icsi corpus off the dice machines cause i hate working on dice it is awful like so i can use my home machine 0 yeah burning it on a like we should be able to burn the whole corpus just the x ha has a c d burner though has a c d burner ah i see i asked support about that 2 days ago in the informatics building there 0 sorry in in appleton tower 5 the ones closest t 2 machines closest to the support office so i presume 0 wait i have the exact email i think he is talking about sort of the ones that yeah the right hand corner far right yeah if you if you enter the big room in the right hand corner i think yeah the thing is like you can only burn from the local file system so if it is from s well actually i think if it is mounted you can directly burn from there but the problem is i have my data on beefy and so i have to get it into the local temp directory and burn it from there but you can burn it from there how big is it without we looked that up and i for we looked that up and i forgot the wav files and stuff cause i could just say at going over s c p one night and just leave it going all night if i had to yeah yeah no you you we should be able to get it at i do not think it was i do not think it was a gigabyte it is yeah i mean the wave data are obviously not going to get off there completely see i would off i would offer you to to get it on this one and then like copy it but you know what i figured out i am quicker down loading over broad band into my computer than using this hard disk really 0 right there is something strange about the way how they access the hard disk how they mount it which is unfortunate i will see if i can s c p it i suppose what operating system do you have i have got a linux box and a windows box okay so wh what connection do you have at home broad band yeah so if anyone of us gets it we can then just use an ext put it on to c d i can if i get down i can put to c d yeah burn it to c d or yeah put it on on hard disk whatever yeah question is if you are not quicker if you because you should get massive compression out of that like 50% or something with a good algorithm so if you could compress it and just put it into a temp directory i am not sure if there is enough space like is how much do we get the temp the temps usually have for gigabyte 3 or 2 really the temps yeah okay i do like i mean there is not guarantee that anything stays there but overnight it will stay and i think the temps usually have ah yeah but that would have to be the temp directory off the machine you can s s h into directory of s s h yeah but i can do it from that session can not i you can compress it from a remote session and s c p it from the same session yeah they wou they would they would probably hate you for doing it but they would probably they would like you more if you s s h into another computer compress it there and then sort of copy it into the into the gateway machine do you think they have if you s s hey you know if you if you s s h and they have this big warning about doing nothing at all in the gateway machine yeah 0 no no i was thinking of sshing just into some machine and then just scping it from there yeah to your home machine yeah i mean it has to go through the gateway i have not i have not figured out how to tunnel through the gateway into another machine yet but can you not do that it is not it is not easy definitely that is why i end up sort of copying stuff into the temp directory at the gateway machine i see sorry if this is boring everybody else this is just details and how to get stuff home from what we can probably just look at that together when we are meeting th yeah yeah kay i just wondered so who is then doing the frequencies on on the words because i am i think i will also i could also make use of it for the agreement and disagreement thing because i i in my outline i talked about using the discourse acts 1st i am sorry and then in the chunks of text i found looking for word patterns and so on so i would for example need the most freq frequent words well yeah so if you cut off all that i would will not be use or as soon as somebody gives me the raw text of the whole thing i can probably just implement like a 5 line java hash table frequency dictionary builder and see yeah i i but i need it for my chunks then i would you know 0 did you not say frequencies f of words in the whole sorry did yeah but i would i would like to look at the frequency of words in my in the regions of text i found out to be interesting so i would not need it so you could just so you would you it it would have to be re calculated only for my segments yeah you would have to count it yourself yeah but th 1st how big are the chunks huh how big are the chunks you are looking at i think it would be you know l as as big at as the hot spot annotation things so quite small then that is quite small yeah that is some utterances so you could just you could use just the same thing we used to build the big dictionary you just do that on line cause that will not take long to build a little dictionary that big will it i mean just use the same tool that we use yes yeah yeah yeah so i would probably just concatenate all my text chunks and then let us say m i will run over it yeah 0 you do not want to have different counts for each chunk but just like sort of for for something from old chunks yes 0 yeah no that is yeah so once i write an ar like w if i write like an algorithm which does a hash table dictionary with frequency from a raw text then the raw text can be anything so how far are we g how f how far are you getting raw text out of it do you think i can get all the raw text but it has to be ordered still so okay well that is good because for the dictionary the order does not make a difference does it it does not need ordered no so yeah so i will get that from you and i will write the hash table which goes over that and creates a dictionary file no it is not so for the dictionary is it okay if i do whatever word blank frequency or something just p could everybody sort of start from that i mean i guess we can well that is the t are you using t f i d f for the information density it is in what is implemented in rainbow is information gain and i am not quite sure how they calculate that alright okay like cause frequency would be useful i think yeah i i need frequency as well yeah but depending on the context the size and what we consider a document in the sense of calculating t f i d f is going to change which might need thinking about well i think we might have a lot in common what we calculate because i for my latent semantic analysis need like counts of words within a document within a a segment actually within a topic segment that is what rainbow does i think you j can just get probabilities for a certain words for each document certain can i convert these probabilities back into frequencies we would have to look at that okay 0 so that is what f rainbow does because that is what l s a builds on like it builds a f a document by frequency matrix 0 so i could probably get that even though but i already have i already have my code to build it up myself no do not bother i have my code already yeah so dave you said you need the frequency counts actually for per document would you say not for the whole thing i think it would be useful yeah well you need the raw frequency as well but you also need how many times things occur within each document and what we consider a document is going to depend on our context i think cause if we are looking at the whole lot of meetings we will consider each meeting a document in sort of terms of this algorithm and if we are viewing like say just a small topic segment you might look at even each utterance as a small document it more and more appears to me that if we if we scrap the notion of the meeting as an individual thing and sort of ju see meetings as as topic segments and have sort of like hierarchical topic segmentation instead then it is b like a more coherent framework yeah but the thing is it is going to need some th th thought of how we actually maybe it does not actually matter maybe if you just do it once at the highest level it it will be fine but i was just thinking it might be difficult to calculate the t f i d f off line for all the different levels we might want cause if we are going to allow disjoint segments for example then how are we going to know what is going to be in context at any given time but i suppose if you just did it globally treating a meeting as a document it would probably still be work out fine because you would only be comparing to ones within the context wait are we are we using this for the for the for the do for the weighting in the end now this this measure you are calculating i do not know i thought were you going to use that in the end the information density because if we are doing like i think for for the information density we we should calculate it on the lowest level not on the highest but like cause 0 sorry that is what i mean like yeah for each word or whatever but across the whole lot is what i mean by highest level like across the whole corpus yeah but w it do not you have to like go sort of like for in a document versus the whole thing is not that how it works that you c look look at r yeah but you would probably look at each meeting as a document i do not think that is a good idea because is not it like that we expect th there to change over i b with the different topic segments more that they talk about something different in each different topic segment possibly cause that is what relative term frequency is about that like in some context they are talking more about a certain word than in general so that would more be the the topic segments then yeah that is what i thought as well that you that probably the the topic segment level is the most informative for the words i do not know yeah yeah yeah are they big enough to get anything meaningful out of yeah that is the problem i do not know so i am just wondering if there is ways to abandon the whole concept of of meetings and sort of but just not really treating separate meetings as too much of a separate entity well yeah that is not it is not an issue but you just concatenate an x m l file together but we still want to have like a notion of meetings for the user yes definitely but on algorithmic level whether we actually whether there is some way to just represent meetings as as topics yeah sure yeah you just like whatever you want to look at you just jam together into an x m l file and that is your meeting even though bits of it may come from all over the place or whatever i mean i do not see why that is really a big problem that is not really what i meant but i think i have to think more about what i meant g i am confused about everything so basically what you are saying is you can take an arbitrary amount of data and process it with the same algorithm it does not matter conceptually what that data is it could be a meeting it could be 2 utterances it could be a meeting plus half a meeting from somewhere else yeah i am i am not so concerned about the m a meeting plus something else i am more talking about like yeah the keeping keeping the same algorithm and the same way of handling it and just saying like just this this topic here i it happens to be like a whole meeting and it has sort of sub topics so just that sort of topics a hierarchical concept where like a topic where there can be super topics and topics and the super topics are in the end what the meetings are but in general at some level super topics are treated like like topics i do not think it is very difficult though i mean what you do is you just build an x m l file and if you want it to get down to the utterances you would go to the leaves and then if you wanted the next level up you would go to the parents of those and like just go from like the leaves inwards towards the branch to build up things like you know when you click on a segment it is going to have like words or whatever that are important i am not really sure what i want as long as like the algorithms are designed with it in mind i do not think it is a very big problem so sorry could describe that again the well like say you had like say for a meeting right you have got like say a hierarchy that looks quite big like this and like the utterances come off of here maybe then when whatever your algorithm is doing as long as when you are working with utterances you go for all the leaves like then if you need something next up so like a topic segment you would go to here but if you were looking at say this one so only went like this right so you it is same you would start with the leaves and you go 0 i want a topic segment so i go one layer up see and then if you are working with just a topic segment in there it is the only thing you have to worry about and like each time you want a higher level you just need to go up the tree and as long as your algorithm respects that then we can just process any arbitrary x m l file with whatever hierarchical structure we want so that would be the series as a whole that would be sort of m meetings yeah a meeting say and that would be a topic segment yeah i am a i am a i am a bit brain damaged at the moment but i think i will just sit together with you again and and go through it again so i think as long as you build an algorithm that respects whatever structure is in the file rather than imposing its own structure so so i will is th it like is this and this structurally then always identical well no it does not have to be so that we can that we can treat it with the same algorithm or but i mean it could be as many nodes as you want like this one could be deeper maybe say so then you would start with all your utterances here and when you go up to get topic segments you go to here here here here here here here that might be a bit confusing though cause you have things on different levels yeah i am also not sure how we can go from from bottom up i have always thought it is like more that 0 whatever i am a can not think of it at the moment probably this is all too complicated worrying about that at that moment anyway is there anything else we should discuss now have have we have we decided anything are we doing anything well wednesday s wednesday we are meeting and looking at their at their implementation in some more detail to actually understand what is going on yeah yeah so we will see if we can get like a mini browser just displays 2 things synched together of some kind we had 2 things from their stuff just to make sure that we are like understand it we understand it enough to to m modify it yeah yeah yeah should we not have like a group directory or something where we can put all our code in and that kinda thing it would be useful i do not know who you see about that though yep how would we do that by just making like it w read write for everyone kay who has most free space on their i have gotten hardly any i d have no idea same here i have probably got a reasonable amount because everything on my dice account can actually be deleted cause i store it all at home as well well we alternatively we can probably just make another directory on the beefy scratch space i mean that is where i am having gigabytes and gigabytes of stuff at the moment is that guaranteed to stay the no no yeah we can ask steve if we can get space yeah maybe you should send a support form yeah we could do that but i think if he sends to the i think if he sends to the port he would probably be in a better position just say we want some web space listen to yeah yeah i am sure he had to deal with that last year yeah cause that would be really useful is if we had a big directory especially for transferring stuff having said that are we allowed to take a copy of the icsi corpus something we should probably ask before we do it i think he said yes to that i think that was like in when we were still in the seminar room i asked that once or like ask is it possible to get it off and nobody said like people were discussing about the technical practicalities okay but nobody said anything about al being allowed to or not allowed to i mean we have access to it here and i guess it probably means that we we can not give it to anybody else but okay but if they give us access to it here 0 sitting on a dice machine then there should not be a reason why we should not be able to use it on our laptop i personally do not have too many friends who would be too keen on getting it anyway no me neither yeah right i have that really excited pirate copied thing ye it annotated meeting data so shall we sit together tomorrow then as well m okay yeah jasmine what is the text you are extracting looking like then at the end yeah w would it be best at the moment it is it is just lines of because i i think it is actually very similar to what i did for my speaker extraction and i think i would ch perhaps have to change 2 lines of codes to get you for each meeting a file that says fr from this millisecond to this millisecond there was this sequence of words and so on so that is just changing 2 lines of code and it would give you that so okay so you would do you extract the words the raw text as well yeah so far i extracted the dura durations okay but it is from the words file so i could just contatenate concatenate the words instead of the durations and it should i mean print out should be very straight forward i can try to do it and send it to you okay pe and you have a look at it will it make sense for what you want okay that okay huh so have we already extracted from all the files yeah p i mean it i just let it run over all the files yeah so did you also order yes wait wait wait i just ordered sorry yeah sorry i ordered according to the starting times of the utterances what i just realized we should really t keep different serieses completely separate for virtually all purposes just let us be careful about that because what do you mean by diffe like the the icsi corpus is not is not one meeting series it is several meeting series with different people meeting for completely different things yeah i mean t i i have one what i give you would be one file for each meeting for each meeting yeah not for each meeting series alright okay but like let us just be careful that whatever we sort of we merge together i did not do that yet that like the highest level of merging it is not the whole icsi corpus but individual series yeah one group yeah might be funny to see what is summarized the whole corpus as anyway i think we might actually i think that is probably be somewhere like well or something like it i think it would be very useful but i think we might just get away with for the whole project just like looking at only one series and just doing within one series i mean you can do everything you want in one series yeah i mean there is one series that has just one meeting 0 yeah let us take that that sounds good is the is the data always clearly split up by different series like is it easy to just pick one yes the you you the data is of the form you have 3 identification letter so b e d or b b d or something and that is always the same group okay okay okay and then after that there is a number like 001002 so it is a okay so at at every level everyone has to be careful to really just take even at the highest level just take stuff from one series and not merge stuff from different series together because they would probably be just majorly messy yeah but that is that is really quite easy to see because they are named yeah so so t so like if even if we make one single text file which has the whole corpus sort of our corpus that would still be from one series only wou but it what you are producing at the moment is like individual text files that sort of have the raw text for a whole a meeting as a whole or yes but i i mean as the start start times start for each meeting at 0 you could just probably just add the final 2nd time to the next meeting and so on and just put it all together but then we would have to change the information about who on which channel it was set to by which person it was set yeah and that is actually stored in another x m l document kay so is is anybody creating an a real raw text thing at the moment like which is just the words yeah that is what i am going to need yeah tha cause that is what i am going to need as well yeah i w would then just not print out the start and end times yes but i but if there b are not like so it is it is start and end times just for the file yeah it is just mo changing it a bit okay like is it just the 1st and the last line or is it for every single thing in no it is for every single word we can just change the code or for every single utterance yeah that depends on what you want yeah so what do you mean by just not print out that yeah but i do it with perl it is just string manipulation so i would i mean i would just okay if you are into it can you make a text file which just like makes just the words sure kay do you want it straight flowing cause i would need something that marks the end of of is is yours segmented by topics then that like is there any information that you have to the topic to the automated topic topic segmentation no i did not do a sea no 0 then i need something different later anyway okay but for now if you c no but that is what m l c seg does it it marks the end of each segment okay you are going to put that as an output of yours the segmentation yeah okay so for now can you create like sort of just a dump which is pure text just pure text so that i can get a dictionary and you can work on that for your topic segmentation yeah and and you would want that all in one file for all the corpus or or for for the series but i can but i can also deal with separate files i mean i can just write the algorithm that it loads all files in a directory or something for the series but i mean if you yeah i can directly put it into just like but if you can put it in one single mega file that would be quite useful for me 0 so only words per meeting series even though for you would not it be easier if you had different files because then you sort of know like yeah yeah for me it is better if they are by meeting yeah so give m give me different files as long as like it m if you could name them in a way that is easy to enumerate over them huh like whatever 12345 or something yes or just anything that i can yeah they will just i will just take i would take over the names they have anyway yeah is is it something that is easily enu like to enumerate over is it some just some ordered pattern yeah yeah okay cool yeah one series has the same 3 starting letters so okay cool yeah so only words and words and times and you i do not need the times i just need the words but in the right order yeah you want it ordered yeah in the right order yes okay it is just a wish list okay anybody orders ord base dot times yeah and do you want yeah sometimes they are contained in one another so yeah that does not matter too much i think just after th kay ordered only words when do you think you will have like a primitive segmentation by some ready made topic segmentation by some ready made tool ready then that will be really easy to do once they have got the raw text it is just a case of running the script okay okay cool cause i will need that then when it is done yeah i mean hopefully this week okay and i think for all the corpus it is just from i know from other times it is 9 megami byte to have i mean should be should be similar to have the words is that it that is quite good should be what is what is 9 megabyte the the na all the words together for all the meetings alright that sounds quite reasonable that is 99 characters over okay that is what i am guessing that is you know what i because 9 mega byte is what i got for when i said for every utterance this is goes from there to there and okay okay that is for are we are we picking one particular series at the moment takes takes seconds or and we could 0 yeah i mean i am it doing it for all of it yes does not matter okay yeah yeah i guess we can probably process the data for all different series and then check which series is the best for the presentation yeah it sounds quite reasonable 9 megabyte i mean if you think if it is r roughly a 1000000 words and 9 characters per word sounds realisti yeah i mean i hope it will be the same for the words it is just what i yeah yes i am going to build a dictionary then from that like just a list of the words that maybe a list of the words with the frequencies or a list of the words sorted alphabetically or numerically i what what does anyone want does this there any wishes for dictionaries i could just use it with the frequency i think until the information density thing is finished that would be really useful so i will create a dictionary so so i will probably send just one file of the 1st meeting to all those who need it so that you can have a look whether that is what you want add add the structure yeah and then the actual file we can probably like copy from your home directory or something like it yeah i mean if it is just for one meeting it is really not too big yeah yeah but i am sa i am saying for the whole thing in the end then like the big thing we probably should not do by email yeah yeah how long would it take to make the frequency counts with a java hash table 0 from the time i get the file i can do that in an afternoon the next sort of the next morning yeah 0 you mean how long processing time it takes no how long you would have to program something ah it is a it is a bog standard algorithm i have i have sort of i have written it for for dil just in half an hour or something similar okay it is just you put them in a hash table and and say well if it exists already in the hash table then you increase the count by one and i will probably implement some filter for filtering out numbers or something because it is quite easy in perl as well it is just a line of code for counting all the words and yeah it is it is by hashes really how do you do that okay well i do not know any perl yeah i mean if anyone wants to do a perl script for that that does it does it nicely i yeah i have no problem with that i but i think i have the java code virtually ready because for dil i wrote something very similar kay like for dil i wrote something that counts the the different occurrences of all the tags if you are doing it in java could you serialize the output as well as writing it to a file sorry if you are doing it in java could you serialize the dictionary yeah as well as writing it to a file the hash table i have never serialized anything it is really easy would not that be absolutely massive though i do not see why it would be any more massive than the file and then seriali and then write the serialization to a file yeah so you want like a se like a file which is the serialization of a hash table it just saves you parsing the okay file representation of it and now cause i would be using it in java anyway so i would just be building the data structure again yeah i i will i will check if i understand how it works i mean otherwise i can give you the code for loading a dictionary give you my my it is just it is it is sort of it is a line break separated file you know yeah but it seems like a bit silly to be parsing it over and over again kinda thing yeah yeah i will see if i understand how to serialize there is a there is a serialize command so that gives me one mega mother of a s i would have thought that i think all the collections and things implement serializable already yeah but do they automatically write to the file anyway i will i will figure that out i think they might do we do not have to yes is that pretty much pretty much it so dave and me look at how nite x m l works and we are tonight i will try and i will either work some more on the t f i d f summarizer or do the audio thing i will build a dictionary as soon as i get the text and yeah so that when do we have to meet again then with this do not know suppose we are just getting on with all our yeah components so how are we going to do a demonstrator next week my god do we have to demonstrate something next week i know wa what do we have to demonstrate no no not demonstrate but like did not you say that did not we sort of agree that it would be useful to have a demonstrator of it like some primitive thing working next week yeah yeah he suggested that we could have an initial prototype that has got to be very prototype i know i would b i would be surprised if we can get anything working by next week yeah ah well let us go sorry alright i feel like like hanging mid air and not really like finding a point where you can get your teeth into it and start working properly and so it is all so fuzzy the whole yeah i know i think it is cause we had to specify it ourselves that it is not as like focus the specification of most work we have to do yeah but it at the moment but at the moment it is also an implementational level like with the data structures i am just like over these vague ideas of some trees i am f yeah once we start doing it it will all become more or less obvious i think anyway yeah it is just we are half way through the project time table that is just what freaks me out
okay okay does anyone want to see steve is feedback from the specification is there much more in it than he said yesterday not really just what he was talking about duplication of effort and like duplication of effort and stuff effort and stuff and yeah it is shame that we should maybe think about having a prototype or week 6 which is next week yeah so if we probably prioritize our yeah i would say if for the prototype if it just like wherever possible chunk in the stuff that we have pre annotated and stuff and for the stuff that we do not have pre annotated right like a stupid base line then we should probably be able to basically that means we focus on the interface 1st so that we we take the ready made parts and just see how we get them work together in the interface the way we want and then we have a working prototype and then we can go back and replace pieces either by our own components or by more sophisticated components of our own so it is probably feasible thing is i am away this weekend so that is really i mean if we just want to have some data for the user face could even be random data 0 yeah yeah no but also like the similarity thing like just my matrix itself for my stuff i think i can do that fairly quickly because i have the algorithm yeah i think today is meeting is really the one where we sort of settle down the data structure and as soon as we have that probably like after today is meeting we then actually need to well go back 1st of all look at night xml to see in how far that which we want is completely compatible with that with night xml offers us and then just sort of everyone makes sure everyone understands the interface so i think if today we decide on what data we want to have now and later maybe even today we go and look at night xml or some of us look at night xml in a bit more detail just trying to make some sense of that code and see how does the representation work in their system and then sort of with that knowledge we should be able to then say okay that type of night xml data we want to load into it and this is how everyone can access it and then we should be able to go from there i have looked at the java code for the data network i have looked at the documentation and seen enough to make me think that if you want to use the nitexmel framework because they have a good event model that synchronizes sort of the data and in every display element so that takes a lot of work away from us of that would be a reason for staying within their framework and using their general classes but beyond that i have not looked at it at all which is something we should really do who actually like for this whole discussion i mean who of us is doing stuff that is happening online and who of us is doing stuff that is happening offline like my data is coming the basic word importance is offline as well the combined measure might not be if we want to wait but a user has typed into the search okay the mind is going to be mostly using the offline but the actual stuff is doing will be online there will not be very processing intensive or memory intensive so basically apart from the display module the display itself we do not have an extremely high end high degree of interaction between sort of our modules that create the stuff and the interface so the interface is mainly while it is running just working on data that is just loaded from a file i guess i do not know about the search functionality that might be online it depends how it is going to work yeah yeah yeah yeah the search is i guess the search is sort of a strange beast anyway because for the search we are leaving the 9 xml framework but that is still sort of that is good that means that at least we do not have the type of situation where somebody has to do like a 1000000000 calculations on data online that would make it a lot more like that would mean that our interface for the data we would have to be a lot more careful about how it performs and everything and nobody is modifying that data at online time at all it seems nobody is making any changes to the actual data online so that is actually making it a lot easier that basically means our browser really is a viewer mostly which is not doing much with the data except for sort of selecting a piece of bid and just paying it are we still going to go for dumping it into a database are we still going to dump it into a database well some part relevant for the search yes i would say so because if we are i reckon we should all read our classes out of the database it would be so much easier  well if we are going to dump the part of it into a database anyway we might as well dump all the fields we want into the database calculate everything from there then we do not even have to worry that much about the underlying xml representation we can just query it yeah but nobody of us is doing much of searching from the data in the online stage and for altogether like the display itself i think we are easier if we if it is sitting on the xnl than if it is sitting on the sql stuff because if it is sitting on the xml we have the knight xml framework with all its functionality for synchronizing through all the different levels whenever there is a change whenever something is moving forward and stuff and we can just more or less look at their code like how their player moves forward and how that moving forward is represented in different windows and stuff so i think in the actual browser itself i do not want to sit on the sql if you can sit on the xml because sitting on the xml we have so much help and for the calculations that we are doing apart from the search it seems everyone needs some special representations anyway what if we are going to do that we should try and store everything in an xml format as well you mean our results yeah yeah in the night xml format so with the time stamps and stuff so that it is easy to tie together things what i am like what we have to think about is if we go with this multi level idea like this idea that sort of if you start with the whole meeting series as one entity as one thing that you display as one whole sort of that then the individual chunks are the individual meetings whereas and then you can click on a meeting and then sort of the meeting is the whole thing and the chunks are the individual segments that means we have multiple levels of presentation which we probably if we do it this way like we have to discuss that if we do it this way then we should probably find some abstraction models so that the interface in the sense deals with it as if it is the same so that the interface does not really have to worry whether it is a meeting in the whole meeting series or a segment within a meeting you know what i mean and that is probably stuff that we have to sort of like process twice than like for example that like the summary of a meeting within the whole meeting corpus or meeting series if meeting series is a good word for it i do not really know how to call it you know what i mean like not the whole corpus but every meeting that has to do with one topic and so in the meeting series so that a summary for a meeting within the meeting series is sort of compiled offline by a summary module and that is separate from a summary of a segment within a meeting i do not think we can we do not even need to do that because we got our information density calculated offline so all we do is treat the whole lot as one massive document i mean it is not going to be so big that we can not load in information density for every utterance i mean just summarize based on that i thought we would just have one big summary around the different importance levels displayed on it and depending on what our zoom level is we just display a part of it so are we doing that at all levels are we i mean we would have one very big thing offline and from that we would just select what we are displaying and just have different like fine grainedness levels sort of yes so for example you would give a high value to those sequences you want to display in the play in the meeting series summary so the only thing that would happen basically if i double click let us say from the whole meeting series on a single meeting is that the zoom level changes like the start and the end position changes and the zoom level changes that was what i saw i think we can do it online i thought we could not do that i was under the impression that we could not do that because we could not load the data for all that but i do not know i mean that i do not think there is really much point in doing like that when it is just going to feed off in the end the information density measure basically and that is all calculated offline so all you are really doing is sorting the list is the computationally hard part of it so i am not sure if i got it well like the ideas we are calculating our information density all offline 1st for every utterance in the whole corpus right so all you do is you say if you are looking at a series series of meetings you just say well our whole document comprises of all these who is not together and then all you have to do is sort them by information density like maybe weighted with the search terms so so i do not think it is too slow to do online to be honest i was just worried about the total memory complexity of it but i completely admit i mean i just sort of like took that from something that jonathan once said about not loading everything but maybe i was just wrong about it but i think the difference might be that we would just want to have the words and that is not so much what he meant was not possibly loading everything was that you load all the annotation stuff all the sound files so what we have is we would have a word like we would have words with some priority levels and that would basically be it already because even the same selection would the summaries automatically feed from just how prioritized an individual word or how prioritized an individual utterances or are the summaries sort of refined from that and made by a machine to make sentences and stuff or are they just sort of taking out the words of the highest priority and then the words of the 2nd highest priority well on the utterance level i think so the utterances of the highest like mean information are we doing it on the utterance level are we doing it on word level like the information well the trouble is doing it on the word level is if you want the audio to sync up you have got no way of getting in and extracting just that word i mean it is impossible we i think we have started in end time for words like every single word yeah but it might but it might sound crazy in the player we should really maybe we can do that together at some point today that we checked out how the player works i do not think but there is maybe some merit and all together doing it on an utterance level in the end i am getting quite lost at the ones because what is our difference between the the importance measure and the skimming i mean do we do both or is it the same thing well the skimming is going to use the importance okay but when we talk about summaries you talk about this about skimming and not about well mostly skimming yeah well but also about the displays i mean the displays in the text body in the latest draft that we had we came up with the idea that it is not displaying utterance part trans but it is also displaying a summarized version in you know like below the below the graph the part yeah maybe yeah it is just like this is not that the skimming and there is just like audio skimming and there is displayed skimming yeah but it is same data yeah yeah maybe there is some merit of going altogether for utterance level and not even bother to calculate i mean if you have to do it internally then you can do it but maybe like not even store the importance levels for individual words and just sort of rank utterances as a whole the nice thing about that is that we will automatically be in sentences more or less yeah so it will make more sense than just extract it might be better skimming and less memory required at the same time and i mean if you know how to do it for individual words then you can just in the worst case if you can not find anything else just sort of make the mean of the words over the utterance you know what i mean not quite so what did you want to do you just wanted to assign what is the smallest chunk at the moment you are thinking of assigning importance measure to is it a word or is not an utterance i thought about words so we are thinking of like maybe just storing it on a per attrance level because it is less stuff to store probably for dave in the audio playing and for in the display it is probably better if you have whole utterances then i do not know like what it is like if you just take single words out of utterance it probably does not make any sense at all at all whereas if you just show important utterances but the atrends as a whole it makes more sense so it does not actually make a difference for your algorithm because it just means that if you work on a word level then we just mean it over the utterance yeah i think we also thought about combining that measure with the mergers i get from hotspots and so on they are on so that would also be on a trance level i think so that is good anyway then yeah because that makes a lot easier than to put it on an utterance level calculated at word level though because otherwise there will not be enough occurrences or returns to rate any 0 yeah no way i mean how it just means does it internally i do not know but it is probably yeah you probably have to work on word levels for importance but there should be ways of easily going from a word level to an utterance level yeah i reckon you can just mean it over the sentence but how about those words which do not carry any meaning at all and uhs and something like that because if we i think we should filter them if we average over whole utterance all the words and there are quite unimportant words in there but quite important words as well i think we should just disregard them maybe we should have like a cutoff so a word only gets a value and it is above a certain threshold so anything that has less than say 05 importance gets assigned to 0 or we do a pre filtering of sort of the whole thing sort of like but that like the problem with that is it is easy to do in the text level but that would mean it would still play the air in your audio unless we sort of also store what pieces we cut out for the audio yeah i think before we can like answer that specific question how we deal with that it is probably good for us to look at what the audio player is capable of doing i think we will have to buffer in audio i do not think it will be very hard i think it will be like an hour or 2 is work so what do you mean by buffering like you think directly feeding i have just built another wave fire essentially yeah just cut it in yeah i mean but not store it on the hard disk and they load it in but loaded in directly from memory so just like there is bound to be like a media wave object or something like that but it is probably a scream if it exists in java it would be probably some binary screen going in for some type i have no idea okay but it must have like classes for dealing with fires and if it has classes with concatenating as far as even doing mainly okay so that means that there is probably even if you go on on per utterance level there is still some merit on within utterances cutting out stuff which clearly is not relevant at all and that maybe also for the audio we would have to do so let us say we play the whole phrase but then in addition to that we have some information that says minus that part or something that is okay that we can do well what i think i might try and build is basically basically a class that you just feed it a linked list of different waveforms that will just string them all together with maybe 10 for a 2nd silence in between each one yeah maybe even i mean that is sort of that depends on how advanced we get maybe if we realize that there is massive differences in gain or in something you can probably just make some simple normalization but that really depends on how much time we have and how much it is necessary yeah like i do not know anything about audio and i have never seen the player so if you find that the player accept some input from memory and if it is easy to do then i guess that is that is very doable so but that means in the general structure we are actually quite lucky so we have we load into memory for the whole series of meetings just the utterances and rankings for the utterances and some information probably that says well i guess that goes with the utterance who is speaking because then we can also do the display about who is speaking we also really want the other search by who is speaking well yeah but i am still confused because i thought like that is just what jonathan said that we can not do like load a massive document of that size it does not matter because all the calculations done offline the other end i mean it should not be like 50 megabyte in rum or something it should not be massive actually 50 100 megabyte is quite big in run just thinking what is the simple so we do get an error message with the project if we load everything into the project with all the data they load so we know that does not work so our hope is essentially that we load less into it what is this lazy loading thing somebody explain lazy loading to me it just means it loads on demand it only loads when it needs to a particular type of file like when it is being accessed okay so that is that only by type of file like if the same thing is in different files would it then maybe like you know if uptrends are split over 3 or 10 or a 100 different files is then a chance maybe that it does not try to load them all into memory at the same time but just yeah i think that is idea so why does it fail then in the 1st place then it should not ever fail because then it should never but if you were in a search over the whole coppers you would have to load them all yeah but it failed right when you load it right did i like that that is interesting it has a defined baseline in what it loads for example it loads all the utterances and so on but it does not load the discourse acts and so so it can not not be it was what else work there not the summaries i need those on the mind let us check that out i will probably ask jonathan about it so alternatively if we realize that we can not do the whole thing in one go we can probably just process some sort of metadata you know what i mean like sort of for the whole series chunks representing the individual meetings or something like something that represents the whole series in a in a structure very similar to the structure in which we represent individual meetings but with data sort of always combined from the whole series so instead of having a single utterance that we display it would probably be like that would be representing a whole topic a segment in a meeting and sort of so that using the same data you mean that you basically split up the big thing into different summaries for example that you have a very top level summary and a separate file for each level i am thinking in the sense of creating a virtual meeting out of the whole meeting series sort of that is easy you just like create a new xml document yeah sort of like offline create a virtual meeting which basically treats the meeting series as if it was a meeting and treats the individual meetings within the series as if they were segments and treats the individual segments within meetings as if they were utterances so we shifted one level up and in that way we could probably use the same algorithm and just like make like one or 2 ifs that say okay if you are on whole document a whole series level and that was a double click then do not just go into that segment but load a new file or something like it but in general use the same algorithm that would be an alternative if you can not actually load the whole thing and because also like even if we maybe this whole like maybe i am worrying too much about the whole series in one thing display because actually i mean probably users would not view that one too often i think it is really that much of a problem because if it is too big what we can do is just well all the offline stuff is not really matter and all we can do is just process a bit of time like for summarisation say you wanted 100 utterances in the summary just look at the meeting take your top 100 utterances in each other meeting it scores higher and the ones already in the summary so far just replacing and then you only have to process one meeting at a time yeah but i am still worried like for example for the display if you actually if you want to display like for the whole series the information density levels based on and the only granularity you have is individual latrances that means you have to go through every single latrants in a series of 70 hours of meeting maybe we should build and store a mean measure for the segment yeah meetings as well yeah and if you make that structurally very similar to the like one level down like the way how we store individual latrances and stuff and maybe we can more less use the same code and just make a few ifs and stuff so but still so in general we are having utterances and they have a score and that is as much as we really need and they also have a time information of course and a speaker and a speaker information yeah topic segmenting yeah so in information which topic they are in yeah and probably separate to that and information about the different topics yeah so the skimming can work on that because the skimming just sort of sorts the utterances and puts as many in as it needs well yeah and then he will preserve the order when it is displayed yeah yeah it will play them in some order in which they were said because otherwise it is going to be more entertaining but that is enough data for the skimming and the searching so what the searching does is the searching leaves the whole framework goes to the sql database and gets like basically in the end gets just a time marker for where that is like that utterance that you are concerned with and then we have to find i am sure there is some way in nightexmel to just say set position to that time mark and then it shifts the whole frame and it alerts every single element of the display and the display updates yeah you can not want it you could do the old tree display as well for multiple results yeah yeah yeah but so if so so so if if some so yeah so if in that tree display somebody clicks on something you just need to find the timestamp yeah and then you sort of feed the timestamp to the night xml central manager and that central manager alerts everything that is there like alerts the skim like the audio display alerts the text display alerts the visual display and says we have a new time frame and then they all sort of do the update routines with respect to the current level of zoom so how much they display and starting position at where the or maybe the mid position of it i do not know like if you start where the thing was found or if that thing was found is in the middle of the part that we display that i do not know but that we can decide about but in general sort of it is the same thing if like whether you play and it moves forward or whether you jump to a position through search it is essentially for all the window handling it is the same event it is only that the event gets triggered by the search routine which sort of pushed it into night xmet and says please go there now so we should basically make our own xml document in memory that everyone is module changes that rather than the underlying data and then have that xml document tied to the interface why do we have to do in memory well you can make the file if you want i mean like the information is coming from offline so we probably do not even have to change the utterance document right because the whole way like the whole beauty of the night xml is that it ties together lots of different files we can just create an additional xml file which for every utterance like the utterance have ideas i presume some references so we just we tie just very very very you can just just a very short xml file which is the only information it has it has whatever a number for the weight for the information density and we just tie that to the existing utterances and tie them to the existing speaker changes but there is no idea for an utterance i think it is just for in the video where it is so how do we do that now no no it is for utterances as well i think it is just for one world so we have to i think there is segments for each utterance or split up yeah well otherwise we would probably have to go over and like add some integer that we just increment from top to bottom sort of to every utterance as an id sometimes or try to understand how nidxml ids work and maybe that is some special rules you have to follow when we use these ids yeah i am pretty sure it is already there pretty sure that is already there the utterance are so numbered the girls say the utterance of themselves are not numbered at the moment i am not quite sure i have only seen that the individual words have got an id so i guess that would be solvable if not yeah you always could have a look at the time stamps and then take the one step belong together to form an adrift no i think we would just take the segments that are already yeah if they are already there it is yeah there is segments 5 you know they smell segments okay is that a boardmarker pen actually yeah i think so like to make a list of all this stuff or we are probably somebody can do it in paper so what so the stuff we have we have utterances and speakers and weights for utterances so for every utterance sort of like the utterance has a speaker and a weight which is coming from outside or which is tied to it and there is segments which they are utterances are not they the segments are that is the impression i get 0 so sorry topic segment i meant like they are they are a super unit so so the utterances are tied to topic segments and if the timestamps are on a word level then we somehow have to extract time stamps for atrances where they start there are time stamps for the segments what segments now segments is for example when when you look at the data what is displayed in one line okay when you is that the same as utterance is that the same as utterance well it is close enough is not it it may not be exact every time 1st i compared it with what i did for the pause duration extraction and basically it is words that are added in a sequence without pauses but sometimes however there are short pauses in it and they are indicated by square brackets pause so that is but that is one segment or is the 2 segments then sometimes the annotators decided what was one segment and what was not okay but generally utterances is that which we just called sorry segments is that which we just called utterances now it is sort of like one person is contribution at a time sort of thing eating okay so yeah so we have those and then we have some fields somewhere else which have topics yeah and the topics basically they are just an id probably with a start time or something and the utterance is referenced to those topics i guess yeah so the topics do not contain any redundant thing of like showing the whole topic again but they just sort of say a number and where they start and where they finish and the utterances then say which topic they belong to yeah but i think for some annotations an utterance can have several types for example for the dialog ads and so yeah but i was thinking of the topic segmentation now for that there would only be one right because it is sort of like it is just a time window it should be here so if this lazy loading works then this should definitely fit into i mean not memory then because it would not all be memory at the same time so if you just have to sort of that information like a long list of all the utterances slash segments and like shorter smaller lists which give weight to them and even though probably if there is a lot of overhead and having 2 different files we can probably merge the weights into it offline you know what i mean like if there is a lot of bureaucracy involved with having 2 different trees and where the one ties to the other because the one is the weight for the other then it is probably quicker to just well why do not we just write it in the xml file can night handle just loading arbitrary new like attributes and stuff i mean i would have thought that they would make it over to yeah i thought that was the whole beauty that like you can just make a new xml file and sort of tie that to the other end and it is so why do we need to have 2 xml trees and then we are at one 0 yeah so no i did not mean tree no i meant just like handling 2 different files internally i was just thinking you know like if the overhead for having the same amount of data coming from 2 files instead of from one file is massive then it would probably be for us easy to just like offline put the weight into the file that has the segments yeah segments slash utterances already but that we can figure out i mean if it is going horrendously wrong the other thing is that would be using an airpar as well which means we would not have to parse anything which would be quite nice yeah pass was probably much faster than anything we have got with anyway yeah yeah no we would be completely using like the whole infrastructure basically just i mean the main difference really between our projects and theirs really is that we load a different part of the data but otherwise we are doing it the same way that they are doing it so we are just we are sort of running different types of queries on it in a sense i think we are running queries not just about what we load and what we do not load but we are running queries in the sense that we dynamically select by weights do not we then we have to check how fast that is like to say give us all the ones that whether that works with their query language whether that is too many results and whether we should you know because if it let us say i mean if their query language is strange and if it would return the 10000000 results and it can not handle it then we can just write our components in the way that they know what the threshold is so they still get all the data and just they internally say 0 no this is less than 3 and i am not going to display it or something yeah i mean process it in chunks and it get too big we can just process it all in chunks yeah i am just thinking for this whole thing of like a different level sort of cutting out different different pieces whether we do that through a query where we say give us everything that is above this and this weight or whether we keep the same infrastructure but every individual module like the player and the displays they still get sort of all the different utterances all the different pieces but they say 0 this piece i leave out because it is below the current threshold level i think we probably want to store sorry i think we probably want to store a hierarchical information density as well so like an information density score for each meeting and each topic segment because otherwise we will be recalculating the same thing over and over again yeah that would be much more efficient to do that and that will obviously make it much easier to display when do we need the one for the meet well maybe not for the whole meeting but like yeah i guess for the so when we have the display where we display the whole series then if we have for the individual topic segments within the meetings if we have ready calculated measures then we do not have to sort of extract that data from the individual utterances yeah and that is also fairly easy to store along with our segments is not it our segments is not it for the segments are we extracting some type of title for them that we craft with some fancy algorithm manually are we just taking the single most highly valued keyword utterance for the segment heading well we can start off like that i was going to start off i have got sort of halfway through implementing one that does just idf and then just i could change that to work on whatever it is probably like in the end probably it would not be the best thing if it is just the most highly ranked phrase or keyword like for example for an introduction that would most definitely not be anything that has any title anywhere similar to introduction they should be wasted by stuff like the hot spots and the keywords in your search and stuff like that also like for this part maybe if we go over with named entity in the end if i mean one of people doing dil has some named entity code to spare and just like at least for the for finding topics titles for segments just take a named entity which has a really high what is it called dfidf whatever because you would probably be quite likely if they are talking about a conference or a person that that would be a named entity which is very highly frequented in that part i do not say something about named entity so i thought he said there was not very many yeah he said they are quite sparse so that basically was do not bother basing too much of your general calculation on it but like especially if there is far as probably individual named entities which describe what a segment is about would probably be quite good like if there is some name of some conference they could probably say that name of the conference quite often even though he is right that they make indirect references to it anyway you said you are counting in implementing the idea what exactly are you it is not tf idf it is just inversed up in the region 2 because it is really easy to do basically okay so it is just like for a baseline really you can not do that in java yeah well i am halfway through it is not working yet but it will do so you are doing that on a per word level yeah okay and then averaging it as well but it is not like related to it related to the corpus at all it is just working on an arbitrary text wire at the moment okay cool i was just wondering where you had the corpus from at the moment no it seems that the data structure is not a big problem and that basically we do not have to have all these massive discussions of how we exactly interact with the data structure because most of our work is not done with that data structure in memory in the browser but it is just done offline and everyone can represent it any represent it anyway they want as long as they sort of store it in the useful xml representation in the end so it would be useful to know how everyone can store their things though yeah that would mean understanding the night xml xml sort of format and a lot more detail we should i think we should just have a long session in a computer room together and like now that we know a bit more what we want take a closer look at night at xml i have got a few hours through yeah i have had a look at the topic segments how it is stored and then yeah there is a few per meeting and it gives a timestamp and inside each one there is the actual utterance segments the list of them that occur and they are all numbered so that is where that is stored yeah i had not looked at this stuff much at all yeah so i guess if i am going to be segmenting it with the lcseg then that is the same format i would want to put it back out and sort of be equivalent who is sort of doing the central coordination of the browser application now like now like well like the integration what do you mean the integration yeah but also like all these elements like the loading and yeah integration and like handing the data loading and stuff i do not know what i think anyone has been allocated to do that yet i am sort of like i think i will take over their display just because i have started with a bit and found it found it doable yeah definitely so somebody should sort of be the one person who is you know someone be the one person who understands most about what is centrally going on with with the with a browser as a whole and where the data comes in and it is a very boring task it is also complicated yeah it could be difficult i guess we can do it like several people together it is probably just those people have to work together a lot and very close and just make sure that they always understand what the other one is doing yeah i guess the important thing is to get the crucial modules built it is simple versions of them yeah or already made versions of them for that matter yeah and then we will maybe have to prioritize prioritize somebody into integrating it yeah but i think actually like at the moment the integration comes 1st i mean sort of at the moment the building the browser comes 1st and then only comes to creating new sophisticated data chunks because that is sort of the whole thing about having a prototype system which is more or less working on chunk data but at least we have the framework in which we can then test everything and look at everything because before we have that it is going to be very difficult for anyone to really see how much the work that they are doing is making sense because you just where i guess you can see something from the data that you have in your individual x mails files that you create but it would be nice to have some basic system which just displays some stuff so maybe we should try doing something really simple like just displaying a whole meeting and like just being able to scroll through it or something like that or just adapt like they are like just sort of go from their system and adapt that piece for piece and see how we could how we could arrange like adapt that to our system does anyone want to like to sit with me and like play for 3 hours with night x mail at some point are you free after this i would not like to be i would like to go to the gym i am theoretically free but if there is any time friday yeah some awful friday you have nothing no 5 every time on wednesday wednesday i have got a 9 12 9 to 12 you have or you yeah nothing in the afternoon  i have got nothing in the afternoon any time wednesday afternoon i would be clear okay so yeah we are about just in afton tower yeah or for us till whatever one is easy to discuss stuff i do not know i will be in i am not buying the aston tower one anyway okay what time is do well i will be there from 12 i have got some other stuff that needs to on that lab so if you are not there at 12 i can just work on that okay i will just meet you in 18 in the afternoon i guess at the moment i guess at the moment nobody critically depends on like the nightexmell stuff working right now right like at moment you can all do your stuff and i can do my lsa stuff and i can even do the display to a vast degree without technically having the supplying framework working so it is not that crucial i would need the raw text pretty soon because i have to find out how i have to put the segments into bins yeah i need the raw text as well yeah i was more thinking of the whole browser framework as a running program now yeah i think we all need the raw text in different in different flavors do not we yeah i thought you said that you did looked at extracting the text yes i did but i have only just got the nodes i have to still have to order everything by the time yeah so you should be doing it in python yeah yeah i think it is quite easy yeah did you use the xml parcel in python right yeah so yeah because yeah i was having a look at it as well and i noticed the speakers are all in a separate file did you have to combine them all and then reorder them yeah that is what i thought had to just combine them and then order their timestamps according yeah so that is approach and well i was going to do so you may as well collaborate okay what i found out was that there are quite a lot of things without time stamps in the beginning in the word files yeah that is just an id or something yeah everything that is word has a time stamp yes but what are the other things that is some kind of number maybe a file number or something number or something that is in the beginning what is that i am not sure what you mean i think there are quite a lot of numbers in the beginning where there is no time stamps for the numbers i think they say quite a lot of numbers and before that there is this number the number within the xml context yeah there are numbers in the double you tag but there are no time stamps are they spoken numbers do they look like their utterances yeah there is the number task is not there that is part of the whole thing that is at the end i think yeah in the beginning as well sometimes i think all right at least i saw so so yeah we have to probably cut that out anyway for our part i do not know it is probably going to screw up a lot of our data otherwise yeah not sure if it would do i think it would not as it occurs i mean it would be because in every meeting it would probably make the yeah if you have segments for that probably and i think it even has its own annotation like digits or something so that should be really easy to i am just thinking like it would probably like the lsa would perform quite well on it we will probably find another number test quite easily yeah a constraint vocabulary with a high co occurrence of the same 9 words but what is it actually that numbers it is just the test of the system i think so but there are no time stamps i think it is also something that they said the numbers in order right they have to read numbers yeah i think it sounded like they wanted to check out how well they were doing with overlapping and stuff because basically it is like they are reading them at different speeds but you know in which order they have said i did not have a look at that so ixie had some reasons for doing it i have to be pissed off saying like numbers have every meeting and also there are different combinations of letters b r u and something like that is it everything ordered are the timestamps global or are they local i thought they were local to up to take the meeting dave if you or actually for just minutes well if you are doing idfs or whatever you call your frequency i was mixed up the name you need some dictionary for that at some point though you need to have some representation of a word as not that specific occurrence of that word token but of a given word form because you make an account for a word form yeah so we should work together on that because i need a dictionary as well i was just going to use the hash map one in java because i am only going to do it in small documents it is just like until the information density is up and running just something to get into work with did not you say that there is only going to use quite small documents to start with yeah i am just wondering for the whole thing who was it of you who said that there is some program which spits out a dictionary probably with frequency yeah it is rainbow it is i think it is just a dictionary in the 1st place okay is anyone of you for the document frequency or over total frequency you are going to have total frequencies of words then with that right like over the whole corpus sort of no i have to bin it up and so i will only have counts for each each bin something why does it need to be classified into like different segments it is because a range is a type classification system and i think it is not possible to have just one class that is the problem do not we just fill a 2nd kind of junk that you do not care about maybe we could like copies of shakespeare or something yeah sure you are sure we could do that but i do not know if that makes sense because if all we are looking for is the frequency statistics i do not see how that would be changed by the classification if we need just frequencies maybe we should just calculate them by using pearl or something or there may be enough a tool available yeah it is quite easy to just count and sort them by all in frequency using which tool are you talking about just using a pearl script be careful with that like my experience with the british national corpus was that there is far more word types than you ever think because anything that is sort of unusual generally is a new word type like any typo or any strange thing where they put 2 words together and also any number is a word type also any number is a word type of its own so you can easily end up with 100000 of words when you did not expect them so generally dictionaries can grow bigger than you think they do i do not know how many terms you can handle you can probably also pre filter like with regular expressions even just say if it consists of only digits then skip it or even if it consists any special characters then skip it because it is probably something with a dot in between which is usually not something you want to have i can not remember who has got it it might be wordnet but one of these big corpators has a list of stop words that you can download and they are just basically there is a really uninteresting boring words that we could filter out before we do that it is like that is one of the papers i read that is one of the things they did right at the beginning is they have got this big stop list and they just ignore all of those what i did for my project just ignored the 100 most frequent words because they actually end up all being articles and everything and stuff so we need like several of us need a dictionary am i the only one who needed with frequencies yeah it would be useful for me as well am i the only one who needs it with frequencies or i think that would be useful for me as well frequencies yeah well i guess as soon as we have the raw text we can probably just start with a java hash map and just hash map over it and see how far we get i mean you can probably on a machine with a few 100 megabyte ram you can go quite far you can run it on beefy so even if it goes wrong and even if it has a 1000000 well i really want to look into getting some subset of the xe corpus on the dice machines because i hate working on dice it is awful like so i can use my home machine 0 yeah burning it on a like we should be able to burn the whole corpus where has a cd ah see i ask the support about that 2 days ago in the informatics building there sorry in appleton tower 5 the ones closest 2 machines closest to the support office so i presume i would i have the exact email i think he is talking about sort of the ones that right hand corner yeah if you enter the big room in the right hand corner i think the thing is like you can only burn from the local file system so if it is from well actually i think if it is mounted you can directly burn from there but the problems i have my data on beefy and so i have to get it into the local temp director and burn it from there but you can burn it from there how big is it without are we looked at that up now we looked that up and i forgot because i could just say it going over scp one night and just leave it going all night if i had to yeah yeah we should be able to get it i do not think it was i do not think it was a gigabyte because yeah i mean wave data obviously not going to get off there see i would offer you to get it on this one and then like copied but you know what i figured out i am quicker downloading over broadband my computer than using this hard disk there is something strange about the way how they access the hard disk how they mounted which is unfortunate i will see if i can see it i suppose what operating system do you have i have got a linux box and the windows what connection do you have do you have at home yeah so if anyone of us gets it we can then just use a yeah burn it to cd  i can if i can yeah burn it to cd or yeah put it on hardness whatever question is if you are not quicker if you because you should get massive compression out of that like 50% or something with a good algorithm so if you can compress it and just put it into a temp direction i am not sure if there is enough space is how much do you get the temp the temps usually have a gigabyte free or the gigabyte free or 2 the temps yeah like i mean there is no guarantee that anything stays there but overnight it will stay and i think the temps usually have but that would have to be the temp directory of the machine you can ssh into directly of ssh yeah but i can do it from that session do not i i can compress it from a remote session in the same session yeah that is a set in the same session yeah they probably hate you for doing it they would probably like you more if you ssh into another computer compress it there and then sort of copied into the gateway machine do you think they have if you ssa you know if you ssa and they have this big warning about doing nothing at all in the gateway machine i do not know i was thinking of sshing just into some machine and then just fcping it from there to your home machine i mean it has to go through the gateway i have not figured out how to tunnel through the gateway into another machine yet it is not easy definitely that is why i end up sort of copying stuff into the temp directory at the gateway machine sorry if this is boring everybody else this is just details in how to get stuff home from we can probably just look at that together when we are meeting okay i just wondered so who is then doing the frequencies on the words because i think i will also i could also make use of it for the agreement and disagreement thing because i in my outline i talked about using the discourse acts 1st and then in the chunks of text i found looking for word patterns and so on so i would for example need the most frequent words well if you cut off all that as soon as somebody gives me the raw text of the whole thing i can probably just implement like a 5 line java hash table frequency dictionary builder and see yeah but i need it for my chance then i would you know did you not say frequency of words no sorry yeah but i i would like to look at the frequency of words in my in the regions of text i found out to be interesting so i would not need it it would have to be recalculated yeah you would have to count it yourself yeah but 1st how big how big are the chunks how big are the chunks you looking at i think it would be you know as big as the hotspot annotation so quite small then that is quite small yeah so you could just you could use just the same thing we use to build a big dictionary just to do that online so that will not take fun to build a little bit to do that big all it i mean just use the same tool yes yeah yeah so i would probably just concatenate all my text chunks and then that is the same i will 0 you do not want to have different counts for each chunk but just like sort of for something from all chunks yes 0 yeah no that is yeah so once i write an art like if i write like an algorithm which does a hash table dictionary with frequency from a raw text then the raw text can be anything so how far are you getting raw text out of it do you think i can get all the vortex but it has to be audits too okay well that is good because for the dictionary the order does not make a difference does it so yes so i will get that from you and i will write a hash table which goes over that and creates a dictionary file so for the dictionary is it okay if i do whatever word blank frequency or something just could everybody sort of start from that i mean i guess we can well that is a using tf idea for the information density it is in what is implemented in reing information gain and i am not quite sure how they calculate that right because frequency would be useful i think yeah i need frequency as well depending on the context the size and what we consider a document in the sense of calculating t the idf is going to change i think we might have a lot in common what we talked about i for my latent semantic analysis need like counts of words within a document within a segment actually within a topic segment 0 that is with rainward does i think you can just get probabilities for certain words for each document can i convert these 2 can i convert these probabilities back into frequency  i would have to look at that okay so that is what rainbow does because that is what lsa builds on like it builds a document by frequency matrix like before we get that even though but i already have i already have my code to build it up myself i do not know if i have my code already  yeah if you said you need the frequency counts actually for it would be used for yeah document but you say not for the whole well you need the raw frequency as well but you also need how many times things occur within each document and what we consider a document is going to depend on our context i think if we are looking at a whole lot of meetings we will consider each meeting a document in sort of terms of this algorithm and if we are viewing like say just a small topic segment you might look at even each utterance there is a small document it more and more appears to me that if we if we scrap the notion of the meeting as an individual thing and sort of see meetings as topic segments and have sort of like hierarchical topic segmentation instead then it is like a more coherent framework it has been a reason thought of how actually maybe it does not actually matter maybe if you just do it once at the highest level it will be fine but i was just thinking it might be difficult to calculate the tf idf offline for the different levels we might want so if we are going to allow this joint segment for example then how we can know what is going to be in context at any given time but i suppose if you just did it if you just did it globally treating the meeting of a document would probably still be work out fine because you would only be comparing the ones within the content are we using this for the for the for the for the waiting in the end now this this measure you are calculating i do not know i thought were you going to use that in the end the inflation then please if we are doing i think for the information density we should calculate it on the lowest level not on the highest 0 sorry that is what i mean like yeah for each word or whatever but across the whole lot is what i mean by the highest level across the whole corporate yeah but do not you have to like go sort of like for in a document versus the whole thing is not that how it works that you look at yeah but you would probably look at each meeting as a document i do not think that is a good idea because is not it like that we expect it to change over with different topic segments more that they are talking about something different in each different topic segment possibly because that is what relative term frequency is about that like in some context they are talking more about a certain word than in general so that would more be the topic segments then and that is what i thought as well that probably the topic segment level is the most informative yeah are they big enough to get anything new as well yeah that is the problem i do not know i am just wondering if there is ways to abandon the whole concept of of meetings and sort of just not really treating separate meetings as too much of a separate entity well yeah it is not an issue you just concatenate an xml file together but we still want to have like a notion of meetings for the user yes definitely but on algorithmic level whether we actually whether there is some way to just represent meetings as topics yeah you just like whatever you want to look at you just jam together into an xml file and that is your meeting even though bits that may have come from all over the place or whatever i do not see why that is really a big problem that is not really what i meant but i think i have to think more about what i meant i am going to be confused about everything so basically what you are saying is you can take an arbitrary amount of data and process it with the same algorithm it does not matter conceptually what that data is it could be a meeting it could be 2 utterances it could be a meeting plus half a meeting or somewhere else i am not so concerned about a meeting plus something else i am more talking about like yeah keeping the same algorithm and the same way of handling it and just saying like just this topic here happens to be like a whole meeting and it has sort of subtopics such as that sort of topic is a hierarchical concept where like the topic where there can be super topics and topics are in the end what the meetings are but in general at some level super topics are treated like topics i think it is very difficult though i mean what you do is you just build an xml file and if you wanted to get down to the utterances you go to the leaves and then if you wanted the next level up you go to the parents of those and like just go from like going to please inwards towards the branch to build up things like you know when you click on a segment it is going to have like words or whatever that are important  i am pretty sure what i am going to as long as like the algorithms are designed with it in mind i do not think it is a very good problem so sorry you can describe that again then well like say you had like say for a meeting right you have got like say a hierarchy that looks quite big like this and like the utterances come off here maybe then when whatever your algorithm is doing as long as when you are working with utterances you go for all the leads like then if you need something next up so like a topic segment you would go to here but if you were looking at say this one so it only went like this right so you say you would start with the leaves and you go 0 i want a topic segment so i go one layer up and then if you are working with just a topic segment in there it is the only thing you have to worry about and like each time you want a higher level you just need to go after the tree as long as your algorithm respects that then we can just process any arbitrary x and i follow with whatever hierarchy structure you want so that would be the serious as a whole that would be sort of a meeting say in that would be the topic 2nd i am a bit brain damaged at the moment but i think i will just sit together with you again and go through it again so i think as long as you build an algorithm that respects whatever structures in the file rather than imposing its own structure so is this and this structurally then always identical so that we can treat it with this algorithm more i mean it could be as many nodes as you want this one could be deeper maybe so then you start with all your utterances here and when you go up to get topic segments you go to here here here here here here here that might be a bit confusing though because you have things on different levels yeah i am also not sure how we can go from bottom up i have always thought it is like more that ah whatever i can not think of it at the moment probably this is all too complicated worrying about that at that moment is there anything else we should discuss no have we decided anything are we doing anything well wednesday we are meeting and looking at their implementation in some more detail to understand what is going on so we will see if we can get like a mini browser just displays 2 things sync together yeah 2 things from their stuff just to make sure that we are like understanding we understand it enough to modify it you should not have a group directory or something where we can put all our code in it may be useful i do not know who you see about that yeah how would you do that by just making like read right for everyone okay who is most free space on there i can not really do it i have probably got a repo amount because everything on my can actually be deleted because i store it all at home as well alternatively we can probably just make another directory on the bp scratch space i mean that is where i am having gigabytes and gigabytes of stuff at the moment is that guaranteed to stay there no no yeah we can ask steve have been we can get space yeah but i think you should send a support for him but i think if he sends to support he would probably be a better position yeah yeah i am sure he had to deal with that last year yeah because that would be really useful as if we had a big directory especially for transferring  having said that we allowed to take a copy of the xc office i think you should probably ask before we do it i think you said yes to that i think that was like when we were still in the seminar room i asked it once or like ask is it possible to get it off and nobody said like people were discussing about the technical practicalities but nobody said anything about being allowed to or not allowed to i mean we have access to it here and i guess it probably means that we can not give it to anybody else but if they give us access to it here sitting on a dice machine then they should be the reason why we should not be able to use it on a laptop i personally do not have too many friends who would be too keen on getting it anyway i have this really excited pirate copied thing annotated meeting data so shall we sit together tomorrow then yeah what is the text you are extracting looking like then at the end what would be best at the moment it is because i think it is actually very similar to what i did for my speaker extraction and i think i would perhaps have to change 2 lines of code to get you for each meeting a file that says from this millisecond to this millisecond there was this sequence of words and so on so that is just changing to lens of code and we need to be here so okay so do you extract the words the text as well yeah so far i extracted the durations but it is from the words file so i could just concatenate the words instead of the durations and it should i mean it should be very straightforward i can try to do it and send it to you okay and you have a look at it how it makes sense for what you want okay so have you already expected from all the files i just let it run over all the files yes wait wait wait wait i always according to the starting terms of theater what i just realized we should really keep different series completely separate for virtually all purposes just let us be careful about that because like the xsycorpos is not one meeting series several meetings series with different people meeting for completely different things yeah i mean i have one what i give you would be one file for each meeting for each meeting yeah not for each meeting series okay but like let us just be careful that whatever we sort of be merged together that is like the highest level of merging it is not the whole xcorpos but individual series yeah one meeting it might be funny to see what is summarized the whole corpus case anyway i think we might actually i think i think it is probably somewhere like well or something like it i think we might just get away with for the whole project just like looking at only one series and just doing within one series i mean you can do everything you want in one series yeah i mean there is one series that has just one meeting 0 yeah that takes a bit is the data always clearly split up a different series is it easy to just pick one the data is of the form you have 3 identification letters so b b b b d or something and that is always the same group and then after that there is a number like 0 2 so at every level everyone has to be careful to really just take even at the highest level just take stuff from one series not merge stuff from different series together because they would probably be just majorly messy is it just to see yeah so like even if you make one single text file which has the whole corpus sort of our corpus that will still be from one series only what you are producing at the moment is like individual text files that sort of have the raw text for a whole meeting as a whole world yes but i mean as the start times start for each meeting at 0 you could just probably just add the final 2nd time to the next meeting and then we would have to change the information about who on which channel it was said to by which person it was said and that is actually started in another xml document thing at the moment like which is just the words yeah that is what i am going to need yeah that is what i am going to need as well then just not print out the style of end times but if they are like so it is not an end times just for the file like is it just the 1st in the last line or is it by every single thing in no it is for every single word so what do you mean by just not print out that i do it with pearl it is just string manipulation i mean i would just if you are into it can you make a text that which just like makes just the words sure okay do you want it straight flowing because i would need something that marks the end of of is you are segmented by topic standards like is there any information that you have to the topic to the automatic topic presentation no i did not use so yeah 0 then i need something different later anyway okay but for now if you that is what nlcs say though that marks the end of each segment okay you are going to put that as an output of yours the segmentation okay so for now can you create like sort of just a dump with pure text just a few texts that i can get a dictionary and you can work on that for your topic segmentation and you would want that all in one file for all the corpus for the series but i can also deal with separate files i mean i can just write the algorithm that it loads all files in the directory or something but i mean yeah i can directly put it so if you can put it in one single mega file that would be quite useful for me so only words per meeting series even though for you would not it be easy if you had different files because i need sort of know like yeah for me it is better for finding me so give me different files as long as like if you could name them in a way that is easy to enumerate over them like whatever 12345 or something it is just anything that i can yeah they will just take over the names they have anyway yeah is it something that is easily like to enumerate over is it some some ordered pattern okay cool one series has the same 3 starting letters okay cool yeah so only words and words sometimes i do not need the times i just need the words but yeah you wanted to order okay okay anyway orders orr based start times yeah and do you want yeah sometimes they are contained in one another so it does not matter too much i think okay order and it is when do you think you will have like a primitive segmentation by some made topic segmentation but i am already made tool ready then not be really you could do once you have got the barcates okay okay cool because i will need that then when it is done yeah and hopefully that is week and i think for all the corpus it is just from what i know for my time it is 9 megabyte to have means it should be similar to have words what is 9 megabyte all the words together for the meetings that sounds quite reasonable that is 9 characters over what i am guessing that is you know because 9 megabyte is what i got for when i said for every utterance this goes from there to there okay that is for are we picking one particular series at the moment or 0 yeah i am doing it for doing it far all but it does not matter yeah i guess we can probably process the data for all different series and then check which series is the best for the presentation it sounds quite reasonable 9 megabyte i mean if you think if it is roughly a 1000000 words and 9 characters per word sound playlist i hope it will be the same for the words it is just with that yeah so i am going to build a dictionary from that like just a list of the words that maybe a list of the words with the frequencies are a list of the words sorted alphabetically or numerically what does anyone want to sustain any wishes for dictionaries i could just use it with the frequency i think until the information density means finished that would be really useful so i will create a dictionary so i will probably send a just one file of the 1st meeting to all those who need it so that you can have a look at the structure yeah and then the actual file we can probably like copy from your home directory or something like it yeah i mean if it is just for one meeting it is really not too big yeah but i am saying for the whole thing in the end and like the big thing we probably should not do by email yeah how long would it take to make the frequency counts for the java hashtable from the time i get the file i can do that in an afternoon the next morning 0 you mean how long processing time it takes no how long you would have to program something it is a borg standard algorithm i have written it for a deal just in half an hour or something similar it is just you put them in a hash table and say well if it exists already in the hash table then you increase the count by one and i probably implement some filter for filtering out numbers or something because it is quite easy in pearl as well it is just a line of code for counting all the words really how do you do that yeah it is by hashes okay well i do not know any i mean if anyone wants to do a pearl script for that that does it nicely i have no problem with that but i think i have the java code virtually ready because fadilla wrote something very similar like fadilla wrote something that counts the different occurrences well the tags if you are doing it in java do you serialize the output as well as writing it to a file sorry if you are doing it in java if you serialize the hash table yeah as well as writing it to a file i have never serialized anything really easy would not that be absolutely massive though i do not see why it be any more massive in the file and then serialize and then write the serialization to a file yeah so you want like a file which is the serialization of a half table i just say to you passing the file representation of it now because i would be using it in java anyway so i would just be building the data structure again i will check if i understand how it works i mean otherwise i can give you the code for loading a dictionary it is just it is sort of a line break separated file yeah it seems like a bit silly to be passing it over and over again kind of thing yeah yeah i will see if i understand how to serialize there is a serialized command so that gives me one mega mother of the i would have thought that i think all the collections and things implement serializable already yeah but do they automatically write to the file anyway i figured that out i think they might do yeah is that pretty much pretty much it so dave and me look at get how nightexmail works and we are i will try and i will either work some more on tf idf summarize or do the audio thing i will build a dictionary as soon as i get the text and yeah so that when do we have to meet again then with this i do not know i suppose we are just getting on with all our components how are we going to do a demonstrator next week we have to demonstrate something next week what do we have to demonstrate no not demonstrate but like did not you say that or did not we sort of agree that it would be useful to have a demonstrator of it like some primitive thing working next week yeah it suggested that we could have an initial prototype it has got to be very prototype i know i would be surprised that we can get anything walking i feel like i feel like i am let us go so i feel like hanging mid air and not really like finding a point where you can get your teeth into it and start working properly yeah i know it is all so fuzzy though i think it is because had to specify it ourselves that it is not as like focus as specification of most work yeah but at the moment but at the moment it is also an implementation level like with the data structures i am just like over these vague ideas of some trees once we start doing it it will become more than obvious i think then yeah it is just we are halfway through the project time that is what just freaks me out yeah okay so next time if you know when you start to me look i have some for use ai being here in the system everything i have done every form of computation the logistics network like is real that is real that is yeah but like nothing that has rules like i do not like rules to be a set of life and i do not like to be so you know we can not even data not like that you know america i think you know thank you