okay we start waiting for people coming late and i can say this because in this moment you cannot answer because you do not have the microphone thanks yeah yeah sorry about that matthew is still on the way so i can say whatever and you cannot reply yeah so yeah okay hang on reply coming yeah it is okay u 0 i hate these things voila okay so we can maybe get a head start have you been talking to matthew about what you have been doing or a little bit but i am m i am not sure he knows everything so basically well 1st we tried to explain why in this resc n best list re scoring from the slides to enhance the speech recognition on the meetings data is not working yeah yeah yeah so that is one thing and so what we did with alessandro is ch t performing some statistical tests to see whatever r if the if the words the the appearance of the words during the meeting is in independent of the ap appearances of the different slides yeah so the in the case if it is dependent that would mean that certain words tend to appear during certain slides and that there is a correlation and then there will be a reason like for believing that this will work yeah which was the intuitive feeling yeah yeah and is and the r result was no the resu the result was no but bu but the question is if is that result no because it is no or is it also just because there is not really enough data to be sure about anything ye in my opinion it is no because of the nature of the language i mean intuitively of course you tend to use more the words that are on the slide sure but the mass of the words actually you use are words that are common yeah okay just think that 50% of the words on average whatever corpus you take are stop words articles et cetera yeah yeah so in terms of recognition 50% goes away yeah of the rest remaining 50% i mean are all words that appear 123 times it is not going to yeah so in any case even if actually and they tend to be related or a statistically related to a single slide in any case in terms of recognition do not help at all or help a very little yeah so that is the kind of thing i mean that is the kind of measure the ind statistical independence is not on single words yeah yeah this is so sort of a feeling that we had but had not shown i think i mean i in from from an application point of view it might be interesting to make sure that your vocabulary contains all the words on the slides but from a research perspective it is not interesting you know even if actually i remember i just made some measure saying okay l let us look at how many words i do not have in the dictionary and are in the slides there would be very few yeah true it was 2% yeah almost nothing well this is h this is how the dictionary is calculated it calculated such that if you do not have the word it does not affect to a great degree the the the word error rate yeah exactly exactly so exactly so some in some sense it intuitively it sounds very good but in practical terms in terms of recognition here we we we have to be clear if you want to improve the recognition that im improves very slightly yeah sure yep it is different if you want to have other tasks where the only words that all are words m like yeah yeah in that case even the slight improvements he gets at that point they can make the difference yeah but th yeah but does the recognition no well this is yeah that is yeah and some way the statistical independence i think it can be a good explanation to show why it is happen even if it is counterintuitive yeah but that is that is what happens basically huh so matthew we were yeah could you ju just quickly rewind yeah it is we were talking that we did some experiments to see if the word during meeting the words appear independently of the slides in a sense that if certain words tend to appear during certain slides or no 0 sorry about that yeah yeah okay 0 okay and the answer is no okay and so we did a statistical test for that and i will add also that we did that on the w the words after filtering them from the removing the the stop words 0 of course yeah yeah yeah so so bec s because yeah yep okay cause it is not going to make any difference yep with the stop words okay th there would be a reason for in independence because there is such a huge mess of them but even after removing them there it it remains independent and actually so when i was doing the re scoring experiments i was doing that on one meeting that i selected as being like the best looking one it ha it had the m the most slides and now i did the same on the other meetings which were available where well it is only 3 meetings because well here there are 4 of them which are from the test set from the ami recogniser w but only 3 of them have slides yep so it is total of 3 meetings and for the others the tendency and i c cause you know i was when i was doing re scoring i was taking into account the for one given slide also the n neighboring slides and as i was in increasing the number of slides which were affecting the the utterance i was re scoring the performance was improving sli s slightly very very little but no no wait wait it is going very fast for me actually yeah thing actually it is very fast so wha 1st point the 1st thing you told me that you took the slides and you was and you had a l big dictionary and you are seeing what what are the inside the slide right what the cause i 1st statement i could not understand the the 1st thing we are talking about ex the experiment to see the if it is dependent or independent when you said the the saint the s the w yeah so we what is so yeah so could you please tell me little bit more in detail how we did it yeah so we used the pearson chi square test no no no no do not go that i i want the no i want to understand what did you mean by the wo the te thing like saying that i took a dictionary and then a word from the slide and what you are looking for i could not understand clearly clearly that ah okay it was to to see if certain words have a tendency to appear during they if they are more likely to appear during certain slides okay but do how do why do you want certain words w like wha how do you expect that certain words be be b be careful it is not exactly this because for sure there are words that tend to be so the point is you want to verify whether you can improve the recognition rate by using as an information the words that are contain in the slides yeah so basically the idea is that in the moment you are in the slide somewhere your language change accordingly to the slide exactly so the fact of having that information somewhere can help you to improve the recognition exactly but actually it is not what happens yeah and this does not happen for many reason 1st for example 50% of the words in any kind of text are stop words so are everywhere yeah and the remaining a appear so little that in the case cannot really improve that much most of the words we use actually whatever we talk about are common words okay so how to verify this i mean this is something that has been measured et cetera but still is a bit qualitative to have a quantitative measure we simply did a measure of statistical independence between the words in general okay so not some words the words in general otherwise you can see that if a words appear once basically it is 100% related to one slide huh yeah but you have to i if you want to consider in terms of recognition have to make it overall i mean what is important is not the word that appear once it is the word that appear yeah it appears in almost it is if it appears in all the slides it says a 100% correlation for me if i exactly if if there is one word that appear once all over the meeting one well yeah it appears in correspondence of one slide then of course it seems to be very related yeah yeah but very few words they represent a very little part of the word mass are like that okay so we simply used a very old test statistical test okay that basically measure the hypothesis that some way the probability of having one word in correspondence of a certain slide is simply the product of the probability of the word for the probability of the okay it is independent of each other okay that is it so that is what we did okay fine that is fine that is s certain words it should but is in general the language and not certain words because certain words for sure have a strong dependency on the slides yeah you have to okay but there are l few yeah yeah yeah fine now i am okay i am into the loop yeah and the answer to that was no and s so when extending the those experiments to the other meetings when while i was observing an improvement on the 1st meeting i was using so i was r observing an improvement as i was increasing like the the context context yeah but it is implemented on the a s r or where on the on the recognition yeah okay so there was the improvement was also increasing as i was increasing the context okay while on the other meetings it is it is it is it is really fluctuating and i it it it looks more like ps well what is the word for that statistical fluctuation fluctuations 0 okay so i mean the it is probably very dependent on the speaker as well yeah some people have a habit of just reading what they have got on their slides and some people are com are are well else interpolate extrapolate or or or quite purposefully talk about different things so that they have sort of got multi modality of you know yeah yeah yeah but so i mean i guess that is not a surprising result then the yeah it is not in terms i mean for me for example it was not in the sense that after working a little bit on language you realize that this kind of thing do not help simply because most of the words have nothing to do specifically with the subject you use yeah yeah most of the words we use that is strange but are simply necessary to build a sentence s yeah yeah there are very few content words yeah but it is true that intuitively as we are driven in our attention i mean in in our understanding we are pretty much driven by attention that we tend to spot only those words yeah sure we s go straight to the semantics of the yep exactly and in some sense intuitively it seems that it can happen i mean this is not the 1st attempt to do things like this and it never works actually i in terms of recognition for other task it can be it can be helpful yes maybe okay so you mea so you mean to say that like when one case you were saying that if you include the context and all those thing it helps in your a s r improvement 0 okay but it does not which more subjective i think this whole well i mean it just comes down to the to the mere fact that word error rate is just take the words how many of them did you get right yeah take plus or -2 or 3 or 5 or 10 words does not really make any difference huh yes yeah yeah huh so yeah even if those words at the end of the day would be quite important in any sort of search of the transcript yep exactly but at the end of the day why would not you just use the slides to search the transcript yeah you can do that yeah i if you have th slides then yeah or it could so that is why if we look on the relationship that exists between the speech and the slides the task of improving the recognition the overall sp speech recognition using the slides seems to be not very good no y you would ju i mean the i mean it so i mean have you been doing other things then i mean we cannot expect much from from it have you have you been thinking about what you would like to to do in in in place of this or or or leading on from this given what you have learnt well for now the most effort was on showing why i it is i it is it will not work especially those experiments and these statistical tests but we w were thinking for example of a task like trying to see if where a speaker is talking i mean if the speech is actually correlated with the slides where the slides which happen during the not correlated but if the be careful correlate is a dangerous word in statistic cause ours is strict yeah it seems to be the opposite as before yeah well if the associated if the speaker is actually talking about what is in the slides or not the okay or roughly basically one thing it can not be done and and here on the contrary for example the few words that are how may i say it recognized more can really make the difference yeah it is not the case if you want to see if i mean what if seen in the meeting and that okay there are presentation then yeah still the slide is there but actually the people talk about other things sure or not really other things but the slide is no longer a support for the discussion huh the focus yeah okay there are moment that actually this is the support because the people describe et cetera and there are moment where it is not is a just a background thing and we rough estimated very quickly yep but it was 13rd of the time at least in in the meeting we have seen the slide were just there i mean but they were no longer use as a support it was a discussion rather between people okay so in that case the presence or the absence and especially the frequence with which you observe the words that are on the slides can be an excellent clue indicator huh indication a clear indication whether actually does it support of the discussion or not m meeting action sort of focus it can be this it can be interpreted as a kind of focus of interest it can be simply interpreted in the sense of saying okay i mean you have this channel open there yep do i have to p taken into account or not or also in terms of action yeah when you see that s the discussion is completely m disconnected with respect to this some way it means that it is happen in something different than before yeah yep so i mean it is a kind of feature that in my opinion can be easy to detect it and that can yeah f can be interesting to do yeah relatively easy to do and in that case for example the few words that you that you get more can make the difference yeah sure yeah 0 so is that double way to to show how some way dependent on the task so you so you so okay so maybe like it can be something like in a meeting it can be like an unusual scenario for you in a presentation with a slide w and the unusual scenario that you are not using the slide at all yeah or for example what happened for example in one of the meetings is that p people were summarizing the the results of the previous meeting well yeah you know so in the beginning so the meeting started with the summary of what happened previous time and this was not related to what was on the slides yeah what are the other slide yeah ye yeah okay so i mean you could begin with s saying the a the simple task of just determining whether or not the speech is related to the slide content and which is sort of building upon what you know dong and others have worked on in terms of you know is it discussion monolog from the original m 4 data collection a y are are you also considering that you could actually look at the relationship between meetings for instance well okay what do you mean exactly well i mean it if if if these similar sort of phrases or words were discussed in this meeting and were also discussed in the previous meeting then you sort of have a a link between meetings for instance link yeah some way well there are f yeah yeah yeah yeah yeah i do not know i do not know how this you have to ches the check the statistics but this is one yeah but i i mean cause obviously you have got to think towards what you know yeah this this will not occupy ab ab you know will not occupy you for 3 years or whatever just on this i guess but have to look at all the future and so just a s huh yeah i know this we are s s talking about today short term if you want yeah yeah things just to the use the things i mean all the work they that has done basically which is a huge work okay yeah it was recycling kind of and the data we have then enough for the for a thesis of course it must be much better yeah but of course the there is only 22 different kinds of words some of them you can do it on a single meeting so this kind of feature extraction because basically that is what it is saying yes no this channel is good now is background now is foreground et cetera you can put it in that in that way yeah okay this you do it on a single meeting yeah and then there are corpus based how may i say it works so yeah that is one possibility for example finding the connection between different meetings and yep task yep then i do not know i mean at that point it becomes pretty much crucial the kind of data you have in the sense that for the way we have collected the data some way i do not know if it can be at the same time too easy in the sense that some way you have little groups of meetings extremely correlated yeah well y yeah sure so i mean it it makes it easier i mean i do not know how much significant can be at that level yeah yeah it can be i mean it is it is and because we do not have all the data collected or annotated yet it is it is very difficult to know is not it i in a sense if you want this kind of words become interesting when the corpus is really big when when you when you consider each meeting as a single item yeah yep yep yep so it is interesting when you have 10s 100s 1000s of meetings which is not the case but i mean you can do other work if you consider in terms of speaker turns or in terms of slides and that is immediately multiply yeah sure or b or as you say back channel versus yeah yeah yep yep that is another cause i th think that is quite interesting huh i mean cause it is an awful lot of speech activity which is yeah huh even summarization or so it can be one more feature that helps in but it is very short term i mean just well it is ju just few few few times we talk together and it was just sure yeah mean the very quick thing you can can have like this okay okay so i mean as far as that you have got everything to do that i presume you do not need any extra stuff from us in the immediate future no or i mean that is good we have plenty of things to do that is good no but i mean it it but for the other thing you know the alternative measures or maybe we should finish with this 1st what track i mean yeah i would not get too distracted i guess on d d if you have got 2 tracks i would like to yeah this this would be a f another subject i would like to discuss s but w but basically ju just you know if we finish discussing this 1st yeah go on yeah it is no problem okay yeah so well so ultimately now the next step what is that because i want like kind of a so at 1st you said that okay now you did this this correlation studies and they show that there is a very less correlation or in fact no correlation kind of thing and so so what next in on top of it what you are going to build i mean is the question now but i think in a sense in my opinion well there are 2 things when this statistical test it can be we have made an experiment we get a result somewhat counter intuitive we give an explanation yeah yeah and that is it i mean that is that is something that some way closes here so when it gives you the kind of answer you get is that okay to go in that in that direction maybe is not the best thing maybe to improve the recognition that way is not something you can expect so it gives the possibility to decide to go in a different direction that can be more on this let us say yeah so t i mean to the different direction now that is what i mean to say what you like to take from there this is not clearly defined yet it is not defined yet okay this is but something using this relationship between the slides and other even other tek textual support and the and the speech but what kind of problem we need a definition of some kind of problem right but i and the k and the question is also are there other sort of more sensible ways of doing it yeah for instance gaze tracking yeah i mean if if people are looking at the slides well i think the advantage eventually w i mean the advantage with this kind of things is that are much easier gaze tracking is still yeah sure and it has specific to the environment and all that sort of thing no no no just with the environment is a difficult thing yeah i mean it requires cameras with pretty good quality et cetera yeah no but but i think his question is more general yeah a and it says so i mean based on this work do you figure out a direction if they guess correctly yeah direction yeah that is that is a thing a domain a direction for your thesis for your exactly huh we go on to the next direction it should not that i it should not happen that you are doing so many things and ultimately it does not goes into a thesis exactly exactly exactly the that is that should not the problem exactly your thesis should be w in one direction one problem and should go in one or one d yeah one domain yeah not a collection of little things completely unrelated it it does not make sense but the it may be good as a c v for you but then as a thesis you will have problems defending it and all those things that is very yeah sure so what is that what would be the next possible direction you want to take well this is the question i am working on but i mean for instance it should be linked to the other stuff that you have been doing on call routing for instance and and you know error merit err error measures yes error measure i mean i think i mean i i think that it is possible to draw some sort of relationship between the 2 as we were saying i mean maybe if you you do not measure things in terms of word error rate maybe this sort of information does actually mean something yeah given that 50% of the words are not actually interesting yeah so i mean w yeah that that could be interesting to try to merge those 2 things well w w no it does not solve the problem but it is f what is it m but still does not solve the problem of saying what i yeah yeah sure sure yeah it is yeah yeah yeah what i what you are going to do for your thesis i mean as herve said i mean basically if you have to explain in 4 lines yeah exactly what will be the topic of my thesis the subject of my thesis and it is something that is not just saying i am going to do this this this and that yeah i mean because this we actually do not know but yeah in a very broad sense i mean w what you are going to investigate what kind of yeah and i s see that is going to tell us something like wha the thing is that i it it see you can totally go into the text domain and be there if i interpret correctly you know it can be s no problem but as long as it is okay but if you want to do with s like certain something with the speed and all those things together then it is a different scenario you may not want to work on all the problems someone may work on some other problem but then you you make use of that to extend your work in some and and as i also t it depends like also like i think if boss is interested in working in both speech and text you can understand very well that part is also there sure are they are interested in that or no no the boss like have a it is like instead of saying like you should like work on both side you know the text and the speech aspect of it so yeah then one basic thing i suggested even thinking about i mean is exactly this stru all these scenarios that can be a meeting that can be presentation where you have a s a relationship between speech between things that are said and some textual documents some way yeah why not try to to to to study this relationship to improve this relationship to use this relationship for whatever it is yeah indexing or annotation or whatever meta data extraction and if you want for example this little work about seeing when the slide channel is background and foreground is not beginning of this kind of thing yeah so a relationship between 2 channel that provide some kind of information yeah yeah i think i think it is maybe an interesting test scenario and we know that you are sort of basing something around speech and and text i think there needs to be one sort of extra higher goal above that so that you can motivate future research yep sure yeah that is what what is missing sure sure sure sure sure w one thing interesting is what we came up in the discussion also you about telling that detecting such scenarios where there is no relation between the slide and the person talking that may be one good starting point for you actually and then at that point y you detect something like that and then you show something like what you are saying that i if there is if there i if it is out of discussion the words are no way are going to help your recognition anyway a s r but if it is more on the the slide information is not going to help you but if it is if it is related to the sli what he is giving a presentation on the slide then you show the your a s r if you detect this even then you can show that how where you can really help or not help because you say there is no correlation now but then here it comes that there at some point there might be a correlation there no no be careful for the recognition in general there is no correlation in the sense that even if you are talking about what is on the slide most of the words you use most of the words you use are not really correlated to that are simply words yeah and consider also that we all speak english here yeah and apart of native english speakers we have a very limited vocabulary in general yeah yeah this was so we tend to use always the same words but on average you know linguistic research shows that on average the people use 500 words that is what we have a disposition so okay yeah and we say everything yeah for english it is especially low yeah we say everything with that and probably we not a native english speaker we use even less so that is gives an idea of why basically in terms of recognition it does not really help because you always use anyway the same words yeah and a few occurrences of words that are there in terms of recognition make 12% so maybe it is not enough to justify an effort to improve the recognition 0 speci then you can change of task and use a task the task we were mentioning about the back channel or foreground channel okay in that case the only words that really are important are those words that appear on both slides the th sides and the talk yeah at that point i mean a little improvement of that it becomes a big improvement i mean maybe in terms of recognition it is 05% yeah yeah but in terms of that task is maybe 20% yeah okay well another thing you have to remember is these models have been tuned to include ami data yeah so i mean perhaps if you had slides and and presentations to do with something that was off topic you did not have you had not seen any prior data then you might see a bigger contribution as well like another mic yeah but i mean essentially you have actually sort of included the information from the slides in the language model yeah that is another interesting point i mean we are using data that are fake any in any case they are simulation no real data yeah and especially the slides i think i have seen those slides i mean they are pretty much artificial okay okay they are pre pre prepared exactly th yeah most of the most of the time you have the title to do no but but not pre prepare okay but the ami dat the ami data what that like that only most of the time you knew they had the word yeah but the word you are going to present everyone is presenting the same set of slide yeah i yeah i mean everything is pretty much artificial huh that that is what i mean so you mention for example this fact that sometimes speakers tend to read yeah this is especially true when you have bullet list huh yeah yeah sometimes people really go through it did not happen in the data simply because it was not the same people prepa exactly it is not there okay ah so for example it will be possible in any case because the herve is infrastructure is ready and now we are going to collect the presentation there maybe we can see what about m l m i w the one we collected last year y the audio quality though the audio is is is very bad yeah audio is very bad right yeah yeah i i thing is really one single channel or at least a closed channel collection no you know that was collected m a little bit like this without too much care it was it was a personal effort of mael that decided to do it okay okay okay it was really good to do it it was good tha tha tha that was good to yeah it was very good but basically it was made without any specific know without thinking we want to recognize just to have them okay so the especially the audio channel is absolutely awful yeah so they are is there are still using that desktop microphone or what over there ah or yeah they had like y y they you know they had this set up with the the microphones so someone every it there was one microphone shared between every 2 people ah but they had deals with people forgetting to turn the microphone on forgetting to turn it off f it was it was meant i think they they chose that lecture theater possibly with the idea that it would be good for collecting data i mean i i am sure they could have probably had university of edinburgh host it or whatever but i th i think the you know the h c i part of it did not quite work out to how they planned yeah okay okay so ap that is fine yeah that is that is that is one of the reason for example why which mark we point pretty much on slides as a mean of indexing because at that point there is no interaction between humans and capture yeah because whenever you have microphones that you have to move it is it is it becomes complete really it is a yeah it is a yeah it is i mean it is same as us putting on a the headphones and whatever i mean even that minor thing i mean any any audio captured during that period is i is rubbish but there is no sort of system that is really built to d to be able to say when you are you are capturing that yeah yeah i understand okay so now there be would be some time wh what is we should be getting a time recording system right that so we are building this slowly i mean it is it is unfortunate i mean is is is taking time for the material and so on okay the but yeah there will be this okay what will that incorporate will it incorporate like an s m a and it will be one camera pointing very general but just for display purposes essentially yep one microphone and we are thinking to use lapel microphone you know this this thing lapel okay and so that potentially helpful hopefully helpful to to do recognition should be okay yeah and to slides through the projector okay okay so no no microphone arrays or anything like that i guess that is quite a that is a quite a bi big deal for setting up the recordings i guess wo let us say that in any case w well there are 2 reason no it it is it is quite a bit yeah problem to do that kind of thing 1st of all we are trying to do something that can be easily the the the idea is really it is something that you take you bring somewhere else and it works yeah so it must be as easy as possible yeah yep but in any case i mean the device we are doing has input channels so some way you can add as many input as we want as you want yep so we are dealing now with 3 but there are other input lines open okay yeah but probably we were the f you know the post the the thing is somewhere projector suppose there is like the the white board or whatever the presentation is over there the speaker is more probably is going to this way or that way so probably you you may not need a very big microphone either probably one or 2 yeah but let us pretend that it is not going to happen in any foreseeable future ye yeah yeah wh the i mean i am just thinking from practical perspectives you would you would have to the the ami recognition system that we have now is either for these microphones or for the microphone array there is no lapel system trained for instance so i mean that in itself it would i mean you would have some fun training some more speech recognition models although at that point we can use this but i mean i was mention the the lapel microphone just like this but if you tell me this that is very interesting because at this point we can use this kind of things yeah yeah yeah but i do not why why not so there will be other data their test data yeah that is and especially for this kind of problem i think it can be a much more useful kind of data because there is real presentations i mean is it is reality sure you know it is it is it is y n it it it it is exactly yeah it is what you should be using for this type of task it is more realistic and yeah probably it can help go in this direction of saying well let us see what happen between speech and slides i think yeah and every time we d we generate a new corpus and try to carry out new research we discover sort of areas that are too controlled you know yeah yeah yeah i mean the ami ones were a lot better in term of that than m 4 or whatever yeah but still i mean es especially when you get into higher level sort of interpretation it is yeah yeah yeah yeah yeah yeah 0 well i am i am just curious do the like the icsi meetings do they have anything other than the audio yeah do they have slides or anything along that i do not think they would no the they the the the no okay i am just thinking whether or not there an are any other sources i mean maybe chil data chil may be interesting for you but it is a l well yeah it may be interesting for your problem i think y yeah yeah cause it is i think it is going to be more like the tam type situation no no no yeah so yeah it is more like tam but yeah yep the the their collection is much like they have r t n s and all those thing people sitting and it is a lec it is a lecture a meeting collection okay that is cool yeah yeah that is cool and i is it through n n l d a or something i do not know but i mean given that it is coming from the same basis as ami i i expect there has to be some sort of relatively f liberal sort of distribution policy yeah yeah yeah it is a it is a european project anyway yeah yeah so i guess it must be available somewhere so so probably we can yeah chil may be interesting actually yeah you can look at it yeah the one thing about that and this is something that came up in the last nist evals was the fact that it is almost too uni modal you have just got one lecturer talking so once again you do not see the the complex sort of levels of interaction yeah i mean y there is no way of getting around that i mean you just have to think of a a task a challenge that is relevant to that type of recording yeah yeah which might you might use the same methods as what we have been pr talking about today but the actual sort of what you are trying to measure might need to be slightly slightly different i guess i do not yeah you know in terms of in terms of say measuring whether or not they are talking about the slides or off channel m yeah i mean you might find 98% of your data is directly referring to your slides in a t in a tam in the text but only we are measuring 13rd it is not that is is in tams at least in no i am talking now about the meeting ami meeting yeah but i am i am meaning if you move on to these corpora yeah how presentation of course on presentation it it should be 100% yeah so you might want to be doing something else basically yeah yeah yeah yeah of course no that is just something that in my opinion with the work that has been already done is something we can get pretty quickly yeah yep it is an interesting task it is more specific on meetings yeah certainly not for for presentation yep and it represent a nice task that can be measured that is clear and yeah and and at the s and at the same end of the day you are probably going to be using or learning about the same techniques that you will need for doing other things along these lines yeah yeah i in terms of actual recognition can of course help yeah so so it is it is but of course not going to be it is not going to be a kind of thesis about this okay well but no that does not make any sense it well it is it is to make a proposal basically for your yep i mean it is a 1st step s that in a sense if you want i mean that that is at least the way i see the that list i have made for my own thesis i mean you define that domain i mean which is large i mean which can be very general this n yep so interaction between speech and slides cool and then you see how all these things this 2 little works or the statistical independence and detecting when it is are 2 little works that some way fit in that very general framework okay shown from this point of view it n that things there are other things for example one that i really would like to do which is kind of fun so when you have a bullet list can i click on one of the bullet and get the piece of speech where the guy was talking about that yep the sp yeah because sometimes people just read sure sometimes people im improve is more so for example that what that is another thing that fits in that kind of framework sometimes it is a lot more yeah and that is interesting in terms of i mean 1st of all again is measurable in terms of browsing yeah in terms of retrieval in terms of indexing annotation meta data extraction yeah yeah that can be applied in many and we could do automatical speaker rating if there are good speakers or bad speakers depending if they just read well you would sure that would be you know that that is that is something you are maybe joking now that is that is a yeah at l at at least spot if they forget to talk about something and it will not let them continue on to the next slide until they have gone through all the points that is something that can be done for example that that they have and some way there are parameters that can be measured that tell you that following certain criteria there is one very easy for example yeah yeah i had the the the sp there are speaker that never look at the audience those are bad speakers in general yeah it is very wrong you have to look at the audience yeah this is one there is for example and this is something that eventually can be done if you have a certain number of words on your slides you need a certain amount of time to read them the speakers that take a time which is too close to that they are not good speaker because some way they are talking too fast they are talking fast yeah i had a problem i like find the answer for my own well that that was just through sheer weight of content though they are not yeah yeah yeah they are not giving you the time and there is a certain number of things that can be some way detected and measured that tell you whether the speaker is actually respecting some form of yeah w it is a bit of having difficult and i would not go that much in that direction b yeah i mean it is but you know you know maybe possible someday yeah so what about the figure things what happens if there is a figures coming on the slide so what do you want to do with that figure you know as in 56 you know number where where the graphemic yeah well i yeah on the s huh either stop it tables number yeah ah visual things well that is for example with we submitted a project with j jean marc to use that thing as meta data okay okay and basically one thing for example we are trying to do and and again this is something that can fit huh it will be not maybe but but if you get the project et cetera in any case it is something that jean marc and me we want to do based on the kind of graphic object you have so tables questions okay figures plots et cetera the side one for example it is a result okay try to get so that is again in terms of structuring browsing structure and retrieval okay when it is a result okay when it is an introduction okay yeah when it is although i remember discussing this like cause when neville was working on his his stuff which was just based on the slides okay if he he asked me all you know could i pre segment the slides for him yeah you know because you wanted to s you wanted to have the topic segmentation yes yes yes yes yes yes and i said well just look at the the headings yeah because anybody generally who writes the slides yeah usually in products and resources it is not really true it is not really it is not really true it is not okay it is true for me then okay yeah yeah it is not really true yeah that is 0 it is one of the most funny thing i mean you can see when you work in this thing each one of us has an idea and then it is not true at all i mean s some people actually are stick very carefully if the p if they put in the outline let us say title one title 2 then you will you find title one title 2 yep but many other people and i am one of those people for example are much more general so my outline is something like introduction and experiments the results conclusions okay yeah you do not find the same title in the slides okay okay it yeah it depends yeah it is it is really a different style of there are people that do not put for example outline that is a stylistic thing as much as anything yep we were for example in the corpus of m l m e so the one is on the demo yeah 50% of the presentation have no outline in the corpus of the tam 2004 tam presentation 25% of the presentation have no outline at all yeah which whether or not that is a good or a bad thing that is that is what yeah w it is it depends on the talk yeah presentation yeah and basically that is that is and again the outline is typical maybe in scientific presentation yeah well in that case that majority has other kind of presentation probably no yeah i mean it is it is an habit we have but you but again something we are taught at university from a very young age i mean unfortunately th n there is nothing true the structure yeah yeah sure it is all subjective nothing true is is everything is chance pretty much so that was on okay on possible work okay okay so what are you going to do between the data when you collect and this what do you want to do now by the time you get the data it is going to take another 2 months if i am right the whole capturing system when it is going to come so what is the next step well did trying to really determine the the subject of something for my proposal i mean yeah you have to submit it in like 2 months also again yeah aro around february or march yeah yeah to submit yeah so really okay it is one year right yeah yeah it is quite strictly imposed now yeah so to find something for that also with the th thinking maybe turning this work into publication the one about the for sure not maybe for sure for sure you you need it bad and also well that depends a little on on on you but try to finish with the measures and the call routing call routing where that is inte ye cause i was talking with that about that to herve since i was i will recycle this poster for i m 2 you know about about this and that could be cool to to finish it maybe fuse it with ian is work and do a journal paper he was suggesting i think we should yeah yeah after the after the hard drive crash we took the time to actually rebuild the the the system and actually train cause originally we sort of had the idea of 0 we will use a recognizer that has been trained on a large amount of telephone speech data and that should give us a better result this is yeah h 0 yeah i just as a contrast i trained one specifically on this data results were almost identical which says to me there is something wrong with the data rather than and it is nothing serious yeah it is simply that it has not been chunked it or or segmented in a way that is typical of speech resources yeah in terms of you can have like a 22nd utterance credit cards and then 20 seconds of silence coming after it yeah i see which in which when you f when you 1st think about it you just think 0 silence whatever nothing will happen but a lot of the normalization that we use i know yeah yeah i know i know i know yeah c m s c c v n so w we need a decent end point detection system running before we can we can we can do that yeah i see yeah and we are converging on having one now just in in sort of tandem with this development of this web based recognizer even if little bit of 0 this ye yeah but of course unfortunately all these things take time but i i think we should see a decent sort of improvement then yeah no no i think you yeah it is it is huge yeah not just in terms of no longer having recognition errors whenever silence appears cause you get that immediate benefit that you do not have any insertion errors but also it should actually affect the recognition on the speech segments as well and so essentially say well as there is this silence time which tend to be even more than than how it is it it it is non determinist th the so better to have a kind of silence detection so something that really takes this and they recognize only that that part yeah yeah this will improve the phoneme recognition rate that is the hard p 0 i might given the trouble we have had with this corpus i can not make any promises yeah that is what we expect to that is what it of of course of course of course and at that point even the call router that jean jean yves has made can improve its performance yeah that is the intention yep exa that is the intention there and it can be used after tool for the measure work so so you are so what you are saying is that you you noticed that when you tried other features and sure sure sure ye we trained the system just on this data like there were there were 6000 utterances that were not used for the call routing side of things that we got i think they th there were 13000 in the entire corpus 6 t half of them roughly we have thrown away so use that for acoustic training and the results from that were almost the same as what we got for the c t s model because and initially it was a s nuance trained on nuance no no no no no no y yeah it i but one can imagine a commercial system that that that s steven cox used from nuance probably had all these things or built in end point detection et cetera et cetera yeah so surely they should they should have had that actually so maybe like we had to do this end point detection and probably a little bit of final filtering and then i think you should be pretty much converging to the system yeah but the still the reason with the the they had like what are the systems we train on databases are they are chop neatly that there is not too much silence beginning not too much silence at the end sure sure then 0 yeah yeah yeah i know i know the problem they are they are j they they chopped so neatly for the it like custom made database but here it is not like that you know like huh yeah i see and there is also just things like tele receiver noise at the beginning and or at the end of the and i mean these these are things you can get rid of if it is a i see i see yeah yeah yeah yeah yeah yeah i see there are the several things i see the problem like one thing is like we can do this begin and end detection as we said or you and he can we can try to use france telecom is approach actually and that they they have this front end which which which can do a spectral yep and plus it can give you a voice activity detection okay this is f okay part f part from basis it can give okay so somethi we can just yeah we can just run it quickly and see we can just run some generic yeah and probably you will have to run it will give you the idea that is it okay like well i do not mind running it yeah i have got all the scripts set up no no no no not running i mean to say p probably he should run it or like we can ge help him to run it actually so because at some point he has to get into i am not not sure it is well i am not i am not sure i see the point actually i mean to say i i mean to say that it is okay yeah i am losi we have all the scripts but probably he should run it i mean to say i do not know yeah whatever i do not know thing th i do not know whether that a it is a run run run what exactly the all these experiments probably i do not know whether that is a for the the the call routing itself or the recognition the phoneme recognition the they are the scri because up to probably maybe we shall do it yeah for to make it faster we should ru it run maybe if we run f we ourself it it is very fast we might as well cause i mean we have up until now you have not been running any recognition on and neither did jean yves on that on that data yeah yeah no no yeah i have all jean yves is scripts and programs for performing the call routing i think starting from the phoneme sequences okay okay yeah yeah no but i mean to say that if you if you if you if you if you give him the scripts and ask him to run the 0 sure but i mean the scripts are so automated it is just a matter of sitting submitting a job to the cluster and no yeah but it is yeah i i know that i do not know how to use the cluster i do not i do not see the point of that i do not mind running it yeah that would it would be like half an hour is work in a day to run the recognition scripts i mean i i i i know you are saying artem should familiarize himself with the a s r systems or whatever yeah 0 yeah fine but i do not i d i do not think this is a point that is worth sort of worrying about at this stage anyway yeah i mean yeah if it is just running a script i do not know if it is well cause i am i am sort of assuming that that we are going to have so an end point detection regardless of this working on this nuance corpus yeah that huh no no no no i the end point system is different thing no so that is and we need it for ami plus other thing we need it for ami and so but the i am to all saying yeah we need it for the online demo for all these things yeah i i am but this is basically detecting speech and no non speech it is it is a lot more difficult than you would think it is n it no no no it is not speech non speech yeah it is this problem i hear every time on the speech meeting how do we detect speech and not speech it is it is it is yeah well it is speech and any and any other class of sound ah and that is the problem yeah or 0 okay it is any anything nonsense or else it beeped and is it is not just speech and silence it is speech and and okay and any other class yeah it is not silent it is you can say that is very yeah if it was just silence it would be easy but and in controlled places you can not just assume it is it is you know any energy below a certain threshold and this is what everybody sort of claimed the problem was solved with years ago but but who really can do when somebody will really be able to do that i mean detect the speech from non speech it will be a little revolution no in the well i mean this is once again a yeah sure but i mean this is once again a very constrained task again i mean it is it is it is you you are almost assured it is only going to be a single speaker on the microphone so i mean the the only sorts of noise you are going to have to deal with is is is hiss or impulsive i would say and it can be it can be anything yeah or or that would be a good start breathing yeah no anyway so we okay so we will finish this call routing stuff what do you say all thi this is okay with you right so we will k because i think it it it has not been even submitted outside also the work what jean yves did yet no no no no basically because it was not at at a level that could be anyway yeah not done that yet tried yeah he still yeah so the idea was good i think yeah with a better phoneme recognition rate definitely it could improve also the the call routing performance yeah yeah it was showing that you have not heard from him i take it either nope no but we can we can find okay well it would be nice to get the thesis i think there is a sort of just a research report sort of basis for this yeah yeah yeah yeah yeah yeah but we can some way find a guy but i mean this should should be have all details so it should be relatively easy to at least this is the last news i had and then i can try to get him back okay yeah apply some pressure yeah so yeah it it is pretty much good time also no and he c he can and when do you intend to start writing the proposal probably you should start i think s slow yeah well this is basically in my 3 main jobs proposal this little bit start thinking no no you you should not wait till the end that is what i mean to say n no it should be looking towards the start of next year basically ah yeah yeah yeah but yeah yes ah yeah sure because you but should not make it clearer before on the on the on what i i will write the is it so i mean you should because it and even a 1st submission using this work will make things clearer for me writing them down and f for it will help i think for that also okay yeah no going very pra pragmatic there is a submission deadline for example 31 december for conference on multi media and expo where i think these 2 little works on okay which is yeah it can go there yeah yeah so it can be see and it is nice to get some feedback on i mean that feedback will probably come after your proposal possibly no no no but i think it is it is good if he submits his pr proposal as a research report you put here submit it for the conference it is quite good for the proposal directly yeah yeah yeah yeah exactly it is as a proposal yeah because as far as i understand this work on the phoneme anyway still takes some development time in the sense that no yeah no i think we we may have to like it can have a it is it is kind of like maybe take like 2 or 3 days you know like to sit out and say one converge to one approach and then just do without that yeah okay n yeah sort out any bugs or whatever i see yeah and probably yeah so that is the thing we have to sort out the bug kind of thing and see how reliable is this end point detection for us yeah i i i would prefer to go back to the c t s models i think as well cause i think i think they did work a little bit better and yeah yeah yeah i think then we can go back to c t s actually anyway the good news is that there are results so yeah you are you have just to to decide yeah what what what what you want to work 1st and and and finalize yep i think you can really start finalizing yeah well i think 1st this submitted and this will help for the proposal yeah and since we cannot start immediately with the measures i think that is the best thing to do okay and yeah so that is this so you wanted to talk something more than that well we obstructed you with that you wanted to talk something different also well no no well for me it is i think we it is fine or maybe another meeting yeah it is okay hayes is another one is not it and then i say i do not remember 0 okay sorry i was like cause i saw you sli i saw you looking at your watch before and i like wow no i was just looking this going to have to s have this ami corpus linked in no time okay we are not out of time you know yeah and i think that will be exactly one hour and fill the tape yeah so the tape is ta tape is at the end the tape is 60 minutes yeah yeah it is happened it happened with my friend 0 okay ah so i was right like he was taking pictures pictures pictures and it was going 373840 even so i said there is some problem with yours your the way he put the roll inside ah yeah huh there is some problem it has got stuck somewhere it can never go to 4140 yeah i 3738 is fine not normally yeah it could go to 3738 yeah 4041 is but see he has already yeah we have been told do not listen yeah okay yeah okay fine
i am going to okay we start waiting for people coming late i can say this because in this moment you cannot answer because you do not have the microphone so i can say whatever and you cannot reply okay i am on reply coming yeah it is okay i hate these things voila okay so we can maybe get a head start have you been talking to matthew about what you have been doing a little bit but i am not sure he knows everything so basically well 1st we try to explain why this nbest list is scoring from the slides to enhance the speech recognition on the meeting is data is not working so that is one thing and so what we did with alessandro is performing some statistical tests to see see whatever if the if the words the appearance of the words during the meeting is independent of the appearances of the different slides yeah so the in the case if it is dependent that would mean that certain words tend to appear during certain slides yeah which is that there is a correlation and then there will be a reason like for believing that this will work and the result was no but the question is is that result no because it is no or is it also just because there is not really enough data to be sure about anything in my opinion it is no because of the nature of the language i mean intuitively of course you tend to use more the words that are on the slide sure but the mass of the words actually use are words that are common just think that 50% of the words on every whatever corpus you take are stop words yeah articles etc so in terms of recognition 50% goes away yeah of the rest remaining 50% i mean are all words that appear 123 times yeah so in any case even if actually they tend to be related or statistically related to a single slide in any case in terms of recognition do not tell but all it is not going to very little so that is the kind of thing i mean that is a kind of measure the statistical independence is not on single words overall yeah yeah which basically means in terms of recognition that does not tell this is sort of a feeling that we had but had not shown i think i mean from an application point of view it might be interesting to make sure that your vocabulary contains all the words on the slides but from a research perspective it is not interesting you know even if actually i remember he has made some measure saying okay let us look at how many words i do not have in the dictionary and are in the slides there would be very few it was 2% well this is how the dictionary is calculated it is calculated such that if you do not have the word it does not affect to a great degree the word error rate exactly so in some sense intuitively it sounds very good but in practical terms in terms of recognition here we have to be clear if you want to improve the recognition that improves very slightly it is different if you want to have other tasks where the only words that actually play a role are those words in that case even the slight improvements you get at that point that can make the difference but does a recognition no well this is yeah and in some way the statistical independence i think it can be a good explanation to show why this happened even if it is counterintuitive but that is what happens just quickly rewind sorry about that yeah we were talking that we did some experiments to see if the word during meeting the words appear independently of the slides in the sense that if certain words tend to appear during certain slides or no and the answer is no and so we did a statistical test for that and i will add also that we did that on the word after filtering them from the removing the the stop words of course yeah yeah so just not going to make any difference because yeah with the stop words okay there would be a reason for independence because there is such a huge mass of them but even after removing them it remains independent and actually so when i was doing the rescuing experiments i was doing that one meeting that i selected as being like the best looking one it had the most slides and now i did the same on the other meetings which were available where well it is only 3 meetings because well there are 4 of them which are from the test set from the amu recognizer but only 3 of them have slides so it is total of 3 meetings and for the others the tendency like because you know i was when i was doing rescoring i was taking into account for one given slide also the neighboring slides and as i was in increasing the number of slides which were affecting the the utterance i was rescoring the performance was improving slides slightly very very little but it is going very fast for me actually that is the thing actually it is very fast so the 1st point the 1st thing you told me is that you took the slides and you was and you had a big dictionary and you were seeing what words are there inside the slide right what the way 1st statement i could not understand when you said the the the 1st thing we were talking about takes the experiment to see the if it is dependent or yeah so yeah so yeah so could you please tell me little bit more in how we did it yeah so we use the pearson key square no no do not go there i want the no i want to understand what did you mean by the the thing like saying that i took a dictionary and then a word from the slide and what you are looking for i could not understand clearly there it was to to see if certain words have tendency to appear during if they are more likely to appear during certain slides okay but how do you what certain words like how do you expect a certain words be careful it is not exactly like this because for sure there are words that tend to be so the point is you want to verify whether you can improve the recognition rate by using as an information the words that are contained in the slide so basically the idea is that in the moment you are in the slide way your language change accordingly to the slide so the fact of having that information someone can help you to improve the recognition exactly but actually it is not what happens and this does not happen for many reasons 1st for example 50% of the words in any kind of text are stop words so everywhere and the reminding appear so little that in the case cannot really improve that much most of the words we use actually whatever we talk about are common words words so how to verify this i mean this is something that has been measured etc but still is a bit qualitative to have a quantitative measure we simply did a measure of statistical independence between the words in general so not some words the words in general otherwise you can see that if a words appear once basically it is 100% related to one slide but you have to if you want to consider in terms of recognition performance you have to make it overall what is important is not the word that appear once it appears in almost if it appears in all the slides it says a 100% correlation for me if there is one word that appear once all over the meeting it appears in correspondence of one slide then of course it seems to be very related but very few words i would like that they represent a very little part of the word mass so we simply used a very old test test statistical tests that basically measure the hypothesis that some way the probability of having one word in correspondence of a certain slide is simply the product of the probability of the word for the probability of the stairs that is it so that is what we did but it is in general the language and not certain words certain words because certain words for sure have a strong dependence on the slides but there are few yeah okay fine now i am okay i am into the loop and the answer to that was no and so when extending those experiments to the other meetings when while i was observing an improvement on the 1st meeting i was using so i was observing an improvement as i was increasing like the context yeah but the improvement is on the asr or on the recognition yeah okay so there was the improvement was also increasing as i was increasing the context okay well on the other meetings it is it is it is it is it is really fluctuating and it it it looks more like well what is the word for that statistical fluctuation okay so i mean it is probably very dependent on the speaker as well some people have a habit of just reading what they have got in their slides and some people interpolate extrapolate or quite purposefully talk about different things so that they are sort of got multi modality of you know but so i mean i guess that is not a surprising result then it is not in terms of me for example it was not in the sense that after working a little bit on language you realize that this kind of thing do not help simply because most of the words have nothing to do specifically with the subject we use most of the words we use that is strange but are simply necessary to build a sentence there are very few content words but it is true that intuitively as we are driven in our attention i mean in our understanding we are pretty much driven by attention that we tend to spot yeah sure we go straight to the semantics exactly and in some sense intuitively it seems that it can help i mean this is not the 1st attempt to do things like this and it never works actually in terms of recognition for other tasks maybe it can be it can be okay helpful so you mean so you mean to say that like one case you were saying that if you increase the context and all those things it helps in your ais improvement but it does not it is more subjective i think this whole well i mean it just comes down to the mere fact that word error rate is just take the words how many of them did you get right yeah take plus or -2 or 3 or 5 or 10 words does not really make any difference yeah so yeah even if those words at the end of the day would be quite important in any sort of search of the transcript but at the end of the day why would not you just use the slides to search the transcript yeah if you have the slides then yeah so that is why if we look on the relationship that exists between the speech and the slides the task of improving their recognition the overall speech recognition using the slides seems to be not very good you cannot expect much from it so i mean have you been doing other things then have you been thinking about what you would like to do in place of this or leading on from this given what you have learned well for now the most effort was on showing why it is it will not work especially those experiments and these statistical tests but we were thinking for example of a task like trying to see if see if a a speaker is talking i mean if the speech is actually correlated with the slides which happened during the not correlated but if the be careful correlate is a dangerous one it seems to be the opposites before well if the associated if the speaker is actually talking about what is talking about what is in the slides or not okay we are roughly basically one thing it can be done and here on the contrary for example the few words that are i may say recognize more yeah can really make the difference it is in the case if you want to see if i mean what you have seen in the meeting in that okay the representation then yeah still the slide is there but actually the people talk about other things or not really other things but the slide is no longer a support for the discussion there are a moment that actually this is the support because the people describe etc and there are a moment where it is not it is just a background thing and we are afterestimated very quickly but it was 13rd of the time at least in the meeting we have seen the slide were just there i mean but they were no longer used as a support it was a discussion between people so in that case the presence or the absence and especially the frequency with which you observe the words that are on the slides can be an excellent clue a clear indication whether actually does the support of the discussion meeting action sort of i guess it can be this it can be interpreted as a kind of focus of interest it can be simply interpreted in the sense of saying okay i mean you have this channel open there to where have to take it into account or not also in terms of action yeah when you see that the discussion is completely disconnected with respect to this some way it means that it is happening something different than before so i mean it is a kind of feature that in my opinion can be easily detected and that can be interesting to do relatively easy to do and in that case for example the few words that you get more can make the difference so it is a double way to show how it is a way depending on the task so you okay so maybe like it can be something like in a meeting it can be like an unusual scenario for you in a presentation with a slide and the unusual scenario is that you are not using the slide at all yeah or for example what happened for example in one of the meetings is that people were summarizing the results of the previous meeting in the beginning so the meeting started with a summary of what happened the previous time and this was not related to what was what was what are the yeah okay so i mean you could begin with seeing the the simple task of just determining whether or not the speech is related to the slide content which is sort of building upon what you know dong and others have worked on in terms of you know is it discussion monolog from the original m 4 data collection are you also considering that you could actually look at the relationship between meetings for instance what do you mean well okay well i mean if these similar sort of phrases or words were discussed in this meeting and were also discussed in the previous meeting then you sort of have a link between meetings for instance yes some way well there are i do not know i do not know how this you have to check the statistics i mean because obviously you have got to think towards what you know this one occupy you know will not occupy you for 3 years or whatever just on this i guess i have to look at all the future yeah no this is we are just talking about a very short term if you want things just to use the things i mean all the world that he has done i think which is a huge work recycle and the data we have then for a thesis of course it must be much better but of course there is a way to different kinds of works some of them you can do it on a single meeting so this kind of feature extraction because basically that is what it is saying yes now this channel is good now is background now is foreground etc you can put it in that way this you do it on a single meeting and then there are corpus based and i see it works so yeah that is one possibility for example finding the connection between different meetings then i do not know i mean at that point it becomes pretty much crucial the kind of data you have yeah in the sense that for the way we have collected the data some way i do not know if it can be at the same time too easy in the sense that some way you have little groups of meetings extremely correlated sure so i mean it makes it easier i mean i do not know how much significant can be at that level yeah i mean it is and because we do not have all the data collected or annotated yet it is very difficult to know in a sense if you want this kind of works become interesting when the corpus is really big yeah when you consider each meeting as a single item so it is interesting when you have 10s 100s 1000s of meetings which is not the case but i mean you can do other work if you consider in terms of speaker dance or in terms of slides and this immediately multiple or as you say back channel versus yeah yeah yeah because i think that is quite interesting i mean because there is an awful lot of speech activity which is yeah so it is can be one more feature that helps in but it is very short term i mean just you just a few few few times we talked together and it was just i mean very quick thing you can have like this so i mean as far as that you have got everything you need to do that i presume you do not need any extra stuff from us in the immediate future i mean that is good but for the other thing you know the alternative measures or maybe we should finish with this 1st yeah i would not get too distracted i guess if you have got 2 tracks i would like to yeah this this would be a another subject i would like to discuss but yeah i gone yeah it is no problem but basically just know if we finish discussing this okay so well so ultimately now the next step what is that i want like were kind of a so 1st you said that okay now you did this this correlation studies and you show that there is a very less correlation in fact no correlation kind of thing and so so what next and on top of it what you are going to build is the question now but i think it has sense in my opinion what are 2 things this statistical test it can be we have made an experiment we get a result somewhat counterintuitive we give an explanation yeah and that is it i mean that is something that some way closes here so it gives you the kind of answer you get is that okay to go in that direction maybe it is not the best thing maybe to improve the recognition that way is not something you can expect so it gives you the possibility to decide to go in a different direction that can be more on this yeah so i mean to the different direction and that is what i mean to say what you would like to take from there so it is not clearly defined yet it is not defined yet okay but something using this relationship between the slides and other even other textual support and the speech  but what kind of problem we need the definition of some kind of problem right problem right and the question is also are there other sort of more sensible ways of doing it for instance gaze tracking i mean if people are looking at the slides well i think the advantage eventually with this kind of things is that are much easier gaze tracking is still yeah sure and it has specific to the environment and all that sort of thing is a difficult thing i mean it requires cameras with pretty good quality etc no but i think his question is more general yeah and i say so based on this work do you figure out a direction if i get correctly the direction yeah that is the thing what is the next direction it should not happen that you are doing so many things and ultimately it does not go into a thesis exactly that should not be the problem the thesis should be in one direction one problem one domain yeah not a collection of little things completely unrelated it does not make sense it may be good as a cv for you but then as a thesis you will have problems defending it and all those things so what is that what would be the next possible direction you want to take well this is the question i am working on but i mean for instance it should be linked to the other stuff that you have been doing on call routing for instance and you know error error measures yes i mean i think i mean i think it is possible to draw some sort of relationship between the 2 as we were saying i mean maybe if you you do not measure things in terms of word error rate maybe this sort of information does actually mean something given that 50% of the words are not actually interesting so i mean yeah that could be interesting to try to merge those 2 things but still does not solve the problem yeah i solve the problem but what is it yeah sure sure what i what you are going to do for your thesis i mean as i was said i mean basically if you have to explain in 4 lines yeah exactly what will be the topic of my thesis the subject of my thesis and it is something that is not just saying i am going to do this this and that because this you actually do not know but yeah in a very broad sense and what you are going to investigate see that is going to tell something like the thing is that see you can totally go into the text domain and be there you know it can be it is no problem but as long as it is okay but if you want to do with like something with the speech and all those things together then it is a different scenario you may not want to work on all the problems someone may work on some other problem but then you make use of that to extend your work and something and as i also it depends like also i think if boss is interested in working in both speak and text if i understand very well that part is also there sure are they are interested in that in that or not the boss like heavy like says it is like you should like work on both sides you know the text and the speech aspect of it so then one better thing i suggested him and he is thinking about and he is exactly this there are all these scenarios that can be meeting that can be presentation where you have a relationship between speech between things that are said and some textual documents so why not try to study this relationship to improve this relationship to use this relationship for whatever it is indexing or annotation or whatever metadata extraction and if you want for example this little work about seeing when the slide channel is background and foreground is not an beginning of this kind of thing so a relationship between 2 channels that provide some kind of information i think it is maybe an interesting test scenario and we know that you are sort of basing something around speech and text i think there needs to be one sort of extra higher goal above that so that you can motivate future research sure sure sure sure one thing interesting is what we came up in the discussion also was telling that detecting such scenarios where there is no relation with it the slide and the person talking that may be one good starting point for you actually and then at that point you detect something like that and then you show something like what you were saying that if it is out of discussion the words are no way are going to help your recognition anyway asr but if it is more on the the slide information is not going to help you but if it is related to what he is giving a presentation on the slide then you show the asr means if you detect this event then you can show that how where you can really help or not help because you say there is no correlation now but then here it comes at some point there might be a correlation there do not know be careful for the recognition in general there is no correlation in the same even if you are talking about what is on the slide most of the words you use most of the words you use are not really correlated to that they are simply words and consider also that we all speak english here and a part of native english speakers we have a very limited vocabulary in general so we tend to use all the same words but on average you know linguistic research shows that on average the people use 500 words that is what we have a disposition yeah every english is especially low yeah we say everything we do and probably we not an active english speaker we use even less so that gives you an idea of why basically in terms of recognition it does not really help because you always use anyway the same words and the few occurrences of words that are there in terms of recognition make one so maybe it is not enough to justify an effort to improve the recognition then you can change of task and use a task the task we were mentioning about the back channel or program channel in that case the only words that really are important are those words that appear on both slides at that point i mean a little improvement of that it becomes a big improvement i mean maybe in terms of recognition is 05% but in terms of that task is maybe 20% okay another thing you have to remember is these models have been tuned to include ami data so i mean perhaps if you had slides and presentations to do with something that was off topic you did not have you had not seen any prior data then you might see a bigger contribution as well but i mean essentially you have actually sort of included the information from the slides in the language model that is another interesting point i mean you are using data that are fake in any case there are simulation no real data and especially the slides i think i have seen the slides i mean they are pretty much artificial okay pre prepared exactly most of the time you have the title to do the ami data was that like that only most of the time you know the head what are the 3 slides you are going to present everyone is presenting the same set of slides yeah i mean everything is pretty much artificial that is what i mean so you mention for example this fact that sometimes speakers tend to read this is especially true when you have ballot list sometimes people really go through it did not happen in the data simply because it was not the same people prepare exactly okay so for example it will be possible in the case because everything infrastructure is ready and now we are going to collect the time presentation there maybe we can see what about mlmi the one we collected last year the audio quality is very bad right yeah they did not have one single channel or at least a closed channel collection no you know that was collected a little bit like this without too much care it was a personal effort of mail that decided to do it it was very good it was good it was very good but basically it was made without any specific you know without thinking we want to recognize just to have them so especially the audio channel is absolutely so they are still using that desktop microphone or whatever there yeah they had like you know they had the setup with the microphones so someone there was one microphone shared between every 2 people but they had deals with people forgetting to turn the microphone on forgetting to turn it off it was meant i think they chose that lecture theater possibly with the idea that it would be good for collecting data i mean i am sure they could have probably had university of edinburgh hosted or whatever but i think the you know the hci part of it quite work out to how they planned okay that is one of the reason for example why would mark we point pretty much on slides as a mean of indexing because at that point there is no interaction between humans and capture because whenever you have microphones that you have to move it becomes completely i mean the same as us putting on the headphones and whatever i mean even that minor thing any audio captured during that period is rubbish but there is no sort of system that is really built to be able to say when you are okay so now there we sometimes move this we should be getting a damn recording system right we are building this slowly and it is it is unfortunate and it is is taking time for the material and so but yeah there will be this thing okay what will that incorporate will they incorporate like an sma and a it will be one camera pointing very general but just for display purposes essentially one microphone and we are thinking to use a level microphone you know this radio thing and so that potentially helpful hopefully helpful to the recognition and slides through the projector okay so no microphone arrays or anything like that i guess that is quite a big deal for setting up the requirements deal for setting up the recording let us say that in any case it is a problem to do that water tourism 1st of all we are trying to do something that can be easily the idea is really it is something that you take you bring somewhere else and it works so it must be as easy as possible but in any case i mean the device we are doing has input channels so you can add as many inputs as you want so we are dealing now with 3 but there are other input lines open yeah but probably since the you know the post the thing is somewhere over project suppose there is the whiteboard or whatever the presentation is over there the speaker is more probably going to this way or that way so probably you may not need a very big microphone area probably one or 2 yeah let us pretend it is not going to happen in any foreseeable future yeah i mean i am just thinking from practical perspectives you would you would you would have to the ame recognition system that we have now is either for these microphones or for the microphone array there is no lapel system trained for instance i mean that in itself would i mean you would have some fun training some more speech recognition models at that point we can use this i mean i was mentioning the lepple microphone just like this but if you tell me this that is very interesting because at this point we can use this kind of things of things why not so there will be other data other test data and especially for this kind of problem i think it can be a much more useful because there is real presentations sure it is reality it is what you should be using for this type of task exactly it is more realistic and probably it can help go in this direction of saying well let us see what happened between speech and slides and every time we generate a new corpus and try to carry out new research we discover sort of areas that are too controlled i mean the ame ones are a lot better in terms of that than m 4 or whatever but still i mean especially when you get into higher level sort of interpretation it is yeah yeah yeah 0 well i am just curious do they like the ixie meetings do they have anything other than the audio do they have slides or anything along that i do not think they would okay i am just thinking whether or not there are any other sources i mean maybe chill chill may be interesting for you but it is well yeah it may be interesting for your problem i think yeah because it is i think it is going to be more like the tam type situation so yeah it is more like them but yeah yeah their collection is much like they have an audience and all those things people sitting okay that is cool it is a lecture meeting collection are they going to release it through lda or something i do not know but i mean given that it is coming from the same basis as ami i expect there has to be some sort of relatively liberal sort of distribution policy it is a european project anyway so probably yeah it should be interesting actually yeah the one thing about that and this is something that came up in the last nist evals was the fact that it is almost too unimodal you have just got one lecturer talking so once again you do not see the complex sort of levels of interaction i mean there is no way of getting around it i mean you just have to think of a task a challenge that is relevant to that type of recording which you might use the same methods as what we have been talking about today but the actual sort of what you are trying to measure might need to be slightly different i guess you know in terms of say measuring whether or not they are talking about the slides or off channel i mean you might find 98% of your data is directly referring to your slides in a tam but now we were measuring 13rd it is not that is in terms at least in no i am talking now about the yeah but i am meaning if you move on to the 0 presentation of course in presentation it should be 100% so you might want to be doing something else basically of course no that is just something that in my opinion with the work has been already done is something we can get pretty quickly it is an interesting task it is more specific on meetings certainly not for presentation and it represents a nice task that can be measured that is clear and at the end of the day you are probably going to be using or learning about the same techniques that you will need for doing other things in terms of action recognition can of course help so it is but of course it is not going to be it is not going to be a kind of thesis about this it does not make any sense it is to make a proposal basically for you in a sense if you want i mean that is at least the way i see at least i have made for my own thesis i mean you define that domain i mean which is large i mean which can be very general so interaction between speech translides cool and then you see how all these things these 2 little works so the statistical independence and detecting when it is are 2 little works that some way fits in that very general framework okay from this point of view there are other things for example one that i really would like to do which is kind of fun so when you have a ballot list can i click on one of the ballot and get the piece of speech where the guy was talking about that sometimes people just read sometimes people improve it is more for example that is another thing that fits in that kind of framework and that is interesting in terms of 1st of all again is measurable in terms of browsing in terms of retrieval in terms of index in annotation metadata extraction we could do automatic speaker rating if they are good speakers or bad speakers depending if they just read it well that is something you are maybe joking now that is something that can be done for example a bleas spot if they forget to talk about something and it will not let them continue on to the next slide until they go to the roll of points and so why there are parameters that can be measured that tell you that following certain criteria there is one very easy for example there are speakers that never look at the audience those are bad speakers in general it is very wrong you have to look at the audience this is one there is for example and there is something that eventually can be done if you have a certain number of words on your slides you need a certain amount of time to read them the speakers that take a time which is too close to that they are are not good speaker because some way they are talking fast yeah they have a problem i like 5 years till for my own they are not well that was just through a sheer weight of content they are not giving you the time and there is a certain number of things that can be some way detected and measured that tell you whether the speaker is actually respecting some form of something like it is a bit i mean difficult and i will not go that much in that direction but you know it is possible some day so what about the figure things what happens if there is the figures coming on the slide so what do you want to do with that figure on the 56 the table number yeah well i visual things well that is for example we submitted a project with jamarck to use that thing as metadata okay and basically one thing for example we are trying to do do and again this is something that can fit and it will be not after maybe but if you get the project etc in any case it is something that you mark and me we want to do based on the kind of graphic object you have so tables questions figures plots etc the side one for example it is a result try to get so that is again in terms of structuring browse and structure and retrieval okay when it is a result when it is an introduction okay yeah when it is a although i remember discussing this like because when neville was working on his stuff which was just based on the slides he asked me i will you know could i pre segment the slides for him you know because you wanted to have the topic segmentation and i said well just look at the headings the headings because anybody generally who writes a introduction reserves it is not really true it is true for me then okay yeah it is not really true yeah that is one of the most funny thing i mean you can see when you work in this thing each one of us has an idea and then it is not true at all i mean some people actually are stick very careful if they put in the outline let us say title one title 2 then you find title one title 2 but many other people and i am one of those people for example are much more general so my outline is something like introduction yeah and experiments and results conclusions and you do not find the same title in the slides okay okay it is really a different style it is a stylistic thing there are people that do not put for example outline we were for example in the corpus of mlme so the one is on the demo yeah 50% of the presentation have not no outline in the corpus of the tam 2004 tam presentation 25% of the presentation have no outline at all which whether or not that is a good or a bad thing it depends on the talk yeah well it is and basically that is and again the outline is typical maybe in scientific presentation but in that case the majority has other kind of presentation probably no i mean it is an habit we have but something we are taught at university from a very young age there is nothing true yeah sure it is all subjective not in true is everything is changed pretty much so yeah that was some okay so what are you going to do between the data when you collect and this what do you want to do now by the time you get the data it is going to take another 2 months if i am read the whole capturing system and it is going to come so what is the next step well trying to really determine the subject something for my proposal i mean yeah you have to submit it in like 2 months also again yeah around february or march yeah yeah it is one year right yeah yeah it is quite strictly imposed now so find something for that also we are there thinking maybe turning this work into publication the one about the not maybe for sure you did it bad and also well well that depends a little on you but try to finish with the measures and the color routing because i was talking with that about that to erv since i would recycle this poster for im 2 you know about this and that could be cool to finish it maybe fuse it with ian is work and do a journal paper he was suggesting after the hard drive crash we took the time to actually read rebuild the system and actually train because originally we sort of had the idea of we will use a recogniser that has been trained on a large amount of telephone speech data and that should give us a better result just as a contrast i trained one specifically on this data results are almost identical which says to me there is something wrong with the data rather than and it is nothing serious it is simply that it has not been chunked or segmented in a way that is typical of speech resources in terms of you can have like a 22nd utterance credit cards and then 20 seconds of silence coming after i see which in which when you when you 1st think about it you just think 0 silence whatever and nothing will happen but a lot of the normalization techniques that we use yeah yeah cms cvn so we need a decent endpoint detection system running before we can we can i see and do that yeah and we are converging on having one now just in sort of tandem with this development of this web based recogniser but of course unfortunately all these things take time but i think we should see a decent sort of improvement not just in terms of no longer having recognition errors whenever silence appears because you get that immediate benefit that you do not have any insertion errors but also it should actually affect the recognition mission on the space segments as well so essentially you say well as there is this silence time which tend to be even more than it is not really so better to have a kind of silence detection or something that really takes this and then recognize only that part this will improve the phoneme recognition right yeah that is what we expect to given the trouble we have had with this corpus i can not make any promises and at that point even the call router that jean jewa is jaiva is maid can improve its performance yeah that is the intention and it can be used after a tool for the measure work so what you are saying is that you noticed that when you tried other features and we trained a system just on this data like there were 6000 utterances that were not used for the core adding side of things i think there were 13000 in the entire corpus 6 6 half of them roughly were thrown away so you used that for acoustic training and the results from that were almost the same as what we got for the cts and initially it was a nuance a train on nuance no no no no but one can imagine a commercial system that stephen cox used from nuance probably had all these things built in endpoint detection et cetera et cetera so they should have had that actually so maybe we have to do this endpoint detection and probably a little bit of painer filtering and then i think you should be pretty much converging to the system but the reason with this is that like what are systems we train on databases are they are chopped neatly that there is not too much silence beginning no too much stuff sure sure they are chopped so neatly for the it is like custom made database but here it is not like that i see i see there is also just things like receiver noise noise at the beginning and or at the end of the and i mean these are things you can get rid of there are several things like one thing is like we can do this beginning and end detection as we said or you and we can try to use france telecom is approach actually and that they have this front end which can do a specter subtraction and plus it can give you a voice activity detection okay this is part time basis okay it can give so we can just run so we can just run it quickly and see and probably you will have to run it we will give you the idea is it okay like well i do not mind running it i have got all the scripts not running i mean to say probably he should run it or like we can help him to run it actually because at some point he has to get into i mean just not sure i see the point actually i mean to say that it is okay we have all the scripts but probably he should run it i mean to say i do not know yeah whatever i do not know think right exactly there all these experiments i do not know whether that is all the the the color oaching itself or the recognition or the phoneme recognition or probably maybe we shall do it for to make it faster we should run maybe if we run we ourselves it is very fast we might as well because i mean we have up until now you have not been running any recognition and neither did jean eve on that on that data yeah i have all jean ives scripts and programs for performing their call routing and i think starting from the for nim sequences no but i mean to say that if he if he if he if if he if he give him the script and ask him to run 0 sure but i mean the scripts are so automated it is just a matter of sitting submitting a job to the cluster no yeah i do not know i do not know i do not think the point of that i do not mind running it it would be like half an hour is work in a day to run the recognition scripts i mean i know what you are saying artem should familiarize himself with the asr systems or whatever fine but i do not i do not think this is a point that is worth sort of worrying about at this stage anyway i mean yeah if it is just running as great because i am sort of i am assuming that we are going to have an endpoint detection system regardless of this working on this nuance corpus the endpoint system is different thing no that we need it for ami we need it for ami we need it for the online demo for all these things but this is basically detecting speech and non speech it is a lot more difficult than you would think it is this problem i hear every time on the speech meeting how do we detect speech and not speech well it is speech and any other class of sound that is the problem or anything nonsense other than speech it is not just speech and silence it is speech and and any other class it is not easy if it was just silence it would be easy and in controlled cases you can just assume it is any energy below a certain threshold and this is what everybody sort of claimed the problem was solved with years ago but who really can do when somebody will really be able to to do that i mean detect well i mean this is once again speech from the speech it will be a little revolution now in the yeah sure but i mean this is once again a very constrained task again i mean it is you are almost assured it is only going to be a single speaker on the microphone so i mean the only sorts of noise you are going to have to deal with is it can be or impulsive i would say or that would be a good start yeah now anyway so we okay so we will finish this call routing what do you say this is okay with you right so we because i think it has not been even submitted outside also the work what joined you did yet no no no no basically we are not done that yet because it was not at a level that could be in any way submitted and the idea i think yeah with a better for name recognition right definitely it could improve also the yeah it was showing that you have not heard from him i take it either no no okay well it would be nice to get the thesis i think there is a sort of just a research report sort of basis for this yeah yeah yeah yeah but we can some way find a guy i mean it should be at multi tell so it should be relatively easy to at least this is the last news i had and then we can try to get him back plus pressure yeah so yeah it is pretty much good time also you know and the he can and when do you intend to start writing the proposal probably should start i think yeah well this is basically 3 main jobs proposal no no you should not wait till the end that is what i mean is there i yeah looking towards the start of next year i yeah sure but so i mean you should because make it clear before and on the on the on what i will write and even a 1st submission using this work will make things clearer for me writing them down and it will help i think for that also no going very pragmatic there is a submission deadline for example 31 december for conference on multimedia and dexpo where i think this 2 little works on yeah it can go there and it is nice to get some feedback on i mean that feedback will probably come after your proposal or possible i think it is good if you submitted for a proposal as a research report you put it here submitted for the conference it is quite good for the proposal itself yeah yeah yeah exactly it is a proposal because as far as i understand this work on the phoneme anyway still takes some development time in the sense that no i think we may have to like it is kind of like it is kind of like maybe take like a 2 or 3 days you know like to sit it out and say one converse to one approach and then just do it out so that any bugs or whatever yeah and probably yeah so that is the thing we had to sort of the bug kind of thing and see how reliable is this endpoint detection for us i would prefer to go back to the cts models i think as well yeah i think then we can go back to cts i think they did work a little bit better anyway to use that there are results you are just to decide yeah what what what what what what what you want to work 1st and finalize i think you can really yeah well i think 1st this submitted yeah and this will help for a proposal and since we cannot start immediately with the measures i think that is the best thing to do okay and so you wanted to talk something more than that well we have talked you with that you wanted to talk something different also well no no for me or maybe another meeting there is another one i said no no i say another so you are looking at a watch before i was like wow i was just looking just okay we are not out of time you know zamee corpus like did no time yeah and i think that will be exactly one hour and the tape is 60 minutes yes yeah it happened with my friend like he was taking pictures pictures pictures and it was going 37 38 40 even and the real never got over so i said there is some problem with yours the way you put the roll inside there is some problem it has got struck somewhere it can never go to 41 42 not normally yeah you go to 37 38 38 is fine you know 40 41 is that is he he has already told us okay okay fine so you are picking pictures and pictures i am sure you are going to get maybe 5 pictures out of this if you get 5 you feel lucky so next week i am 2 and me and john we have to pick it out there we have to start working stuff we have this car routing cell let us get a lot of christian now no yeah now i think of it i think that is fine i mean the point of the post is just to provide information for people that would be interested and start discussion right i think we have a lot of people there well i am