okay okay so b maybe i will go back to your questions yes so you were talking about the 21st question yeah okay and a little bit about same people moving with the clustering yeah and this also allows you to separate people huh and the questions i did not talk about was going from location to identity of the person that is quite different you you need to use a signal you have separated and do the type of analysis i mentioned from the spectrum you can transform the the magnitude spectrum and build model of the person or actually look at the pitch no so this one yeah pitch at least do not ask me how it is done i i know the theory so yeah but pitch is pitch is not is not enough sometimes actually you can t perhaps just just is not it the pitch and the frequency the base frequency of the i it is a frequency yeah yeah it is a frequency of your vibration here th there is one frequency that is the base from for yeah and then it is transformed through the mouth that is the usual model and if you have both you can well you can compare this to to one stored and so get out which person it is okay both pardon the pitch and and the freque what and well the things that make the sou the voice okay s and you can measure measure that yeah or i yeah it is not necessarily done explicitly but it is equivalent to that yeah yeah so again f f t is useful to do this type of analysis i am sorry i did not really get the point of comparing the pitch the pitch is the is the frequency that is okay yeah but it is quite personal and the way you transform it v by the mouth yeah you were comparing what with what and and the the holes in your head everything the the nose and so on huh okay okay yeah and that is that is really unique for ev well almost unique for every person so so and you were talking about comparing what with what well once you get this information your frequency and your vocal contract or how it is called yeah yeah y you can use this information to compare to to to the frames here okay but you want to have the information because the the mouth you move it it it is an information equivalent to that no but it is maybe n you do not need to do the whole complicated modeling but to scan t all the user no no no no no no that that that everything is done through through the voice here you take a voice of someone and then you can get out this information okay it is it is not p it is clear it is not perfect but s it is enough to to to use it for for different purposes and okay so you have the p the pitch and you have information about the voice dynamic or yeah rate of speech is also possibility okay how fast you speak that is quite personal also but it varies over time depending on emotion that also might be interesting for you detecting emotion emotion the involvement in a discussion yeah yeah if you are saying 0 yeah yeah we should and us yeah yeah so these measures are a way to quantify that okay and i is it expensive to do in c p u you mean not not that well as far as i know p no no no it is not but in fact if y you speak of a lot of things that are not expensive at the end you have something that is actual what w w what is m what is more expensive is to take the decision finally so your decision can be for example who is it or if the person moved which is probably the most complicated thing like the person goes away then comes back 10 minutes later sits at a different place you will need to build statistical models of the person identity using these measures so the ones he mentioned i do not know if you are familiar with that is it called p c m i thi well it is it is not a technology s yeah it is more or less used this to compress audio yeah yeah i have to i want to check it i can show it you afterwards so not really familiar well i did a course about it but i should not say it too loud because the professor is working here at idiap no okay so you are the expert yeah i should reread what i signed there so careful you are recorded okay s okay so all this stuff is done from the f f t yeah so you have to do f f t because otherwise also location to identity all i was saying is that these different measures are a way to evaluate the identity of the person okay so if with location all you can do is extract segments of speech for example but it will not tell you that these are the same person it is just a location but to get a cluster the location that is a place where noise is regularly coming from yeah yeah yeah so that might be good for example for 10 minutes but then the person might move or a different person might come but for instance for a setting like here it would be enough because we are not moving yeah it is well unless sometimes somebody goes there or yeah okay right yes but we are targeting a place where we do not know i where it will be more flexible so we need more than just location if we want to yeah yes but it well it i am even not sure if it is that important to know which person is talking because if if it i think about a table or so something physical yeah you just want to know in in which corner of the table speech is coming and not which person produced the speech so i agree i it it is im maybe just yeah it i it yes it is important for detecting patterns because if you say let us say w how much the person have spoken over the the 5 last minutes yeah yeah but it is not really related to the ah yeah i see it is you need a bit both so it if someone come and yes but that you can do with location if it is only on the short term yes yes i think it may it may be okay but it depends how long y on on the off line yeah i would say you might use these measures in a simple way if you do it on line like to give feedback but you you mentioned that another side of your project is also to analyze off line so there you might want to use a this is we had a student here he finished one or 2 years ago but he left his software and his software is precisely to do that to cluster at another level to cluster these small clusters of speech group them by person automatically from th from the location y no from the these measures pitch etcetera 0 okay so sound so i this is only based on you give the spectrum and okay yeah exactly so i would say from location at the lower level you can get small parts of speech and then at the next level possibly off line you can group them into a speech segment i am hoping to have enough time to try doing that before i finish my thesis yes so if i manage to do some practically usable software i will tell you when are you planning to finish your in 56 months abou approximately okay but the last ones are always the more extensible not really more busy i would say yes yeah so you said you want to do separation but ex exactly yeah i do not know how mu maybe we would be more interested to know that it is not the same people more than knowing that it is the same people yeah okay or to detect that someone is more that was not talking before is talking now okay i i said we want t to know to to do separation just because of the voice recognition ah you want to do voice recognition i this is not the yes you might want you might might want to do yes okay yes no but but in fact we have yes we are designing a noise sensitive table just f from your ring but we are also designing a prototype to analyze the conversations okay so we might add there a lot of features yeah so yeah once you have locations it is not a big problem okay and i do not know perhaps perhaps if you have 2 people speaking speaking at the same time i do not know if voice recognition of an such a stream is real no no you should separate them 1st yeah yeah okay so yes so that is fine so the type of things i presented can lead to separation sure it is only a very small step to add yes it is not every important feature for now but yeah yeah although it might be if you want to ex extract these features like pitch and rate of speech or energy simply from each person you might have to do some ba basic separation cause quite often people do not realize that they talk at the same time interrupt each other yes so even if you do not want to recognize a speech fully at least you need yeah you need to separate it and another what i do not remember it is okay you can email later yeah so okay yes also to to extract the context yeah because if you know the words that are spoken you can let us say someone say he is angry about the word he is seen okay a semantic context yeah yes yeah yeah then you mi you might want to add this kind of this is not in my project but that are further directions no but i think once you have the voice for a person you can start to do this and once people have speech recognition or far enough to t yes professor was interested to know when people were talking about him ah someone is talking about me let us listen so it is yeah you could detect one word yeah yes keyword spotting yeah and have a special events trigger if and lighting a on his office so okay so are you done w with what you wanted to presented us in regards with these questions yeah i think so yeah i i will just say if you want i can point to 3 papers but no i think it is better if i just send you the link it is probably simpler sure the link sure 3 papers about one im is about this sector based stuff like the localisation and detection okay one is about how to take the decision that somebody is active or not it sounds simple at 1st you think you just put the threshold and that is it but the problem is the it is n no problem no it is it does not sound simple but yes you can yes it no you can do that no more time for problems there is no problem but i if you want your your system to work in different conditions like cafeteria library okay the environment will be quite different so a single fixed threshold value might be a problem sure yes in fact for now we are using a kind of calibration but it is still s yeah so you can do that automatically this kind of things i do not know if you are doing that already but yes okay yes but in fact you have to sit in front of the microphone and it is yeah people one speak and people 2 and people 3 and you are just subtracting the the level that you you kept in the other microphones okay so i have another paper which might be interesting for that for a single channel calibration okay and i have code on line so for this particular one okay and the last one i would say is the clustering to yeah the clustering of the different locations over time so that you can get small clusters of speech there are other ways to do it but i think it is it is important because in spontaneous speech all these words are quite small so you need to do that adequately yeah but otherwise yeah i am done okay just a last question when you mentioned it if i am speaking very how do you say it yes i am just speaking one word every every 12nd yes a is the the event that is detected is will be a sum of small parts slowly yeah yeah there will not be okay many small events basically yeah so you yes you have to to reprocess to to say let us say yes there is a very small interval you have to consider it yeah basically i mean it depends because this is yes this is one this can be considered as one interv intervention yeah even if somebody says just yes i do not know if it is important for you or not it might be it might be might this is not a very easy problem for the 1st application we will do with prototypes it will be more about quantity of speech or how long yeah how much someone speak but when we will go to final details yeah if i am talking and someone say yes yes yes from time to time it show that he is participating yeah and obviously it makes different that if someone is talking and one is replying never nor moving or nor having eyes open y yeah yes so i i will send you links some of the papers are long but you can just briefly look yeah okay go through them okay we can go and see olivier unless you want to talk about something else no i think i think we has already have plenty to read no i think does this work email i am done huh does it work if i put in my email and well we can try try but no if it is not i guess you would have to install it and so on we have yeah we have 0 c r but maybe not on this computer no no no you see i thought you can put in here your email address and and just do this and then it mails me yeah that is good idea yeah it is a good idea yeah you see i want too much and the application you are using all of them y every time you are using matlab or you are programming everything is based based on matlab yeah yeah yeah once i i coded the most expensive part in c just to see so you can have a mix of both from matlab you call c and back sure yes and after all matlab is also programming so it and you can code java but the day you want to do on line real time stuff nah i would not would not trust it you can choose matlab yes sure okay you have to code it directly to the machine so thank you for the information my sheet is very filled up i will take yours i am writing so only me can read it is a kind of secret language okay
it is good it is good if you are if you are getting out okay yes so you were talking about the 21st questions yeah okay and a little bit about same people moving yeah with the clustering and this also allows you to separate people huh the questions i did not talk about was going from location to an identity of the person that is quite different you need to use the signal you have separated and do the type of analysis i mentioned from the spectrum you can transform the magnitude spectrum and build model of the person i will actually get the peach now yeah pitch at least do not know i ask me how it is done but i know the theory it is not enough sometimes you can perhaps just is the pitch and the frequency the base frequency it is a frequency yeah yeah it is a frequency of your vibration here there is one frequency that is the base for me yeah and then it is transformed through the mouth that is the usual model okay and if you have both you can well you can compare this to one store stored and so get out how much person it is on the beach and the what and the fruget and what things that make the sound and you can measure that yeah yeah it is not necessarily done explicitly but it is equivalent to that yeah so again fft is useful to do this type of analysis i am sorry i did not really get the point of comparing the pitch the pitch is the is a frequency that is yeah but it is quite personal okay and the way you transform it by the mouth yeah you were comparing what with what the holes in your head everything the nose and that is really unique for almost unique for every person and you were talking about comparing what with what well once you get this information your frequency and your vocal contract what is called you can use this information to compare it but you want to have the information because the mouse you move it it is the information equivalent to that it is maybe you do not need to do the whole complicated modeling to scan all the user to have the no no no no everything is done through the voice here you take a voice of someone and then you can get out this information it is not clear it is not perfect clear it is not perfect but it is enough to use it for different purposes and you have the pitch and you have information about the voice dynamic or yeah right of speech is also possibility okay how fast you speak that is quite personal also but it varies over time depending on emotion that also might be interesting for you detecting emotion emotion the involvement in the discussion yeah yeah if you are saying 0 yeah yeah yeah so these measures are a way to quantify that is it expensive to do not that well as far as i know no no it is not but in fact if you speak of a lot of things that are not expensive at the end you have something what is more expensive is to take the decision finally finally so your decision can be for example who is it or if the person moved which is probably the most complicated thing like the person goes away then comes back 10 minutes later sits at a different place you will need to build statistical models of the person identity using these measures so the one he mentioned i do not know if you are familiar with that is it called pcm well it is not technology it is more or less used this to compress audio i have to check it i can show it you afterwards so not really familiar well i did a course about it but i okay so you are the expert i should not say it too loud because the professor is working here at the video careful you are recounted okay i should read what i signed today so all this stuff is done from the fft yeah so we have to do fft because otherwise also location to identity all i was saying is that these different measures are a way to evaluate the identity of the person okay so with location all you can do is extract segments of speech for example but it will not tell you that these are the same person it is just a location but you get a cluster of the location that is a place where noise is regularly coming from yeah so that might be good for example for 10 minutes but then a person might move or a different person might come but for instance for a setting like here to be enough because we are not moving it is well unless sometimes somebody goes there okay right but we are targeting a place where we do not know it where it will be more flexible so yeah we need more than just location if we want to yes i am even not sure if it is that important to know which person is talking because if you think about about table as something physical yeah you just want to know in which cone of the table i agree maybe the beach is coming and not which person produces speech cell yes it is important for detecting patterns because if you say let us say how much the person have spoken over the 5 last minutes yeah but it is not really related to the ah yeah i see it is if someone come and but that you can do with location if it is only on the short term yes yes i think it may be okay but it depends on the off line yeah i would say you might use these measures in a simple way if you do it online like to give feedback but you mentioned that another side of your project is also to analyze offline so there you might want to use we had a student here he finished one or 2 years ago but he left his software and his software is precisely to do that to cluster at another level to cluster these small clusters of speech group them by person automatically from the location no from these measures pitch etc so so this is only based on the spectrum and yeah exactly so i would say from location at the lower level you can get small parts of speech and then at the next level possibly offline you can group them into speech segment i am hoping to have enough time to try doing that before i finish my thesis so if i manage to do some practically usable software i will tell you when are you planning to finish your in 56 months approximately but the last months are always done more extensible naturally more busy i would say so you said you want to do separation but maybe we would be more interested you know that it is not the same people more than knowing that it is the same people or to detect that someone is more that was not talking before talking before he is talking now okay i said we want to know to do separation just because of the voice recognition are you want to do voice this is not the you might want yes yes okay might want to do everything but in fact we are designing a no sensitive table just for mirroring but we are also designing a prototype to analyze the conversations so we might add a lot of features so yeah once you have locations it is not a big problem okay i do not know perhaps if you have 2 people speaking at the same time i do not know if voice recognition on the switch stream is a no no you should separate them 1st okay yeah yeah so that is why so the type of things i presented can lead to separation sure it is only a very small step to add yes it is not a very important feature for now but follow but yeah yeah although it might be if you want to extract these features like pitch and the rate of speech or energy simply from each person you might have to do some basic separation because quite often people do not realize but they talk at the same time interrupt each other so even if you do not want to recognize the speech fully at least you need to separate it yeah and the what what would i say i do not remember it is okay you can email later so yeah so okay yes also to extract the context yeah because if you know the words that are spoken you can let us say someone say he is angry about the word a semantic context yes yeah yeah then that are that are theirder directions it is not in my project but no but i think once you have to voice for a person you can yes i started to do this and the ones are people of speech requisitional professor dylan blur was interested to know when people were talking about him yeah so it is a someone is talking about me let us listen yeah you could detect one word yes keyword spotting and have a special event trigger if and lighting a lead on his office so okay so are you done with what you wanted to present it as yeah with these questions yeah i will just say if you want i can point to 3 papers sure but no i think it is better if i just send you the link the link sure it is probably simpler so 3 papers about one is about this sector based stuff like the localization and detection one is about how to take the decision that somebody is active or not it sounds simple that 1st you think you just put the threshold and that is it but the problem is no problem no you can do that for a problem problem but if you want your system to work in different conditions like cafeteria library the environment will be quite different so a single fixed result value might be a problem yes in fact for now we are using a kind of calibration but yeah it is still so you can do that automatically these kind of things yes i do not know if you are doing that already but yes but in fact you have to see it in front of the microphone and let us say people one speak and people 2 and people 3 and just subtracting the level that you kept in the other microphones so i have another paper which might be interesting for that for a single channel calibration and i have a code online okay so for this particular one the last one i would say is the clustering to yeah the clustering of the different locations over time so that you can get small clusters of speech there are other ways to do it but i think it is it is important because in spontaneous speech all these words are quite small so you need to do that adequately but otherwise yeah i am done okay just a last question when you mentioned it if i am speaking very do you see it yes i am just speaking one word every 12nd slowly the event that is detected is a will be a sum of small parts there will not be many small events basically yes you have to reprocess to say let us say yes yes there is a very small interval you have to consider it yeah basically i mean it depends because this is yes this is one this can be considered as one intervention yeah even if somebody says just yes i do not know if it is important for you or not it might be it might be right this is not very easy problem for the 1st application we will do with prototypes it will be more about quantity of speech or how long or much someone speaks but when we will go to finer details if i am talking on someone say yes yes yes from time to time it shows that he is participating and obviously it makes different that if someone is talking and one is replying never and no moving or no having a hi it is open yes so i will send you links some of the papers are long but you can just briefly look okay we can go and see olivier unless you want to talk about something else no i think i does this work email i think we already have plenty to it does it work if i put in my email well we can try but no it is not i guess you would have to install it and so we have ocr but maybe not on this computer no no you see i thought you can put in here your mail address and just do this and then it mails me and it mails me no that is a good idea yeah it is a good idea yeah i see i want too much the application you are using all of them every time you are using matlab or programming yeah everything is based on matlab once i coded the most expensive part in c just to c so you can have a mix of both from matlab you call c and back back sure yes i love to all my tab is also programming so it but the day you want to do online real time stuff you can not use my tab never i would not trust it okay you have to code it there you are key to the machine my shit is very filled of i will take yours i am writing so only me can read that is a kind of secret language okay you have not you have no you have to be if you want to let me i want to if you have a guardon i would leave the he would have to ask some question on the material stop he is there he is there bastian d hawr okay he will be here he comes here here okay that is a little so we will stay here