Toward Multimodal Notation for Mathematics a first step
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bernareggi, Cristian
Parasiliti Provenza, Loredana
Notation is a tool of thought in reasoning and communication. However, to become an effective tool of thought, each notation expression must be properly perceived. Blind and partially sighted persons run into difficulty in working with spatially represented mathematical expressions.
 Good afternoon, I'm Lorda, I'm from the University of Milan. In this talk, I'm Christian, who presents our course of multi-model notation for mathematics as a tool of code for both visually impaired and blind people. Then we illustrate a first step for multi-model notation in the case of restructuring. Notation is widely recognized as a tool of code for both using both reasoning and communication in several scientific contexts, chemistry and particularly mathematics. However, to become a tool of code, each notation expression must be properly perceived. Blind and partially silent people run into difficulty in making, exploring and understanding mathematical communication and scientific concepts designed to be visually represented on paper or screen, for example. Our goal is to develop new multi-model notation for mathematics to enable blind and partially silent people to perceive and reflect on mathematical concepts that are specially presented to manipulate expression such notation and to communicate with silent people as well as each other their reasoning and experience with mathematical notation. In our work, we conceived multi-model notation for mathematics as a digital medium for thinking and communicating mathematical concepts that adopt a mechanical language of mathematics and that mask its expression perceptible as a set of multi-model signals, both in visual and cognitive contexts, for example. Multi-model notation for mathematics is needed to overcome the flows of existing notation for blind and partially silent people. Central notation, in this central notation, of course mainly sequential axis and it does not permit diamond relation. On the other hand, speech and whole notation supports no apparent method representation of concepts. In short, such notation is partial and passive to all the thoughts. According to our approach, each notation expression is located in a virtual free space. Each user can experience and have a box where the outside walls are ultimately perceived to constrain the user within the box while having user orientation within this box. The 3D space can also be perceived as a audio box where audio signals are imported users about a reserve position in the space and the space can also be perceived as a visual box where visual signals denote the box components and a visual proxy denotes the user position in the space. The 3D space is the virtual environment where the user can create and manage mathematical expression through diamond relation. The interaction with 3D space and mathematical expression in multi-model notation occurs through serrated values, obviously keywords, screen, loudspeakers, and passive devices, specifically the phantom device. We have chosen to facilitate and to permit a sufficiently wide 3D workspace and to create a force feedback. As a first step toward multi-model notation for mathematics, we have posted our study on graph structures which are mathematical expressions that represent various scientific concepts from calculus to automatics and forth. Specifically, we have developed a prototype that supports visual impaired and studied people to create and explore multi-model expressions representing graph structures. Now, Christian will show you how to create and explore graph structures through our prototype. Just to create a graph in a 3D dimension. I can move in a box which can be perceived through the force feedback. I can place notes in this workspace, for example, I can move wherever and insert a note freely or I can insert another note. I can connect these notes by engine. To find the second note, I can start the creation of an engine and I have to find the second note. Here, once I found the time, I can connect it. To find the note, I used an exploration modality which is a driven exploration modality. I was attracted by force directly to the note present in the workspace. According to the distance, they are sorted according to the distance from the starting point. I can then create another example, another engine and another engine. For example, I can start, I move, I place a note and I connect with the start in the source node. I can also label the notes, just to give them a label. I can write a label, for example, if it is a molecular structure, I can insert the name of a camp. When moving in the structure, I can ask for the name of the note in the game by using the Phantom device. Just about the exploration, I have just used the exploration modality based on force attraction. It was for notes, so it means I was directly attracted to the notes in the workspace. I have another modality, another mode to explore the ground. In particular, from a note, I can be interested in understanding which are the directions where edges are going out or are entering the note. To do that, I have an exploration modality, which is the two rock edges, and I can activate it, and then I am attracted along the edge and back. Doing that, I am able, at the end, to find the right direction to leave the note and to reach the neighbor note. It is also possible to do some other operations, for example, you can sell or rename and so on, which can be shown in a trial demonstration. Let's come here to... As we have seen, we have used these three modalities, three, two rock notes and two rock edges, and I just wanted to give another view of the workflow to design and develop this system. This prototype was, at first, developed according to a usability cycle, and two sets of experiments were done. At first, with this prototype, which aimed at assessing if the optical representation of ground elements was available, and the second series of experiments was done to especially assess the new tools of edges and of notes, which are the exploration modes we introduced. They were performed with pre-designed students, which were experts both in the optical device and also in the work style of computer science. Let's come to some results. These results show that the work with the exploration of the tools, the exploration by tools with two rock notes and two rock edges, decreased the time to detect, to find, to locate an object in the workspace, or to locate an edge, and where edges were present around an object. Also, when a user was really exploring, he was not sure or constrained, he was not sure to have found all the notes in the workspace, but using tools at the end was sure because he or she was attracted to all the elements present in the workspace, notes or edges. For the world, we focused especially on the evaluation of other evaluations with the sighted persons. The use of guided tools also in other situations, for example, when there are not connected sub-graphs, or when just to be driven through sub-graphs, which are not connected to treat them as a sense of note, as a one-note, not just a sense of note. So, just specific tools of this kind of exploration. Thank you for your attention. Thank you very much. How much does that device have to cost? The cost of an active device. The one at your hand. The one at my hand is about 2,000 euros. For example, the five devices which can be a choice is about 200 dollars. It can be improved the cost effectiveness by using a different active device with the same principles. Our speakers finished a little early, so we have time for two questions from the audience. I'll hear one question. So those early evaluation results, that was you trying it out, it wasn't another person trying it, or was that the group of people? The first one went to groups of other groups, this one was with five blind persons, and the second one after now is with three blind persons. We still have time for another question. A difficult conceptual problem for sighted users in graph theories distinguishes between graphs that are potentially planar and bedridden as they aren't. What would the corresponding situation be here? So within visualization in graph theory for sighted users, is distinguishing between graphs which are planar and bedridden, or not obviously so, and ones that are not planar and bedridden, is the same contrast physical here with your active device? So visualization at the moment is a problem, but with the device, with the active device, if the graph is not planar, we can represent the problem.
 Good afternoon, I'm Lorda, I'm from the University of Milan.  In this talk, I'm Christian, who presents our course of  multi-model notation for mathematics as a tool of code for both visually impaired and blind people.  Then we illustrate a first step for multi-model notation in the case of restructuring.  Notation is widely recognized as a tool of code for both using both reasoning and communication  in several scientific contexts, chemistry and particularly mathematics.  However, to become a tool of code, each notation expression must be properly perceived.  Blind and partially silent people run into difficulty in making, exploring and understanding  mathematical communication and scientific concepts designed to be visually represented  on paper or screen, for example.  Our goal is to develop new multi-model notation for mathematics to enable blind and partially silent people  to perceive and reflect on mathematical concepts that are specially presented  to manipulate expression such notation and to communicate with silent people as well as each other  their reasoning and experience with mathematical notation.  In our work, we conceived multi-model notation for mathematics as a digital medium  for thinking and communicating mathematical concepts that adopt a mechanical language of mathematics  and that mask its expression perceptible as a set of multi-model signals,  both in visual and cognitive contexts, for example.  Multi-model notation for mathematics is needed to overcome the flows of existing notation  for blind and partially silent people.  Central notation, in this central notation, of course mainly sequential axis and it does not  permit diamond relation.  On the other hand, speech and whole notation supports no apparent method representation of concepts.  In short, such notation is partial and passive to all the thoughts.  According to our approach, each notation expression is located in a virtual free space.  Each user can experience and have a box where the outside walls are ultimately perceived  to constrain the user within the box while having user orientation within this box.  The 3D space can also be perceived as a audio box where audio signals are imported users  about a reserve position in the space and the space can also be perceived as a visual box  where visual signals denote the box components and a visual proxy denotes the user position in the space.  The 3D space is the virtual environment where the user can create and manage  mathematical expression through diamond relation.  The interaction with 3D space and mathematical expression in multi-model notation  occurs through serrated values, obviously keywords, screen, loudspeakers,  and passive devices, specifically the phantom device.  We have chosen to facilitate and to permit a sufficiently wide 3D workspace  and to create a force feedback.  As a first step toward multi-model notation for mathematics, we have posted our study on  graph structures which are mathematical expressions that represent various scientific concepts  from calculus to automatics and forth.  Specifically, we have developed a prototype that supports visual impaired and studied people  to create and explore multi-model expressions representing graph structures.  Now, Christian will show you how to create and explore graph structures through our prototype.  Just to create a graph in a 3D dimension.  I can move in a box which can be perceived through the force feedback.  I can place notes in this workspace, for example, I can move wherever and insert a note freely  or I can insert another note.  I can connect these notes by engine.  To find the second note, I can start the creation of an engine and I have to find the second note.  Here, once I found the time, I can connect it.  To find the note, I used an exploration modality which is a driven exploration modality.  I was attracted by force directly to the note present in the workspace.  According to the distance, they are sorted according to the distance from the starting point.  I can then create another example, another engine and another engine.  For example, I can start, I move, I place a note and I connect with the start in the source node.  I can also label the notes, just to give them a label.  I can write a label, for example, if it is a molecular structure, I can insert the name of a camp.  When moving in the structure, I can ask for the name of the note in the game by using the Phantom device.  Just about the exploration, I have just used the exploration modality based on force attraction.  It was for notes, so it means I was directly attracted to the notes in the workspace.  I have another modality, another mode to explore the ground.  In particular, from a note, I can be interested in understanding which are the directions where edges are going out or are entering the note.  To do that, I have an exploration modality, which is the two rock edges, and I can activate it, and then I am attracted along the edge and back.  Doing that, I am able, at the end, to find the right direction to leave the note and to reach the neighbor note.  It is also possible to do some other operations, for example, you can sell or rename and so on, which can be shown in a trial demonstration.  Let's come here to...  As we have seen, we have used these three modalities, three, two rock notes and two rock edges, and I just wanted to give another view of the workflow to design and develop this system.  This prototype was, at first, developed according to a usability cycle, and two sets of experiments were done.  At first, with this prototype, which aimed at assessing if the optical representation of ground elements was available,  and the second series of experiments was done to especially assess the new tools of edges and of notes, which are the exploration modes we introduced.  They were performed with pre-designed students, which were experts both in the optical device and also in the work style of computer science.  Let's come to some results. These results show that the work with the exploration of the tools, the exploration by tools with two rock notes and two rock edges,  decreased the time to detect, to find, to locate an object in the workspace, or to locate an edge, and where edges were present around an object.  Also, when a user was really exploring, he was not sure or constrained, he was not sure to have found all the notes in the workspace,  but using tools at the end was sure because he or she was attracted to all the elements present in the workspace, notes or edges.  For the world, we focused especially on the evaluation of other evaluations with the sighted persons.  The use of guided tools also in other situations, for example, when there are not connected sub-graphs, or when just to be driven through sub-graphs,  which are not connected to treat them as a sense of note, as a one-note, not just a sense of note.  So, just specific tools of this kind of exploration.  Thank you for your attention.  Thank you very much.  How much does that device have to cost?  The cost of an active device.  The one at your hand.  The one at my hand is about 2,000 euros.  For example, the five devices which can be a choice is about 200 dollars.  It can be improved the cost effectiveness by using a different active device with the same principles.  Our speakers finished a little early, so we have time for two questions from the audience.  I'll hear one question.  So those early evaluation results, that was you trying it out, it wasn't another person trying it, or was that the group of people?  The first one went to groups of other groups, this one was with five blind persons,  and the second one after now is with three blind persons.  We still have time for another question.  A difficult conceptual problem for sighted users in graph theories distinguishes between graphs that are potentially planar and bedridden as they aren't.  What would the corresponding situation be here?  So within visualization in graph theory for sighted users, is distinguishing between graphs which are planar and bedridden,  or not obviously so, and ones that are not planar and bedridden,  is the same contrast physical here with your active device?  So visualization at the moment is a problem, but with the device, with the active device, if the graph is not planar, we can represent the problem.
Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__0_611.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__612_1091.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__1092_1535.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__1536_2207.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__2208_2867.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__2868_3743.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__3744_3983.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__3984_5735.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__5736_6635.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__6636_7691.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__7692_8063.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__8064_8963.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__8964_11015.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__11016_11555.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__11556_11879.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__11880_14759.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__14760_15095.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__15096_15251.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__15252_17255.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__17256_19151.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__19152_20339.png Toward_Multimodal_Notation_for_Mathematics__a_first_step__480p_6fps_H264-128kbit_AAC__20340_24024.png