Donald Norman's work involves the advocacy of user-centered design. His books all have the underlying purpose of furthering the field of design, from doors to computers. Norman has recently been controversial in saying that the design research community has had little impact in the innovation of products, and that whereas academics can help in refining existing products, it is technologists that accomplish the breakthroughs.

An example of Neilsen's work in this area is the 'Test your design' paper where he takes a basic web site and works with the designers to improve it. The changes he makes are usually quite simple, for instance, the overall impression is less cluttered, especially with text. Nielsen is adamant with clients that they need to rethink their approach to textual communication on Web sites and emphasize powerful headlines, summaries, and bullet points, not long blocks of text that annoy and put off readers. Gouma's notes: "On the original version, you'll note that there are hundreds more words. There's a quarter of the words and number of ideas on the newer site. That was definitely based on feedback." There were a series of other changes:

reality, role play, strategy games (such as economic modellers), interviewing skills and intelligent tutors. If Prensky is to be followed completely we as educators may overestimate the abilities and skills of the Digital Natives and their skill at accessing and manipulating software which may camouflage their reduced ability to understand and be creative within a totally new environment. Having the prior knowledge of a range of software may though lessen the learning curve and enable rapid uptake of any new technology, and students may not necessarily judge which technology is best for a specific purpose or to be able to assess

The design decisions that will be reviewed in this chapter concern the choices that designers of multimedia and large resources have to make. There are generally in terms of the restrictions they place on movement, the structure of the package and what the user is allowed to do within the package. There is a great deal of work in the human computer interaction field that investigates and researches on these areas. Examples of good current thinking in this area are Shneiderman, the textbook by Dix, Finlay, Abowd and Beale, and that of Benyon, Turner and Turner. These core works move on from the earlier work of gurus such as Norman and Jacob Nielsen who were the original proponents of the need to organise the resource properly. Neilsen thought that all designs need to be tested – early

Donald Norman's work involves the advocacy of user-centered design. His books all have the underlying purpose of furthering the field of design, from doors to computers. Norman has recently been controversial in saying that the design research community has had little impact in the innovation of products, and that whereas academics can help in refining

An example of Neilsen's work in this area is the 'Test your design' paper where he takes a basic web site and works with the designers to improve it. The changes he makes are usually quite simple, for instance, the overall impression is less cluttered, especially with text. Nielsen is adamant with clients that they need to rethink their approach to textual communication on Web sites and emphasize powerful headlines, summaries, and bullet points, not long blocks of text that annoy and put off readers. Gouma's notes: "On the original version, you'll note that there are hundreds more words. There's a quarter of the words and number of ideas on the newer site. That was definitely based on feedback." There

The colour of the navigation bar and words on it shifted to a more readable blue, and

More effort was made to distinguish in different places between registration, which is

The testimonial photograph was made smaller and dropped on the page, and the eyes

It's clear from the headline and the treatment of the top left-hand box that that's where

 Finally, one change (not apparent here) went to the heart of the process by which users were guided through the site. Originally, Ideas.com featured "wizards" that used

aimed at first-time visitors, and log-in, which is for returning visitors.

and often, rather than the industry norm of testing once the design is complete.

existing products, it is technologists that accomplish the breakthroughs.

how useful a new technology is for a specific educational activity.

**5.2 Design decisions**

were a series of other changes:

one "starts" on this page.

the language was changed to become clearer.

are averted to keep from trapping users' eyes.

shortcuts based on user responses to various questions. However, the wizards did not test well. "We learned they actually made it too simple," says Goumas. "Users wanted more hand-holding and walking-through the process."

Work by Shneiderman has further revolutionized specific areas in particular that of health and health records. Since 1991 his major focus has been information visualisation, beginning with his dynamic queries and starfield display research that led to the development of Spotfire. Shneiderman developed the treemap concept which inspired his research and commercial implementations. The University of Maryland's Treemap 4.0, developed in cooperation with Catherine Plaisant, remains available for educational and research purposes. Shneiderman advised Smartmoney which implemented the widely used MarketMap for stock market analyses. Later information visualisation work includes the LifeLines project for exploring a patient history, and its successor project, PatternFinder, which enables searches across electronic medical records. Searching for patterns in numerical time series data was enabled by three versions of TimeSearcher, which was applied for stock market, auction, genomic, weather, and other data.

The Hierarchical Clustering Explorer supports the discovery of features in multidimensional data, especially for gene expression data, using the powerful rank-by-feature framework. Three current projects focus on network visualisation: Network Visualisation by Semantic Substrates, SocialAction, and NodeXL. These tools are being applied for citation analysis and social network analysis, especially for the iOpener project, STICK (Science Technology Innovation Concept Knowledge-base), and ManyNets (explores & visualises many networks at once) projects. A new direction is based on applying social media to national priorities such as the 911.gov article in Science, which led to work on emergency and disaster response: Community Response Grids. Raising awareness of the need for expanded research was accomplished by the iParticipate report.

#### **5.3 Navigational patterns research**

Research on navigational patterns investigated patterns that other researchers had recognized with users in multimedia, hypertext and Internet resources. Research undertaken by Horney (1993), and Canter, River and Storrs (1985), both recorded and discussed the navigational paths of individual users. Research by Simpson and McKnight (1990) on how users preferred to navigate looked at different structures within a hypertext system. They varied the structures and cues for the subjects from alphabetical indexing to hierarchical structuring, using typographical cues and giving provision for position indicators. Their results showed users had preferences for hierarchies. When researching different navigation methods, Henderson (1993) commented that the prevalent characteristic of the modern worldview was the dependence on the conceptual view of information as being hierarchical, and that this conflicted with an alternative conceptual view, that of time, which produced a linear and sequential pattern. It was this dichotomy that Henderson considered made the investigation of multimedia packages more difficult, as users wanted to use the resource hierarchically, but moved through it sequentially. The user's perspective and gradual acquisition of navigational skills and awareness were emphasised by McKnight, Dillon and Richardson (1990): "Acquisition of navigational knowledge proceeds through several development phases from the initial identification of landmarks in the environment to a fully formed mental map" (p.69).

Multimedia Design Decisions, Visualisations and the User's Experience 165

three tasks within the same extensive encyclopaedic package. For each of these tasks a 2D chart was produced. From these navigation charts (60 in all) each users navigation patterns

These charts also enabled comparison of different users across each task. Using the scanned records of each user's progression through the software and these navigation charts, it was possible to create a range of the different types of navigation pattern found. The list of navigation types was then compared to the other patterns, which are shown in the list below. From these scanned records or audit trails of each user and the subsequent charts it was possible to build taxonomy of navigation patterns. These included nine main types

The following navigation patterns were recognised from the initial empirical work and these were then further tested in the second study. These navigational patterns were put into a

1. Linear - path on one level, using tools e.g. index, time line, or word search, one

2. Linear extra - paths lead away from basic linear pattern, returning to the linear path,

3. Circular - initially recognised as linear, but circular if completed, one/two-way dependent on software design, may be represented by an ellipse/multifaceted shape

4. Star - movement initially linear but implies a change in level, going into second level

5. Star extra - a development or extension of the star pattern, movement into the second or third level of the package, i.e. into an additional level, beyond the usual star pattern,

6. Hierarchical - movement down the hierarchy, with a possible return along the same path, to go down one or more branches of a tree structure. Progressing one way down the structure and across to the next branch of the tree, can be two-way, can return to the

7. Hierarchical - extra - movement along multiple hierarchies usually with different subject/themes, usually in the same way, returning to the same tree or continuing onto

8. Complex - Chaotic - movement follows a series of different paths, usually in rapid succession, random and erratic navigation, frequent changes of route and searching

9. Complex - Planned - sequence of moves following established path, sometimes including definite patterns, using a mixture of different types, but following an ordered

The list of navigational patterns above shows all the subdivisions of each pattern type, these should mean that the navigational patterns are clear and that the classification can be used with many different types of digital resources. This is especially useful for educational resources when the user may not be aware of how the package/ resource are structured or

one or two way, and the extra depth may only be used for part of the route.

usually at the same place or the next node to their original leaving point.

and preferences were assessed.

(with several sub types).

direction

e.g. an octagon.

**7. Recognising navigational patterns**

original starting point, retracing path.

method, may be a mixture of the above types.

a linked or associated tree.

how to find the information required.

route.

definitive list and a graphical version of the patterns was prepared.

areas from the first level and returning, one way or two way.

This area brings in another area of my research that of using landmarks within multimedia and Internet research, so that users can navigate within the resource and be aware of their physical place within this resource. This is in much the same way as a games user can fix their position in a game either by geographical landmarks or by placing themselves within a maze or 3D map. The concept of a mental map related to navigation patterns to the mental processes involved in working through software, and with physical awareness of where the user was in the package. Shum (1990), discussing navigating in multimedia and the user's acquisition of spatial knowledge as a two-step process, produced similar results to Simpson and McKnight (1990). Shum explained the first step as the acquisition of route knowledge, where the information was context dependent, and the second step as the acquisition of map knowledge, in which the individual understood the global spatial relationships, navigation was then world centred.

Using audit trails or server log files it is possible to back up this more esoteric research with objective behavioural data however, the data files are not easy to capture, analyse and interpret. The greatest problem to these scanned and audio files though is the time that it takes to analyse these sufficiently in order that robust statements can be made backed by actual data. This problem has been recognized by other researchers (c.f. Spiliopoulou, et al. 1999) who developed methodological procedures for visualizing and analysing log file data for data mining purposes. However, there are a few methods that have been developed which would enable the navigational behaviour of individuals to be analysed using the methods of the psychological researcher, such as using a range of different task demands and conditions (Unz & Hesse, 1999).

#### **6. Investigating navigational patterns**

Research was conducted into the methods users employ in navigating through multimedia and internet resources. The three elements of the research into these patterns consisted of:

A taxonomy of multimedia packages – concentrating on the navigation paths and available tools

A First Study – of school pupils using multimedia packages (2-3 packages for each pair – 23 pairs)

A Second Study – of adults using a specific encyclopaedic multimedia resource – 20 individuals.

The taxonomy was useful as it revealed how the design of a range of multimedia packages affected how these resources could be used. The second study on schoolchildren fits in well with the scope of this chapter as the children could be termed digital natives but as they were young, they had not experienced the full effect of the digital revolution, in terms of access to and use of computers, mobile devices and the Internet. The third study looked at adults and although they ranged from novice to expert, they ranged from being digital natives to the digital immigrant range of Prensky's work (2001). An interesting point here is that novices were able to benefit from the more experienced experts and it is possible that digital immigrants could learn usage and navigation techniques from digital immigrants in the same way. The other key aspect of this research is the nature and amount of learning that took place and how the navigation patterns used affected what information was retrieved and the sections that were used within the resources. The most important of these three areas of study for this chapter is the last of these. In this research, each user was given three tasks within the same extensive encyclopaedic package. For each of these tasks a 2D chart was produced. From these navigation charts (60 in all) each users navigation patterns and preferences were assessed.

These charts also enabled comparison of different users across each task. Using the scanned records of each user's progression through the software and these navigation charts, it was possible to create a range of the different types of navigation pattern found. The list of navigation types was then compared to the other patterns, which are shown in the list below. From these scanned records or audit trails of each user and the subsequent charts it was possible to build taxonomy of navigation patterns. These included nine main types (with several sub types).

### **7. Recognising navigational patterns**

164 Interactive Multimedia

This area brings in another area of my research that of using landmarks within multimedia and Internet research, so that users can navigate within the resource and be aware of their physical place within this resource. This is in much the same way as a games user can fix their position in a game either by geographical landmarks or by placing themselves within a maze or 3D map. The concept of a mental map related to navigation patterns to the mental processes involved in working through software, and with physical awareness of where the user was in the package. Shum (1990), discussing navigating in multimedia and the user's acquisition of spatial knowledge as a two-step process, produced similar results to Simpson and McKnight (1990). Shum explained the first step as the acquisition of route knowledge, where the information was context dependent, and the second step as the acquisition of map knowledge, in which the individual understood the global spatial relationships, navigation

Using audit trails or server log files it is possible to back up this more esoteric research with objective behavioural data however, the data files are not easy to capture, analyse and interpret. The greatest problem to these scanned and audio files though is the time that it takes to analyse these sufficiently in order that robust statements can be made backed by actual data. This problem has been recognized by other researchers (c.f. Spiliopoulou, et al. 1999) who developed methodological procedures for visualizing and analysing log file data for data mining purposes. However, there are a few methods that have been developed which would enable the navigational behaviour of individuals to be analysed using the methods of the psychological researcher, such as using a range of different task demands

Research was conducted into the methods users employ in navigating through multimedia and internet resources. The three elements of the research into these patterns consisted of: A taxonomy of multimedia packages – concentrating on the navigation paths and available

A First Study – of school pupils using multimedia packages (2-3 packages for each pair – 23

A Second Study – of adults using a specific encyclopaedic multimedia resource – 20

The taxonomy was useful as it revealed how the design of a range of multimedia packages affected how these resources could be used. The second study on schoolchildren fits in well with the scope of this chapter as the children could be termed digital natives but as they were young, they had not experienced the full effect of the digital revolution, in terms of access to and use of computers, mobile devices and the Internet. The third study looked at adults and although they ranged from novice to expert, they ranged from being digital natives to the digital immigrant range of Prensky's work (2001). An interesting point here is that novices were able to benefit from the more experienced experts and it is possible that digital immigrants could learn usage and navigation techniques from digital immigrants in the same way. The other key aspect of this research is the nature and amount of learning that took place and how the navigation patterns used affected what information was retrieved and the sections that were used within the resources. The most important of these three areas of study for this chapter is the last of these. In this research, each user was given

was then world centred.

tools

pairs)

individuals.

and conditions (Unz & Hesse, 1999).

**6. Investigating navigational patterns**

The following navigation patterns were recognised from the initial empirical work and these were then further tested in the second study. These navigational patterns were put into a definitive list and a graphical version of the patterns was prepared.

The list of navigational patterns above shows all the subdivisions of each pattern type, these should mean that the navigational patterns are clear and that the classification can be used with many different types of digital resources. This is especially useful for educational resources when the user may not be aware of how the package/ resource are structured or how to find the information required.

Multimedia Design Decisions, Visualisations and the User's Experience 167

The chapter includes the detail of both the initial design and of how the improvements affected the usage, together with tips for future design decisions. It is intended that some visualisations of how users have navigated through specific packages will be included, as well as visual images of different methods of designing the package to allow easier

The Open University have developed a multimedia package for the course 'Homer: Poetry and Society', which looks at the nature of the relationship between the Homeric poems and

The course is taught currently through print (poems, commentaries, essays, study guide), video (of archaeological sites, site plans, excavations, material objects) and audio (readings of poetry, expert discussions). In the format of these narrative media, a section of the teaching will have quite a complex structure, with different media (printed study guide, video of archaeological site, students' notes, published text) carrying different parts of the

navigation and use.

**8.3 The Homer package**

the archaeological data on Ancient Greece.

1. Hypothetical question – in the Guide 2. Critique of assumption – in the Guide 3. Presentation of evidence - view the Video 4. Elicited student's response – put into the Notes 5. Interpretation of evidence – checking in the Guide

storyline. An example of this would be, for example:

6. Contradictory evidence – reading about other evidence - Text 7. Interpretation of evidence – reviewing interpretation - Guide

8. Synthesis at higher level – being able to synthesise the information - Guide

Fig. 2. Screen shot from the Open University Carbon Cycle

#### **8. Recognising design decisions and the effect of these on how users use the software ‒ Good design practice and problem designs**

#### **8.1 British Library Medieval Realms**

The excellent British Library's package has been developed over a long period to include: historical information, real documents, music, arts and poetry but had a complex interface. Aimed at school pupils, the package assumed knowledge in many of these specific fields in order to be able to select information. During testing of the package with the planned age group this knowledge was not apparent and they spent a lot of time trying to work out how to find the material. Giving the developers an idea of how complex they found the package, and the types of searches or information the users needed, allowed a total redesign of the interface. Keeping the high quality underlying information base meant that the redesign was specifically targeted at the interface. The redesign made the resource easier to use and with far greater hit rates for the relevant searches.

Fig. 1. Screen shot from Medieval Realms, British Library

Perseus is an interactive guide to Classical Greece which includes various items such as maps, photographs of sites, datasets of coins and sculpture, as well as texts in Ancient Greek and translations. This has been designed for university students, but again the sources are very text based. The resource has now been developed into a website, called the Perseus Digital Library.

Perhaps one of the problems of moving from a library or text resource is how to make the resource more interesting and the methods of linking different types of media to make a full multimedia product.

There are many other multimedia packages with a historical or archaeological theme. One of these is Viking World produced by the York Jorvik Centre as a commercial venture. This uses a large image database but presently has little information on each of the artefacts. Other products from the British Library include Inventors and Inventions, The Patent Express Jukebox and the Electronic Beowulf. The latter of which has received critical acclaim.

#### **8.2 The Open University Carbon Cycle**

The Carbon Cycle software was part of a larger Science introductory course from the OU comprising over 100 CD's. The carbon cycle was receiving mixed responses as users had spent a long time exploring all the options before eventually going through the one correct route to achieve the required completion of the task. Redesigning the interface meant that the majority of students followed the most suitable route and much less time was taken to complete the section of the work.

The chapter includes the detail of both the initial design and of how the improvements affected the usage, together with tips for future design decisions. It is intended that some visualisations of how users have navigated through specific packages will be included, as well as visual images of different methods of designing the package to allow easier navigation and use.

Fig. 2. Screen shot from the Open University Carbon Cycle

#### **8.3 The Homer package**

166 Interactive Multimedia

The excellent British Library's package has been developed over a long period to include: historical information, real documents, music, arts and poetry but had a complex interface. Aimed at school pupils, the package assumed knowledge in many of these specific fields in order to be able to select information. During testing of the package with the planned age group this knowledge was not apparent and they spent a lot of time trying to work out how to find the material. Giving the developers an idea of how complex they found the package, and the types of searches or information the users needed, allowed a total redesign of the interface. Keeping the high quality underlying information base meant that the redesign was specifically targeted at the interface. The redesign made the resource easier to use and

Perseus is an interactive guide to Classical Greece which includes various items such as maps, photographs of sites, datasets of coins and sculpture, as well as texts in Ancient Greek and translations. This has been designed for university students, but again the sources are very text based. The resource has now been developed into a website, called the Perseus

Perhaps one of the problems of moving from a library or text resource is how to make the resource more interesting and the methods of linking different types of media to make a full

There are many other multimedia packages with a historical or archaeological theme. One of these is Viking World produced by the York Jorvik Centre as a commercial venture. This uses a large image database but presently has little information on each of the artefacts. Other products from the British Library include Inventors and Inventions, The Patent Express

The Carbon Cycle software was part of a larger Science introductory course from the OU comprising over 100 CD's. The carbon cycle was receiving mixed responses as users had spent a long time exploring all the options before eventually going through the one correct route to achieve the required completion of the task. Redesigning the interface meant that the majority of students followed the most suitable route and much less time was taken to

Jukebox and the Electronic Beowulf. The latter of which has received critical acclaim.

**8. Recognising design decisions and the effect of these on how users use**

**the software ‒ Good design practice and problem designs**

**8.1 British Library Medieval Realms**

with far greater hit rates for the relevant searches.

Fig. 1. Screen shot from Medieval Realms, British Library

Digital Library.

multimedia product.

**8.2 The Open University Carbon Cycle**

complete the section of the work.

The Open University have developed a multimedia package for the course 'Homer: Poetry and Society', which looks at the nature of the relationship between the Homeric poems and the archaeological data on Ancient Greece.

The course is taught currently through print (poems, commentaries, essays, study guide), video (of archaeological sites, site plans, excavations, material objects) and audio (readings of poetry, expert discussions). In the format of these narrative media, a section of the teaching will have quite a complex structure, with different media (printed study guide, video of archaeological site, students' notes, published text) carrying different parts of the storyline. An example of this would be, for example:

Multimedia Design Decisions, Visualisations and the User's Experience 169

States (2010). The software consists of over 400 desert scenes, with narrated information, images, sound clips and video. The Gila field centre allows the user to discover more about the desert including ecology, climate and geography. The user can learn what you should take with you on a field trip, conduct experiments, look up information and run their own slide show. The CD also features 100 video sequences from the BBC Natural History Unit, 800 sound clips and narrated tours, 1,300 photographs and illustrations and allows both a scrapbook and print option as well 'satellite links' to other deserts and laboratory

This package was designed with a strong route defined basis. The routes were selected from a compass point system. The user could choose to start off North, South, West or East. However as most users are very organised into a top to bottom and left to right orientation in terms of movement through material, the most popular initial route was North. Giving people a choice without being aware of their natural preferences is not helpful, as in this instance introductory information was given for each of the routes. If everyone using the package is going to select a first route, i.e. North then this route and this route only, could have the introductory information. Giving multiple repetitions of the basic information (or even of similar interfaces) is likely to either annoy the user or to totally put them off using

An alternative design would be for the introductory section to appear which ever route is chosen first, but not repeated in subsequent route choices. This though would involve

experiments

the package further.

Fig. 4. Screen shot from the Sonoran Desert

Fig. 3. Screen shot from A295 Homer: Poetry and Society

From the illustration of the Homer multimedia package it can be seen that the structure of the work packages relates to each week of the course and these have been organized as an introduction and then six weekly sessions.

The icons under the Trojan horse image are links into other areas of the package. These include detailed archaeological tools, showing the user how these are employed and the methods used with them, maps and bird's eye views of the sites, notes and historical information as well as all the poems and associated works. This resource is an amazing, complex resource for Homer. However the way it has been designed with a controlling tutorial workload restricts the use the student can make of it. For instance only the tools which are needed for each week are introduced and certain areas are restricted until the course has arrived at this stage. This limits the amount of serendipity for the student and puts all the resources in the hands of the programme director. This also effectively blocks reuse as it would be necessary to go through the course again. This would have been better designed with the information resource available and the course as a prescribed route for first time users. I would suspect that the majority of the users would do this the first time but not be prepared to go through this again, hence limiting the use and usability of the package. The amount of work which has gone into developing it is therefore wasted and should have been allowed to augment and support the learning rather than control it.

#### **8.4 The Sonoran Desert**

The Sonoran desert multimedia package has been produced by the BBC and is advertised as being able to 'Explore one of the world's most fascinating deserts, accompanied by an experienced desert guide'. The Desert multimedia has been arranged as if the user was going on a multimedia field trip to the Cactus Desert of Arizona, south western United

From the illustration of the Homer multimedia package it can be seen that the structure of the work packages relates to each week of the course and these have been organized as an

The icons under the Trojan horse image are links into other areas of the package. These include detailed archaeological tools, showing the user how these are employed and the methods used with them, maps and bird's eye views of the sites, notes and historical information as well as all the poems and associated works. This resource is an amazing, complex resource for Homer. However the way it has been designed with a controlling tutorial workload restricts the use the student can make of it. For instance only the tools which are needed for each week are introduced and certain areas are restricted until the course has arrived at this stage. This limits the amount of serendipity for the student and puts all the resources in the hands of the programme director. This also effectively blocks reuse as it would be necessary to go through the course again. This would have been better designed with the information resource available and the course as a prescribed route for first time users. I would suspect that the majority of the users would do this the first time but not be prepared to go through this again, hence limiting the use and usability of the package. The amount of work which has gone into developing it is therefore wasted and should have been allowed to augment and support the learning rather than control it.

The Sonoran desert multimedia package has been produced by the BBC and is advertised as being able to 'Explore one of the world's most fascinating deserts, accompanied by an experienced desert guide'. The Desert multimedia has been arranged as if the user was going on a multimedia field trip to the Cactus Desert of Arizona, south western United

Fig. 3. Screen shot from A295 Homer: Poetry and Society

introduction and then six weekly sessions.

**8.4 The Sonoran Desert**

States (2010). The software consists of over 400 desert scenes, with narrated information, images, sound clips and video. The Gila field centre allows the user to discover more about the desert including ecology, climate and geography. The user can learn what you should take with you on a field trip, conduct experiments, look up information and run their own slide show. The CD also features 100 video sequences from the BBC Natural History Unit, 800 sound clips and narrated tours, 1,300 photographs and illustrations and allows both a scrapbook and print option as well 'satellite links' to other deserts and laboratory experiments

This package was designed with a strong route defined basis. The routes were selected from a compass point system. The user could choose to start off North, South, West or East. However as most users are very organised into a top to bottom and left to right orientation in terms of movement through material, the most popular initial route was North. Giving people a choice without being aware of their natural preferences is not helpful, as in this instance introductory information was given for each of the routes. If everyone using the package is going to select a first route, i.e. North then this route and this route only, could have the introductory information. Giving multiple repetitions of the basic information (or even of similar interfaces) is likely to either annoy the user or to totally put them off using the package further.

Fig. 4. Screen shot from the Sonoran Desert

An alternative design would be for the introductory section to appear which ever route is chosen first, but not repeated in subsequent route choices. This though would involve

Multimedia Design Decisions, Visualisations and the User's Experience 171

interviews were held with the users to determine why they decided to follow a specific route and to ask questions such as had they found what they were looking for, was there sufficient depth and if they were aware of specific types of routes they were using. One interesting outcome of this was that the younger children thought they had investigated a large proportion of each of the resources they had used, whereas in fact they had been

In some resources a key issue was whether or not the resource was at the right level or had sufficient content to both interest them and hold their attention. In some packages notably the Eyewitness series the information stopped at the point where they were becoming most interested – so it was rather too shallow. If this is likely to be the case then linking the resource to more detailed web pages or sites may be an answer, allowing the keen student to develop their knowledge further. In some cases the front end (the interface) was too simplistic – such as the Peanuts cartoon interface on a math resource. Younger students liked the cartoon interface, but found the math underneath too difficult, and getting the answers wrong repeatedly meant they rapidly dropped out of the package. In the opposite way – a too simple front end with complex information underneath – the resource The Way Things Work, again had a cartoon front end and simple cartoon like diagrams of how they worked, but the physics described was of a much higher standard. This package also had a workshop front end which made a series of noises – very upsetting to students trying to work nearby and causing laughter and playful behaviour from its devotees! Again testing this in a school environment would have immediately shown up these problems – even the

Further development of the navigation tools would necessitate knowing information on each node and where the information is found within the resource. This information proved difficult to find for commercial resources and so a new resource would need to be developed around this basis. Research by McGuffin and Schraefel (2003) called Hyperstructures has involved representing information on the web. Hyperstructures which include component models such as Zzstructures and mSpaces are described using graph theory. These structures use hypertext structures like the web and hypertext links to show links between pages but also the multiple relationships between information within the pages. The INTENTS project from the University of Dublin (2007) involves the design of a series of intelligent knowledge based tools to assist in the construction, navigation and management of hypertext documents. Examples of these tools are 1) Creation of a framework which gives conceptual structure to a document corpus which can then be interrogated by an intelligent hypertext browser or 2) A knowledge based browser, using metadata from other tools in the series, as well as document management and authoring tools, and user behaviour tools. Developments of this nature would allow further

exploration of individual's preferences and behaviour when using navigation tools.

and extent of what they would prefer to have in their own toolkit.

The next part of the research was to develop a list of the most essential navigational tools to be used by designers in the construction of multimedia/Internet resources. This list gives an overview of the necessary component parts of good quality interactive multimedia. These resources are a good example of how digital natives could be used to determine the nature

through relatively small amounts of the whole resource.

children were aware of them and stated that it was interesting but noisy.

**11. Navigational tools ‒ Investigating navigation tools**

getting the student/user to register with the package before using it. Most of the people using educational packages would be doing this as part of their courses this might not be an issue, but for courses where a high throughput is needed this might create a time delay. This sort of decision is particularly important for software designed for younger children, who might spend some valuable time filling out a form and have less time in the interactive package. In primary schools where pupils need to share limited resources this could be difficult balance to resolve. Obviously if the package performs some sort of testing or assessment then the registering of the user is important, or in some cases essential, as with online or lifelong learning courses where progress is measured electronically. However in these cases it should be possible to keep the registration required to a minimum.

#### **9. Problems with design decisions**

This section deals with examples of software where the design of the package has caused problems or significantly affected how the resource is used.

Investigating large resources has highlighted a series of problems that are common with both multimedia and large resources generally. Both of these resources need to be fully indexed and it is necessary to be able to search for and find, specific items reasonably accurately and quickly. Many of the problems of using the resources are based in the slow speed of searching and the poor quality of the results obtained. Much of this is due to the key wording and other selection techniques, but some is due to poor structural design of the package. In examples such as the British Library Medieval Realms, the initial package was very difficult to navigate and the detail involved did not match the age range of the material it was targeted at. Hence areas within the resource such as the Black Death which were taught to second and third year secondary students required a real knowledge of the period and the events and people in order to access the information. Giving feedback to the British Library meant that they went through a period of testing and then of redesigning the original interface to make it more comprehensible to the pupils it was meant to help.

The other main issue of any large and rich resource is that of serendipity. Rather than going to a specific area of the resource it is helpful for beginners to obtain an idea of the wealth and depth of the resource. Allowing students to browse or offering a quick tour, or better still a random review of the material may allow them to view or be interested in something they might not otherwise have discovered. One of the issues with using specific software in a school context is both the limited number of computers with the software – which restricts the amount of time they can spend on it, but also the time constraints of having to find the right material to answer the questions as quickly as possible. The result of this is that each student's experience of using the resource is likely to be similar, hurried and very rapid with no browsing or discovery time.

#### **10. Visualisation**

Another area of interest in this research is visualisations of how the users have approached the software, comments on these and developing tools for visualisation.

In the main research work each of the pairs and the individuals were mapped using the software. This was done by recording a scanned progress of the user through the software so that each step they took and the time taken on each screen was recorded. Additionally

getting the student/user to register with the package before using it. Most of the people using educational packages would be doing this as part of their courses this might not be an issue, but for courses where a high throughput is needed this might create a time delay. This sort of decision is particularly important for software designed for younger children, who might spend some valuable time filling out a form and have less time in the interactive package. In primary schools where pupils need to share limited resources this could be difficult balance to resolve. Obviously if the package performs some sort of testing or assessment then the registering of the user is important, or in some cases essential, as with online or lifelong learning courses where progress is measured electronically. However in

This section deals with examples of software where the design of the package has caused

Investigating large resources has highlighted a series of problems that are common with both multimedia and large resources generally. Both of these resources need to be fully indexed and it is necessary to be able to search for and find, specific items reasonably accurately and quickly. Many of the problems of using the resources are based in the slow speed of searching and the poor quality of the results obtained. Much of this is due to the key wording and other selection techniques, but some is due to poor structural design of the package. In examples such as the British Library Medieval Realms, the initial package was very difficult to navigate and the detail involved did not match the age range of the material it was targeted at. Hence areas within the resource such as the Black Death which were taught to second and third year secondary students required a real knowledge of the period and the events and people in order to access the information. Giving feedback to the British Library meant that they went through a period of testing and then of redesigning the

original interface to make it more comprehensible to the pupils it was meant to help.

The other main issue of any large and rich resource is that of serendipity. Rather than going to a specific area of the resource it is helpful for beginners to obtain an idea of the wealth and depth of the resource. Allowing students to browse or offering a quick tour, or better still a random review of the material may allow them to view or be interested in something they might not otherwise have discovered. One of the issues with using specific software in a school context is both the limited number of computers with the software – which restricts the amount of time they can spend on it, but also the time constraints of having to find the right material to answer the questions as quickly as possible. The result of this is that each student's experience of using the resource is likely to be similar, hurried and very rapid with

Another area of interest in this research is visualisations of how the users have approached

In the main research work each of the pairs and the individuals were mapped using the software. This was done by recording a scanned progress of the user through the software so that each step they took and the time taken on each screen was recorded. Additionally

the software, comments on these and developing tools for visualisation.

these cases it should be possible to keep the registration required to a minimum.

**9. Problems with design decisions**

no browsing or discovery time.

**10. Visualisation**

problems or significantly affected how the resource is used.

interviews were held with the users to determine why they decided to follow a specific route and to ask questions such as had they found what they were looking for, was there sufficient depth and if they were aware of specific types of routes they were using. One interesting outcome of this was that the younger children thought they had investigated a large proportion of each of the resources they had used, whereas in fact they had been through relatively small amounts of the whole resource.

In some resources a key issue was whether or not the resource was at the right level or had sufficient content to both interest them and hold their attention. In some packages notably the Eyewitness series the information stopped at the point where they were becoming most interested – so it was rather too shallow. If this is likely to be the case then linking the resource to more detailed web pages or sites may be an answer, allowing the keen student to develop their knowledge further. In some cases the front end (the interface) was too simplistic – such as the Peanuts cartoon interface on a math resource. Younger students liked the cartoon interface, but found the math underneath too difficult, and getting the answers wrong repeatedly meant they rapidly dropped out of the package. In the opposite way – a too simple front end with complex information underneath – the resource The Way Things Work, again had a cartoon front end and simple cartoon like diagrams of how they worked, but the physics described was of a much higher standard. This package also had a workshop front end which made a series of noises – very upsetting to students trying to work nearby and causing laughter and playful behaviour from its devotees! Again testing this in a school environment would have immediately shown up these problems – even the children were aware of them and stated that it was interesting but noisy.

#### **11. Navigational tools ‒ Investigating navigation tools**

Further development of the navigation tools would necessitate knowing information on each node and where the information is found within the resource. This information proved difficult to find for commercial resources and so a new resource would need to be developed around this basis. Research by McGuffin and Schraefel (2003) called Hyperstructures has involved representing information on the web. Hyperstructures which include component models such as Zzstructures and mSpaces are described using graph theory. These structures use hypertext structures like the web and hypertext links to show links between pages but also the multiple relationships between information within the pages. The INTENTS project from the University of Dublin (2007) involves the design of a series of intelligent knowledge based tools to assist in the construction, navigation and management of hypertext documents. Examples of these tools are 1) Creation of a framework which gives conceptual structure to a document corpus which can then be interrogated by an intelligent hypertext browser or 2) A knowledge based browser, using metadata from other tools in the series, as well as document management and authoring tools, and user behaviour tools. Developments of this nature would allow further exploration of individual's preferences and behaviour when using navigation tools.

The next part of the research was to develop a list of the most essential navigational tools to be used by designers in the construction of multimedia/Internet resources. This list gives an overview of the necessary component parts of good quality interactive multimedia. These resources are a good example of how digital natives could be used to determine the nature and extent of what they would prefer to have in their own toolkit.

Multimedia Design Decisions, Visualisations and the User's Experience 173

for further enhancement and expansion. This community is based on lecturers and researchers in the arts and humanities area. The Arch project is now developing this resource further, to include both an interface for the community to add and develop content, but also a series of demonstrators to show what is possible when using large resources. The

The demonstrators for this project, include an outline of the tools presented, but also an interactive reconstruction of a house/museum format to store the exhibits. Previous work on the use of metaphors in multimedia has shown that although there are problems in using familiar structures such as a house that some users fail to realise that each room has a different function. The aim of this project is to help with the navigation and visualisation of the information resource but to make this discovery process interactive. The research will include both diagrams and images of these navigational routes, but also how these tools can be used and the type of structure for the resources that would aid discovery and usage in

Analysing multimedia and Internet resources has highlighted several problems, which were common across many resources. These include: insufficient or inadequate capabilities for navigation, the lack of awareness of where the user is in the package, and misconceptions of the amount of the resource they had investigated. Many of these problems could be resolved with better interfaces and more adequate testing; however allowing the users to develop navigational skills and providing them with a toolkit to do this may have longer term and more developmental benefits. These navigation tools are also meant to give an ability to the user to navigate on a physical level, and to do physical actions such as they might do in exploring a physical resource. Common actions in this environment would be walk, look around, zoom, pan, orbit, examine (as well as pick up/ inspect) fly – or bird's eye view, and turntable. Using this type of physical approach may lessen the need for specific navigational tools but support a toolbox or palette of navigational tools approach and the time investment needed is reduced if these tools are re-usable over a wide array of resources.

The standard types of navigational tool hence include indexes, search engines, maps, plans, timelines, histories and directional devices. There is however, a whole range of more innovative tools that could be developed for multimedia and Internet resources. The development of Open Source software has meant that it would be possible to develop a set of tools, which conform to XML or Java formats. This would mean that these tools could be freely available and added in with the minimum of effort. The newer tools include breadcrumbs, searchlights, beacons, landmarks, 3-D mapping, trackers, location finders, compasses, and route maps. Breadcrumbs are paths selected by the user of the key screens they have visited and can be set up to record different aspects of the search or information retrieval, Searchlights and beacons allow selected searches to be highlighted. There are ranges of direction/ map finders from games software where compass type location finders, and 2-D plans of the whole resource, help locate the user and allow them to find specific

The previous sections detailed current research on navigation patterns, new navigation tools and an outline of possible directions in visualizing these with some visualisation examples

**14. Recommendations for design and user assistance**

project also brings together research on tools and suitable software packages.

education.

areas of the resource.

across a range of disciplines.

	-

This list can be used by designers as a check to ensure that as many of these as feasible are built into the new software package. Some of these are reasonably straightforward but other items such as the intelligent tutor involve more detailed construction and facilities and substantial input from the tutor(s) to make this effective. If the navigation tools are produced as a series of device independent software tools then these could be added onto each resource and the user allowed to select from the available palette.

#### **12. Navigational tools ‒ and how these relate to the resources used ‒ Producing new navigation tools**

The newer navigational tools consist of the following: beacons, landmarks, breadcrumb trails, searchlights, intelligent agents, 2D and 3D maps and interactive guides. Beacons allow significant items or areas to be highlighted or lit up for particular students, or for whole classes or for specific search areas. Landmarks relate to the resource in the same way as physical resources, by providing points at which users can recognise that they have been there or they have gone past this point. These can be colour markers or set up as points within the resources so that users can find them again easily. A simple map could involve reproducing a schematic image of these and putting these landmarks on to a virtual map of the entire resource. Searchlights can be used with or without a search engine and can highlight just the relevant section or be used within areas where specific references are made to the search term. Breadcrumb trails enable a series of markers (with the breadcrumbs left at strategic points, i.e. whenever a user flags this area or screen, or would want to return to it. These trails could be set up either with different markers/ objects or colours or for specific trails. For instance, while searching for a particular set of images the user may find several that are useful for another project and they could then use several different breadcrumb trails while researching in the same resource. The example used is of a geographical resource with breadcrumb tails in different colours. It should be possible to create a similar navigation tool within a multimedia or internet based resource so that individual routes through the resource are detailed as a distinct route and can then be viewed on an overview map. New technologies such as geo-location, the facilities of smart phones and usage knowledge by digital natives could all be used to make learning more interesting, relevant and available ubiquitously and whenever needed.

#### **13. Recommendations ‒ General**

New research using this information on preferred navigation and tools usage, together with the visualisation of the routes has been applied to new resources. The large catalogue of Arts and Humanities material created by the Intute project based at Oxford has created a large body of Internet accessible web sites with information on their quality and content and their applicability to UK HE courses. This resource is now being made available to a community

1. Navigational tool Diagrams or Map 8. Individual student records/histories

6. Activities - Worksheet/ Task 14. Expert guide, tour, induction/ tutorial

This list can be used by designers as a check to ensure that as many of these as feasible are built into the new software package. Some of these are reasonably straightforward but other items such as the intelligent tutor involve more detailed construction and facilities and substantial input from the tutor(s) to make this effective. If the navigation tools are produced as a series of device independent software tools then these could be added onto

The newer navigational tools consist of the following: beacons, landmarks, breadcrumb trails, searchlights, intelligent agents, 2D and 3D maps and interactive guides. Beacons allow significant items or areas to be highlighted or lit up for particular students, or for whole classes or for specific search areas. Landmarks relate to the resource in the same way as physical resources, by providing points at which users can recognise that they have been there or they have gone past this point. These can be colour markers or set up as points within the resources so that users can find them again easily. A simple map could involve reproducing a schematic image of these and putting these landmarks on to a virtual map of the entire resource. Searchlights can be used with or without a search engine and can highlight just the relevant section or be used within areas where specific references are made to the search term. Breadcrumb trails enable a series of markers (with the breadcrumbs left at strategic points, i.e. whenever a user flags this area or screen, or would want to return to it. These trails could be set up either with different markers/ objects or colours or for specific trails. For instance, while searching for a particular set of images the user may find several that are useful for another project and they could then use several different breadcrumb trails while researching in the same resource. The example used is of a geographical resource with breadcrumb tails in different colours. It should be possible to create a similar navigation tool within a multimedia or internet based resource so that individual routes through the resource are detailed as a distinct route and can then be viewed on an overview map. New technologies such as geo-location, the facilities of smart phones and usage knowledge by digital natives could all be used to make learning more

2. Known material 9. Customobility issues 3. History 11. Curriculum research 4. Audit Trails - navigational paths 12. Off-line facilities 5. Time line 13. Internet links

7. User Levels 15. Intelligent Tutor

each resource and the user allowed to select from the available palette.

interesting, relevant and available ubiquitously and whenever needed.

New research using this information on preferred navigation and tools usage, together with the visualisation of the routes has been applied to new resources. The large catalogue of Arts and Humanities material created by the Intute project based at Oxford has created a large body of Internet accessible web sites with information on their quality and content and their applicability to UK HE courses. This resource is now being made available to a community

**Producing new navigation tools**

**13. Recommendations ‒ General**

**12. Navigational tools ‒ and how these relate to the resources used ‒**

for further enhancement and expansion. This community is based on lecturers and researchers in the arts and humanities area. The Arch project is now developing this resource further, to include both an interface for the community to add and develop content, but also a series of demonstrators to show what is possible when using large resources. The project also brings together research on tools and suitable software packages.

The demonstrators for this project, include an outline of the tools presented, but also an interactive reconstruction of a house/museum format to store the exhibits. Previous work on the use of metaphors in multimedia has shown that although there are problems in using familiar structures such as a house that some users fail to realise that each room has a different function. The aim of this project is to help with the navigation and visualisation of the information resource but to make this discovery process interactive. The research will include both diagrams and images of these navigational routes, but also how these tools can be used and the type of structure for the resources that would aid discovery and usage in education.

Analysing multimedia and Internet resources has highlighted several problems, which were common across many resources. These include: insufficient or inadequate capabilities for navigation, the lack of awareness of where the user is in the package, and misconceptions of the amount of the resource they had investigated. Many of these problems could be resolved with better interfaces and more adequate testing; however allowing the users to develop navigational skills and providing them with a toolkit to do this may have longer term and more developmental benefits. These navigation tools are also meant to give an ability to the user to navigate on a physical level, and to do physical actions such as they might do in exploring a physical resource. Common actions in this environment would be walk, look around, zoom, pan, orbit, examine (as well as pick up/ inspect) fly – or bird's eye view, and turntable. Using this type of physical approach may lessen the need for specific navigational tools but support a toolbox or palette of navigational tools approach and the time investment needed is reduced if these tools are re-usable over a wide array of resources.

The standard types of navigational tool hence include indexes, search engines, maps, plans, timelines, histories and directional devices. There is however, a whole range of more innovative tools that could be developed for multimedia and Internet resources. The development of Open Source software has meant that it would be possible to develop a set of tools, which conform to XML or Java formats. This would mean that these tools could be freely available and added in with the minimum of effort. The newer tools include breadcrumbs, searchlights, beacons, landmarks, 3-D mapping, trackers, location finders, compasses, and route maps. Breadcrumbs are paths selected by the user of the key screens they have visited and can be set up to record different aspects of the search or information retrieval, Searchlights and beacons allow selected searches to be highlighted. There are ranges of direction/ map finders from games software where compass type location finders, and 2-D plans of the whole resource, help locate the user and allow them to find specific areas of the resource.

#### **14. Recommendations for design and user assistance**

The previous sections detailed current research on navigation patterns, new navigation tools and an outline of possible directions in visualizing these with some visualisation examples across a range of disciplines.

Multimedia Design Decisions, Visualisations and the User's Experience 175

Following these recommendations would mean that better, more user oriented software would be produced that was easy and intuitive to use and allowed rapid and successful

The doctoral research quoted in this chapter was completed with a scholarship from the Engineering and Physical Science Research Council (EPSRC). My supervisors for the doctorate were Professor Josie Taylor, Dr Ann Jones and Professor John Richardson from the

Benyon, D Turner, P and Turner, S (2005) Designing Interactive Systems: People, Activities,

Canter, D., Rivers, R., & Storrs, G. (1985) Characterising user navigation through complex

Chambers, E. & Rae, J. (1999) Evaluation of the Homer CD-ROM: Final Report. Milton

Dix, A, Finlay, J Abowd, G and Beale, R (2003) Human-Computer Interaction, Prentice Hall,

Fenley, S (2004) Metaphors in interactive multimedia. Workshop paper for Human

Fenley, S. (2006) *Navigational patterns in interactive multimedia*. PhD thesis, Open University Frand, J. L. (2000) The information-age mindset. Changes in students and implications for higher education. *EDUCAUSE Review,* 35(5), 15-24. ERM0051.pdf a/a Horney, M.A. (1993) Case Studies of Navigational Patterns in Constructive Hypertext.

Henderson, L. (1993) Interactive multimedia computer courseware and culturally

Laurillard, D (1998) Multimedia and the learner's experience of narrative. Computers &

McGuffin, M. J. & Schraefel, M.C. (2004) A Comparison of Hyperstructures: Zzstructures,

McKnight, C., Dillon, A.P. & Richardson, J.H., Eds. (1993) *Hypertext: A Psychological*

Nielsen, J (2002) Building Web Sites With Depth (Web Techniques, Feb 2001)By Dr. Dobb's

Nielsen, J (1995) Navigating Large Information Spaces In *Multimedia and Hypermedia*. San

appropriate ways of learning. In C. Latchem, J. Williamson, & L. Henderson-Lancett (Eds.), *Interactive multimedia: Practice and promise*, 189-203 London: Kogan

mSpaces, and Polyarchies. In: ACM Conference on Hypertext and Hypermedia,

Computer Interaction Conference (HCI2003), University of Bath. Internet

data structures. *Behaviour and Information Technology*, 4 (2), 93-102

Contexts, Technologies Addison Wesley, London

http://kn.open.ac.uk/public/getfile.cfm?documentfileid=111

Laurillard, D, ( ) Multimedia and the learner's experience of narrative

2004, Santa Cruz, California, USA, 153-162

*Perspective*, Ellis Horwood, Chichester, England

Keynes: Open University.

publication Spring 2004.

*Computers & Education* 20 (3), 257-270

Jacob Nielsen web site http://www.useit.com/

Education, 31, (2), 229-242

Diego, CA: Academic Press

searches in the minimum of time.

**16. Acknowledgments**

Open University.

**17. References**

London

Page

Journal

These new areas of research in visualisation which would have significant impact when applied to multimedia and internet resources and these techniques would allow an expansion of the possible views or investigations possible within interactive media resources. Much of this work is being done in the field of social networking. In this research the connections that people make over work or research is mapped and different methods of visualising these as maps, charts, patterns etc. give a different perspective on potential analysis methods. The key point of the chapter is the need to assess navigation abilities of students, even of digital natives, to develop their skills and abilities in this area and to give them a set of device independent navigation tools that they can use in many different resources.

The benefits of the visualisations are that these are easy to understand and to remember and a graphical image is easier to retain in the memory. Allowing students to link work areas together and to see the benefit of the work they have done is very important. The Internet will remain a source of information resources and users need to be fully equipped to use these and to make the best use of limited time. Allowing them the facility to re-use information and skills is equally essential and developing an appropriate set of tools into a tool bag or palette would be a useful skill set to take with them into employment or further study.

#### **15. Conclusions and a way forward for designers**

The conclusions from the research described in this chapter cover several areas, such as user preferences, potential navigation routes, testing designs and providing both flexibility and generic tools to assist the user.

Firstly, in the user preference areas, the research has shown that each individual user has very distinct preferences in how they navigate. Most users would perform some sort of linear route and most would do some hierarchical searching. Some users wanted to complete every section, or to cover all the screens in a particular sector, however the expert users were usually less systematic and would search for exactly what they wanted. It is therefore important to design different routes and paths through the resource.

Secondly the potential navigation routes in some multimedia and Internet resources were very restricted often with only one main route, forcing users to go through the resource in a specific way. This may be acceptable for introductory sessions but becomes very limiting in terms of exploring or self discovery modes. Ideally several different ways of navigating should be built in to the resource, with at least linear and hierachical routes being available.

Thirdly echoing Neilsen's work it is essential that developers test the design and the resource at every stage of its development, with the same age and experience for the target audience.

Finally producing a palette or toolkit of suitable navigation and visualisation tools that could be generic would be very useful for most users. Using the same tools over a series of resources would be less time consuming and far more productive, as users would not have to relearn another method of navigating. Ideally these tools should be created to work with a whole range of different software packages.

Following these recommendations would mean that better, more user oriented software would be produced that was easy and intuitive to use and allowed rapid and successful searches in the minimum of time.

#### **16. Acknowledgments**

The doctoral research quoted in this chapter was completed with a scholarship from the Engineering and Physical Science Research Council (EPSRC). My supervisors for the doctorate were Professor Josie Taylor, Dr Ann Jones and Professor John Richardson from the Open University.

#### **17. References**

174 Interactive Multimedia

These new areas of research in visualisation which would have significant impact when applied to multimedia and internet resources and these techniques would allow an expansion of the possible views or investigations possible within interactive media resources. Much of this work is being done in the field of social networking. In this research the connections that people make over work or research is mapped and different methods of visualising these as maps, charts, patterns etc. give a different perspective on potential analysis methods. The key point of the chapter is the need to assess navigation abilities of students, even of digital natives, to develop their skills and abilities in this area and to give them a set of device independent navigation tools that they can use in many different

The benefits of the visualisations are that these are easy to understand and to remember and a graphical image is easier to retain in the memory. Allowing students to link work areas together and to see the benefit of the work they have done is very important. The Internet will remain a source of information resources and users need to be fully equipped to use these and to make the best use of limited time. Allowing them the facility to re-use information and skills is equally essential and developing an appropriate set of tools into a tool bag or palette would be a useful skill set to take with them into employment or further

The conclusions from the research described in this chapter cover several areas, such as user preferences, potential navigation routes, testing designs and providing both flexibility and

Firstly, in the user preference areas, the research has shown that each individual user has very distinct preferences in how they navigate. Most users would perform some sort of linear route and most would do some hierarchical searching. Some users wanted to complete every section, or to cover all the screens in a particular sector, however the expert users were usually less systematic and would search for exactly what they wanted. It is

Secondly the potential navigation routes in some multimedia and Internet resources were very restricted often with only one main route, forcing users to go through the resource in a specific way. This may be acceptable for introductory sessions but becomes very limiting in terms of exploring or self discovery modes. Ideally several different ways of navigating should be built in to the resource, with at least linear and hierachical routes being available. Thirdly echoing Neilsen's work it is essential that developers test the design and the resource at every stage of its development, with the same age and experience for the target

Finally producing a palette or toolkit of suitable navigation and visualisation tools that could be generic would be very useful for most users. Using the same tools over a series of resources would be less time consuming and far more productive, as users would not have to relearn another method of navigating. Ideally these tools should be created to work with

therefore important to design different routes and paths through the resource.

**15. Conclusions and a way forward for designers**

resources.

study.

audience.

generic tools to assist the user.

a whole range of different software packages.

http://kn.open.ac.uk/public/getfile.cfm?documentfileid=111

**10**

*1,2Japan 3USA*

**Digital Scope on 2D Communication Sheet**

Bing Zhang1, Youiti Kado1, Kiyohiko Hattori2 and Jiang Yu Zheng3

In recent years, multimedia data have explosively increased in the web environment for education, investigation and exploration. At the same time, augment reality research has paid a great attention to providing views and contents in a real setting to enhance the perception of environments and spaces. A basic function position aware display is demanded in such systems for many applications. Examples include finding locationspecific data in a map [1-5], exploring media related to an object [6-7], viewing process at a spot, watching structure and diagnosing a part of human body, simulating various effects in

Position indexing has been realized on normal computer screen by reading various input devices. However, such systems are not sufficient for indexing details from a large map or hierarchical maps. The windows showing different media such as a map and a street view are usually displayed separately and it requires some effort to correspond the location between two windows [2]. This design has been generalized to many machine interfaces with large scale where the hand movement is tracked by cameras and the detected position drives scenes on another separate screen. In the tabletop projections, two views with different resolutions and media are combined [8-10] with a projector and sensor systems. In this chapter, we construct a platform on which users can watch details just as a magnificent

To develop a new communication technology that complements wired and wireless communications, a special physical medium has been developed to perform both data and power transmission inside a 2D sheet [11]-[13]. Through a connector put on the top of sheet, an electromagnetic (EM) proximity connection is obtained to receive and send the electromagnetic signals, while moving the connector mounted device away from the sheet disconnects the transmission [14]. The size of a sheet can be as wide as several meters and is inexpensive in production. The signal transmission is different from the wireless communication that overwhelms the entire 3D space; it covers a wide surface or on partitioned surfaces to facilitate information transmission and visualization. The sheets can

**1. Introduction**

a designed space, etc.

by touching a special sheet.

**for Location-Specific Multimedia Service**

*1National Institute of Information and Communication Technologies*

*2University of Electro-Communications*

*3Indiana University Purdue University Indianapolis,*

### **Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service**

Bing Zhang1, Youiti Kado1, Kiyohiko Hattori2 and Jiang Yu Zheng3 *1National Institute of Information and Communication Technologies 2University of Electro-Communications 3Indiana University Purdue University Indianapolis, 1,2Japan 3USA*

#### **1. Introduction**

176 Interactive Multimedia

Nielsen, J (1999) Designing Web Usability: The Practice of Simplicity, by New Riders

Oblinger, D. (2003) Boomers, Gen-Xers & Millennials.Understanding the new students. *EDUCAUSE Review,* 38(4), 37-47. www.educause.edu/ir/library/pdf Prensky, M. (2003) Digital game-based learning Computers in Entertainment (CIE), 2003 -

Simpson, A., & McKnight, C. (1990) Navigation in hypertext: structural cues and mental

Spiliopoulou, M., Faulstich, L. C., and Winkler, K. (1999) A Data Miner analysing the

Unz, D., & Hesse, F. (1999) The use of hypertext for learning, *J. of Educational Computing*

maps. In R. McAleese and C. Green (eds.) *Hypertext: State of the Art*. Oxford:

Navigational Behaviour of Web Users. In *Proc. of the Workshop on Machine Learning*

Prensky, M. (2005) *Listen to the natives Educational Leadership*, 2005 - centre4.core-ed.net Shneiderman, B Plaisant, C Cohen, M Jacobs, S (2010) Designing the User Interface: Strategies for Effective Human-Computer Interaction, Pearson, London Shum, S. (1990) *Real and Virtual Spaces: Mapping from Spatial Cognition to Hypertext*.

Publishing, Indianapolis

Hypermedia, 2 (2), 133-58

*Research*, 20(3) 279-295

portal.acm.org

Intellect, 73-83

Norman, D (1988) The Design of everyday things Newprint 2002

*in User Modelling of the ACAI'99* Int. Conf., Greece.

In recent years, multimedia data have explosively increased in the web environment for education, investigation and exploration. At the same time, augment reality research has paid a great attention to providing views and contents in a real setting to enhance the perception of environments and spaces. A basic function position aware display is demanded in such systems for many applications. Examples include finding locationspecific data in a map [1-5], exploring media related to an object [6-7], viewing process at a spot, watching structure and diagnosing a part of human body, simulating various effects in a designed space, etc.

Position indexing has been realized on normal computer screen by reading various input devices. However, such systems are not sufficient for indexing details from a large map or hierarchical maps. The windows showing different media such as a map and a street view are usually displayed separately and it requires some effort to correspond the location between two windows [2]. This design has been generalized to many machine interfaces with large scale where the hand movement is tracked by cameras and the detected position drives scenes on another separate screen. In the tabletop projections, two views with different resolutions and media are combined [8-10] with a projector and sensor systems. In this chapter, we construct a platform on which users can watch details just as a magnificent by touching a special sheet.

To develop a new communication technology that complements wired and wireless communications, a special physical medium has been developed to perform both data and power transmission inside a 2D sheet [11]-[13]. Through a connector put on the top of sheet, an electromagnetic (EM) proximity connection is obtained to receive and send the electromagnetic signals, while moving the connector mounted device away from the sheet disconnects the transmission [14]. The size of a sheet can be as wide as several meters and is inexpensive in production. The signal transmission is different from the wireless communication that overwhelms the entire 3D space; it covers a wide surface or on partitioned surfaces to facilitate information transmission and visualization. The sheets can

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 179

Fig. 2. Two dimensional communication sheet. (Top) 2D sheet construction; (Bottom) The

Meanwhile, the connector is an antenna by which an electromagnetic wave is extracted from or inserted into the 2D sheet. An example of the connector was designed and proposed by Yamahira et al. [14]. The proposed connector consists of metal, dielectric material with the relative permittivity of 10.5, and a subminiature type-A (SMA) connector. The radius of the connector was also determined so that the reflection of the electromagnetic wave from the

Besides the communication, the power can also be supplied through the 2D communication sheet. A device such as PDA can be put anywhere on the sheet to work without extra wires and batteries, which forms a very flexible system of distributed displays. The sheet can be manufactured in a very low cost affordable to be put on table top, walls, and poster boards. The connector is an antenna by which an electromagnetic wave is extracted from or inserted into the 2D sheet. The connector is manufactured in a small piece to contact the sheet for receiving and passing signals. Figure 3 shows a set of connector (each piece is 25x70x3mm and weights 2g) and a small display (LCD of 800x480 pixels) that can be put on any position

Fig. 3. Digital scope by a lightweight high-resolution display receiving data signal through the 2D sheet. The connector for the display contacts the sheet to receive the power signal.

electromagnetic wave seeping out from the 2D sheet.

connector to a cable is minimized at the frequency of 2.4 GHz.

of the communication sheet to receive data and power signals.

be laid on a table, wall, and floor in living and working spaces. By putting a display device onto the sheet, we can look into the information at that position by receiving the signal from a computer transmitted via the sheet.

We design and build a system that can enlarge details at a location or show related media data without directing viewer's focus to separated screens or windows. The system has a function of magnifier or scope to pin-point locations with a small display on a surface. Moreover, because there has no successful example of curved screens that can fit on a curved body for clicking and display, we can then wrap a surface with this two dimensional sheet for signal transmission. By placing a small high-resolution LCD on the sheet, we will be able to see position specific information and media detailed in resolution in contrast to a global map laid on the surface. In this chapter, we introduce the design and implementation of a platform that can look into a 2D or even 3D space by touching the surface sheet with a small display. Several tasks will also be examined to evaluate the performance of the platform. Figure 1 shows such an idea to magnify details at designated locations of an atlas.

Fig. 1. Digital scope on a communication sheet to view media contents at indicated locations in an atlas.

#### **2. The sheet as communication medium**

A 2D communication system (2DCS) consists of two components: a sheet and a connector. Figure 2 illustrates the basic structure of the 2D sheet, which contains three layers in order: conductive layer, dielectric layer and mesh conductive layer from the bottom. The conductive fabric is usually copper or aluminum, whereas the dielectric material is polystyrene. With this layered composition, an electromagnetic wave can be confined within the 2D sheet depending on the relatively permittivity of dielectric layer and mesh size of conductive layer. The interval and width of mesh structure are determined from the wave length of the electromagnetic wave. In the case of 2.4GHz microwave, the interval and width of the stripes in mesh structure are designed to be 7mm and 1mm respectively, as shown in Fig.2 (Top). The electromagnetic field seeped out from the sheet is called as "evanescent wave" as shown as Fig.2 (Bottom), which spreads only to the vicinity of the sheet. When the connector is brought close to the sheet, the energy is transmitted to the connector through the evanescent wave.

be laid on a table, wall, and floor in living and working spaces. By putting a display device onto the sheet, we can look into the information at that position by receiving the signal from

We design and build a system that can enlarge details at a location or show related media data without directing viewer's focus to separated screens or windows. The system has a function of magnifier or scope to pin-point locations with a small display on a surface. Moreover, because there has no successful example of curved screens that can fit on a curved body for clicking and display, we can then wrap a surface with this two dimensional sheet for signal transmission. By placing a small high-resolution LCD on the sheet, we will be able to see position specific information and media detailed in resolution in contrast to a global map laid on the surface. In this chapter, we introduce the design and implementation of a platform that can look into a 2D or even 3D space by touching the surface sheet with a small display. Several tasks will also be examined to evaluate the performance of the platform. Figure 1 shows such an idea to magnify details at designated locations of an atlas.

Fig. 1. Digital scope on a communication sheet to view media contents at indicated locations

A 2D communication system (2DCS) consists of two components: a sheet and a connector. Figure 2 illustrates the basic structure of the 2D sheet, which contains three layers in order: conductive layer, dielectric layer and mesh conductive layer from the bottom. The conductive fabric is usually copper or aluminum, whereas the dielectric material is polystyrene. With this layered composition, an electromagnetic wave can be confined within the 2D sheet depending on the relatively permittivity of dielectric layer and mesh size of conductive layer. The interval and width of mesh structure are determined from the wave length of the electromagnetic wave. In the case of 2.4GHz microwave, the interval and width of the stripes in mesh structure are designed to be 7mm and 1mm respectively, as shown in Fig.2 (Top). The electromagnetic field seeped out from the sheet is called as "evanescent wave" as shown as Fig.2 (Bottom), which spreads only to the vicinity of the sheet. When the connector is brought close to the sheet, the energy is transmitted to the connector through

a computer transmitted via the sheet.

in an atlas.

the evanescent wave.

**2. The sheet as communication medium**

Fig. 2. Two dimensional communication sheet. (Top) 2D sheet construction; (Bottom) The electromagnetic wave seeping out from the 2D sheet.

Meanwhile, the connector is an antenna by which an electromagnetic wave is extracted from or inserted into the 2D sheet. An example of the connector was designed and proposed by Yamahira et al. [14]. The proposed connector consists of metal, dielectric material with the relative permittivity of 10.5, and a subminiature type-A (SMA) connector. The radius of the connector was also determined so that the reflection of the electromagnetic wave from the connector to a cable is minimized at the frequency of 2.4 GHz.

Besides the communication, the power can also be supplied through the 2D communication sheet. A device such as PDA can be put anywhere on the sheet to work without extra wires and batteries, which forms a very flexible system of distributed displays. The sheet can be manufactured in a very low cost affordable to be put on table top, walls, and poster boards. The connector is an antenna by which an electromagnetic wave is extracted from or inserted into the 2D sheet. The connector is manufactured in a small piece to contact the sheet for receiving and passing signals. Figure 3 shows a set of connector (each piece is 25x70x3mm and weights 2g) and a small display (LCD of 800x480 pixels) that can be put on any position of the communication sheet to receive data and power signals.

Fig. 3. Digital scope by a lightweight high-resolution display receiving data signal through the 2D sheet. The connector for the display contacts the sheet to receive the power signal.

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 181

Fig. 5. The signals from two positioning sets are alternatively sent to the receivers by using the wireless Zigbee switch controller. (a) The positioning set 1 is turned on switched by

orientation of the display. The temporal output of the vector provides a trace of orientation moving on top of the communication sheet as depicted in Fig. 6. The drawbacks of this approach are the necessity of the additional sensor devices to generate and sense the infrared and ultrasound waves, as well as the influence from surrounding objects and human bodies, because the reflection of beams from other objects makes the signal noisy

Fig. 6. Obtaining location and orientation of a display device on the 2D sheet.

Zigbee controller, (b) The positioning set 2 is turned on.

when reaching the receivers.

#### **3. Position detection on communication sheet**

To develop the location-specific multimedia applications for 2D communication system, it is highly important to accurately estimate the position where the display is placed over the sheet. Here, two position identification approaches are examined in our platform: (a) position sensing the arrival time difference between the infrared and ultrasound waves, and (b) position sensing the arrival phase difference between the multiple radio inputs.

#### **3.1 Positioning by sensing arrival time difference**

The first approach uses infrared and ultrasound technologies. Two kinds of transmitters are adopted to send the infrared and ultrasonic waves, which are mounted on the display. A sensor bar including two kinds of receivers sensing the infrared and ultrasonic waves is set up on one side of the 2D communication sheet. Upon receiving the infrared and ultrasonic signals from transmitters, the distance between display and sensor bar is acquired from the arrival time difference of two waves. Figure 4 shows the mechanism that the relative position of transmitter is calculated from *S0, S1*, and *W*, which are the distances between the transmitter and ultrasonic receiver, transmitter and another ultrasonic receiver, and two ultrasonic receivers, respectively, as follows

$$\mathbf{y} = \frac{\mathbf{s}\_0^2 - \mathbf{s}\_1^2 + \mathbf{W}^2}{\mathbf{2}^\*\mathbf{W}} \tag{1}$$

$$\mathbf{x} = \sqrt{\mathbf{s}\_0^2 - \mathbf{y}^2} \tag{2}$$

Through the above Trilateration, the location of the transmitter can be determined.

Fig. 4. Acquiring the relative position by the arrival time difference between the infrared and ultrasound waves.

To obtain the orientation of the display in addition to the position, two positioning sets are mounted on two corners of the display. Since the sensor bar cannot simultaneously receive the location information from two positioning sets, a relay switch is built into the display side, as shown in Fig. 5. The relay switch on the display receives the ZigBee signal from the computer via the communication sheet and then controls the turn of the transmitters in sending pulses. The positions of transmitters are linked as a vector that defines the

To develop the location-specific multimedia applications for 2D communication system, it is highly important to accurately estimate the position where the display is placed over the sheet. Here, two position identification approaches are examined in our platform: (a) position sensing the arrival time difference between the infrared and ultrasound waves, and

The first approach uses infrared and ultrasound technologies. Two kinds of transmitters are adopted to send the infrared and ultrasonic waves, which are mounted on the display. A sensor bar including two kinds of receivers sensing the infrared and ultrasonic waves is set up on one side of the 2D communication sheet. Upon receiving the infrared and ultrasonic signals from transmitters, the distance between display and sensor bar is acquired from the arrival time difference of two waves. Figure 4 shows the mechanism that the relative position of transmitter is calculated from *S0, S1*, and *W*, which are the distances between the transmitter and ultrasonic receiver, transmitter and another ultrasonic receiver, and two

> 22 2 0 1 ssW <sup>y</sup> 2\*W

> > 2 2

Through the above Trilateration, the location of the transmitter can be determined.

Fig. 4. Acquiring the relative position by the arrival time difference between the infrared

To obtain the orientation of the display in addition to the position, two positioning sets are mounted on two corners of the display. Since the sensor bar cannot simultaneously receive the location information from two positioning sets, a relay switch is built into the display side, as shown in Fig. 5. The relay switch on the display receives the ZigBee signal from the computer via the communication sheet and then controls the turn of the transmitters in sending pulses. The positions of transmitters are linked as a vector that defines the

(1)

<sup>0</sup> x s y (2)

(b) position sensing the arrival phase difference between the multiple radio inputs.

**3. Position detection on communication sheet**

**3.1 Positioning by sensing arrival time difference**

ultrasonic receivers, respectively, as follows

and ultrasound waves.

Fig. 5. The signals from two positioning sets are alternatively sent to the receivers by using the wireless Zigbee switch controller. (a) The positioning set 1 is turned on switched by Zigbee controller, (b) The positioning set 2 is turned on.

orientation of the display. The temporal output of the vector provides a trace of orientation moving on top of the communication sheet as depicted in Fig. 6. The drawbacks of this approach are the necessity of the additional sensor devices to generate and sense the infrared and ultrasound waves, as well as the influence from surrounding objects and human bodies, because the reflection of beams from other objects makes the signal noisy when reaching the receivers.

Fig. 6. Obtaining location and orientation of a display device on the 2D sheet.

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 183

change with time, which changes in the range from 0 to 360 angular degrees. From a practical point of view, however, since there exist reflected waves from the edges of the 2D sheet, the EM absorber is attached to the sides of 2D sheet to mitigate the deviation of the

Our proposed position identifying method works in two stage processes: the first stage is the database creation stage by preliminary experiment measurement, and the second stage is the position determination stage by comparing the online data with the database of phase difference at each electrode. In the database creation stage, the pilot signal is sent from reference points on the 2D sheet to obtain the mapping database of RSS and phase difference. After constructing the mapping database, the maximum likelihood is utilized to

Because the radio signal in the 2D communication sheet is less influenced by the outside noises, the RSS captured by the electrode shows a stable distribution with respect to the position of target device. Thus, a likelihood method can be employed to estimate the corresponding positions of target device. However, the inner factors such as reflections at the sides of the sheet or shadows caused by the connectors of the target device influence the propagation of the EM wave to some extent. To confirm such influence level, we have conducted some simulations concerning radio propagation in the 2D communication sheet where reflections and shadowing effects are taken into account [16]. The results show that the positioning method using the arrival phase difference provide the adequate estimation of location, and the less the reflection coefficient, the better the accuracy. We also evaluate our proposed method by performing actual position measurement on a 30 cm x 30 cm area, where the phase differences are measured by eight array electrodes corresponding to 2.44GHz pilot signal. In order to identify the position by sensing the arrival phase difference between any neighboring two electrodes, we first calibrate the phase patterns at all possible positions of target device placed on the top of communication sheet. At each position p, we record an array of multi-receiver output, and memorize them as database. The measuring process is done by a motor driven X-Y translator moving to every position on the sheet. The experiment results reveal that our method can detect position at resolution of 2.5 cm square

Fig. 8. Difference of received phase and signal strength in the 2D sheet.

locate device at the online position determination stage.

RSS at electrodes.

grid perfectly.

#### **3.2 Positioning by sensing arrival phase difference**

To avoid the interference to the other radio communications performed outside of the sheet, the second approach sends signal within the communication sheet. To aim at a simple system, instead of the utilization of the extra infrared and ultrasound waves mentioned in subsection 3.1, we directly employ the microwaves applied in the 2D communication system for position estimation. We employ a small signal emitter mounted on the scope device that is called target device, and attach the multiple microwave receiving electrodes on the sides of the communication sheet, as shown in Fig. 7. As the microwave runs much faster than the ultrasonic wave, even when the time clock is 3.4GHz achieved by the chip-scale atomic clock in one of DARPA projects in 2007 [15], the precision of the measured distance computed by the arrival time of radio beacon is about 10 cm. Instead of directly measuring the arrival time of microwave, we here examine another approach in which we measure the phase difference arriving at the different abovementioned electrodes, by transmitting the pilot signal from the target device.

Fig. 7. Positioning of target device using phase difference of electrode array on 2D Communication sheet.

When a sinusoidal wave at radio frequency is transmitted from a targeted device via a coupler on the 2D sheet, it is propagated through the 2D sheet as a cylindrical wave. Figure 8 schematically shows how the phase shift and difference of received signal strengths (RSS) occur between two electrodes at different distance ݀1 and ݀2 from the target device on a 2D sheet under the ideal condition. The waveform in the 2D sheet can be approximated by the following equation.

$$\mathbf{f} = \mathbf{A} \left( \mathbf{d} \right) \mathbf{e}^{i \left( \mathbf{k} \mathbf{d} - \alpha \mathbf{t} + \mathcal{Q} \right)} \tag{5}$$

Where ݀ is the distance from the target, ܣ)݀ (is amplitude corresponding to the distance, ݇ is wave number associated with the wavelength, ߱ is the angular frequency of the waveform, ݐ is the time and ߶is the initial phase. The phase difference between the electrodes does not

To avoid the interference to the other radio communications performed outside of the sheet, the second approach sends signal within the communication sheet. To aim at a simple system, instead of the utilization of the extra infrared and ultrasound waves mentioned in subsection 3.1, we directly employ the microwaves applied in the 2D communication system for position estimation. We employ a small signal emitter mounted on the scope device that is called target device, and attach the multiple microwave receiving electrodes on the sides of the communication sheet, as shown in Fig. 7. As the microwave runs much faster than the ultrasonic wave, even when the time clock is 3.4GHz achieved by the chip-scale atomic clock in one of DARPA projects in 2007 [15], the precision of the measured distance computed by the arrival time of radio beacon is about 10 cm. Instead of directly measuring the arrival time of microwave, we here examine another approach in which we measure the phase difference arriving at the different abovementioned electrodes, by transmitting the pilot

Fig. 7. Positioning of target device using phase difference of electrode array on 2D

When a sinusoidal wave at radio frequency is transmitted from a targeted device via a coupler on the 2D sheet, it is propagated through the 2D sheet as a cylindrical wave. Figure 8 schematically shows how the phase shift and difference of received signal strengths (RSS)

sheet under the ideal condition. The waveform in the 2D sheet can be approximated by the

i(kd <sup>ω</sup>t ) f Ade

߱

ܣ)݀ ݀1 and ݀

is the initial phase. The phase difference between the electrodes does not

(3)

(is amplitude corresponding to the distance,

is the angular frequency of the waveform,

2 from the target device on a 2D

݇is

ݐ

**3.2 Positioning by sensing arrival phase difference**

signal from the target device.

Communication sheet.

following equation.

Where ݀

is the time and

occur between two electrodes at different distance

is the distance from the target,

wave number associated with the wavelength,

߶

change with time, which changes in the range from 0 to 360 angular degrees. From a practical point of view, however, since there exist reflected waves from the edges of the 2D sheet, the EM absorber is attached to the sides of 2D sheet to mitigate the deviation of the RSS at electrodes.

Fig. 8. Difference of received phase and signal strength in the 2D sheet.

Our proposed position identifying method works in two stage processes: the first stage is the database creation stage by preliminary experiment measurement, and the second stage is the position determination stage by comparing the online data with the database of phase difference at each electrode. In the database creation stage, the pilot signal is sent from reference points on the 2D sheet to obtain the mapping database of RSS and phase difference. After constructing the mapping database, the maximum likelihood is utilized to locate device at the online position determination stage.

Because the radio signal in the 2D communication sheet is less influenced by the outside noises, the RSS captured by the electrode shows a stable distribution with respect to the position of target device. Thus, a likelihood method can be employed to estimate the corresponding positions of target device. However, the inner factors such as reflections at the sides of the sheet or shadows caused by the connectors of the target device influence the propagation of the EM wave to some extent. To confirm such influence level, we have conducted some simulations concerning radio propagation in the 2D communication sheet where reflections and shadowing effects are taken into account [16]. The results show that the positioning method using the arrival phase difference provide the adequate estimation of location, and the less the reflection coefficient, the better the accuracy. We also evaluate our proposed method by performing actual position measurement on a 30 cm x 30 cm area, where the phase differences are measured by eight array electrodes corresponding to 2.44GHz pilot signal. In order to identify the position by sensing the arrival phase difference between any neighboring two electrodes, we first calibrate the phase patterns at all possible positions of target device placed on the top of communication sheet. At each position p, we record an array of multi-receiver output, and memorize them as database. The measuring process is done by a motor driven X-Y translator moving to every position on the sheet. The experiment results reveal that our method can detect position at resolution of 2.5 cm square grid perfectly.

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 185

Fig. 10. Zooming in a map across media for location-specific information visualization. (left)

The system is designed for multi-users, i.e., multiple persons can look at different information at separated locations in the map, which cannot be achieved by a flat screen or touch panel. To meet this goal, multiple media streams have to be sent to the corresponding multiple displays. The design of the platform is depicted in Fig. 11. Here we can employ

Two approaches are examined here. The first one uses location-free-servers which receive analogue signals from PC output, and convert them to digital signal in IEEE 802.11A (5.2GHz). In such a framework, the same number of location-free servers as the number of users is needed to send multi-channel signals. Heavy graphics or video contents may even need multiple PCs to render scenes on separated displays. We feed the signals into the sheet through the clip connectors (antenna), and the signals reach everywhere in the sheet for

The second approach is to use Windows sub-monitors to distribute different contents to displays. Multiple USB LAN output are fed into the sheet by clip connectors and sent to individual displays. The displays that have USB as their input are connected to antenna and is required to register their IDs on the PC first via the communication sheet. This approach requests less resource but can only be used for a few users because of the limitation in the

from map to scene, (right) how a bird grows up.

different channels to send signals to individual displays.

**4.2 Data transmission to multi-users**

displays to pick up.

rendering power of a single PC.

#### **4. Digital scope system on the sheet**

#### **4.1 Scope display for visualizing location-specific contents**

A large and coarse level map is overlaid on the communication sheet or is projected onto it, and the small display device is placed on the sheet. The printed map has no influence on the electromagnetic wave. Because the communication sheet is possible to be made transparent, the coarse map can also be projected from back of the sheet. Hence, the sheet can be attached on screen or table glass as well with the map projected from back or bottom respectively.

The map overlaid on the sheet can be an atlas of world, city, convention site, exhibition room, designed interior space, human body, etc [3]-[5]. The map can also be a table of items as a brief index of categories. Such a map or table can cover a large area or a big collection that a normal PC screen has insufficient resolution to include. In contrast to the map, the underlying media data to display can be selected as those with drastic differences from the map in resolution or type in order to achieve the special effect of the platform. For example, a street view can be displayed according to the position and orientation of the scope display on a road map. Moving the scope display along a street in the map will update the street view in it. This setting can benefit virtual navigation in a large city, which is not flexible on a normal PC window that displays only a limited part of map and the corresponding street view side-by-side. Because the small display of the scope shows details without changing the large map, the user has a good understanding on his/her global location during the exploration of in-depth data.

Another advantage of the scope display is the possibility for visualizing subjects in different modes or media. Figure 9 shows a table for a children's museum on what it becomes when the egg hatched, and how the living species grow up with the scope displays. The map can also be zoomed in on the display and eventually cross different media to visualize a spot, as displayed in Fig. 10. This type of system best fits into a quiz-and-answer diagram and arouses people's curiosity in exploring topics of interest. User can glance over a large category in exposition and then dig into details by placing the display at an interesting spot.

Fig. 9. Scoping details from a category using the communication sheet.

Fig. 10. Zooming in a map across media for location-specific information visualization. (left) from map to scene, (right) how a bird grows up.

#### **4.2 Data transmission to multi-users**

184 Interactive Multimedia

A large and coarse level map is overlaid on the communication sheet or is projected onto it, and the small display device is placed on the sheet. The printed map has no influence on the electromagnetic wave. Because the communication sheet is possible to be made transparent, the coarse map can also be projected from back of the sheet. Hence, the sheet can be attached on screen or table glass as well with the map projected from back or bottom respectively.

The map overlaid on the sheet can be an atlas of world, city, convention site, exhibition room, designed interior space, human body, etc [3]-[5]. The map can also be a table of items as a brief index of categories. Such a map or table can cover a large area or a big collection that a normal PC screen has insufficient resolution to include. In contrast to the map, the underlying media data to display can be selected as those with drastic differences from the map in resolution or type in order to achieve the special effect of the platform. For example, a street view can be displayed according to the position and orientation of the scope display on a road map. Moving the scope display along a street in the map will update the street view in it. This setting can benefit virtual navigation in a large city, which is not flexible on a normal PC window that displays only a limited part of map and the corresponding street view side-by-side. Because the small display of the scope shows details without changing the large map, the user has a good understanding on his/her global location during the

Another advantage of the scope display is the possibility for visualizing subjects in different modes or media. Figure 9 shows a table for a children's museum on what it becomes when the egg hatched, and how the living species grow up with the scope displays. The map can also be zoomed in on the display and eventually cross different media to visualize a spot, as displayed in Fig. 10. This type of system best fits into a quiz-and-answer diagram and arouses people's curiosity in exploring topics of interest. User can glance over a large category in exposition and then dig into details by placing the display at an interesting spot.

Fig. 9. Scoping details from a category using the communication sheet.

**4. Digital scope system on the sheet**

exploration of in-depth data.

**4.1 Scope display for visualizing location-specific contents**

The system is designed for multi-users, i.e., multiple persons can look at different information at separated locations in the map, which cannot be achieved by a flat screen or touch panel. To meet this goal, multiple media streams have to be sent to the corresponding multiple displays. The design of the platform is depicted in Fig. 11. Here we can employ different channels to send signals to individual displays.

Two approaches are examined here. The first one uses location-free-servers which receive analogue signals from PC output, and convert them to digital signal in IEEE 802.11A (5.2GHz). In such a framework, the same number of location-free servers as the number of users is needed to send multi-channel signals. Heavy graphics or video contents may even need multiple PCs to render scenes on separated displays. We feed the signals into the sheet through the clip connectors (antenna), and the signals reach everywhere in the sheet for displays to pick up.

The second approach is to use Windows sub-monitors to distribute different contents to displays. Multiple USB LAN output are fed into the sheet by clip connectors and sent to individual displays. The displays that have USB as their input are connected to antenna and is required to register their IDs on the PC first via the communication sheet. This approach requests less resource but can only be used for a few users because of the limitation in the rendering power of a single PC.

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 187

Fig. 12. Power supply via the communication sheet to small devices including speakers, LED

The large map underlying the small display solves the problem of location/orientation perception in multiwindow system on PCs. For multi-user on the same sheet, we prepare multiple displays connected to a PC for media presentation. The current response time for video clip, however, is a bottleneck that affects the usability of the system. A high

Based on the estimated positioning data of the display with the reconstructed MIMIO system, the location-specific media details can be magnified and displayed as a digital scope to learning materials, catalogs, tourists guide, X-ray image of building, and various other purposes. For example, as shown in Fig.13, we have developed a digital scope system called

performance PC and customized software will be developed to solve this problem.

Fig. 13. Digital scope system called "World Corridor".

lights, monitors, etc.

Fig. 11. The system design in which the multimedia signals are sent to individual displays for detailed content visualization according to the positions of displays on the communication sheet.

#### **5. System construction for education and exposition**

The system platform uses a communication sheet originally developed for communication and power supply, which has 2mm thickness and a mesh interval of 7mm. Figure 12 shows power supply via the communication sheet to small devices including speakers, LED lights, monitors, etc. Currently, the communication sheet can be printed at a width of 1m and a length of 2m for experiments. The cost of such a sheet is low and will be dropped further in mass production for home and office use.

The digital scope display is a Century 4.3 inch screen driven by USB input and the resolution is 800\*480 pixels. The connector developed is attached to the display from back and it contacts the sheet to pick up signals via the communication sheet. We use a position sensing device reconstructed from MIMIO [17], which has the granularity of 100dpi. A circuit is designed to switch between multiple infrared-ultrasound generators to send out pulses alternatively. The signal reaching the receivers is then sent to computer via USB connection. The position estimation on the computer is programmed with MIMIO SDK. The frequency to compute the position of one generator is 87Hz, and the Zigbee switching between two positioning sets is 10Hz.

We have experimented on several types of media contents and demonstrated the functioning system. The contents include a world map with 42 topics in famous sightseeing cities and spots, six categories of living species (41 species ranging from animals to plants) and their growing process exhibited in museum. The system is a new type of interface for users to move on a large map space and peek in detailed information at different locations.

Fig. 11. The system design in which the multimedia signals are sent to individual displays

The system platform uses a communication sheet originally developed for communication and power supply, which has 2mm thickness and a mesh interval of 7mm. Figure 12 shows power supply via the communication sheet to small devices including speakers, LED lights, monitors, etc. Currently, the communication sheet can be printed at a width of 1m and a length of 2m for experiments. The cost of such a sheet is low and will be dropped further in

The digital scope display is a Century 4.3 inch screen driven by USB input and the resolution is 800\*480 pixels. The connector developed is attached to the display from back and it contacts the sheet to pick up signals via the communication sheet. We use a position sensing device reconstructed from MIMIO [17], which has the granularity of 100dpi. A circuit is designed to switch between multiple infrared-ultrasound generators to send out pulses alternatively. The signal reaching the receivers is then sent to computer via USB connection. The position estimation on the computer is programmed with MIMIO SDK. The frequency to compute the position of one generator is 87Hz, and the Zigbee switching

We have experimented on several types of media contents and demonstrated the functioning system. The contents include a world map with 42 topics in famous sightseeing cities and spots, six categories of living species (41 species ranging from animals to plants) and their growing process exhibited in museum. The system is a new type of interface for users to move on a large map space and peek in detailed information at different locations.

for detailed content visualization according to the positions of displays on the

**5. System construction for education and exposition**

mass production for home and office use.

between two positioning sets is 10Hz.

communication sheet.

Fig. 12. Power supply via the communication sheet to small devices including speakers, LED lights, monitors, etc.

The large map underlying the small display solves the problem of location/orientation perception in multiwindow system on PCs. For multi-user on the same sheet, we prepare multiple displays connected to a PC for media presentation. The current response time for video clip, however, is a bottleneck that affects the usability of the system. A high performance PC and customized software will be developed to solve this problem.

Based on the estimated positioning data of the display with the reconstructed MIMIO system, the location-specific media details can be magnified and displayed as a digital scope to learning materials, catalogs, tourists guide, X-ray image of building, and various other purposes. For example, as shown in Fig.13, we have developed a digital scope system called

Fig. 13. Digital scope system called "World Corridor".

Digital Scope on 2D Communication Sheet for Location-Specific Multimedia Service 189

simultaneously. Moreover, the communication sheet has the location dependent property as compared to wireless in the air; a display placed over different sheets can receive different signals. By partitioning a large space into sheet patches that correspond to regions of interest, a display can receive region-specific data on the patches and will show nothing if it is away from the sheet. This also improves the security in the information access in a large space. The future works include the accuracy improvement of the position sensing on the sheet, which will increase the granularity of the map for dense location information retrieval; lightweight display design for mobility on the sheet; building stereo display with a lenticular sheet for visualizing 3D urban scenes on the map; and adding touching panel on the display to allow viewing different media. We will further consider extending the map to curved surfaces on human body or various objects. For example, partitioned sheets can be wrapped on each part of a manikin or on surfaces of a car to form 3D surface maps. By contacting displays onto surfaces, people will be able to see internal structures, which will

This chapter describes a multimedia interface platform on the communication sheet named digital scope for visualizing location-specific information.  The sensed position of each display is sent to the server and the related media data are transmitted to the display via the communication sheet. Functioning as a scope, users can move the small displays over a large map and look into details at different locations. The multi-channel media data are distributed to facilitate multi-user's interaction. Because of the large map, the locationrelated data are more perceivable in a global layout than in a single screen as current PC that has to present different media at a location side-by-side. The platform is also power efficient than a wall of monitors in achieving the same goal. It will enhance many applications in

[1] J. Y. Zheng, X. Wang, " Pervasive Views: Area exploration and guidance using extended

[2] J. Y. Zheng, B. Zhang, H. Cai," Pervasive scene map on wireless devices for city

[3] H. Cai, J. Y. Zheng," Key views for visualizing large spaces ". Journal of Visual

[4] J. Y. Zheng, M. Shi, Mapping cityscapes into cyberspace for visualization, J. of

[5] C. Pinhanez, M. Podlaseck, R. Kjeldsen, A. Levas, G. Pingali, N. Sukaviriya, "Ubiquitous

[6] M. Podlaseck, C. Pinhanez, N. Alvarado, M. Chan, E. Dejesus, "On interfaces projected

interactive displays in a retail environment", SIGGRAPH'03 Sketches. San Diego,

Communication and Image Representation 20(6): 420-427 (2009).

Visualization & Computer Animation, 16(2), 97-107, 2005.

onto real-world objects", CHI'03. Florida, 2003

image media ", ACM Multimedia 05, 986-995, 2005.

Google Map: http://maps.google.com.

navigation ", NBiS 2009, 75-82.

help education in engineering and medical areas.

**7. Conclusion**

education and exposition.

California.

**8. References**

"World Corridor." Along the "World Corridor," we can enjoy an around-the-world trip with family and friends while staying at home. Depending on the position where the display is placed, the scenery image of the place on the map can be gradually magnified by zooming in; details of living creatures in the sightseeing spots can also be visualized. We have also developed another digital scope system for children's education materials. Figure 14 shows a picture book of various eggs and species on what it will become when it grows up. For example, as shown in Fig. 14 , when the display is placed in the middle of 2D sheet, the motion pictures are displayed on how the baby chicken were hatched from eggs, grow up and finally develop to adult chicken.

Fig. 14. Digital scope system for children's education materials on what it will become when it grows up.

#### **6. Discussion and future work**

The proposed digital scope platform is an interface composed of a physical medium and multiple displays to reveal details behind a global map or a general category. It separates the presentation of a map and in-depth data to different media, one on printed paper and others on small screens. This reduces the burden in using a large display or projector in presenting both large map and multiple scope views. A few servers can handle multi-user tasks, which is more flexible than employing a wall of monitors or an array of projectors in conventional setting. It is also energy saving because the signal spread within the communication sheet has less loss than in the air, and in our case only low power signal is sent for communication. The entire system is also convenient to be set anywhere on flat surfaces.

The scope displays are not wired to ensure their free movement on the 2D surface. On the other hand, this platform can send multi-channel signals in the closed sheet by utilizing wider bandwidth than wireless. Multi-channel video can be distributed to different displays simultaneously. Moreover, the communication sheet has the location dependent property as compared to wireless in the air; a display placed over different sheets can receive different signals. By partitioning a large space into sheet patches that correspond to regions of interest, a display can receive region-specific data on the patches and will show nothing if it is away from the sheet. This also improves the security in the information access in a large space. The future works include the accuracy improvement of the position sensing on the sheet, which will increase the granularity of the map for dense location information retrieval; lightweight display design for mobility on the sheet; building stereo display with a lenticular sheet for visualizing 3D urban scenes on the map; and adding touching panel on the display to allow viewing different media. We will further consider extending the map to curved surfaces on human body or various objects. For example, partitioned sheets can be wrapped on each part of a manikin or on surfaces of a car to form 3D surface maps. By contacting displays onto surfaces, people will be able to see internal structures, which will help education in engineering and medical areas.

#### **7. Conclusion**

188 Interactive Multimedia

"World Corridor." Along the "World Corridor," we can enjoy an around-the-world trip with family and friends while staying at home. Depending on the position where the display is placed, the scenery image of the place on the map can be gradually magnified by zooming in; details of living creatures in the sightseeing spots can also be visualized. We have also developed another digital scope system for children's education materials. Figure 14 shows a picture book of various eggs and species on what it will become when it grows up. For example, as shown in Fig. 14 , when the display is placed in the middle of 2D sheet, the motion pictures are displayed on how the baby chicken were hatched from eggs, grow up

Fig. 14. Digital scope system for children's education materials on what it will become when

The proposed digital scope platform is an interface composed of a physical medium and multiple displays to reveal details behind a global map or a general category. It separates the presentation of a map and in-depth data to different media, one on printed paper and others on small screens. This reduces the burden in using a large display or projector in presenting both large map and multiple scope views. A few servers can handle multi-user tasks, which is more flexible than employing a wall of monitors or an array of projectors in conventional setting. It is also energy saving because the signal spread within the communication sheet has less loss than in the air, and in our case only low power signal is sent for communication. The

The scope displays are not wired to ensure their free movement on the 2D surface. On the other hand, this platform can send multi-channel signals in the closed sheet by utilizing wider bandwidth than wireless. Multi-channel video can be distributed to different displays

entire system is also convenient to be set anywhere on flat surfaces.

and finally develop to adult chicken.

it grows up.

**6. Discussion and future work**