Question: Handheld projectors for multi-user interaction in shared spaces.

This paper explores a range of multi-finger and whole hand gestural interaction methods designed for multi-user tabletop displays, capitalizing on recent advancements in sensing technology that facilitate concurrent input from multiple users. Despite the existence of a few illustrative techniques [17], contemporary user interfaces have largely underutilized this enhanced input capability. The authors introduce a plethora of gestural interaction methods that mimic and expand upon the actions typically executed on physical tabletops. Beyond gestural inputs, the paper delves into interaction and visualization strategies that cater to shared spaces, awareness, and privacy concerns. These methodologies are showcased through a prototype application, dubbed RoomPlanner, aimed at room furniture layout design. This application serves as a practical demonstration of how the proposed techniques can enhance collaborative user experiences on multi-user tabletop displays.

In this paper we present a novel hardware and software platform (STARS) to realize computer augmented tabletop games that unify the strengths of traditional board games and computer games. STARS game applications preserve the social situation of traditional board games and provide a tangible interface with physical playing pieces to facilitate natural interaction. The virtual game components offer exciting new opportunities for game design and provide richer gaming experiences impossible to realize with traditional media. This paper describes STARS in terms of the hardware setup and the software platform used to develop and play STARS games. The interaction design within STARS is discussed and sample games are presented with regard to their contributions to enhancing user experience. Finally, realworld experiences with the platform are reported.

JackIn is a new human-human communication framework for connecting two or more people. With first-person view video streaming from a person (called Body) wearing a transparent head-mounted display and a head-mounted camera, the other person (called Ghost) participates in the shared first-person view. With JackIn, people's activities can be shared and assistance or guidance can be given through other peoples expertise. This can be applied to daily activities such as cooking lessons, shopping navigation, education in craft-work or electrical work, and sharing experiences of sporting and live events. For a better viewing experience with frist-person view, we developed the out-of-body view in which first-person images are integrated to construct a scene around a Body, and a Ghost can virtually control the viewpoint to look around the space surrounding the Body. We also developed a tele-pointing gesture interface. We conducted an experiment to evaluate how effective this framework is and found that Ghosts can understand the spatial situation of the Body.

Contemporary museums are much more than places devoted to the placement and the exhibition of collections and artworks; indeed, they are nowadays considered as a privileged means for communication and play a central role in making culture accessible to the mass audience. One of the keys to approach the general public is the use of new technologies and novel interaction paradigms. These means, which bring with them an undeniable appeal, allow curators to modulate the cultural proposal by structuring different courses for different user profiles. Immersive Virtual reality (VR) is probably one of the most appealing and potentially effective technologies to serve this purpose; nevertheless, it is still quite uncommon to find immersive installations in museums. Starting from our 10 years’ experience in this topic, and following an in-depth survey about these technologies and their use in cultural contexts, we propose a classification of VR installations, specifically oriented to cultural heritage applications, based on their features in terms of interaction and immersion. On the basis of this classification, aiming to provide a tool for framing VR systems which would hopefully suggest indications related to costs, usability and quality of the sensorial experience, we analyze a series of live examples of which we point out strengths and weak points. We then summarize the current state and the very next future, identifying the major issues that prevent these technologies from being actually widespread, and outline proposals for a more se of pervasive and effective u

The Participatory Simulations Project explores how a new kind of collaborative learning environment, which is supported by small, wearable computers, can facilitate collaborative theory-building and lead to a richer understanding of scientific experimentation. In a Participatory Simulation, participants become "agents" in a full-scale simulation. Unlike previous work, Participatory Simulations combines the notion of a microworld with the affordances of real world experience. By involving a large number of people (typically between 15 and 30) in a "life-sized" experience, the project brings a microworld off of the computer screen and into the physical world. In particular, this study explored the use of a series of Participatory Simulations in a high school Biology class. The students became "agents" in a simulation of a disease infecting their community. The results of this project show that Participatory Simulations fully engage participants in the simulation, facilitate collaborative problem solving, provide a substrate for collaboratively designing and running experiments, and support the definition of new vocabulary to discuss the underlying rules of the simulation. Thesis Advisor: Mitchel Resnick Associate Professor of Media Arts and Sciences Fukutake Career Development Professor of Research in Education This research was sponsored by the National Science Foundation (grant # CDA-9616444), the Lego Group, and the Things That Think and Digital Life consortia. Participatory Simulations: Building Collaborative Understanding through Immersive Dynamic Modeling

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize information robots collect (e.g., a camera stream) with contextual knowledge of how the robot is moving in the environment. We explore how advances in augmented reality (AR) technologies are creating a new design space for mediating robot teleoperation by enabling novel forms of intuitive, visual feedback. We prototype several aerial robot teleoperation interfaces using AR, which we evaluate in a 48-participant user study where participants completed an environmental inspection task. Our new interface designs provided several objective and subjective performance benefits over existing systems, which often force users into an undesirable paradigm that divides user attention between monitoring the robot and monitoring the robot»s camera feed(s).

This paper proposes a dynamic acoustic field generation system for a spot audio towards particular person indoors. Spot audio techniques have been explored by generating the ultrasound beams toward the target person in certain area however everyone in this area can hear the sound. Our system recognizes the positions of each person indoor using motion capture and 3D model data of the room. After that we control direction of parametric speaker in real-time so that sound reach only particular person by calculating the reflection of sound on surfaces such as wall and ceiling. We calculate direction of parametric speaker using a beam tracing method. We present generating methods of dynamic acoustic field in our system and conducted the experiments on human factor to evaluate performance of proposed system.

Foldios are foldable interactive objects with embedded input sensing and output capabilities. Foldios combine the advantages of folding for thin, lightweight and shape-changing objects with the strengths of thin-film printed electronics for embedded sensing and output. To enable designers and end-users to create highly custom interactive foldable objects, we contribute a new design and fabrication approach. It makes it possible to design the foldable object in a standard 3D environment and to easily add interactive high-level controls, eliminating the need to manually design a fold pattern and low-level circuits for printed electronics. Second, we contribute a set of printable user interface controls for touch input and display output on folded objects. Moreover, we contribute controls for sensing and actuation of shape-changeable objects. We demonstrate the versatility of the approach with a variety of interactive objects that have been fabricated with this framework.

A major difficulty in writing Single Display Groupware (co-present collaborative) applications is getting input from multiple devices. We introduce MID, a Java package that addresses this problem and offers an architecture to access advanced events through Java. In this paper, we describe the features, architecture and limitations of MID. We also briefly describe an application that uses MID to get input from multiple mice: KidPad.

Traditional indoor laser scanning trolley/backpacks with multi-laser scanner, panorama cameras, and an inertial measurement unit (IMU) installed are a popular solution to the 3D indoor mapping problem. However, the cost of those mapping suits is quite expensive, and can hardly be replicated by consumer electronic components. The consumer RGB-Depth (RGB-D) camera (e.g., Kinect V2) is a low-cost option for gathering 3D point clouds. However, because of the narrow field of view (FOV), its collection efficiency and data coverages are lower than that of laser scanners. Additionally, the limited FOV leads to an increase of the scanning workload, data processing burden, and risk of visual odometry (VO)/simultaneous localization and mapping (SLAM) failure. To find an efficient and low-cost way to collect 3D point clouds data with auxiliary information (i.e., color) for indoor mapping, in this paper we present a prototype indoor mapping solution that is built upon the calibration of multiple RGB-D sensors to construct an array with large FOV. Three time-of-flight (ToF)-based Kinect V2 RGB-D cameras are mounted on a rig with different view directions in order to form a large field of view. The three RGB-D data streams are synchronized and gathered by the OpenKinect driver. The intrinsic calibration that involves the geometry and depth calibration of single RGB-D cameras are solved by homography-based method and ray correction followed by range biases correction based on pixel-wise spline line functions, respectively. The extrinsic calibration is achieved through a coarse-to-fine scheme that solves the initial exterior orientation parameters (EoPs) from sparse control markers and further refines the initial value by an iterative closest point (ICP) variant minimizing the distance between the RGB-D point clouds and the referenced laser point clouds. The effectiveness and accuracy of the proposed prototype and calibration method are evaluated by comparing the point clouds derived from the prototype with ground truth data collected by a terrestrial laser scanner (TLS). The overall analysis of the results shows that the proposed method achieves the seamless integration of multiple point clouds from three Kinect V2 cameras collected at 30 frames per second, resulting in low-cost, efficient, and high-coverage 3D color point cloud collection for indoor mapping applications.

Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality.
 The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.


Question: Handheld projectors for multi-user interaction in shared spaces.

This paper introduces a groundbreaking concept for computer users dealing with extensive visual documents, such as expansive layouts, detailed blueprints, or intricate maps, who require simultaneous access to both an overview and detailed information. Traditionally, users had to compromise between using large screens for comprehensive views or resorting to visualization techniques to manage space. However, high-resolution "wall-size" screens are often prohibitively expensive or require substantial space, while visualization methods either necessitate switching between multiple views or introduce visual distortion. To address these limitations, we propose a novel display solution: focus plus context screens. These screens integrate a high-resolution display with a larger, lower-resolution display, enabling users to view content at different scales without compromising the integrity of the image. The display content maintains its scaling, with resolution adjustments based on the display region. Focus plus context screens offer a practical alternative for tasks that currently employ overviews or fisheye views, delivering a single, undistorted view without the need for visualization techniques. To illustrate the concept's feasibility, we have developed a prototype that seamlessly merges an LCD screen with a projection screen. We showcase four applications that have been adapted to demonstrate the versatility and potential of focus plus context screens in enhancing user experience and efficiency.

In this paper we present an augmented reality (AR) application based on the QR Code. The system can extract the information embedded in a QR Code and show the information in an extended 3D form with the QR Code being the traditional AR marker. Traditional AR systems often use a particularly designed pattern (the marker) to recover the 3D scene structure and identify the object to be displayed on the scene. In these systems, the marker is used only for tracking and identification. They do not convey any other information. Consequently, the applications of these systems are limited and often a registration procedure is required for a new marker.
 QR Code has the advantage of large information capacity and is similar to an AR marker in appearance. Thus, more interesting and useful applications can be developed by combining the QR Code with the traditional AR system. In this paper, we combine these two techniques to develop a product demo system. A QR Code is pasted on the package of a product and then a 3D virtual object is displayed on the QR Code. This system allows the customer to visualize the product via a more direct and interactive way. Our system demonstrates the success of using QR Code as the AR marker to a particular application and we believe it can bring more convenient to our life in the future.

How can designers of programming interfaces, interactive tools, and rich social environments enable more people to be more creative more often?

In this paper, we explore the challenges in applying and investigate methodologies to improve direct-touch interaction on intangible displays. Direct-touch interaction simplifies object manipulation, because it combines the input and display into a single integrated interface. While traditional tangible display-based direct-touch technology is commonplace, similar direct-touch interaction within an intangible display paradigm presents many challenges. Given the lack of tactile feedback, direct-touch interaction on an intangible display may show poor performance even on the simplest of target acquisition tasks. In order to study this problem, we have created a prototype of an intangible display. In the initial study, we collected user discrepancy data corresponding to the interpretation of 3D location of targets shown on our intangible display. The result showed that participants performed poorly in determining the z-coordinate of the targets and were imprecise in their execution of screen touches within the system. Thirty percent of positioning operations showed errors larger than 30mm from the actual surface. This finding triggered our interest to design a second study, in which we quantified task time in the presence of visual and audio feedback. The pseudo-shadow visual feedback was shown to be helpful both in improving user performance and satisfaction.

Computer users working with large visual documents, such as large layouts, blueprints, or maps perform tasks that require them to simultaneously access overview information while working on details. To avoid the need for zooming, users currently have to choose between using a sufficiently large screen or applying appropriate visualization techniques. Currently available hi-res "wall-size" screens, however, are cost-intensive, space-intensive, or both. Visualization techniques allow the user to more efficiently use the given screen space, but in exchange they either require the user to switch between multiple views or they introduce distortion.In this paper, we present a novel approach to simultaneously display focus and context information. Focus plus context screens consist of a hi-res display and a larger low-res display. Image content is displayed such that the scaling of the display content is preserved, while its resolution may vary according to which display region it is displayed in. Focus plus context screens are applicable to practically all tasks that currently use overviews or fisheye views, but unlike these visualization techniques, focus plus context screens provide a single, non-distorted view. We present a prototype that seamlessly integrates an LCD with a projection screen and demonstrate four applications that we have adapted so far.

In this paper, we present a novel rendering method which integrates reflective or refractive objects into a differential instant radiosity (DIR) framework usable for mixed-reality (MR) applications. This kind of objects are very special from the light interaction point of view, as they reflect and refract incident rays. Therefore they may cause high-frequency lighting effects known as caustics. Using instant-radiosity (IR) methods to approximate these high-frequency lighting effects would require a large amount of virtual point lights (VPLs) and is therefore not desirable due to real-time constraints. Instead, our approach combines differential instant radiosity with three other methods. One method handles more accurate reflections compared to simple cubemaps by using impostors. Another method is able to calculate two refractions in real-time, and the third method uses small quads to create caustic effects. Our proposed method replaces parts in light paths that belong to reflective or refractive objects using these three methods and thus tightly integrates into DIR. In contrast to previous methods which introduce reflective or refractive objects into MR scenarios, our method produces caustics that also emit additional indirect light. The method runs at real-time frame rates, and the results show that reflective and refractive objects with caustics improve the overall impression for MR scenarios.

In recent years, projection mapping has attracted much attention in a variety of fields. Generally, however, the objects in projection mapping are limited to rigid and static or quasistatic objects. Dynamic projection mapping onto a deformable object could remarkably expand the possibilities. In order to achieve such a projection mapping, it is necessary to recognize the deformation of the object even when it is occluded. However, it is still a challenging problem to achieve this task in real-time with low latency. In this paper, we propose an efficient, high-speed tracking method utilizing high-frame-rate imaging. Our method is able to track an array of dot markers arranged on a deformable object even when there is external occlusion caused by the user interaction and self-occlusion caused by the deformation of the object itself. Additionally, our method can be applied to a stretchable object. Dynamic projection mapping with our method showed robust and consistent display onto a sheet of paper and cloth with a tracking performance of about 0.2 ms per frame, with the result that the projected pattern appeared to be printed on the deformable object.

This demonstration presents a novel form of mid-air tactile display, LaserStroke, that makes use of a laser irradiated on the elastic medium attached to the skin. LaserStroke extends a laser device with an orientation control platform and a magnetic tracker so that it can elicit tapping and stroking sensations to a user's palm from a distance. LaserStroke offers unique tactile experiences while a user freely moves his/her hand in midair.

We present an augmented reality system based on Kinect for on-line handbag shopping. The users can virtually try on different handbags on a TV screen at home. They can interact with the virtual handbags naturally, such as sliding a handbag to different positions on their arms and rotating a handbag to see it from different angles. The users can also see how the handbags fit them in different virtual environments other than the current real background. We describe the technical details for building such a system and demonstrate the experimental results.

This video presents insights into the usability challenges present in consumer VR Head-Mounted Displays regarding a users' capability to interact with and be aware of reality. We demonstrate how these issues can be overcome through selectively incorporating necessary elements of reality into VR, as a user engages with reality. We term this approach Engagement-Dependent Augmented Virtuality.

Passive stereo vision is widely used as a range sensing technology in robots, but suffers from dropouts: areas of low texture where stereo matching fails. By supplementing a stereo system with a strong texture projector, dropouts can be eliminated or reduced. This paper develops a practical stereo projector system, first by finding good patterns to project in the ideal case, then by analyzing the effects of system blur and phase noise on these patterns, and finally by designing a compact projector that is capable of good performance out to 3m in indoor scenes. The system has been implemented and has excellent depth precision and resolution, especially in the range out to 1.5m.


Question: Handheld projectors for multi-user interaction in collaborative environments.

Embark on a journey into the realm of contemporary information retrieval, a topic that has recently piqued the interest of many. Are you ready to join this intellectual adventure? Indeed, you should be. Engaging with this subject will not weigh heavily on your leisure time. Despite some individuals' belief that delving into complex topics is a daunting task, you must trust in your capacity to grasp and enjoy such material. The notion of difficulty may arise when you're uncertain about which subjects to explore, or occasionally, when your reading material fails to captivate your interest sufficiently.

Let us consider a scenario where a group of individuals is interacting through the use of handheld projectors, a concept that is gaining traction in the field of collaborative technology. This innovative method of multi-user engagement enables seamless sharing of information and experiences among participants. By employing handheld projectors, users can project content onto various surfaces, facilitating interactive sessions that are both dynamic and engaging. This technology not only enhances the collaborative process but also opens up new avenues for social interaction and creative collaboration. In a world where connectivity and shared experiences are increasingly valued, the integration of handheld projectors in multi-user settings represents a significant advancement in facilitating group activities and communal learning experiences.

This paper considers the ways in which the interactive whiteboard may support and enhance pedagogic practice through whole-class teaching within literacy and numeracy. Data collected from observations of whole-class lessons, alongside individual interviews and focus group discussions with class teachers and Initial Teacher Education students, has provided opportunities to consider the potential of such technology to facilitate a more creative approach to whole-class teaching. The data suggests that, in the first instance, the special features of information and communications technology such as interactivity, ‘provisionality,’ speed, capacity and range enhance the delivery and pace of the session. This research seems to indicate that it is the skill and the professional knowledge of the teacher who mediates the interaction, and facilitates the development of pupils’ creative responses at the interface of technology, which is critical to the enhancement of the whole-class teaching and learning processes. Introduction The globalising phenomenon of information and communication technologies (ICT) is a distinct characteristic of modern times. The speed and immediacy of ICT, coupled with opportunities for increased information flow through multiple routes of communication, suggest that we are living in a time of unprecedented change, with ICT affecting the way we live and function as individuals and as a society (Castells, 2004). Within the context of education there are some technologies that appear to have attracted more interest than others; however, the degree to which they have been British Journal of Educational Technology Vol 39 No 1 2008 84–96 doi:10.1111/j.1467-8535.2007.00703.x © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. successfully integrated into the classroom environment has been varied. In recent years, there has been a growing level of interest in the electronic or interactive whiteboard (IWB), well documented by the educational press. Such technology is generally comprised of a triangulation between data projector, computer and an electronic screen. This allows an individual to interact with software at the front of a class rather than from the computer. Effectively, the computer screen is projected onto the electronic whiteboard and presented to the class with the teacher, or perhaps student, selecting, activating and interacting with the programs. At a time in England when the government has promoted whole-class interactive teaching, particularly within Literacy and Numeracy, access to IWB technology through targeted government funding is also increasing, and the IWB is steadily becoming a feature of most numeracy and literacy lessons. In January 2004, Charles Clarke, the Secretary of State for Education in England, announced that, in addition to the £25 million previously made available to schools in September 2003; a further £25 million would be released for the purchase of IWBs. This technology is therefore likely to become a key resource in most schools. Introduction of new technologies such as this within the classroom context raises questions regarding the ways in which pedagogic practice may be supported and enhanced; this being the focus of this study, specifically, the links between three areas; whole-class direct teaching, creativity and the integration of technology. As IWBs are becoming more familiar within the educational market, additional challenges, which imply new priorities, have arisen from recent demands for ‘a much stronger emphasis on creative and cultural education and a new balance in teaching and in the curriculum.’ (The National Advisory Committee on Creative and Cultural Education [NACCCE], 1999) The teacher educator is therefore faced with a complex set of demands that require resolution in terms of pedagogic practice. The aim of this study is to investigate how IWBs can provide opportunities for creativity in teaching and learning, particularly within whole-class lessons, by drawing upon observations of, and discussions with, classroom practitioners and ITE students. In so doing, the study will contribute to debates within educational fields (Bleakley, 2004; Craft, 2005; NACCCE, 1999) regarding the notion of creativity in primary teaching and learning. Current developments in creativity, interactivity and whole-class direct teaching The current focus on direct whole-class teaching, particularly in mathematics, developed in response to concerns in England about the level of children’s performance in English and mathematics compared to those in other countries. In particular, in the Third International Mathematics and Science Study of 1999 (Mullis et al, 2000) England was placed low in basic number skills although as Muijs and Reynolds (2001) note, faring better in geometry and problem solving. Research conducted by Professor David Reynolds, chair of the newly established Numeracy Task Force, singled out a substantial amount of direct whole-class teaching as a key feature of mathematics sessions in the top-attaining countries (Reynolds & Farrell, 1996), these findings The use of the interactive whiteboard in literacy and maths: a case study 85 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. echoing those of previous studies, (Galton & Croll, 1980; Rosenshine, 1979), and more recently, Bierhoff (1996), Bierhoff and Prais (1995) and Burghes (1995). In response to these concerns about attainment and the related research findings, the National Literacy Strategy (NLS; DfEE, 1998) and National Numeracy Strategy (NNS) were introduced into most state-funded primary schools; the NLS in 1998 and the NNS in 1999, both providing a clear framework for the content of the curriculum and recommended teaching approaches. In both Literacy and Numeracy, sessions were expected to consist of a substantial amount of direct, whole-class teaching. A wholeclass approach, it was claimed, enables the teacher to interact more with each pupil; adapt activities quickly in response to pupils’ responses; use errors and misconceptions as a teaching point for the whole class and keep pupils on task for longer periods of time (Muijs & Reynolds, 2001). However, Muijs and Reynolds also noted that this approach was not necessarily the best to use in all circumstances. Brophy and Good (1986) are cited as finding this method more suited for teaching rules, procedures and basic skills, especially to younger pupils. Less structured and teacher-directed approaches, Muijs and Reynolds (2001) suggest, would be more appropriate when the aims of the lesson are more complex or open-ended (eg, developing students’ thinking skills) With the IWB featuring widely in whole-class teaching, there is a concern that its full interactive potential may not be explored through this structured, teacher-directed approach as the teaching and modelling of rules, procedures and basic skills is likely to take precedence over more complex and cognitively demanding activities. Where whole-class interactive teaching in mathematics lessons is concerned, the Numeracy Task Force stressed that ‘it is the quality of the whole class teaching which determines its success’ (Department for Education and Employment [DfEE], 1999) rather than the whole-class approach per se. They note that whole-class teaching should not be viewed as a purely transmission-based approach, with children given instruction regarding knowledge and understanding in a product-driven mode of teaching. Rather, it should be seen as an interactive, process-oriented approach to learning with the quality of the interaction within the classroom being of prime importance, maximised effectively and efficiently through good whole-class teaching. The NNS goes on to state that ‘interaction is a two-way process in which pupils are expected to play an active part’ and that ‘high quality interactive teaching is oral, interactive and lively’, which is fundamental to a social constructivist view of learning (Vygotsky, 1978). This may be a laudable claim, however; Muijs and Reynolds (2001) caution that with direct whole-class teaching, pupils may find it easy to adopt a more passive role, becoming too dependent on the teacher and failing to develop independent learning skills. Nevertheless, it may be within a more social, interactive and lively learning environment that the IWB could be seen to make a valuable contribution. At present, there is a limited amount of research available that focuses specifically upon the IWB and associated pedagogy, however, Smith, Hardman and Higgins (2006) have undertaken a substantial study involving observations made in primary schools of 184 lessons in literacy and numeracy conducted over a 2-year period. The study suggests that, although the use of IWBs engages the pupils and sessions are generally faster in 86 British Journal of Educational Technology Vol 39 No 1 2008 © 2007 The Authors. Journal compilation © 2007 British Educational Communications and Technology Agency. their pace of delivery, the underlying pedagogy of whole-class teaching appears to remain unaffected, with teacher-led recitation and emphasis upon recall dominating proceedings. Previous research has highlighted the motivational impact of IWBs upon pupils, with the large screen, the multimedia capability and the element of fun enhancing the presentational aspects of a lesson (Glover & Miller, 2001; Levy, 2002). Essentially, there appears to be the potential for enhancements in whole-class teaching and learning through the use of IWBs if pedagogic practice were to adapt and change through creative and innovative use of the particular features of this new technology. Creativity Creativity has been described as ‘the word of the moment’ (Bruce, 2004, p. vi

A s the effects of Moore's law cause computing systems to become cheaper and more plentiful, a new problem arises: increasingly, the bottleneck in computing is not its disk capacity, processor speed, or communication bandwidth, but rather the limited resource of human attention. Human attention refers to a user's ability to attend to his or her primary tasks, ignoring system-generated distractions such as poor performance and failures. By exploiting plentiful computing resources to reduce user distraction, Project Aura is creating a system whose effectiveness is considerably greater than that of other systems today. Aura is specifically intended for pervasive computing environments involving wireless communication , wearable or handheld computers, and smart spaces. Human attention is an especially scarce resource in such environments, because the user is often preoccupied with walking, driving, or other real-world interactions. In addition, mobile computing poses difficult challenges such as intermittent and variable-bandwidth connectivity, concern for battery life, and the client resource constraints that weight and size considerations impose. To accomplish its ambitious goals, research in Aura spans every system level: from the hardware, through the operating system, to applications and end users. Underlying this diversity of concerns, Aura applies two broad concepts. First, it uses proactivity, which is a system layer's ability to anticipate requests from a higher layer. In today's systems , each layer merely reacts to the layer above it. Second, Aura is self-tuning: layers adapt by observing the demands made on them and adjusting their performance and resource usage characteristics accordingly. Currently, system-layer behavior is relatively static. Both of these techniques will help lower demand for human attention. To illustrate the kind of world we are trying to create, we present two hypothetical Aura scenarios. Although these might seem far-fetched today, they represent the kind of scenarios we expect to make commonplace through our research. In the first scenario, Jane is at Gate 23 in the Pitts-burgh airport, waiting for her connecting flight. She has edited many large documents and would like to use her wireless connection to email them. Unfortunately , bandwidth is miserable because many passengers at Gates 22 and 23 are surfing the Web. Aura observes that, at the current bandwidth, Jane won't be able to finish sending her documents before her flight departs. Consulting the airport's wireless network bandwidth service and flight schedule service, Aura discovers that wireless bandwidth is excellent at Gate 15, and that there are …

Although the exploration of design alternatives is crucial for interaction designers and customization is required for end-users, the current development tools for physical computing devices have focused on single versions of an artifact. We propose the parametric design of devices including their enclosure layouts and programs to address this issue. A Web-based design tool called f3.js is presented as an example implementation, which allows devices assembled from laser-cut panels with sensors and actuator modules to be parametrically created and customized. It enables interaction designers to write code with dedicated APIs, declare parameters, and interactively tune them to produce the enclosure layouts and programs. It also provides a separate user interface for end-users that allows parameter tuning and dynamically generates instructions for device assembly. The parametric design approach and the tool were evaluated through two user studies with interaction designers, university students, and end-users.

Augmented Reality (AR) is an emerging visualization technology that promises to change the way people see the world. The main goal of AR is to “enhance” a person’s vision of the real world with useful information about the surrounding environment. Primarily this can be done through the use of a portable computer such as a laptop and a head-mounted display (HMD) unit. When such technology is made available to scientists and engineers, Augmented Reality can provide them with the freedom to do scientific analysis at the actual location of an experiment. With this idea in mind, the goal of this paper is to investigate the development of an AR system as a visualization tool for structural engineers. This particular application will focus on monitoring the “health” of the Kings Stormwater Bridge structure. Unfortunately, since AR is still in its infancy, much of the research devoted to it has focused mainly on technical issues rather than on the development of how this technology would be used by scientists. This has led to the development of “functional” AR systems that primarily can only be operated by the developer of the system ([1], [2], [3], [12], [14]). This paper avoids this trend by focusing on the design of the visualization system so that it is useful for structural engineers. As a result, this should produce a more robust Augmented Reality system that can be used by many different users. Thus, it is the goal of this paper to show how scientists would benefit by designing a visualization system with their needs in mind.

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize information robots collect (e.g., a camera stream) with contextual knowledge of how the robot is moving in the environment. We explore how advances in augmented reality (AR) technologies are creating a new design space for mediating robot teleoperation by enabling novel forms of intuitive, visual feedback. We prototype several aerial robot teleoperation interfaces using AR, which we evaluate in a 48-participant user study where participants completed an environmental inspection task. Our new interface designs provided several objective and subjective performance benefits over existing systems, which often force users into an undesirable paradigm that divides user attention between monitoring the robot and monitoring the robot»s camera feed(s).

Several common systems satisfy some but not all of the VR definition above. Flight simulators provide vehicle tracking, not head tracking, and do not generally operate in binocular stereo. Omnimax theaters give a large angle of view [8], occasionally in stereo, but are not interactive. Head-tracked monitors [4][6] provide all but a large angle of view. Head-mounted displays (HMD) [7][13] and BOOMs [9] use motion of the actual display screens to achieve VR by our definition. Correct projection of the imagery on large screens can also create a VR experience, this being the subject of this paper. This paper describes the CAVE (CAVE Automatic Virtual Environment) virtual reality/scientific visualization system in detail and demonstrates that projection technology applied to virtual-reality goals achieves a system that matches the quality of workstation screens in terms of resolution, color, and flicker-free stereo. In addition, this format helps reduce the effect of common tracking and system latency errors. The off-axis perspective projection techniques we use are shown to be simple and straightforward. Our techniques for doing multi-screen stereo vision are enumerated, and design barriers, past and current, are described. Advantages and disadvantages of the projection paradigm are discussed, with an analysis of the effect of tracking noise and delay on the user. Successive refinement, a necessary tool for scientific visualization, is developed in the virtual reality context. The use of the CAVE as a one-to-many presentation device at SIGGRAPH '92 and Supercomputing '92 for computational science data is also mentioned. Previous work in the VR area dates back to Sutherland [12], who in 1965 wrote about the “Ultimate Display.” Later in the decade at the University of Utah, Jim Clark developed a system that allowed wireframe graphics VR to be seen through a headmounted, BOOM-type display for his dissertation. The common VR devices today are the HMD and the BOOM. Lipscomb [4] showed a monitor-based system in the IBM booth at SIGGRAPH '91 and Deering [6] demonstrated the Virtual Portal, a closets ized three-wal l project ion-based system, in the Sun Microsystems' booth at SIGGRAPH '92. The CAVE, our projection-based VR display [3], also premiered at SIGGRAPH '92. The Virtual Portal and CAVE have similar intent, but different implementation schemes.

Augmented Reality has revolutionized the way of looking at actuality. The colossal developments in this area have led to visualizations beyond desktop. Unreal objects can be augmented on real surfaces. Augmented Reality has converged the knowledge of computer vision, virtual reality, image-processing, human-to-computer interaction and more such areas. It has found applications in various expanses like Games, Entertainment, Military, Navigation and many more. One such application could be in Library Management System. Earlier systems for book tracking involved immobile computers that only had the shelf numbers nested with book id to locate a book. A borrower needed to ask the Librarian/support staff to look up the shelf number from the system so as to find a book or simply do it manually by himself by searching each shelf. The application of Augmented Reality in Library Administration is useful in automating the proximity location tracking of a book and providing alternatives to the borrower in the books context. The borrower is navigated through the library till he/she reaches the shelf where the book is kept. The spine of the book serves as the index of the required book and contextual information is augmented once the book is located. Thus Augmented Reality can prove immensely helpful for Library Administration.

The use of the gesture system in our daily life as a natural human-human interaction has inspired the researchers to simulate and utilize this gift in human-machine interaction which is appealing and can take place the bore interaction ones that existed such as television, radio, and various home appliances as well as virtual reality will worth and deserve its name. This kind of interaction ensures promising and satisfying outcomes if applied in systematic approach, and supports unadorned human hand when transferring the message to these devices which is easiest, comfort and desired rather than the communication that requires frills to deliver the message to such devices. With the rapid emergence of 3d applications and virtual environments in computer system the need for a new type of interaction device arises. This is because the traditional devices such as mouse, keyboard and joystick become inefficient and cumbersome within this virtual environments.in other words evolution of user interfaces shapes the change in the HumanComputer Interaction (HCI).Intuitive and naturalness characteristics of “Hand Gestures” in the HCI have been the driving force and motivation to develop an interaction device which can replace current unwieldy tools. A survey on the methods of analysing, modelling and recognizing hand gestures in the context of the HCI is provided in this paper.

This paper discusses a technology that is expected to widely spread in the near future, which is the Sixth Sense Technology. With this new technology yet to be introduced in the market, different implementation approaches are discussed. Such approaches are demonstrated through other inventions similar to the Sixth Sense (SS). They all fall into the same category, which is Augmented Reality Technologies. The paper focuses on the possible applications and opportunities of such technology. Furthermore, the implementation approaches are compared and the pros/cons of each approach are explored. Most importantly, technical challenges and open issues regarding each implementation approach are brought out. Furthermore, predictions are made on which approach is expected to succeed in the coming years. Finally, solutions are discussed to improve the Sixth Sense Technology with regards to its implementation approaches to insure new and better ways of human-computer interaction.

In this paper we describe the design evolution of a novel technology that collects and displays presence information to be used in the homes of older adults. The first two iterations, the Ambient Plant and Presence Clock, were designed for higher socio-economic status (SES) older adults, whereas the Check-In Tree was designed for low SES older adults. We describe how feedback from older adult participants drove our design decisions, and give an in-depth account of how the Check-In Tree evolved from concept to a final design ready for in situ deployment.


Question: Handheld projectors for multi-user interaction in collaborative environments.

Introducing the Cyberguide: A Mobile, Context-Aware Tour Guide for Enhanced User Experience

In anticipation of future computing landscapes that liberate users from the limitations of desktop environments, applications must harness contextual data, including location, to offer more sophisticated services. This paper details the Cyberguide project, which aims to develop prototypes of a mobile, context-aware tour guide. By utilizing the user's current and historical locations, the Cyberguide aspires to replicate the nuanced services provided by a human tour guide. We outline the design and features of various Cyberguide prototypes, which have been tailored for both indoor and outdoor use across multiple handheld platforms. Additionally, we delve into the overarching research challenges that have surfaced during our development of context-aware applications for mobile environments, highlighting the innovative potential of such systems in enhancing user experience.

**Question**: Evaluating the impact of renewable energy on electricity markets

**Documents**: The impact of wind power on electricity prices: The case of the Irish Single Electricity Market
This paper examines the impact of wind power on electricity prices in the Irish Single Electricity Market (SEM). The SEM is a wholesale electricity market that operates in both the Republic of Ireland and Northern Ireland. The SEM has one of the highest wind penetrations in the world, with wind power accounting for over 20% of total electricity generation. Using a unique data set that includes hourly electricity prices, wind power generation, and other relevant variables, we estimate the impact of wind power on electricity prices using a vector autoregression (VAR) model. Our results suggest that wind power has a statistically significant impact on electricity prices, with a 1% increase in wind power generation leading to a 0.1% decrease in electricity prices. This finding has important implications for policy makers and market participants, as it suggests that increasing the share of wind power in the electricity mix can lead to lower electricity prices.

**Similar Documents**: Assessing the Influence of Wind Power on Electricity Prices: Insights from the Irish Single Electricity Market

This study investigates the effect of wind power on electricity prices within the context of the Irish Single Electricity Market (SEM), a wholesale market encompassing both the Republic of Ireland and Northern Ireland. Notably, the SEM boasts one of the highest global wind power penetrations, with wind energy contributing over 20% to total electricity generation. By employing a comprehensive dataset that encompasses hourly electricity prices, wind power generation, and pertinent variables, we apply a Vector Autoregression (VAR) model to quantify the influence of wind power on electricity prices.

This article reports empirical findings from four inter-related studies, with an emphasis on collocated sharing. Collocated sharing remains important, using both traditional and emerging image-related technologies. Co-present viewing is a dynamic, improvisational construction of a contingent, situated interaction between story-teller and audience. The concept of performance, as articulated differently by Erving Goffman and Judith Butler, is useful understand the enduring importance of co-present sharing of photos and the importance of oral narratives around images in enacting identity and relationships. Finally, we suggest some implications for both HCI research and the design of image-related technologies. & 2009 Elsevier Ltd. All rights reserved.

Media Architecture is reaching a level of maturity at which we can identify tools and approaches for addressing the main challenges for HCI practitioners working in this field. While previous influential contributions within Media Architecture have identified challenges for designers and offered case studies of specific approaches, here, we (1) provide guidance on how to tackle the domain-specific challenges of Media Architecture design -- pertaining to the interface, integration, content, context, process, prototyping, and evaluation -- on the basis of the development of numerous installations over the course of seven years, and thorough studies of related work, and (2) present five categories of tools and approaches -- software tools, projection, 3D models, hardware prototyping, and evaluation tools -- developed to address these challenges in practice, exemplified through six concrete examples from real-life cases.

This thesis presents WiTrack, a system that tracks the 3D motion of a user from the radio signals reflected off her body. It works even if the person is occluded from the WiTrack device or in a different room. WiTrack does not require the user to carry any wireless device, yet its accuracy exceeds current RF localization systems, which require the user to hold a transceiver. Empirical measurements with a WiTrack prototype show that, on average, it localizes the center of a human body to within a median of 10 to 13 cm in the x and y dimensions, and 21 cm in the z dimension. It also provides coarse tracking of body parts, identifying the direction of a pointing hand with a median of 11.20. WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device. Thesis Supervisor: Dina Katabi Title: Professor of Computer Science and Engineering

A s the effects of Moore's law cause computing systems to become cheaper and more plentiful, a new problem arises: increasingly, the bottleneck in computing is not its disk capacity, processor speed, or communication bandwidth, but rather the limited resource of human attention. Human attention refers to a user's ability to attend to his or her primary tasks, ignoring system-generated distractions such as poor performance and failures. By exploiting plentiful computing resources to reduce user distraction, Project Aura is creating a system whose effectiveness is considerably greater than that of other systems today. Aura is specifically intended for pervasive computing environments involving wireless communication , wearable or handheld computers, and smart spaces. Human attention is an especially scarce resource in such environments, because the user is often preoccupied with walking, driving, or other real-world interactions. In addition, mobile computing poses difficult challenges such as intermittent and variable-bandwidth connectivity, concern for battery life, and the client resource constraints that weight and size considerations impose. To accomplish its ambitious goals, research in Aura spans every system level: from the hardware, through the operating system, to applications and end users. Underlying this diversity of concerns, Aura applies two broad concepts. First, it uses proactivity, which is a system layer's ability to anticipate requests from a higher layer. In today's systems , each layer merely reacts to the layer above it. Second, Aura is self-tuning: layers adapt by observing the demands made on them and adjusting their performance and resource usage characteristics accordingly. Currently, system-layer behavior is relatively static. Both of these techniques will help lower demand for human attention. To illustrate the kind of world we are trying to create, we present two hypothetical Aura scenarios. Although these might seem far-fetched today, they represent the kind of scenarios we expect to make commonplace through our research. In the first scenario, Jane is at Gate 23 in the Pitts-burgh airport, waiting for her connecting flight. She has edited many large documents and would like to use her wireless connection to email them. Unfortunately , bandwidth is miserable because many passengers at Gates 22 and 23 are surfing the Web. Aura observes that, at the current bandwidth, Jane won't be able to finish sending her documents before her flight departs. Consulting the airport's wireless network bandwidth service and flight schedule service, Aura discovers that wireless bandwidth is excellent at Gate 15, and that there are …

How can designers of programming interfaces, interactive tools, and rich social environments enable more people to be more creative more often?

Stereoscopic head-mounted displays (HMD's) provide an effective capability to create dynamic virtual environments. For a user of such environments, virtual objects would be displayed ideally at the appropriate distances, and natural concordant accommodation and convergence would be provided. Under such image display conditions, the user perceives these objects as if they were objects in a real environment. Current HMD technology requires convergent eye movements. However, it is currently limited by fixed visual accommodation, which is inconsistent with real-world vision. A prototype multiplanar volumetric projection display based on a stack of laminated planes was built for medical visualization as discussed in a paper presented at a 1999 Advanced Research Projects Agency workshop (Sullivan, Advanced Research Projects Agency, Arlington, Va., 1999). We show how such technology can be engineered to create a set of virtual planes appropriately configured in visual space to suppress conflicts of convergence and accommodation in HMD's. Although some scanning mechanism could be employed to create a set of desirable planes from a two-dimensional conventional display, multiplanar technology accomplishes such function with no moving parts. Based on optical principles and human vision, we present a comprehensive investigation of the engineering specification of multiplanar technology for integration in HMD's. Using selected human visual acuity and stereoacuity criteria, we show that the display requires at most 27 equally spaced planes, which is within the capability of current research and development display devices, located within a maximal 26-mm-wide stack. We further show that the necessary in-plane resolution is of the order of 5 microm.

We present a simple, fast solution for reflectance acquisition using tools that fit into a pocket. Our method captures video of a flat target surface from a fixed video camera lit by a hand-held, moving, linear light source. After processing, we obtain an SVBRDF.
 We introduce a BRDF chart, analogous to a color "checker" chart, which arranges a set of known-BRDF reference tiles over a small card. A sequence of light responses from the chart tiles as well as from points on the target is captured and matched to reconstruct the target's appearance.
 We develop a new algorithm for BRDF reconstruction which works directly on these LDR responses, without knowing the light or camera position, or acquiring HDR lighting. It compensates for spatial variation caused by the local (finite distance) camera and light position by warping responses over time to align them to a specular reference. After alignment, we find an optimal linear combination of the Lambertian and purely specular reference responses to match each target point's response. The same weights are then applied to the corresponding (known) reference BRDFs to reconstruct the target point's BRDF. We extend the basic algorithm to also recover varying surface normals by adding two spherical caps for diffuse and specular references to the BRDF chart.
 We demonstrate convincing results obtained after less than 30 seconds of data capture, using commercial mobile phone cameras in a casual environment.

This paper contributes to the nascent body of literature on pointing performance in Virtual Environments (VEs), comparing gaze- and hand-based pointing. Contrary to previous findings, preliminary results indicate that gaze-based pointing is slower than hand-based pointing for distant objects.

Various interaction techniques have been developed in the field of virtual and augmented reality. Whereas techniques for object selection, manipulation, travel, and wayfinding have already been covered in existing taxonomies in some detail, application control techniques have not yet been sufficiently considered. However, they are needed by almost every mixed reality application, e.g. for choosing from alternative objects or options. For this purpose a great variety of distinct three-dimensional (3D) menu selection techniques is available. This paper surveys existing 3D menus from the corpus of literature and classifies them according to various criteria. The taxonomy introduced here assists developers of interactive 3D applications to better evaluate their options when choosing, optimizing, and implementing a 3D menu technique. Since the taxonomy spans the design space for 3D menu solutions, it also aids researchers in identifying opportunities to improve or create novel virtual menu techniques. r 2006 Elsevier Ltd. All rights reserved.

In this paper we present a novel wearable navigation system along with implicit HCI (iHCI) model, where interaction with technology is dissolved into a day-today activity. In this type of HCI model a computer takes the input and tries to output an action that is a proactive anticipation of next action of a user. Usually, in urban areas people use voice assisted navigation systems or navigation guidelines displayed on a mobile phone. Some navigation systems are already installed on car dashboard, which needs explicit attention in order to make driving decisions. A navigation system using haptic perception to guide a user throughout a journey is the key contribution of this paper. It does not ask for explicit user attention and demonstrates the indolent form of technological interaction. This wearable device is an index finger sleeve, which consists of vibrator modules, Bluetooth communication module and Microcontroller Unit (MCU). A working prototype has been built and tested.


Question: Handheld projector-based multi-user interaction methods.

Incorporating Paper into the Digital Workspace: A Seamless Integration

The enduring appeal of paper in a digital age presents unique challenges and opportunities for enhancing user interaction. Visionaries once predicted the advent of the "paperless office," but the reality has been quite different. Paper remains a preferred medium due to its readability, affordability, universal acceptance, tactile qualities, and portability [5]. Despite technological advancements, paperwork in offices has seen a six-fold increase since 1970, with a 20% annual growth rate [14]. Electronic documents, while convenient, cannot fully replicate the inherent benefits of paper, leading to a dual existence of "paper pushing" and "pixel pushing" desks [10].

The divide between these two workspaces, though often interconnected, remains largely isolated. Printers and scanners serve as the primary conduits for transitioning documents between physical and digital formats, but these processes are time-consuming and inconvenient. Wang's Freestyle system [3] aimed to bridge this gap by coexisting with partially paper-based processes, emphasizing the minimization of printing and scanning to prevent a complete reversion to paper-based systems. The choice between paper and electronic documents can be a complex trade-off, but what if there was no need to choose?

Imagine a space where documents can exist simultaneously in both paper and electronic forms. Instead of immersing the user in the virtual world of the computer, we could augment the user's real-world environment with computational capabilities, creating a Computer Augmented Environment for paper [8]. This approach does not seek to replace paper with computers but rather to enhance paper with the power of computation. A prime example of this is the Xerox PaperWorks product, which introduces a fax-based paper user interface (UI) to a storage and retrieval system. Through this system, ordinary paper forms gain some electronic document properties, albeit with the limitation of fax machine speed and the associated delays in scanning and printing.

The integration of paper into the digital workspace offers a promising avenue for multi-user interaction using handheld projectors. By leveraging the tactile and portable nature of paper while harnessing the computational power of digital devices, we can create a more seamless and user-friendly interface for collaborative work environments. This approach not only respects the enduring appeal of paper but also enhances its functionality in the digital age.

A study of a professional touring mixed reality performance called Desert Rain yields insights into how performers orchestrate players' engagement in an interactive experience. Six players at a time journey through an extended physical and virtual set. Each sees a virtual world projected onto a screen made from a fine water spray. This acts as a traversable interface, supporting the illusion that performers physically pass between real and virtual worlds. Live and video-based observations of Desert Rain, coupled with interviews with players and the production team, have revealed how the performers create conditions for the willing suspension of disbelief, and how they monitor and intervene in the players experience without breaking their engagement. This involves carefully timed performances and “off-face” and “virtual” interventions. In turn, these are supported by the ability to monitor players' physical and virtual activity through asymmetric interfaces.

In this paper we present a novel hardware and software platform (STARS) to realize computer augmented tabletop games that unify the strengths of traditional board games and computer games. STARS game applications preserve the social situation of traditional board games and provide a tangible interface with physical playing pieces to facilitate natural interaction. The virtual game components offer exciting new opportunities for game design and provide richer gaming experiences impossible to realize with traditional media. This paper describes STARS in terms of the hardware setup and the software platform used to develop and play STARS games. The interaction design within STARS is discussed and sample games are presented with regard to their contributions to enhancing user experience. Finally, realworld experiences with the platform are reported.

JackIn is a new human-human communication framework for connecting two or more people. With first-person view video streaming from a person (called Body) wearing a transparent head-mounted display and a head-mounted camera, the other person (called Ghost) participates in the shared first-person view. With JackIn, people's activities can be shared and assistance or guidance can be given through other peoples expertise. This can be applied to daily activities such as cooking lessons, shopping navigation, education in craft-work or electrical work, and sharing experiences of sporting and live events. For a better viewing experience with frist-person view, we developed the out-of-body view in which first-person images are integrated to construct a scene around a Body, and a Ghost can virtually control the viewpoint to look around the space surrounding the Body. We also developed a tele-pointing gesture interface. We conducted an experiment to evaluate how effective this framework is and found that Ghosts can understand the spatial situation of the Body.

In this paper, we propose and implement a multi-layered visualization technique in 3D space called Multi-Layered Visualization for Multi-Dimensional Data (MLMD) with its corresponding interaction techniques, for visualizing multi-dimensional data. Layers of point based plots are stacked and connected in a virtual visualization cube for comparison between different dimension settings. Viewed from the side, the plot layers intrinsically form parallel coordinates plots. MLMD integrates point based plots and parallel coordinates compactly so as to present more information at a time to help data investigation. Pertinent user interactions for MLMD method to enable convenient manipulation are designed. By using MLMD and its matching interaction techniques, proper dimension settings and in-depth data perception can be achieved.

We describe an augmented reality, optical see-through display based on a DMD chip with an extremely fast (16 kHz) binary update rate. We combine the techniques of post-rendering 2-D offsets and just-in-time tracking updates with a novel modulation technique for turning binary pixels into perceived gray scale. These processing elements, implemented in an FPGA, are physically mounted along with the optical display elements in a head tracked rig through which users view synthetic imagery superimposed on their real environment. The combination of mechanical tracking at near-zero latency with reconfigurable display processing has given us a measured average of 80 μs of end-to-end latency (from head motion to change in photons from the display) and also a versatile test platform for extremely-low-latency display systems. We have used it to examine the trade-offs between image quality and cost (i.e. power and logical complexity) and have found that quality can be maintained with a fairly simple display modulation scheme.

This thesis presents WiTrack, a system that tracks the 3D motion of a user from the radio signals reflected off her body. It works even if the person is occluded from the WiTrack device or in a different room. WiTrack does not require the user to carry any wireless device, yet its accuracy exceeds current RF localization systems, which require the user to hold a transceiver. Empirical measurements with a WiTrack prototype show that, on average, it localizes the center of a human body to within a median of 10 to 13 cm in the x and y dimensions, and 21 cm in the z dimension. It also provides coarse tracking of body parts, identifying the direction of a pointing hand with a median of 11.20. WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device. Thesis Supervisor: Dina Katabi Title: Professor of Computer Science and Engineering

Interactive touchscreens such as tablet PCs (TPC) and interactive whiteboards (IWB) are becoming more and more common in classrooms around the world. To date, very little research has been conducted on the impact of the new technology on physics education. This systematic literature review aims to investigate research on what impact tablet PCs and interactive whiteboards might have on the education in upper Secondary School. The review was performed in response to the following questions: 1. What is the influence of IWBs and TPCs on students’ active participation in physics education? 2. How can an IWB or TPC improve students’ learning about physics concepts? 3. How can educational research on touchscreen technology help inform effective teaching strategies in physics education? To respond to the questions of the study, relevant research about interactive whiteboards and/or tablet PCs was consulted and analysed. Twelve articles were located, mainly through the ERIC and Scopus databases, but also through Google Scholar. The included articles reported empirical research about physics education with interactive whiteboards or tablet PCs. The results from the articles indicate that interactive touchscreens might help improve learners’ active participation in physics education. Interactive whiteboards can, for example, be used to display interactive simulations during group work, something students are found to appreciate and easily engage in. A tablet PC can be used in the same way, but also allows students to receive anonymous support and feedback from the teacher during class which seems to be beneficial for learning. Results show that it is possible to improve students’ understanding of physics concepts by using interactive whiteboards or tablet PCs. However, further research is required to compare results from students using touch technology and students taught in traditional manner to be able to draw any general conclusions about observed learning effects.

This film demonstrates a tool for visualizing the development history of software projects as an interactive animation, showing developers working on the hierarchical file-directory structure of a project over the course of its development.

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize information robots collect (e.g., a camera stream) with contextual knowledge of how the robot is moving in the environment. We explore how advances in augmented reality (AR) technologies are creating a new design space for mediating robot teleoperation by enabling novel forms of intuitive, visual feedback. We prototype several aerial robot teleoperation interfaces using AR, which we evaluate in a 48-participant user study where participants completed an environmental inspection task. Our new interface designs provided several objective and subjective performance benefits over existing systems, which often force users into an undesirable paradigm that divides user attention between monitoring the robot and monitoring the robot»s camera feed(s).

Contemporary museums are much more than places devoted to the placement and the exhibition of collections and artworks; indeed, they are nowadays considered as a privileged means for communication and play a central role in making culture accessible to the mass audience. One of the keys to approach the general public is the use of new technologies and novel interaction paradigms. These means, which bring with them an undeniable appeal, allow curators to modulate the cultural proposal by structuring different courses for different user profiles. Immersive Virtual reality (VR) is probably one of the most appealing and potentially effective technologies to serve this purpose; nevertheless, it is still quite uncommon to find immersive installations in museums. Starting from our 10 years’ experience in this topic, and following an in-depth survey about these technologies and their use in cultural contexts, we propose a classification of VR installations, specifically oriented to cultural heritage applications, based on their features in terms of interaction and immersion. On the basis of this classification, aiming to provide a tool for framing VR systems which would hopefully suggest indications related to costs, usability and quality of the sensorial experience, we analyze a series of live examples of which we point out strengths and weak points. We then summarize the current state and the very next future, identifying the major issues that prevent these technologies from being actually widespread, and outline proposals for a more se of pervasive and effective u


Question: A time-limited self-attention mechanism for automatic speech recognition.

In this paper, we present the Transformer, a novel architecture for sequence transduction tasks that relies exclusively on attention mechanisms, eliminating the need for recurrent or convolutional layers. Traditional models in this domain typically involve complex structures, such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs), paired with an attention mechanism to bridge the encoder and decoder. Our proposed model, the Transformer, breaks from this convention by using only attention mechanisms, demonstrating superior performance and efficiency in training. Through experiments on two key machine translation tasks, we establish the Transformer's effectiveness. Notably, on the WMT 2014 English-to-German translation task, the Transformer achieves a BLEU score of 28.4, outperforming previous state-of-the-art models, including ensembles, by more than 2 BLEU points. On the English-to-French translation task from the same WMT 2014 dataset, the Transformer sets a new single-model state-of-the-art BLEU score of 41.2 after just 4.5 days of training on eight GPUs, a markedly reduced training cost compared to the leading models in the literature. Moreover, the Transformer's versatility is underscored by its successful application to English constituency parsing, showcasing its generalizability across different tasks with varying training data sizes.

Engineering automatic speech recognition (ASR) for speech to speech (S2S) translation systems, especially targeting languages and domains that do not have readily available spoken language resources, is immensely challenging due to a number of reasons. In addition to contending with the conventional data-hungry speech acoustic and language modeling needs, these designs have to accommodate varying requirements imposed by the domain needs and characteristics, target device and usage modality (such as phrase-based, or spontaneous free form interactions, with or without visual feedback) and huge spoken language variability arising due to socio-linguistic and cultural differences of the users. This paper, using case studies of creating speech translation systems between English and languages such as Pashto and Farsi, describes some of the practical issues and the solutions that were developed for multilingual ASR development. These include novel acoustic and language modeling strategies such as language adaptive recognition, active-learning based language modeling, class-based language models that can better exploit resource poor language data, efficient search strategies, including N-best and confidence generation to aid multiple hypotheses translation, use of dialog information and clever interface choices to facilitate ASR, and audio interface design for meeting both usability and robustness requirements

In this paper, we explore the possibility of leveraging Residual Networks (ResNet), a powerful structure in constructing extremely deep neural network for image understanding, to improve recurrent neural networks (RNN) for modeling sequential data. We show that for sequence classification tasks, incorporating residual connections into recurrent structures yields similar accuracy to Long Short Term Memory (LSTM) RNN with much fewer model parameters. In addition, we propose two novel models which combine the best of both residual learning and LSTM. Experiments show that the new models significantly outperform LSTM.

Automatic speech and speaker recognition are traditionally treated as two independent tasks and are studied separately. The human brain in contrast deciphers the linguistic content, and the speaker traits from the speech in a collaborative manner. This key observation motivates the work presented in this paper. A collaborative joint training approach based on multitask recurrent neural network models is proposed, where the output of one task is backpropagated to the other tasks. This is a general framework for learning collaborative tasks and fits well with the goal of joint learning of automatic speech and speaker recognition. Through a comprehensive study, it is shown that the multitask recurrent neural net models deliver improved performance on both automatic speech and speaker recognition tasks as compared to single-task systems. The strength of such multitask collaborative learning is analyzed, and the impact of various training configurations is investigated.

The attentional mechanism has proven to be effective in improving end-to-end neural machine translation. However, due to the intricate structural divergence between natural languages, unidirectional attention-based models might only capture partial aspects of attentional regularities. We propose agreement-based joint training for bidirectional attention-based end-to-end neural machine translation. Instead of training source-to-target and target-to-source translation models independently, our approach encourages the two complementary models to agree on word alignment matrices on the same training data. Experiments on ChineseEnglish and English-French translation tasks show that agreement-based joint training significantly improves both alignment and translation quality over independent training.

One of the key problems in spoken language understanding (SLU) is the task of slot filling. In light of the recent success of applying deep neural network technologies in domain detection and intent identification, we carried out an in-depth investigation on the use of recurrent neural networks for the more difficult task of slot filling involving sequence discrimination. In this work, we implemented and compared several important recurrent-neural-network architectures, including the Elman-type and Jordan-type recurrent networks and their variants. To make the results easy to reproduce and compare, we implemented these networks on the common Theano neural network toolkit, and evaluated them on the ATIS benchmark. We also compared our results to a conditional random fields (CRF) baseline. Our results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.

In automatic speech recognition on mobile devices, very often what a user says strongly depends on the particular context he or she is in. The n-grams relevant to the context are often not known in advance. The context can depend on, for example, particular dialog state, options presented to the user, conversation topic, location, etc. Speech recognition of sentences that include these n-grams can be challenging, as they are often not well represented in a language model (LM) or even include out-of-vocabulary (OOV) words. In this paper, we propose a solution for using contextual information to improve speech recognition accuracy. We utilize an on-the-fly rescoring mechanism to adjust the LM weights of a small set of n-grams relevant to the particular context during speech decoding. Our solution handles out of vocabulary words. It also addresses efficient combination of multiple sources of context and it even allows biasing class based language models. We show significant speech recognition accuracy improvements on several datasets, using various types of contexts, without negatively impacting the overall system. The improvements are obtained in both offline and live experiments.

Automatic Speech Recognition (ASR) has historically been a driving force behind many machine learning (ML) techniques, including the ubiquitously used hidden Markov model, discriminative learning, structured sequence learning, Bayesian learning, and adaptive learning. Moreover, ML can and occasionally does use ASR as a large-scale, realistic application to rigorously test the effectiveness of a given technique, and to inspire new problems arising from the inherently sequential and dynamic nature of speech. On the other hand, even though ASR is available commercially for some applications, it is largely an unsolved problem - for almost all applications, the performance of ASR is not on par with human performance. New insight from modern ML methodology shows great promise to advance the state-of-the-art in ASR technology. This overview article provides readers with an overview of modern ML techniques as utilized in the current and as relevant to future ASR research and systems. The intent is to foster further cross-pollination between the ML and ASR communities than has occurred in the past. The article is organized according to the major ML paradigms that are either popular already or have potential for making significant contributions to ASR technology. The paradigms presented and elaborated in this overview include: generative and discriminative learning; supervised, unsupervised, semi-supervised, and active learning; adaptive and multi-task learning; and Bayesian learning. These learning paradigms are motivated and discussed in the context of ASR technology and applications. We finally present and analyze recent developments of deep learning and learning with sparse representations, focusing on their direct relevance to advancing ASR technology.

This work addresses the use of deep neural networks (DNNs) in automatic language identification (LID) focused on short test utterances. Motivated by their recent success in acoustic modelling for speech recognition, we adapt DNNs to the problem of identifying the language in a given utterance from the short-term acoustic features. We show how DNNs are particularly suitable to perform LID in real-time applications, due to their capacity to emit a language identification posterior at each new frame of the test utterance. We then analyse different aspects of the system, such as the amount of required training data, the number of hidden layers, the relevance of contextual information and the effect of the test utterance duration. Finally, we propose several methods to combine frame-by-frame posteriors. Experiments are conducted on two different datasets: the public NIST Language Recognition Evaluation 2009 (3 s task) and a much larger corpus (of 5 million utterances) known as Google 5M LID, obtained from different Google Services. Reported results show relative improvements of DNNs versus the i-vector system of 40% in LRE09 3 second task and 76% in Google 5M LID.

We are interested in the problem of adaptive learning in the context of automatic speech recognition (ASR). In this paper, we propose an active learning algorithm for ASR. Automatic speech recognition systems are trained using human supervision to provide transcriptions of speech utterances. The goal of Active Learning is to minimize the human supervision for training acoustic and language models and to maximize the performance given the transcribed and untranscribed data. Active learning aims at reducing the number of training examples to be labeled by automatically processing the unlabeled examples, and then selecting the most informative ones with respect to a given cost function for a human to label. In this paper we describe how to estimate the confidence score for each utterance through an on-line algorithm using the lattice output of a speech recognizer. The utterance scores are filtered through the informativeness function and an optimal subset of training samples is selected. The active learning algorithm has been applied to both batch and on-line learning scheme and we have experimented with different selective sampling algorithms. Our experiments show that by using active learning the amount of labeled data needed for a given word accuracy can be reduced by more than 60% with respect to random sampling.

This paper analyzes the behavior of stackaugmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, contextfree language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.


Question: A time-limited self-attention mechanism for automatic speech recognition.

This paper presents the development of Kaldi, an open-source toolkit designed for speech recognition research, freely available to the public. Kaldi offers a comprehensive speech recognition system built on finite-state transducers, facilitated by the OpenFst library, along with extensive documentation and scripts for constructing full-scale recognition systems. Implemented in C++, the core library of Kaldi enables modeling with customizable phonetic-context dimensions, supporting both subspace Gaussian mixture models (SGMM) and standard Gaussian mixture models for acoustic modeling, complemented by the standard linear and affine transforms. Under the permissive Apache License v2.0, Kaldi is accessible to a broad audience, encompassing both academic and commercial sectors, due to its nonrestrictive licensing terms.

State-of-the-art sequence labeling systems traditionally require large amounts of taskspecific knowledge in the form of handcrafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both wordand character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data preprocessing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks — Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both datasets — 97.55% accuracy for POS tagging and 91.21% F1 for NER.

Sophisticated gated recurrent neural network architectures like LSTMs and GRUs have been shown to be highly effective in a myriad of applications. We develop an un-gated unit, the statistical recurrent unit (SRU), that is able to learn long term dependencies in data by only keeping moving averages of statistics. The SRU’s architecture is simple, un-gated, and contains a comparable number of parameters to LSTMs; yet, SRUs perform favorably to more sophisticated LSTM and GRU alternatives, often outperforming one or both in various tasks. We show the efficacy of SRUs as compared to LSTMs and GRUs in an unbiased manner by optimizing respective architectures’ hyperparameters in a Bayesian optimization scheme for both synthetic and realworld tasks.

Nowadays, neural networks play an important role in the task of relation classification. In this paper, we propose a novel attention-based convolutional neural network architecture for this task. Our model makes full use of word embedding, part-of-speech tag embedding and position embedding information. Word level attention mechanism is able to better determine which parts of the sentence are most influential with respect to the two entities of interest. This architecture enables learning some important features from task-specific labeled data, forgoing the need for external knowledge such as explicit dependency structures. Experiments on the SemEval-2010 Task 8 benchmark dataset show that our model achieves better performances than several stateof-the-art neural network models and can achieve a competitive performance just with minimal feature engineering.

We propose “Areas of Attention”, a novel attentionbased model for automatic image captioning. Our approach models the dependencies between image regions, caption words, and the state of an RNN language model, using three pairwise interactions. In contrast to previous attentionbased approaches that associate image regions only to the RNN state, our method allows a direct association between caption words and image regions. During training these associations are inferred from image-level captions, akin to weakly-supervised object detector training. These associations help to improve captioning by localizing the corresponding regions during testing. We also propose and compare different ways of generating attention areas: CNN activation grids, object proposals, and spatial transformers nets applied in a convolutional fashion. Spatial transformers give the best results. They allow for image specific attention areas, and can be trained jointly with the rest of the network. Our attention mechanism and spatial transformer attention areas together yield state-of-the-art results on the MSCOCO dataset.

Although NeuroCRF, an augmented Conditional Random Fields (CRF) model whose feature function is parameterized as a Feed-Forward Neural Network (FF NN) on word embeddings, has soundly outperformed traditional linear-chain CRF on many sequence labeling tasks, it is held back by the fact that FF NNs have a fixed input length and therefore cannot take advantage of the full input sentence. We propose to address this issue by replacing the FF NN with a Long Short-Term Memory (LSTM) NN, which can summarize an input of arbitrary length into a fixed dimension representation. The resulting model obtains F1=89.28 on WikiNER dataset, a significant improvement over the NeuroCRF baseline’s F1=87.58, which is already a highly competitive result.

Recently a variety of LSTM-based conditional language models (LM) have been applied across a range of language generation tasks. In this work we study various model architectures and different ways to represent and aggregate the source information in an endto-end neural dialogue system framework. A method called snapshot learning is also proposed to facilitate learning from supervised sequential signals by applying a companion cross-entropy objective function to the conditioning vector. The experimental and analytical results demonstrate firstly that competition occurs between the conditioning vector and the LM, and the differing architectures provide different trade-offs between the two. Secondly, the discriminative power and transparency of the conditioning vector is key to providing both model interpretability and better performance. Thirdly, snapshot learning leads to consistent performance improvements independent of which architecture is used.

Traditional convolutional neural network (CNN) based query classification uses linear feature mapping in its convolution operation. The recurrent neural network (RNN), differs from a CNN in representing word sequence with their ordering information kept explicitly. We propose using a deep long-short-term-memory (DLSTM) based feature mapping to learn feature representation for CNN. The DLSTM, which is a stack of LSTM units, has different order of feature representations at different depth of LSTM unit. The bottom LSTM unit equipped with input and output gates, extracts the first order feature representation from current word. To extract higher order nonlinear feature representation, the LSTM unit at higher position gets input from two parts. First part is the lower LSTM unit’s memory cell from previous word. Second part is the lower LSTM unit’s hidden output from current word. In this way, the DLSTM captures the nonlinear nonconsecutive interaction within n-grams. Using an architecture that combines a stack of the DLSTM layers with a tradition CNN layer, we have observed new state-of-the-art query classification accuracy on benchmark data sets for query classification.

We introduce a neural machine translation model that views the input and output sentences as sequences of characters rather than words. Since word-level information provides a crucial source of bias, our input model composes representations of character sequences into representations of words (as determined by whitespace boundaries), and then these are translated using a joint attention/translation model. In the target language, the translation is modeled as a sequence of word vectors, but each word is generated one character at a time, conditional on the previous character generations in each word. As the representation and generation of words is performed at the character level, our model is capable of interpreting and generating unseen word forms. A secondary benefit of this approach is that it alleviates much of the challenges associated with preprocessing/tokenization of the source and target languages. We show that our model can achieve translation results that are on par with conventional word-based models.

In multi-turn dialogs, natural language understanding models can introduce obvious errors by being blind to contextual information. To incorporate dialog history, we present a neural architecture with Speaker-Sensitive Dual Memory Networks which encode utterances differently depending on the speaker. This addresses the different extents of information available to the system — the system knows only the surface form of user utterances while it has the exact semantics of system output. We performed experiments on real user data from Microsoft Cortana, a commercial personal assistant. The result showed a significant performance improvement over the state-of-the-art slot tagging models using contextual information.

This paper describes the technical and system building advances made to the Google Home multichannel speech recognition system, which was launched in November 2016. Technical advances include an adaptive dereverberation frontend, the use of neural network models that do multichannel processing jointly with acoustic modeling, and Grid-LSTMs to model frequency variations. On the system level, improvements include adapting the model using Google Home specific data. We present results on a variety of multichannel sets. The combination of technical and system advances result in a reduction of WER of 8-28% relative compared to the current production system.