Question: A practical task assessment of warehouse order selection utilizing head-mounted displays.

The maturation of Pick-by-Vision: assessing an augmented reality-enhanced picking system in a practical warehouse setting
Order picking represents a critical and labor-intensive component of logistics operations, where human dexterity and adaptability outshine automated alternatives. The integration of head-mounted displays (HMDs) into order picking systems, facilitated by Augmented Reality (AR), has emerged as a transformative approach to optimize information presentation and enhance worker efficiency.
This study chronicles the evolution of an AR-supported picking system, dubbed Pick-by-Vision, designed to augment the capabilities of warehouse personnel. The system's efficacy was rigorously evaluated through an empirical task analysis conducted in a live warehouse environment. Key performance indicators, including pick rates and accuracy, alongside qualitative measures of user satisfaction and system usability, were meticulously recorded.
The empirical findings underscore the significant advantages of Pick-by-Vision in streamlining industrial order picking processes. Notably, the system demonstrated substantial improvements in pick speed and accuracy, corroborating the potential of AR to revolutionize warehouse operations by providing workers with intuitive, real-time visual cues. Moreover, the user feedback was overwhelmingly positive, highlighting the system's ease of use and its potential to reduce cognitive load and training time for new staff. These outcomes collectively attest to the transformative impact of AR on enhancing the efficiency and effectiveness of warehouse order picking tasks.

In this paper we propose a theoretical framework of distributed representations and a methodology of representational analysis for the study of distributed cognitive tasksÑtasks that require the processing of information distributed across the internal mind and the external environment. The basic principle of distributed representations is that the representational system of a distributed cognitive task is a set of internal and external representations, which together represent the abstract structure of the task. The basic strategy of representational analysis is to decompose the representation of a hierarchical task into its component levels so that the representational properties at each level can be independently examined. The theoretical framework and the methodology are used to analyze the hierarchical structure of the Tower of Hanoi problem. Based on this analysis, four experiments are designed to examine the representational properties of the Tower of Hanoi. Finally, the nature of external representations is discussed. This research was supported by a grant to Donald Norman and Edwin Hutchins from the Ames Research Center of the National Aeronautics & Space Agency, Grant NCC 2-591 in the Aviation Safety/Automation Program, technical monitor, Everett Palmer. Additional support was provided by funds from the Apple Computer Company and the Digital Equipment Corporation to the Affiliates of Cognitive Science at

Augmented reality technology is a unique medium that can be helpful for young children's entertainment and education, but in order to achieve the benefits of this technology, augmented reality experiences need to be appropriately designed for young children's developing physical and cognitive skills. In the present study we investigated how 5-10 year-old children react to typical handheld augmented reality interaction techniques such as crosshair selection and finger selection, in AR environments that require them to change perspective or not. Our analysis shows significant impacts of age upon AR performance, with young children having slower selection times, more tracking losses, and taking longer to recover tracking. Significant differences were also found between AR interaction technique conditions, with finger selection being faster than crosshair selection, and interactions which required changes in perspective taking longer, generating more tracking losses, and more errors in selection. Furthermore, by analyzing children's performance in relation to metrics of physical and cognitive development, we identified correlations between AR interaction techniques performance and developmental tests of spatial relations, block construction and visuomotor precision. Gender differences were analyzed but no significant effects were detected.

During the usage of conventional stereo microscope, repetitive head movement are often inevitable for the users to retrieve information away from the microscope. Moreover, risks of surgeons loosing focus and increasing fatigue could also arise during microsurgeries. Therefore, Scope+, a stereoscopic video see-through augmented reality system was created not only to solve the problems mentioned above, but also to improve users stereo microscopic experience.

In this paper, we introduce the first validation framework of an indoor navigation system for blind and visually impaired (BVI) users, which is a significant step toward the development of cost effective indoor way-finding solutions for BVI users. The BVI users require detailed landmark-based navigation instructions that will help them arrive at the chosen destination independently. These users will interact with the navigation instructions on a smartphone using an accessible user interface. The validation framework includes the following three main components: 1) virtual reality-based simulation that simulates a BVI user traversing and interacting with the physical environment, developed using Unity game engine; 2) generation of action codes that emulate the avatar movement in the virtual environment, developed using a natural language processing parser of the navigation instructions; and 3) accessible user interface, which enables the user to access the navigation instructions, developed using Sikuli script library. We introduce a case study that illustrates the use of the validation tool using PERCEPT system. It is our strong belief that the validation framework we provide in this paper will encourage other developers to invent indoor navigation systems for BVI users. In addition, we would like to mention that this tool is the first step of validating an indoor navigation system and should be followed by trials with human subjects.

The deployment of human gaze has been almost exclusively studied independent of any specific ongoing task and limited to two-dimensional picture viewing. This contrasts with its use in everyday life, which mostly consists of purposeful tasks where gaze is crucially involved. To better understand deployment of gaze under such circumstances, we devised a series of experiments, in which subjects navigated along a walkway in a virtual environment and executed combinations of approach and avoidance tasks. The position of the body and the gaze were monitored during the execution of the task combinations and dependence of gaze on the ongoing tasks as well as the visual features of the scene was analyzed. Gaze distributions were compared to a random gaze allocation strategy as well as a specific "saliency model." Gaze distributions showed high similarity across subjects. Moreover, the precise fixation locations on the objects depended on the ongoing task to the point that the specific tasks could be predicted from the subject's fixation data. By contrast, gaze allocation according to a random or a saliency model did not predict the executed fixations or the observed dependence of fixation locations on the specific task.

Current input device taxonomies and other frameworks typically emphasize the mechanical structure of input devices. We suggest that selecting an appropriate input device for an interactive task requires looking beyond the physical structure of devices to the deeper perceptual structure of the task, the device, and the interrelationship between the perceptual structure of the task and the control properties of the device. We affirm that perception is key to understanding performance of multidimensional input devices on multidimensional tasks. We have therefore extended the theory of processing of percetual structure to graphical interactive tasks and to the control structure of input devices. This allows us to predict task and device combinations that lead to better performance and hypothesize that performance is improved when the perceptual structure of the task matches the control structure of the device. We conducted an experiment in which subjects performed two tasks with different perceptual structures, using two input devices with correspondingly different control structures, a three-dimensional tracker and a mouse. We analyzed both speed and accuracy, as well as the trajectories generated by subjects as they used the unconstrained three-dimensional tracker to perform each task. The result support our hypothesis and confirm the importance of matching the perceptual structure of the task and the control structure of the input device.

The design of office technology relies upon underlying conceptions of human organization and action. The goal of building office information systems requires a representation of office work and its relevant objects. The concern of this paper is that although system designers recognize the centrality of procedural tasks in the office, they tend to ignore the actual work involved in accomplishing those tasks. A perspicuous instance of work in an accounting office is used to recommend a new line of research into the practical problems of office work, and to suggest preliminary implications of that research for office systems design.

Several common systems satisfy some but not all of the VR definition above. Flight simulators provide vehicle tracking, not head tracking, and do not generally operate in binocular stereo. Omnimax theaters give a large angle of view [8], occasionally in stereo, but are not interactive. Head-tracked monitors [4][6] provide all but a large angle of view. Head-mounted displays (HMD) [7][13] and BOOMs [9] use motion of the actual display screens to achieve VR by our definition. Correct projection of the imagery on large screens can also create a VR experience, this being the subject of this paper. This paper describes the CAVE (CAVE Automatic Virtual Environment) virtual reality/scientific visualization system in detail and demonstrates that projection technology applied to virtual-reality goals achieves a system that matches the quality of workstation screens in terms of resolution, color, and flicker-free stereo. In addition, this format helps reduce the effect of common tracking and system latency errors. The off-axis perspective projection techniques we use are shown to be simple and straightforward. Our techniques for doing multi-screen stereo vision are enumerated, and design barriers, past and current, are described. Advantages and disadvantages of the projection paradigm are discussed, with an analysis of the effect of tracking noise and delay on the user. Successive refinement, a necessary tool for scientific visualization, is developed in the virtual reality context. The use of the CAVE as a one-to-many presentation device at SIGGRAPH '92 and Supercomputing '92 for computational science data is also mentioned. Previous work in the VR area dates back to Sutherland [12], who in 1965 wrote about the “Ultimate Display.” Later in the decade at the University of Utah, Jim Clark developed a system that allowed wireframe graphics VR to be seen through a headmounted, BOOM-type display for his dissertation. The common VR devices today are the HMD and the BOOM. Lipscomb [4] showed a monitor-based system in the IBM booth at SIGGRAPH '91 and Deering [6] demonstrated the Virtual Portal, a closets ized three-wal l project ion-based system, in the Sun Microsystems' booth at SIGGRAPH '92. The CAVE, our projection-based VR display [3], also premiered at SIGGRAPH '92. The Virtual Portal and CAVE have similar intent, but different implementation schemes.

This paper describes a mapping system that acquires 3D object models of man-made indoor environments such as kitchens. The system segments and geometrically reconstructs cabinets with doors, tables, drawers, and shelves, objects that are important for robots retrieving and manipulating objects in these environments. The system also acquires models of objects of daily use such glasses, plates, and ingredients. The models enable the recognition of the objects in cluttered scenes and the classification of newly encountered objects. Key technical contributions include (1) a robust, accurate, and efficient algorithm for constructing complete object models from 3D point clouds constituting partial object views, (2) feature-based recognition procedures for cabinets, tables, and other task-relevant furniture objects, and (3) automatic inference of object instance and class signatures for objects of daily use that enable robots to reliably recognize the objects in cluttered and real task contexts. We present results from the sensor-based mapping of a real kitchen.

In recent years, population aging becomes a serious problem. To decrease the demand for labor when navigating visitors in museums, exhibitions, or libraries, this research designs an automatic museum robotic guide which integrates image and gesture recognition technologies to enhance the guided tour quality of visitors. The robot is a self-propelled vehicle developed by ROS (Robot Operating System), in which we achieve the automatic driving based on the function of lane-following via image recognition. This enables the robot to lead guests to visit artworks following the preplanned route. In conjunction with the vocal service about each artwork, the robot can convey the detailed description of the artwork to the guest. We also design a simple wearable device to perform gesture recognition. As a human machine interface, the guest is allowed to interact with the robot by his or her hand gestures. To improve the accuracy of gesture recognition, we design a two phase hybrid machine learning-based framework. In the first phase (or training phase), k-means algorithm is used to train historical data and filter outlier samples to prevent future interference in the recognition phase. Then, in the second phase (or recognition phase), we apply KNN (k-nearest neighboring) algorithm to recognize the hand gesture of users in real time. Experiments show that our method can work in real time and get better accuracy than other methods.


Question: An empirical task evaluation of warehouse order picking utilizing head-mounted displays.

This study delves into the order batching problem (OBP) within the context of a 2-block rectangular warehouse, focusing on the optimization of order picking tours to minimize the average throughput time of orders. Given that orders typically follow a Poisson process and the routing strategy employed is the established S-shape heuristic, we undertake a detailed analysis to ascertain the mean and variance of the order picker's travel time. Leveraging these statistical moments, we then estimate the average throughput time for a random order, which in turn facilitates the calculation of the optimal size for a picking batch. Our findings, corroborated through simulation, reveal that this method offers a high degree of accuracy. Moreover, the simplicity of the approach ensures its practical applicability, making it a viable solution for real-world warehouse operations.

This paper explores a user interface technique which augments an immersive head tracked display with a hand-held miniature copy of the virtual environment. We call this interface technique the Worlds in Miniature (WIM) metaphor. In addition to the first-person perspective offered by a virtual reality system, a World in Miniature offers a second dynamic viewport onto the virtual environment. Objects may be directly manipulated either through the immersive viewport or through the three-dimensional viewport offered by the WIM. In addition to describing object manipulation, this paper explores ways in which Worlds in Miniature can act as a single unifying metaphor for such application independent interaction techniques as object selection, navigation, path planning, and visualization. The WIM metaphor offers multiple points of view and multiple scales at which the user can operate, without requiring explicit modes or commands. Informal user observation indicates that users adapt to the Worlds in Miniature metaphor quickly and that physical props are helpful in manipulating the WIM and other objects in the environment.

Usability evaluation of an information visualization technique can only be done by the joint evaluation of both the visual representation and the interaction techniques. This work proposes task models as a key element for carrying out such evaluations in a structured way. We base our work on a taxonomy abstracting from rendering functions supported by information visualization techniques. CTTE is used to model these abstract visual tasks as well as to generate scenarios from this model for evaluation purposes. We conclude that the use of task models allows generating test scenarios which are more effective than informal and unstructured evaluations.

This work explores the feasibility of using sensors embedded in Google Glass, a head-mounted wearable device, to measure physiological signals of the wearer. In particular, we develop new methods to use Glass's accelerometer, gyroscope, and camera to extract pulse and respiratory rates of 12 participants during a controlled experiment. We show it is possible to achieve a mean absolute error of 0.83 beats per minute (STD: 2.02) for heart rate and 1.18 breaths per minute (STD: 2.04) for respiration rate when considering different combinations of sensors. These results included testing across sitting, supine, and standing still postures before and after physical exercise.

The tourism sector has found virtual reality technology to be a good way to market travel destinations for consumers. In this paper, we describe two user studies with three web-based 3D virtual travel destination marketing portals. These three portals were developed to support and attract wintertime tourism into the region by offering a possibility to experience in advance, a virtual snowy scenery with different activities, for example downhill skiing. In both user studies with 21 subjects the focus was on user experience with the 3D virtual travel destination marketing portals. In the second study also the visual design aspects within these portals were studied. Our studies indicate that 3D virtual travel destination marketing portals can enhance 2D web pages, if they offer the possibility to explore the location freely and through different kinds of virtual activities. Also our studies support prior findings of the efficiency of glow effect for indicating interactive 3D elements within a 3D virtual environment.

We describe a novel system for rapid prototyping of laserbased optical and holographic layouts. Users of this optical prototyping tool – called the Illuminating Light system – move physical representations of various optical elements about a workspace, while the system tracks these components and projects back onto the workspace surface the simulated propagation of laser light through the evolving layout. This application is built atop the Luminous Room infrastructure, an aggregate of interlinked, computer-controlled projector-camera units called I/O Bulbs. Philosophically, the work embodies the emerging ideas of the Luminous Room and builds on the notions of ‘graspable media’. We briefly introduce the I/O Bulb and Luminous Room concepts and discuss their current implementations. After an overview of the optical domain that the Illuminating Light system is designed to address, we present the overall system design and implementation, including that of an intermediary toolkit called voodoo which provides a general facility for object identification and tracking.

Digital Technology is constantly improving as information becomes wireless. These advances demand more wearable and mobile form factors for products that access information. A product that is wearable should have wearability. This paper explores the concept of dynamic wearability, through design research. Wearability is defined as the interaction between the human body and the wearable object. Dynamic wearability extends that definition to include the human body in motion. Our research has been to locate, understand, and define the spaces on the human body where solid and flexible forms can rest-without interfering with fluid human movement. The result is a set of design guidelines embodied in a set of wearable forms. These wearable forms describe the three dimensional spaces on the body best suited for comfortable and unobtrusive wearability by design.

We present the design and construction of a handheld mirror simulation device. The perception of the world reflected through a mirror depends on the viewer's position with respect to the mirror and the 3-D geometry of the world. In order to simulate a real mirror on a computer screen, images of the observed world, consistent with the viewer's position, must be synthesized and displayed in realtime. Our system is build around a LCD screen manipulated by the user, a single camera fixed on the screen, and a tracking device. The continuous input video stream and tracker data is used to synthesize, in real-time, a continuous video stream displayed on the LCD screen. The synthesized video stream is a close approximation of what the user would see on the screen surface if it were a real mirror. Our system provides a generic interface for applications involving rich, first-person interaction, such as the Virtual Daguerreotype.

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).

Bringing real objects into the virtual world has been shown to increase usability and presence in virtual reality applications. This paper presents a system to generate a real time virtual reconstruction of real world user interface elements for use in a head mounted display based driving simulator. Our system uses sensor fusion algorithms to combine data from depth and color cameras to generate an accurate, detailed, and fast rendering of the user's hands while using the simulator. We tested our system and show in our results that the inclusion of the participants real hands, the wheel, and the shifter in the virtual environment increases the immersion, presence, and usability of the simulation. Our system can also be used to bring other real objects into the virtual world, especially when accuracy, detail, and real time updates are desired.

We introduce a set of nine measures to characterize viewers' visual attention patterns for multi-screen TV. We apply our measures during an experiment involving nine screen layouts with two, three, and four TV screens, for which we report new findings on visual attention. For example, we found that viewers need an average discovery time up to 4.5 seconds to visually fixate four screens, and their perceptions of how long they watched each screen are substantially accurate, i.e., we report Pearson correlations up to .892 with measured eye tracking data. We hope our set of new measures (and the companion toolkit to compute them automatically) will benefit the community as a first step toward understanding visual attention for emerging multi-screen TV applications.


Question: An empirical analysis of warehouse order picking tasks utilizing head-mounted displays.

This paper explores the integration of a wearable mobile computing system within Adtranz, a cutting-edge train maintenance and diagnosis platform, focusing on its mobile information and communication capabilities. The system, featuring wearable/mobile devices enhanced by wireless technology, significantly boosts the efficiency and precision of maintenance tasks. Maintenance staff on-site can now seamlessly connect with a remote helpdesk or expertise center, exchanging digital data, audio, and visual information, thanks to this technological advancement. The paper delves into the working prototype's functionalities and presents research findings that underscore the system's transformative impact on maintenance and collaboration processes.

Will Augmented Reality (AR) allow us to access digital information, experience others' stories, and thus explore alternate realities? AR has recently attracted attention again due to the rapid advances in related multimedia technologies as well as various glass-type AR display devices. However, in order to widely adopt AR in alternated realities, it is necessary to improve various core technologies and integrate them into an AR platform. Especially, there are several remaining technical challenges such as 1) real-time recognition and tracking of multiple objects while generating an environment map, 2) organic user interface with awareness of user's implicit needs, intentions or emotion as well as explicit requests, 3) immersive multimodal content augmentation and just-in-time information visualization, 4) multimodal interaction and collaboration during augmented telecommunication, etc. In addition, in order to encourage user engagement and enable an AR ecosystem, AR standards should be established that support creating AR content, capturing user experiences, and sharing the captured experiences.

Introduction Given the fact that the driving task will remarkably change by the increase of assistance and information systems in the future innovative display technologies play a central role for the ergonomic realization of the driver vehicle interaction. Increasing assistance and even automation of the driving task do not lead to a decrease or disappearance of visual information but instead request new and in some cases revolutionary concepts to close the loop between driver, vehicle and traffic environment. Augmenting information in contact analog head-up-displays for navigation and driver assistance is a very promising approach. Replacement of mirrors via camera monitor systems is a further example. Free programmable cluster instruments in combination with HUD promise to resolve the problem of information density produced by an increasing amount of ADAS and IVIS functionality.

This paper surveys the field of augmented reality (AR), in which 3D virtual objects are integrated into a 3D real environment in real time. It describes the medical, manufacturing , visualization, path planning, entertainment, and military applications that have been explored. This paper describes the characteristics of augmented reality systems, including a detailed discussion of the tradeoffs between optical and video blending approaches. Registration and sensing errors are two of the biggest problems in building effective augmented reality systems, so this paper summarizes current efforts to overcome these problems. Future directions and areas requiring further research are discussed. This survey provides a starting point for anyone interested in researching or using augmented reality.

We present MRTouch, a novel multitouch input solution for head-mounted mixed reality systems. Our system enables users to reach out and directly manipulate virtual interfaces affixed to surfaces in their environment, as though they were touchscreens. Touch input offers precise, tactile and comfortable user input, and naturally complements existing popular modalities, such as voice and hand gesture. Our research prototype combines both depth and infrared camera streams together with real-time detection and tracking of surface planes to enable robust finger-tracking even when both the hand and head are in motion. Our technique is implemented on a commercial Microsoft HoloLens without requiring any additional hardware nor any user or environmental calibration. Through our performance evaluation, we demonstrate high input accuracy with an average positional error of 5.4 mm and 95% button size of 16 mm, across 17 participants, 2 surface orientations and 4 surface materials. Finally, we demonstrate the potential of our technique to enable on-world touch interactions through 5 example applications.

or more than a decade researchers have tried to create intuitive computer interfaces by blending reality and virtual reality. The goal is for people to interact with the digital domain as easily as with the real world. Various approaches help us achieve this—in the area of tangible interfaces, we use real objects as interface widgets; in augmented reality, researchers overlay 3D virtual imagery onto the real world; and in VR interfaces , we entirely replace the real world with a computer generated environment. As Milgram pointed out, 1 these types of computer interfaces can be placed along a continuum according to how much of the user's environment is computer generated (Figure 1). Tangible interfaces lie far to the left on this reality–virtuality line, while immersive virtual environments are at the right extremity. Most current user interfaces exist as discrete points along this continuum. However, human activity can't always be broken into discrete components and for many tasks users may prefer to move seamlessly along the reality–virtuality continuum. This proves true when interacting with 3D graphical content, either creating virtual models or viewing them. For example, if people want to experience a virtual scene from different scales, then immer-sive virtual reality may be ideal. If they want to have a face-to-face discussion while viewing the virtual scene, an augmented reality interface may be best. 2 The MagicBook project is an early attempt to explore how we can use a physical object to smoothly transport users between reality and virtuality. Young children often fantasize about flying into the pages of a fairy tale and becoming part of the story. The MagicBook project makes this fantasy a reality using a normal book as the main interface object. People can turn the pages of the book, look at the pictures, and read the text without any additional technology (Figure 2a). However, if a person looks at the pages through an augmented reality display, they see 3D virtual models appearing out of the pages (Figure 2b). The models appear attached to the real page so users can see the augmented reality scene from any perspective by moving themselves or the book. The virtual content can be any size and is animated, so the augmented reality view is an enhanced version of a traditional 3D pop-up book. Users can change the virtual models by turning the book pages. When they see a scene they particularly like, …

This paper describes the UJI Librarian Robot, a mobile manipulator that is able to autonomously locate a book in an ordinary library, and grasp it from a bookshelf, by using eye-in-hand stereo vision and force sensing. The robot is only provided with the book code, a library map and some knowledge about its logical structure and takes advantage of the spatio-temporal constraints and regularities of the environment by applying disparate techniques such as stereo vision, visual tracking, probabilistic matching, motion estimation, multisensor-based grasping, visual servoing and hybrid control, in such a way that it exhibits a robust and dependable performance. The system has been tested, and experimental results show how it is able to robustly locate and grasp a book in a reasonable time without human intervention.

We have been developing a wearable robot that directly and physically supports human movement. A ldquomuscle suitrdquo for the arm that will provide muscular support for the paralyzed or those otherwise unable to move unaided is one example. Whereas low back pain is one of the most serious issue for the manual worker and in this paper, we focus on developing the support system for forward tilting of the upper body. In this paper, the concept and mechanical structure of the support system is shown. From the experimental result to keep forward tilting posture with load indicates our system is very effective for keeping forward tilting posture and it becomes clear that more than 60% reduction of muscular power is possible.

The cave automatic virtual environment (CAVE) system is one of the most fully immersive systems for virtual reality environments. By providing users with realistic perception and immersive experience, CAVE systems have been widely used in many fields, including military, education, health care, entertainment, design, and others. In this paper, we focus on the design applications in the CAVE. The design applications involve many interactions between the user and the CAVE. However, the conventional interaction tool, the wand, cannot provide fast and convenient interactions. In this paper, we propose vDesign, a CAVE-based virtual design environment using hand interactions. The hand interactions in vDesign are classified into menu navigation and object manipulations. For menu navigation, we define two interactions: activating the main menu and selecting a menu item. For object manipulations, we define three interactions: moving, rotating, and scaling an object. By using the proposed hand interactions, we develop the functions of image segmentation and image composition in vDesign. With the image segmentation function, the designer can select and cut the interested objects from different images. With the image composition function, the designer can manipulate the segmented objects and combine them as a composite image. We implemented the vDesign prototype in CAVE and conducted experiments to evaluate the interaction performance in terms of manipulation time and distortion. The experimental results demonstrated that the proposed hand interactions can provide faster and more accurate interactions compared to the traditional wand interactions.

Virtual Reality (VR) has finally caught the attention of major technology companies, giving it a significant boost towards becoming market-ready. VR has the capacity to create profoundly engaging experiences by synthesizing highly detailed sensory input in real time, making games the most obvious applications of VR. While the promises of VR may be exciting due to the problems it can solve and new experiences it can offer, there are still numerous issues to be addressed in order to achieve maximal immersion in VR. In this paper, we present the issues encountered during open testing of our applications on commercially available VR goggles and hand input device. The inability to effectively use the hands and feet to control and navigate the environment degrades the level of immersion of VR. With users preferring reliability over intuitiveness, we explore the use of gaze as a substitute input device. Results of experiments conducted on numerous volunteers showed significant improvement in terms of user control.

A useful starting point for designing advanced graphical user interfaces is the Visual InformationSeeking Mantra: Overview first, zoom and filter, then details-on-demand. But this is only a starting point in trying to understand the rich and varied set of information visualizations that have been proposed in recent years. This paper offers a task by data type taxonomy with seven data types (1-, 2-, 3-dimensional data, temporal and multi-dimensional data, and tree and network data) and seven tasks (overview, zoom, filter, details-on-demand, relate, history, and extract).


Question: A large-scale automated software diversity technique—revisiting program evolution.

This paper reveals the feasibility of executing return-oriented programming (ROP) attacks without the explicit use of return instructions on both x86 and ARM architectures. We demonstrate that by leveraging specific instruction sequences that mimic the functionality of return instructions, we can construct Turing-complete gadget sets. These sequences are found with adequate frequency in large-scale libraries on (x86) Linux and (ARM) Android platforms, enabling the creation of sophisticated ROP attacks.

The absence of return instructions in our attacks has significant repercussions for several contemporary defense mechanisms against ROP. These include defenses that monitor the frequency of return instruction usage in the instruction stream, those that enforce the last-in, first-out (LIFO) discipline for the return-address stack, and compiler modifications aimed at reducing the reliance on return instructions. Our findings challenge the effectiveness of these defenses and underscore the need for more robust security measures against ROP attacks that do not rely on conventional return instruction patterns.

Address-space randomization is a technique used to fortify systems against buffer overflow attacks. The idea is to introduce artificial diversity by randomizing the memory location of certain system components. This mechanism is available for both Linux (via PaX ASLR) and OpenBSD. We study the effectiveness of address-space randomization and find that its utility on 32-bit architectures is limited by the number of bits available for address randomization. In particular, we demonstrate a <i>derandomization attack</i> that will convert any standard buffer-overflow exploit into an exploit that works against systems protected by address-space randomization. The resulting exploit is as effective as the original exploit, although it takes a little longer to compromise a target machine: on average 216 seconds to compromise Apache running on a Linux PaX ASLR system. The attack does not require running code on the stack.
 We also explore various ways of strengthening address-space randomization and point out weaknesses in each. Surprisingly, increasing the frequency of re-randomizations adds at most 1 bit of security. Furthermore, compile-time randomization appears to be more effective than runtime randomization. We conclude that, on 32-bit architectures, the only benefit of PaX-like address-space randomization is a small slowdown in worm propagation speed. The cost of randomization is extra complexity in system support.

Programming community-based question-answering (PCQA) websites such as Stack Overflow enable programmers to find working solutions to their questions. Despite detailed posting guidelines, duplicate questions that have been answered are frequently created. To tackle this problem, Stack Overflow provides a mechanism for reputable users to manually mark duplicate questions. This is a laborious effort, and leads to many duplicate questions remain undetected. Existing duplicate detection methodologies from traditional community based question-answering (CQA) websites are difficult to be adopted directly to PCQA, as PCQA posts often contain source code which is linguistically very different from natural languages. In this paper, we propose a methodology designed for the PCQA domain to detect duplicate questions. We model the detection as a classification problem over question pairs. To extract features for question pairs, our methodology leverages continuous word vectors from the deep learning literature, topic model features and phrases pairs that co-occur frequently in duplicate questions mined using machine translation systems. These features capture semantic similarities between questions and produce a strong performance for duplicate detection. Experiments on a range of real-world datasets demonstrate that our method works very well; in some cases over 30% improvement compared to state-of-the-art benchmarks. As a product of one of the proposed features, the association score feature, we have mined a set of associated phrases from duplicate questions on Stack Overflow and open the dataset to the public.

Bugs due to data races in multithreaded programs often exhibit non-deterministic symptoms and are notoriously difficult to find. This paper describes RaceTrack, a dynamic race detection tool that tracks the actions of a program and reports a warning whenever a suspicious pattern of activity has been observed. RaceTrack uses a novel hybrid detection algorithm and employs an adaptive approach that automatically directs more effort to areas that are more suspicious, thus providing more accurate warnings for much less over-head. A post-processing step correlates warnings and ranks code segments based on how strongly they are implicated in potential data races. We implemented RaceTrack inside the virtual machine of Microsoft's Common Language Runtime (product version v1.1.4322) and monitored several major, real-world applications directly out-of-the-box,without any modification. Adaptive tracking resulted in a slowdown ratio of about 3x on memory-intensive programs and typically much less than 2x on other programs,and a memory ratio of typically less than 1.2x. Several serious data race bugs were revealed, some previously unknown.

A dynamic program slice is that part of a program that ‘‘affects’’ the computation of a variable of interest during program execution on a specific program input. Dynamic program slicing refers to a collection of program slicing methods that are based on program execution and may significantly reduce the size of a program slice because run-time information, collected during program execution, is used to compute program slices. Dynamic program slicing was originally proposed only for program debugging, but its application has been extended to program comprehension, software testing, and software maintenance. Different types of dynamic program slices, together with algorithms to compute them, have been proposed in the literature. In this paper we present a classification of existing dynamic slicing methods and discuss the algorithms to compute dynamic slices. In the second part of the paper, we compare the existing methods of dynamic slice computation.  1998 Elsevier Science B.V. All rights reserved.

Software change impact analysis (CIA) is a technique for identifying the effects of a change, or estimating what needs to be modified to accomplish a change. Since the 1980s, there have been many investigations on CIA, especially for code-based CIA techniques. However, there have been very few surveys on this topic. This article tries to fill this gap. And 30 papers that provide empirical evaluation on 23 code-based CIA techniques are identified. Then, data was synthesized against four research questions. The study presents a comparative framework including seven properties, which characterize the CIA techniques, and identifies key applications of CIA techniques in software maintenance. In addition, the need for further research is also presented in the following areas: evaluating existing CIA techniques and proposing new CIA techniques under the proposed framework, developing more mature tools to support CIA, comparing current CIA techniques empirically with unified metrics and common benchmarks, and applying the CIA more extensively and effectively in the software maintenance phase. Copyright © 2012 John Wiley & Sons, Ltd.

This paper presents an approach to automatic discovery of functions in Genetic Programming. The approach is based on discovery of useful building blocks by analyzing the evolution trace, generalizing blocks to define new functions, and finally adapting the problem representation onthe-fly. Adaptating the representation determines a hierarchical organization of the extended function set which enables a restructuring of the search space so that solutions can be found more easily. Measures of complexity of solution trees are defined for an adaptive representation framework. The minimum description length principle is applied to justify the feasibilityof approaches based on a hierarchy of discovered functions and to suggest alternative ways of defining a problem’s fitness function. Preliminary empirical results are presented.

Backdoors in software systems probably exist since the very first access control mechanisms were implemented and they are a well-known security problem. Despite a wave of public discoveries of such backdoors over the last few years, this threat has only rarely been tackled so far.
 In this paper, we present an approach to reduce the attack surface for this kind of attacks and we strive for an automated identification and elimination of backdoors in binary applications. We limit our focus on the examination of server applications within a client-server model. At the core, we apply variations of the delta debugging technique and introduce several novel heuristics for the identification of those regions in binary application that backdoors are typically installed in (i.e., authentication and command processing functions). We demonstrate the practical feasibility of our approach on several real-world backdoors found in modified versions of the popular software tools ProFTPD and OpenSSH. Furthermore, we evaluate our implementation not only on common instruction set architectures such as x86-64, but also on commercial off-the-shelf embedded devices powered by a MIPS32 processor.

With a large and rapidly changing codebase, Google software engineers are constantly paying interest on various forms of technical debt. Google engineers also make efforts to pay down that debt, whether through special Fixit days, or via dedicated teams, variously known as janitors, cultivators, or demolition experts. We describe several related efforts to measure and pay down technical debt found in Google's BUILD files and associated dead code. We address debt found in dependency specifications, unbuildable targets, and unnecessary command line flags. These efforts often expose other forms of technical debt that must first be managed.

Business processes support the modeling and the implementation of software as workflows of local and inter-process activities. Taking over structuring and composition, evolution has become a central concern in software development. We advocate it should be taken into account as soon as the modeling of business processes, which can thereafter be made executable using process engines or model-to-code transformations. We show here that business process evolution needs formal analysis in order to compare different versions of processes, identify precisely the differences between them, and ensure the desired consistency. To reach this objective, we first present a model transformation from the BPMN standard notation to the LNT process algebra. We then propose a set of relations for comparing business processes at the formal model level. With reference to related work, we propose a richer set of comparison primitives supporting renaming, refinement, propertyand context-awareness. Thanks to an implementation of our approach that can be used through a Web application, we put the checking of evolution within the reach of business process designers.

We identify three types of attack on the intellectual property contained in software, and three corresponding technical defenses. A potent defense against reverse engineering is obfuscation, a process that renders software unintelligible but still functional. A defense against software piracy is watermarking, a process that makes it possible to determine the origin of software. A defense against tampering is tamper-proofing, so that unauthorized modifications to software (for example to remove a watermark) will result in non-functional code. We briefly survey the available technology for each type of defense.


Question: A large-scale automated approach to software diversity—revisiting program evolution.

The advent of simultaneous multithreading, notably through Intel's "Hyper-Threading" technology in their Pentium 4 processors, has popularized the concept of sharing execution resources among multiple threads. This sharing, intended for efficiency and cost-effectiveness, goes beyond execution units to include the memory caches, raising significant security concerns. Our investigation reveals that the shared access to memory caches not only facilitates a high-bandwidth covert channel between threads but also enables a malevolent thread, potentially with restricted privileges, to surveil the execution of a cohabiting thread. This surveillance can, in numerous instances, lead to the theft of cryptographic keys. To address these vulnerabilities, we offer recommendations to processor designers, operating system developers, and cryptographic software authors on strategies to mitigate or entirely eliminate this form of attack. This includes enhanced cache management techniques and the implementation of security protocols that can thwart such malicious activities without compromising the performance benefits of simultaneous multithreading.

Semantic Differencing for Numerical Programs Nimrod Partush Eran Yahav

Context: Source code reuse has been widely accepted as a fundamental activity in software development. Recent studies showed that StackOverflow has emerged as one of the most popular resources for code reuse. Therefore, a plethora of work proposed ways to optimally ask questions, search for answers and find relevant code on StackOverflow. However, little work studies the impact of code reuse from Stack-

We present MUSE (MUtation-baSEd fault localization technique), a new fault localization technique based on mutation analysis. A key idea of MUSE is to identify a faulty statement by utilizing different characteristics of two groups of mutants-one that mutates a faulty statement and the other that mutates a correct statement. We also propose a new evaluation metric for fault localization techniques based on information theory, called Locality Information Loss (LIL): it can measure the aptitude of a localization technique for automated fault repair systems as well as human debuggers. The empirical evaluation using 14 faulty versions of the five real-world programs shows that MUSE localizes a fault after reviewing 7.4 statements on average, which is about 25 times more precise than the state-of-the-art SBFL technique Op2.

Deep learning has made significant breakthroughs in various fields of artificial intelligence. Advantages of deep learning include the ability to capture highly complicated features, weak involvement of human engineering, etc. However, it is still virtually impossible to use deep learning to analyze programs since deep architectures cannot be trained effectively with pure back propagation. In this pioneering paper, we propose the “coding criterion” to build program vector representations, which are the premise of deep learning for program analysis. Our representation learning approach directly makes deep learning a reality in this new field. We evaluate the learned vector representations both qualitatively and quantitatively. We conclude, based on the experiments, the coding criterion is successful in building program representations. To evaluate whether deep learning is beneficial for program analysis, we feed the representations to deep neural networks, and achieve higher accuracy in the program classification task than “shallow” methods, such as logistic regression and the support vector machine. This result confirms the feasibility of deep learning to analyze programs. It also gives primary evidence of its success in this new field. We believe deep learning will become an outstanding technique for program analysis in the near future.

The increased availability of data created as part of the sof tware development process allows us to apply novel analysis techniques on the data and use the results to guide t he process’s optimization. In this paper we describe variou s data sources and discuss the principles and techniques of data mi ning as applied on software engineering data. Data that can b e mined is generated by most parts of the development process: requi rements elicitation, development analysis, testing, debu gging, and maintenance. Based on this classification we survey the mini ng approaches that have been used and categorize them accord ing to the corresponding parts of the development process and th e task they assist. Thus the survey provides researchers wit h a concise overview of data mining techniques applied to softw are engineering data, and aids practitioners on the selecti on of appropriate data mining techniques for their work.

Security practitioners use the attack surface of software systems to prioritize areas of systems to test and analyze. To date, approaches for predicting which code artifacts are vulnerable have utilized a binary classification of code as vulnerable or not vulnerable. To better understand the strengths and weaknesses of vulnerability prediction approaches, vulnerability datasets with classification and severity data are needed. The goal of this paper is to help researchers and practitioners make security effort prioritization decisions by evaluating which classifications and severities of vulnerabilities are on an attack surface approximated using crash dump stack traces. In this work, we use crash dump stack traces to approximate the attack surface of Mozilla Firefox. We then generate a dataset of 271 vulnerable files in Firefox, classified using the Common Weakness Enumeration (CWE) system. We use these files as an oracle for the evaluation of the attack surface generated using crash data. In the Firefox vulnerability dataset, 14 different classifications of vulnerabilities appeared at least once. In our study, 85.3% of vulnerable files were on the attack surface generated using crash data. We found no difference between the severity of vulnerabilities found on the attack surface generated using crash data and vulnerabilities not occurring on the attack surface. Additionally, we discuss lessons learned during the development of this vulnerability dataset.

During maintenance developers cannot read the entire code of large systems. They need a way to get a quick understanding of source code entities (such as, classes, methods, packages, etc.), so they can efficiently identify and then focus on the ones related to their task at hand. Sometimes reading just a method header or a class name does not tell enough about its purpose and meaning, while reading the entire implementation takes too long. We study a solution which mitigates the two approaches, i.e., short and accurate textual descriptions that illustrate the software entities without having to read the details of the implementation. We create such descriptions using techniques from automatic text summarization. The paper presents a study that investigates the suitability of various such techniques for generating source code summaries. The results indicate that a combination of text summarization techniques is most appropriate for source code summarization and that developers generally agree with the summaries produced.

With a large and rapidly changing codebase, Google software engineers are constantly paying interest on various forms of technical debt. Google engineers also make efforts to pay down that debt, whether through special Fixit days, or via dedicated teams, variously known as janitors, cultivators, or demolition experts. We describe several related efforts to measure and pay down technical debt found in Google's BUILD files and associated dead code. We address debt found in dependency specifications, unbuildable targets, and unnecessary command line flags. These efforts often expose other forms of technical debt that must first be managed.

One of the key challenges of developers testing code is determining a test suite's quality -- its ability to find faults. The most common approach is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high absolute coverage as a stopping rule. In testing research, suite quality is often evaluated by a suite's ability to kill mutants (artificially seeded potential faults). Determining which criteria best predict mutation kills is critical to practical estimation of test suite quality. Previous work has only used small sets of programs, and usually compares multiple suites for a single program. Practitioners, however, seldom compare suites --- they evaluate one suite. Using suites (both manual and automatically generated) from a large set of real-world open-source projects shows that evaluation results differ from those for suite-comparison: statement (not block, branch, or path) coverage predicts mutation kills best.

This tutorial paper considers the issues of low-level software security from a language-based perspective, with the help of concrete examples. Four examples of low-level software attacks are covered in full detail; these examples are representative of the major types of attacks on C and C++ software that is compiled into machine code. Six examples of practical defenses against those attacks are also covered in detail; these defenses are selected because of their effectiveness, wide applicability, and low enforcement overhead.


Question: A large-scale automated software diversity technique—revisiting program evolution.

Introducing Self-Randomizing Instruction Addresses: Binary Stirring for Enhanced Security

In the realm of software security, the static nature of instruction addresses in application binary code presents a significant vulnerability. Unlike library code that benefits from address space layout randomization (ASLR), application binary code remains predictably structured, offering attackers a pathway to craft sophisticated shell codes by leveraging these static instruction gadgets. This paper unveils binary stirring, an innovative technique designed to empower x86 native code with the capability to self-randomize its instruction addresses upon each execution.

Binary stirring, or STIR, targets legacy x86 binary code without the need for source code, debug symbols, or relocation information. The core output of STIR is a transformed binary where the basic block addresses are determined dynamically at load time. This dynamic addressing ensures that while an attacker might identify code gadgets in one instance, the instruction addresses in subsequent instances are entirely unpredictable, significantly complicating attack strategies.

To achieve this level of protection, STIR employs a suite of binary transformation techniques that are transparent to the user. These techniques are particularly crucial for safeguarding large-scale, complex applications that may feature challenging binary characteristics such as computed jumps, code-data interleaving, OS callbacks, dynamic linking, and others. Notably, evaluations on both Windows and Linux platforms reveal that binary stirring introduces a manageable 1.6% overhead on average to application runtimes, demonstrating its practicality and efficiency in enhancing software security.

In summary, binary stirring represents a significant advancement in software diversity and security by enabling the self-randomization of instruction addresses in legacy x86 binary code, thereby thwarting attacks that rely on the predictability of these addresses.

This tutorial paper considers the issues of low-level software security from a language-based perspective, with the help of concrete examples. Four examples of low-level software attacks are covered in full detail; these examples are representative of the major types of attacks on C and C++ software that is compiled into machine code. Six examples of practical defenses against those attacks are also covered in detail; these defenses are selected because of their effectiveness, wide applicability, and low enforcement overhead.

Malware is notoriously difficult to combat because it appears and spreads so quickly. In this article, we describe the design and implementation of CWSandbox, a malware analysis tool that fulfills our three design criteria of automation, effectiveness, and correctness for the Win32 family of operating systems

The discovery of vulnerabilities in source code is a key for securing computer systems. While specific types of security flaws can be identified automatically, in the general case the process of finding vulnerabilities cannot be automated and vulnerabilities are mainly discovered by manual analysis. In this paper, we propose a method for assisting a security analyst during auditing of source code. Our method proceeds by extracting abstract syntax trees from the code and determining structural patterns in these trees, such that each function in the code can be described as a mixture of these patterns. This representation enables us to decompose a known vulnerability and extrapolate it to a code base, such that functions potentially suffering from the same flaw can be suggested to the analyst. We evaluate our method on the source code of four popular open-source projects: LibTIFF, FFmpeg, Pidgin and Asterisk. For three of these projects, we are able to identify zero-day vulnerabilities by inspecting only a small fraction of the code bases.

Code smell is an indication of a software designing problem. The presence of code smells can have a severe impact on the software quality. Smells basically refers to the structure of the code which violates few of the design principals and so has negative effect on the quality of the software. Larger the source code, more is its presence. Software needs to be reliable, robust and easily maintainable so that it can minimize the cost of its development as well as maintenance. Smells may increase the chances of failure of the system during maintenance. A SLR has been performed based on the search of digital libraries that includes the publications since 1999 to 2016. 60 research papers are deeply analyzed that are most relevant. The objective of this paper is to provide an extensive overview of existing research in the field of bad smells, identify the detection techniques and correlation between the detection techniques, in addition to find the name of the code smells that need more attention in detection approaches. This SLR identified that code clone (code smell) receives most research attention. Our findings also show that very few papers report on the impact of code bad smells. Most of the papers focused on the detection techniques and tools. A significant correlation between detection techniques has been calculated. There are four code smells that are not yet detected are Primitive Obsession, Inappropriate Intimacy, Incomplete library class and Comments.

Buffer overflow is one of the most common types of software security vulnerabilities. Although researchers have proposed various static and dynamic techniques for buffer overflow detection, buffer overflow attacks against both legacy and newly-deployed software systems are still quite prevalent. Compared with dynamic detection techniques, static techniques are more systematic and scalable. However, there are few studies on the effectiveness of state-of-the-art static buffer overflow detection techniques. In this paper, we perform an in-depth quantitative and qualitative study on static buffer overflow detection. More specifically, we obtain both the buggy and fixed versions of 100 buffer overflow bugs from 63 real-world projects totalling 28 MLoC (Millions of Lines of Code) based on the reports in Common Vulnerabilities and Exposures (CVE). Then, quantitatively, we apply Fortify, Checkmarx, and Splint to all the buggy versions to investigate their false negatives, and also apply them to all the fixed versions to investigate their false positives. We also qualitatively investigate the causes for the false-negatives and false-positives of studied techniques to guide the design and implementation of more advanced buffer overflow detection techniques. Finally, we also categorized the patterns of manual buffer overflow repair actions to guide automated repair techniques for buffer overflow. The experiment data is available at http://bo-study.github.io/Buffer-Overflow-Cases/.

Malware clustering and classification are important tools that enable analysts to prioritize their malware analysis efforts. The recent emergence of fully automated methods for malware clustering and classification that report high accuracy suggests that this problem may largely be solved. In this paper, we report the results of our attempt to confirm our conjecture that the method of selecting ground-truth data in prior evaluations biases their results toward high accuracy. To examine this conjecture, we apply clustering algorithms from a different domain (plagiarism detection), first to the dataset used in a prior work’s evaluation and then to a wholly new malware dataset, to see if clustering algorithms developed without attention to subtleties of malware obfuscation are nevertheless successful. While these studies provide conflicting signals as to the correctness of our conjecture, our investigation of possible reasons uncovers, we believe, a cautionary note regarding the significance of highly accurate clustering results, as can be impacted by testing on a dataset with a biased cluster-size distribution.

Regression testing is an expensive but necessary maintenance activity performed on modified software to provide confidence that changes are correct and do not adversely affect other portions of the softwore. A regression test selection technique choses, from an existing test set, thests that are deemed necessary to validate modified software. We present a new technique for regression test selection. Our algorithms construct control flow graphs for a precedure or program and its modified version and use these graphs to select tests that execute changed code from the original test suite. We prove that, under certain conditions, the set of tests our technique selects includes every test from the original test suite that con expose faults in the modified procedfdure or program. Under these conditions our algorithms are safe. Moreover, although our algorithms may select some tests that cannot expose faults, they are at lease as precise as other safe regression test selection algorithms. Unlike many other regression test selection algorithms, our algorithms handle all language constructs and all types of program modifications. We have implemented our algorithms; initial empirical studies indicate that our technique can significantly reduce the cost of regression testing modified software.

Software visualization studies techniques and methods for graphically representing different aspects of software. Its main goal is to enhance, simplify and clarify the mental representation a software engineer has of a computer system. During many years, visualization in 2D space has been actively studied, but in the last decade, researchers have begun to explore new 3D representations for visualizing software. In this article, we present an overview of current research in the area, describing several major aspects like: visual representations, interaction issues, evaluation methods and development tools. We also perform a survey of some representative tools to support different tasks, i.e., software maintenance and comprehension, requirements validation and algorithm animation for educational purposes, among others. Finally, we conclude identifying future research directions.

The increased availability of data created as part of the sof tware development process allows us to apply novel analysis techniques on the data and use the results to guide t he process’s optimization. In this paper we describe variou s data sources and discuss the principles and techniques of data mi ning as applied on software engineering data. Data that can b e mined is generated by most parts of the development process: requi rements elicitation, development analysis, testing, debu gging, and maintenance. Based on this classification we survey the mini ng approaches that have been used and categorize them accord ing to the corresponding parts of the development process and th e task they assist. Thus the survey provides researchers wit h a concise overview of data mining techniques applied to softw are engineering data, and aids practitioners on the selecti on of appropriate data mining techniques for their work.

Programming community-based question-answering (PCQA) websites such as Stack Overflow enable programmers to find working solutions to their questions. Despite detailed posting guidelines, duplicate questions that have been answered are frequently created. To tackle this problem, Stack Overflow provides a mechanism for reputable users to manually mark duplicate questions. This is a laborious effort, and leads to many duplicate questions remain undetected. Existing duplicate detection methodologies from traditional community based question-answering (CQA) websites are difficult to be adopted directly to PCQA, as PCQA posts often contain source code which is linguistically very different from natural languages. In this paper, we propose a methodology designed for the PCQA domain to detect duplicate questions. We model the detection as a classification problem over question pairs. To extract features for question pairs, our methodology leverages continuous word vectors from the deep learning literature, topic model features and phrases pairs that co-occur frequently in duplicate questions mined using machine translation systems. These features capture semantic similarities between questions and produce a strong performance for duplicate detection. Experiments on a range of real-world datasets demonstrate that our method works very well; in some cases over 30% improvement compared to state-of-the-art benchmarks. As a product of one of the proposed features, the association score feature, we have mined a set of associated phrases from duplicate questions on Stack Overflow and open the dataset to the public.


Question: A large-scale automated software diversity technique—revisiting program evolution.

This paper introduces an innovative approach to bolster operating system (OS) security through the implementation of efficient and fine-grained address space randomization (ASR). Given the recent surge in kernel-level vulnerabilities and the lack of robust mechanisms to safeguard the OS from targeted attacks, there has been a growing demand for comprehensive defensive strategies. Our proposed ASR design is tailored specifically for the OS environment, offering an effective shield against both traditional and emerging threats, including return-oriented programming attacks. To contextualize our solution, we examine the disparities between application-level ASR and OS-level ASR, highlighting that certain assumptions underlying application-level ASR no longer hold in the OS domain. Notably, the prevention of information leakage assumes paramount importance in this new setting. We demonstrate that our ASR strategy surpasses existing state-of-the-art solutions in terms of performance and security, without disrupting the software distribution model. Moreover, we unveil a pioneering live rerandomization strategy, which we found to be crucial for enhancing OS security. Our empirical evaluation reveals that our techniques introduce minimal runtime performance overhead (less than 5% on average across SPEC and syscall-intensive benchmarks) and a negligible increase in runtime memory footprint (approximately 15% during benchmark execution). We are confident that our techniques can significantly elevate OS security standards without compromising performance and reliability.

Title of dissertation: DEEP ANALYSIS OF BINARY CODE TO RECOVER PROGRAM STRUCTURE Khaled ElWazeer, Doctor of Philosophy, 2014 Dissertation directed by: Professor Rajeev Barua Department of Electrical and Computer Engineering Reverse engineering binary executable code is gaining more i nterest in the research community. Agencies as diverse as anti-virus companies, se curity consultants, code forensics consultants, law-enforcement agencies and nati onal security agencies routinely try to understand binary code. Engineers also often need to d ebug, optimize or instrument binary code during the software development process. In this dissertation, we present novel techniques to extend he capabilities of existing binary analysis and rewriting tools to be more scalab le, handling a larger set of stripped binaries with better and more understandable outp uts as well as ensuring correct recovered intermediate representation (IR) from binar ies such that any modified or rewritten binaries compiled from this representation work correctly. In the first part of the dissertation, we present techniques t o recover accurate function boundaries from stripped executables. Our techniques as opposed to current techniques ensure complete live executable code coverage, high quality recovered code, and functional behavior for most application binaries. We use s tatic and dynamic based techniques to remove as much spurious code as possible in a safe ma nner that does not hurt code coverage or IR correctness. Next, we present static tec hniques to recover correct prototypes for the recovered functions. The recovered prot otypes include the complete set of all arguments and returns. Our techniques ensure corr ct behavior of rewritten binaries for both internal and external functions. Finally, we present scalable and precise techniques to reco ver l cal variables for every function obtained as well as global and heap variables . Different techniques are represented for floating point stack allocated variables an d memory allocated variables. Data type recovery techniques are presented to declare mean ingful data types for the detected variables. Our data type recovery techniques can rec over integer, pointer, structural and recursive data types. We discuss the correctness of the r ecovered representation. The evaluation of all the methods proposed is conducted on Se condWrite, a binary rewriting framework developed by our research group. An imp ortant metric in the evaluation is to be able to recompile the IR with the recovered info rmation and run it producing the same answer that is produced when running the original ex ecutable. Another metric is the analysis time. Some other metrics are proposed to meas ure the quality of the IR with respect to the IR with source code information availabl e. DEEP ANALYSIS OF BINARY CODE TO RECOVER PROGRAM STRUCTURE

Software vulnerabilities represent a major cause of cybersecurity problems. The National Vulnerability Database (NVD) is a public data source that maintains standardized information about reported software vulnerabilities. Since its inception in 1997, NVD has published information about more than 43,000 software vulnerabilities affecting more than 17,000 software applications. This information is potentially valuable in understanding trends and patterns in software vulnerabilities, so that one can better manage the security of computer systems that are pestered by the ubiquitous software security flaws. In particular, one would like to be able to predict the likelihood that a piece of software contains a yet-to-be-discovered vulnerability, which must be taken into account in security management due to the increasing trend in zero-day attacks. We conducted an empirical study on applying data-mining techniques on NVD data with the objective of predicting the time to next vulnerability for a given software application. We experimented with various features constructed using the information available in NVD, and applied various machine learning algorithms to examine the predictive power of the data. Our results show that the data in NVD generally have poor prediction capability, with the exception of a few vendors and software applications. By doing a large number of experiments and observing the data, we suggest several reasons for why the NVD data have not produced a reasonable prediction model for time to next vulnerability with our current approach.

During maintenance developers cannot read the entire code of large systems. They need a way to get a quick understanding of source code entities (such as, classes, methods, packages, etc.), so they can efficiently identify and then focus on the ones related to their task at hand. Sometimes reading just a method header or a class name does not tell enough about its purpose and meaning, while reading the entire implementation takes too long. We study a solution which mitigates the two approaches, i.e., short and accurate textual descriptions that illustrate the software entities without having to read the details of the implementation. We create such descriptions using techniques from automatic text summarization. The paper presents a study that investigates the suitability of various such techniques for generating source code summaries. The results indicate that a combination of text summarization techniques is most appropriate for source code summarization and that developers generally agree with the summaries produced.

Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been proposed for many years, few empirical studies have been conducted to examine their actual usefulness in predicting fault-proneness. Objective. We aim to provide an in-depth understanding of the ability of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction, i.e. their effectiveness in helping practitioners find post-release faults when taking into account the effort needed to test or inspect the code. Method. We use the most commonly used code and process metrics, including size, structural complexity, Halstead's software science, and code churn metrics, as the baseline metrics. First, we employ principal component analysis to analyze the relationships between slice-based cohesion metrics and the baseline metrics. Then, we use univariate prediction models to investigate the correlations between slice-based cohesion metrics and post-release fault-proneness. Finally, we build multivariate prediction models to examine the effectiveness of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction when used alone or used together with the baseline code and process metrics. Results. Based on open-source software systems, our results show that: 1) slice-based cohesion metrics are not redundant with respect to the baseline code and process metrics; 2) most slice-based cohesion metrics are significantly negatively related to post-release fault-proneness; 3) slice-based cohesion metrics in general do not outperform the baseline metrics when predicting post-release fault-proneness; and 4) when used with the baseline metrics together, however, slice-based cohesion metrics can produce a statistically significant and practically important improvement of the effectiveness in effort-aware post-release fault-proneness prediction. Conclusion. Slice-based cohesion metrics are complementary to the most commonly used code and process metrics and are of practical value in the context of effort-aware post-release fault-proneness prediction.

Developers often run tests to check that their latest changes to a code repository did not break any previously working functionality. Ideally, any new test failures would indicate regressions caused by the latest changes. However, some test failures may not be due to the latest changes but due to non-determinism in the tests, popularly called flaky tests. The typical way to detect flaky tests is to rerun failing tests repeatedly. Unfortunately, rerunning failing tests can be costly and can slow down the development cycle.
 We present the first extensive evaluation of rerunning failing tests and propose a new technique, called DeFlaker, that detects if a test failure is due to a flaky test without rerunning and with very low runtime overhead. DeFlaker monitors the coverage of latest code changes and marks as flaky any newly failing test that did not execute any of the changes. We deployed DeFlaker live, in the build process of 96 Java projects on TravisCI, and found 87 previously unknown flaky tests in 10 of these projects. We also ran experiments on project histories, where DeFlaker detected 1, 874 flaky tests from 4, 846 failures, with a low false alarm rate (1.5%). DeFlaker had a higher recall (95.5% vs. 23%) of confirmed flaky tests than Maven's default flaky test detector.

Support for generic programming was added to the Java language in 2004, representing perhaps the most significant change to one of the most widely used programming languages today. Researchers and language designers anticipated this addition would relieve many long-standing problems plaguing developers, but surprisingly, no one has yet measured how generics have been adopted and used in practice. In this paper, we report on the first empirical investigation into how Java generics have been integrated into open source software by automatically mining the history of 40 popular open source Java programs, traversing more than 650 million lines of code in the process. We evaluate five hypotheses and research questions about how Java developers use generics. For example, our results suggest that generics sometimes reduce the number of type casts and that generics are usually adopted by a single champion in a project, rather than all committers. We also offer insights into why some features may be adopted sooner and others features may be held back.

Bugs due to data races in multithreaded programs often exhibit non-deterministic symptoms and are notoriously difficult to find. This paper describes RaceTrack, a dynamic race detection tool that tracks the actions of a program and reports a warning whenever a suspicious pattern of activity has been observed. RaceTrack uses a novel hybrid detection algorithm and employs an adaptive approach that automatically directs more effort to areas that are more suspicious, thus providing more accurate warnings for much less over-head. A post-processing step correlates warnings and ranks code segments based on how strongly they are implicated in potential data races. We implemented RaceTrack inside the virtual machine of Microsoft's Common Language Runtime (product version v1.1.4322) and monitored several major, real-world applications directly out-of-the-box,without any modification. Adaptive tracking resulted in a slowdown ratio of about 3x on memory-intensive programs and typically much less than 2x on other programs,and a memory ratio of typically less than 1.2x. Several serious data race bugs were revealed, some previously unknown.

Frameworks and libraries offer reusable and customizable functionality through Application Programming Interfaces (APIs). Correctly using large and sophisticated APIs can represent a challenge due to hidden assumptions and requirements. Numerous approaches have been developed to infer properties of APIs, intended to guide their use by developers. With each approach come new definitions of API properties, new techniques for inferring these properties, and new ways to assess their correctness and usefulness. This paper provides a comprehensive survey of over a decade of research on automated property inference for APIs. Our survey provides a synthesis of this complex technical field along different dimensions of analysis: properties inferred, mining techniques, and empirical results. In particular, we derive a classification and organization of over 60 techniques into five different categories based on the type of API property inferred: unordered usage patterns, sequential usage patterns, behavioral specifications, migration mappings, and general information.

We present a tool, called sif, for finding all similar files in a large file system. Files are considered similar if they have significant number of common pieces, even if they are very different otherwise. For example, one file may be contained, possibly with some changes, in another file, or a file may be a reorganization of another file. The running time for finding all groups of similar files, even for as little as 25% similarity, is on the order of 500MB to 1GB an hour. The amount of similarity and several other customized parameters can be determined by the user at a post-processing stage, which is very fast. Sif can also be used to very quickly identify all similar files to a query file using a preprocessed index. Application of sif can be found in file management, information collecting (to remove duplicates), program reuse, file synchronization, data compression, and maybe even plagiarism detection.

The discovery of vulnerabilities in source code is a key for securing computer systems. While specific types of security flaws can be identified automatically, in the general case the process of finding vulnerabilities cannot be automated and vulnerabilities are mainly discovered by manual analysis. In this paper, we propose a method for assisting a security analyst during auditing of source code. Our method proceeds by extracting abstract syntax trees from the code and determining structural patterns in these trees, such that each function in the code can be described as a mixture of these patterns. This representation enables us to decompose a known vulnerability and extrapolate it to a code base, such that functions potentially suffering from the same flaw can be suggested to the analyst. We evaluate our method on the source code of four popular open-source projects: LibTIFF, FFmpeg, Pidgin and Asterisk. For three of these projects, we are able to identify zero-day vulnerabilities by inspecting only a small fraction of the code bases.


Question: A large-scale automated software diversity technique: revisiting program evolution.

This paper illustrates the successful exploitation of cache-timing vulnerabilities to recover the Advanced Encryption Standard (AES) key from known-plaintext timings obtained from a network server operating on a separate machine. The vulnerability highlighted in this attack is inherent to the AES design itself, rather than being an issue with the specific AES library employed by the server. Crafting AES software that is both high-speed and constant-time, thus immune to such timing attacks, is a significant challenge on the majority of prevalent general-purpose computing platforms. The paper delves into the intricacies of these challenges, elucidating the various obstacles that must be overcome to achieve secure and efficient AES implementations.

We present four datasets that are focused on the general roles of OSS peer review members. With data mined from both an integrated peer review system and code source repositories, our rich datasets comprise of peer review data that was automatically recorded. Using the Android project as a case study, we describe our extraction methodology, the datasets and their application used for three separate studies. Our datasets are available online at http://sdlab.naist.jp/reviewmining/.

The threat that attacks cyberspace is known as malware. In order to infect the technologic devices that are attacked, malware needs to evade the different antivirus systems. To avoid detection, an obfuscation technique must be applied so malware is updated and ready to be performed. No obstant, the technique implementation presents difficulties in terms of its required ability, evasion tests and infection functionality that turn outs to be a problem to keep malware updated. Therefore, a procedure is proposed that allows applying AVFUCKER or DSPLIT techniques. The purpose is to optimize the required technical means, reduce the antivirus analysis and malware functionality check times.

This paper contains a summary of the talk given by the author on the occasion of the MODELS 2013 most influential paper award. The talk discussed the original paper as published in 2003, the research work done by others afterwards and the author’s personal reflection on the award. 1 Version Control of Software and System Models There are two main usage scenarios for design models in software and system development: models as sketches, that serve as a communication aid in informal discussions, and models as formal artifacts, to be analyzed, transformed into other artifacts, maintained and evolved during the whole software and system development process. In this second scenario, models are valuable assets that should be kept in a trusted repository. In a complex development project, these models will be updated often and concurrently by different developers. Therefore, there is a need for a version control system for models with optimistic locking. This is a system to compare, merge and store all versions of all models created within a development project. We can illustrate the use of a version control system for models as follows. Let us assume that the original model shown at the top of Figure 1 is edited simultaneously by two developers. One developer has decided that the subclass B is no longer necessary in the model. Simultaneously, the other developer has decided that class C should have a subclass D. The problem is to combine the contributions of both developers into a single model. This is the model shown at the bottom of Fig. 1. We presented the basic algorithms to solve this problem in the original paper published in the proceedings of the UML 2003 conference [1]. The proposed solution is based on calculating the final model as the merge of the differences between the original and the edited models. Figure 2 shows an example of the difference of two models, in this case the difference between the models edited by the developers and the original model. The result of the difference is not always a model, in a similar way that the difference between two natural numbers is not a natural number but a negative one. An example of this is shown in the bottom B C A Original Model

Software engineering has been striving for years to improve the practice of software development and maintenance. Documentation has long been prominent on the list of recommended practices to improve development and help maintenance. Recently however, agile methods started to shake this view, arguing that the goal of the game is to produce software and that documentation is only useful as long as it helps to reach this goal.On the other hand, in the re-engineering field, people wish they could re-document useful legacy software so that they may continue maintain them or migrate them to new platform.In these two case, a crucial question arises: "How much documentation is enough?" In this article, we present the results of a survey of software maintainers to try to establish what documentation artifacts are the most useful to them.

As a traditional dynamic language, Python is increasingly used in various software engineering tasks. However, due to its flexibility and dynamism, Python is a particularly challenging language to write code in and maintain. Consequently, Python programs contain code smells which indicate potential comprehension and maintenance problems. With the aim of supporting refactoring strategies to enhance maintainability, this paper describes how to detect code smells in Python programs. We introduce 11 Python smells and describe the detection strategy. We also implement a smell detection tool named Pysmell and use it to identify code smells in five real world Python systems. The results show that Pysmell can detect 285 code smell instances in total with the average precision of 97.7%. It reveals that Large Class and Large Method are most prevalent. Our experiment also implies Python programs may be suffering code smells further.

Malware is code designed for a malicious purpose, such as obtaining root privilege on a host. A malware detector identifies malware and thus prevents it from adversely affecting a host. In order to evade detection, malware writers use various obfuscation techniques to transform their malware. There is strong evidence that commercial malware detectors are susceptible to these evasion tactics. In this paper, we describe the design and implementation of a malware transformer that reverses the obfuscations performed by a malware writer. Our experimental evaluation demonstrates that this malware transformer can drastically improve the detection rates of commercial malware detectors.

Malicious code (or Malware) is defined as software that fulfills the deliberately harmful intent of an attacker. Malware analysis is the process of determining the behavior and purpose of a given Malware sample (such as a virus, worm, or Trojan horse). This process is a necessary step to be able to develop effective detection techniques and removal tools. Currently, Malware analysis is mostly a manual process that is tedious and time-intensive. To mitigate this problem, a number of analysis tools have been proposed that automatically extract the behavior of an unknown program by executing it in a restricted environment and recording the operating system calls that are invoked. The problem of dynamic analysis tools is that only a single program execution is observed. Unfortunately, however, it is possible that certain malicious actions are only triggered under specific circumstances (e.g., on a particular day, when a certain file is present, or when a certain command is received). In this paper, we propose a system that allows us to explore multiple execution paths and identify malicious actions that are executed only when certain conditions are met. This enables us to automatically extract a more complete view of the program under analysis and identify under which circumstances suspicious actions are carried out. Our experimental results demonstrate that many Malware samples show different behavior depending on input read from the environment. Thus, by exploring multiple execution paths, we can obtain a more complete picture of their actions.

In this article we summarize the strengths and weaknesses of empirical research in software engineering. We argue that in order to improve the current situation we must create better studies and draw more credible interpretations from them. We finally present a roadmap for this improvement, which includes a general structure for software empirical studies and concrete steps for achieving these goals: designing better studies, collecting data more effectively, and involving others in our empirical enterprises.

Information retrieval (IR) approaches have proven useful in recovering traceability links between free-text documentation and source code. IR-based traceability recovery approaches produce ranked lists of traceability links between pieces of documentation and source code. These traceability links are then pruned using various strategies and, finally, validated by human experts. In this paper we propose two contributions to improve the precision and recall of traceability links and, thus, reduces the required human experts' manual validation effort. First, we propose a novel approach, Trust race, inspired by Web trust models to improve the precision and recall of traceability links: Trust race uses intractability recovery approach to obtain a set of traceability links, which rankings are then re-evaluated using a set of other traceability recovery approaches. Second, we propose a novel traceability recovery approach, His trace, to identify traceability links between requirements and source code through CVS/SVN change logs using a Vector Space Model (VSM). We combine a traditional recovery traceability approach with His trace to build Trust race in which we use Histraceas one expert adding knowledge to the traceability links extracttedfrom CVS/SVN  change logs. We apply Trustrace on two case studies to compare its traceability links with those recovered using only the VSM-based approach, in terms of precision and recall. We show that Trustrace improves with statistical significance the precision of the traceability links while also improving recall but without statistical significance.

After a program has crashed and terminated abnormally, it typically leaves behind a snapshot of its crashing state in the form of a core dump. While a core dump carries a large amount of information, which has long been used for software debugging, it barely serves as informative debugging aids in locating software faults, particularly memory corruption vulnerabilities. A memory corruption vulnerability is a special type of software faults that an attacker can exploit to manipulate the content at a certain memory. As such, a core dump may contain a certain amount of corrupted data, which increases the difficulty in identifying useful debugging information (e.g. , a crash point and stack traces). Without a proper mechanism to deal with this problem, a core dump can be practically useless for software failure diagnosis. In this work, we develop CREDAL, an automatic tool that employs the source code of a crashing program to enhance core dump analysis and turns a core dump to an informative aid in tracking down memory corruption vulnerabilities. Specifically, CREDAL systematically analyzes a core dump potentially corrupted and identifies the crash point and stack frames. For a core dump carrying corrupted data, it goes beyond the crash point and stack trace. In particular, CREDAL further pinpoints the variables holding corrupted data using the source code of the crashing program along with the stack frames. To assist software developers (or security analysts) in tracking down a memory corruption vulnerability, CREDAL also performs analysis and highlights the code fragments corresponding to data corruption.
 To demonstrate the utility of CREDAL, we use it to analyze 80 crashes corresponding to 73 memory corruption vulnerabilities archived in Offensive Security Exploit Database. We show that, CREDAL can accurately pinpoint the crash point and (fully or partially) restore a stack trace even though a crashing program stack carries corrupted data. In addition, we demonstrate CREDAL can potentially reduce the manual effort of finding the code fragment that is likely to contain memory corruption vulnerabilities.