Question: Innovativeness in Tertiary Education: Employing Creative Thinking in Analyzing Textbook Chapters.

This paper explores an associative framework to understand the creative process, aiming to elucidate the fundamental mechanisms that underpin creativity across diverse domains, such as art and science. The discussion unfolds as follows: (a) Initially, we articulate a definition of creative thinking grounded in associative terms, delineating three pathways through which creative solutions can emerge—serendipity, similarity, and mediation. (b) This definition paves the way for identifying individual difference variables that could enhance creative performance, enabling a more nuanced understanding of the factors that contribute to creativity. (c) Stemming from our conceptualization of the creative process, we propose an operational definition that can be translated into a testable framework. This test, along with preliminary research findings, will be outlined to provide empirical insights into the nature of creativity. (d) The paper culminates in a speculative discussion on how certain experimentally manipulable variables might influence the creative process. The study of creative individuals and the processes they employ to manifest their creativity continues to captivate researchers and practitioners alike, offering rich avenues for further exploration.

This case study examines the physical aspects of a particular university classroom, and what affect specific changes to the classroom had on the perceptions of students, instructors and observers regarding the room as an effective learning space. We compare survey and focus group data collected from students taking courses in the classroom prior to changes to the physical environment with comparable data from students taking courses in the same classroom after specific changes had been made. Immediately following changes to the classroom, notable increases were observed in reported perceptions of student satisfaction with the physical environment, including perceptions of the classroom as a more effective and engaging learning space. Similar perceptions of improvement as a teaching-learning space were reported by instructors and observers. However, subsequent follow-up data collection and analyses suggested little if any sustained increase in perceptions of efficacy of the room as a learning space; indeed, most reported variables returned to baseline levels. The implications of these findings and their relevance to classroom design nevertheless may provide insight regarding the manner in which physical space might support or even enhance teaching and learning. Keywords: learning spaces, active learning, classrooms, teaching and learning environment, classroom design For a number of years there has been an on-going pedagogical shift in higher education away from a traditional content delivery model of instruction to more active models of learning in which students play more involved and interactive roles within the classroom (Cornell, 2002; Brown, 2006). This movement has been coupled with the recognition that the traditional university classroom, with its unidirectional design and tiered, fixed theatre-like seating, is insufficient to accommodate what have increasingly become more varied teaching and learning practices. This growing realization that as the nature of teaching and learning evolves so too must teaching and learning spaces has, in recent years, resulted in a heightened interest among scholars in the examination of classroom space and, more specifically, the inquiry into the connection between classroom design and pedagogy and learning (Brooks, 2011). This study presents findings from an on-going research project examining classroom spaces on a small university campus. The present case study, which is one component of the larger project, compares perceptions of a specific classroom as an effective learning space before and after relatively extensive changes were made to the physical environment of the room. These changes were guided by feedback from students and instructors as well as by existing literature on innovative classroom designs. In particular, we were interested in assessing if these changes 1
  University
  of
  Lethbridge,
  Lethbridge,
  Alberta,
  Canada

Purpose The purpose of this research is to understand the mechanisms of knowledge management (KM) for innovation and provide an approach for enterprises to leverage KM activities into continuous innovation. Design/methodology/approach – By reviewing the literature from multidisciplinary fields, the concepts of knowledge, KM and innovation are investigated. The physical, human and technological perspectives of KM are distinguished with the identification of two core activities for innovation: knowledge creation and knowledge usage. Then an essential requirement for continuous innovation –an internalization phase is defined. The systems thinking and human-centred perspectives are adopted for providing a comprehensive understanding about the mechanisms of KM for innovation. Findings – A networking process of continuous innovation based on KM is proposed by incorporating the phase of internalization. According to the three perspectives of KM, three sources of organizational knowledge assets in innovation are identified. Then based on the two core activities of innovation, a meta-model and a macro process of KM are proposed to model the mechanisms of KM for continuous innovation. Then, in order to operationalize the KM mechanisms, a hierarchical model is constructed by integrating three sources of knowledge assets, the meta-model and the macro process into the process of continuous innovation. This model decomposes the complex relationships between knowledge and innovation into four layers. Practical implications – According to the lessons learned about KM practices in previous research, the three perspectives of KM should collaborate with each other for successful implementation of KM projects for innovation; and the hierarchical model provides a suitable architecture to implement systems of KM for innovation. Originality/value – The meta-model and macro process of KM explain how the next generation of KM can help the value creation and support the continuous innovation from the systems thinking perspective. The hierarchical model illustrates the complicate knowledge dynamics in the process of continuous innovation.

The potential of Serious Games (SGs) in education is widely recognized, and their adoption is significant in particular in children instruction. However, the deployment rate of SGs in higher education (HE) and their proper insertion in meaningful curricula is still quite low. This paper intends to make a first step in the direction of a better characterization of the pedagogical effectiveness of SGs in HE, by providing a qualitative analysis based on our field experience using three games for entrepreneurship, that we have studied in the light of two well established pedagogical paradigms, such as the Revised Bloom’s taxonomy and the Kolb’s Learning stages. In general, we observe that SGs address several goals of the Bloom’s taxonomy, in particular at the lower levels. Moreover, the cyclical nature of the business simulations can be directly mapped to the sequential steps described by Kolb. However, our analysis also shows that SGs have still to significantly evolve in order to become an effective and efficient tool that could be successfully and reliably used in HE. In the light of our experience, we also propose a schema for a proper integration of SGs by supporting different goals in different steps of a formal education process, Our study finally suggests directions for future research in the field.

Educational technology research has moved through several stages or “ages”, focusing at the beginning on the content to be learned, then on the format of instructional messages and, finally on the interaction between computers and students. The present paper reviews the research in technology-based learning environments in order to give both a historical perspective on educational technology research and a view of the current state of this discipline. We conclude that: 1) trends in educational technology research were forged by the evolution of learning theories and the technological changes, 2) a clear shift from the design of instruction to the design of learning environments can be noticed; 3) there is a positive effect of educational technology on learning, but the size of the effect varies considerably; 4) learning is much more dependent on the activity of the learner than on the quantity of information and processing opportunities provided by the environment.

This chapter reviews basic issues of theory and method in approaches to research on teaching that are alternatively called ethnographic, qualitative, participant observational, case study, symbolic interactionist, phenomenological, constructivist, or interpretive. These approaches are all slightly different, but each bears strong family resemblance to the others. The set of related approaches is relatively new in the field of research on teaching. The approaches have emerged as significant in the decade of the 1960s in England and in the 1970s in the United States, Australia, New Zealand, and Germany. Because interest in these approaches is so recent, the previous editions of the Handbook of Research on Teaching do not contain a chapter devoted to participant observational research. Accordingly, this chapter attempts to describe r~s and their theoretical _nresup~s in considerable detail and does not attempt an exhaustive review of the rapidly growing literature in the field. Such a review will be appropriate for the next edition of this handbook. From this point on I will use the term interpretive to refer to the whole family of approaches to participant observational research. I adopt this term for three reasons: (a) It is more inclusive than many of the others (e.g., ethnography, case study); (b) W Blake

Whereas it is well documented that student question asking is infrequent in classroom environments, there is little research on questioning processes during tutoring. The present study investigated the questions asked in tutoring sessions on research methods (college students) and algebra (7th graders). Student questions were approximately 240 times as frequent in tutoring settings as classroom settings, whereas tutor questions were only slightly more frequent than teacher questions. Questions were classified by (a) degree of specification , (b) content, and (c) question-generation mechanism to analyze their quality. Student achievement was positively correlated with the quality of student questions after students had some experience with tutoring, but the frequency of questions was not correlated with achievement. Students partially self-regulated their learning by identifying knowledge deficits and asking questions to repair them, but they need training to improve these skills. We identified some ways that tutors and teachers might improve their question-asking skills. specializations are discourse processes and questioning mechanisms.

As the frequency with which Classroom Response Systems (CRSs) are used is increasing, it becomes more and more important to define the affordances and limitations of these tools. Currently existing literature is largely either anecdotal or focuses on comparing CRS and non-CRS environments that are unequal in other aspects as well. In addition, the literature primarily describes situations in which the CRS is used to provide an individual as opposed to a group response. This article points to the need for a concerted research effort, one that rigorously explores conditions of use across diverse settings and pedagogies.

This paper proposes quiz-style information presentation for interactive systems as a means to improve user understanding in educational tasks. Since the nature of quizzes can highly motivate users to stay voluntarily engaged in the interaction and keep their attention on receiving information, it is expected that information presented as quizzes can be better understood by users. To verify the effectiveness of the approach, we implemented read-out and quiz systems and performed comparison experiments using human subjects. In the task of memorizing biographical facts, the results showed that user understanding for the quiz system was significantly better than that for the read-out system, and that the subjects were more willing to use the quiz system despite the long duration of the quizzes. This indicates that quiz-style information presentation promotes engagement in the interaction with the system, leading to the improved user understanding.

The fields of learning theory and instructional design are in the midst of a scientific revolution in which their objectivist philosophical foundations are being replaced by a constructivist epistemology. This article describes the assumptions of a constructivist epistemology, contrasts them with objectivist assumptions, and then describes instructional systems that can support constructive learning at a distance Limitations of Distance Learning Technologies In an effort to supplement or replace live face-to-face instruction, technologicany mediated distance learning has more often than not merely replicated the ineffective methods that limit learning in face-to-face classrooms (Turoff 1995). Too often, potentially interactive technologies are used to present one-way lectures to students in remote locations. However, we believe that the most valuable activity in a classroom of any kind is the opportunity for students to work and interact together and to build and become part of a community of scholars and practitioners (Selfe and Eilola 1989; Bates 1990; Seaton 1993; Nalley 1995). A good learning experience is one in which a student can "master new knowledge and skills, criticany examine assumptions and beliefs, and engage in an invigorating, collaborative quest for wisdom and personal, holistic development" (Eastmond and Ziegahn 1995, 59). Technology used in distance education should facilitate these "good learning experiences" in an "extended classroom model" rather than broadcast teacher-centered lectures and demonstrations (Burge and Roberts 1993). A significant impediment to this goal is the fact that many teachers and instructional designers come to distance education from traditional backgrounds, bringing with them assumptions about teaching and learning that are not theory-based and do not translate well to technologicany mediated instruction (Schieman, Taere, and McLaren 1992).

Australian universities have traditionally relied on government funding to support undergraduate teaching. As the government has adopted the ‘user-pays’ principle, universities have been forced to look outside their traditional market to expand the undergraduate, post-graduate and international offerings. Alternate delivery methods in many universities have utilised web-based instruction as a basis for this move because of three perceptions: access by the target market is reasonably significant, it is a cost-effective method of delivery, and it provides global access. Since the mid sixties, the trend for both on-campus teaching and teaching at a distance has been to use behaviourist instructional strategies for subject development, which rely on the development of a set of instructional sequences with predetermined outcomes. These models, whilst applicable in a behaviourist environment, are not serving instructional designers well when the theoretical foundation for the subject outcomes is based on a constructivist approach to learning, since the constructivist group of theories places less emphasis on the sequence of instruction and more emphasis on the design of the learning environment. (Jonassen, 1994. p 35). In a web-based environment this proves to be even more challenging. This paper will review current research in design goals for web-based constructivist learning environments, and a move towards the development of models. The design of two web-based subjects will be explored in the context of the design goals developed by Duffy and Cunningham (1996 p 177) who have produced some basic assumptions that they call “metaphors we teach by”. The author seeks to examine the seven goals for their relevance to the instructional designer through the examination of their relevance to the web-based subjects, both of which were framed in constructivist theory.


Question: The relationship between Minimum Description Length induction, Bayesian inference, and Kolmogorov complexity.

This paper delves into the realm of computable numbers, defined as real numbers whose decimal expressions can be calculated using finite methods. While ostensibly focused on computable numbers, the exploration extends easily to computable functions of integral or real variables, as well as computable predicates. Despite the diverse applications, the underlying challenges remain consistent across these domains. To streamline the discussion, computable numbers have been selected for detailed examination due to their simpler analytical framework. A comprehensive overview of the interrelationships among computable numbers, functions, and predicates is forthcoming. This will encompass a theoretical framework for functions of a real variable, articulated through the lens of computable numbers. By my definition, a number attains computable status if its decimal representation can be generated by a mechanical process. This foundational concept paves the way for a deeper investigation into the Minimum Description Length principle, Bayesian methodologies, and the intricacies of Kolmogorov Complexity, elucidating the computational underpinnings that govern the succinctness and predictability of mathematical entities.

Our inspiration comes from Nell (Never Ending Language Learning), a computer program running at Carnegie Mellon University to extract structured information from unstructured web pages. We consider the problem of semi-supervised learning approach to extract category instances (e.g. country(USA), city(New York)) from web pages, starting with a handful of labeled training examples of each category or relation, plus hundreds of millions of unlabeled web documents. Semisupervised approaches using a small number of labeled examples together with many unlabeled examples are often unreliable as they frequently produce an internally consistent, but nevertheless, incorrect set of extractions. We believe that this problem can be overcome by simultaneously learning independent classifiers in a new approach named Coupled Bayesian Sets algorithm, based on Bayesian Sets, for many different categories and relations (in the presence of an ontology defining constraints that couple the training of these classifiers). Experimental results show that simultaneously learning a coupled collection of classifiers for random 11 categories resulted in much more accurate extractions than training classifiers through original Bayesian Sets algorithm, Naive Bayes, BaS-all and Coupled Pattern Learner (the category extractor used in NELL).

It is argued that underlying the Church-Turing hypothesis there is an implicit physical assertion. Here, this assertion is presented explicitly as a physical principle: ‘every finitely realizable physical system can be perfectly simulated by a universal model computing machine operating by finite means’. Classical physics and the universal Turing machine, because the former is continuous and the latter discrete, do not obey the principle, at least in the strong form above. A class of model computing machines that is the quantum generalization of the class of Turing machines is described, and it is shown that quantum theory and the ‘universal quantum computer’ are compatible with the principle. Computing machines resembling the universal quantum computer could, in principle, be built and would have many remarkable properties not reproducible by any Turing machine. These do not include the computation of non-recursive functions, but they do include ‘quantum parallelism’, a method by which certain probabilistic tasks can be performed faster by a universal quantum computer than by any classical restriction of it. The intuitive explanation of these properties places an intolerable strain on all interpretations of quantum theory other than Everett’s. Some of the numerous connections between the quantum theory of computation and the rest of physics are explored. Quantum complexity theory allows a physically more reasonable definition of the ‘complexity’ or ‘knowledge’ in a physical system than does classical complexity theory. Current address: Centre for Quantum Computation, Clarendon Laboratory, Department of Physics, Parks Road, OX1 3PU Oxford, United Kingdom. Email: david.deutsch@qubit.org This version (Summer 1999) was edited and converted to L TEX by Wim van Dam at the Centre for Quantum Computation. Email:wimvdam@qubit.org 1 Computing machines and the Church-Turing principle The theory of computing machines has been extensively developed during the last few decades. Intuitively, a computing machine is any physical system whose dynamical evolution takes it from one of a set of ‘input’ states to one of a set of ‘output’ states. The states are labelled in some canonical way, the machine is prepared in a state with a given input label and then, following some motion, the output state is measured. For a classical deterministic system the measured output label is a definite functionf of the prepared input label; moreover the value of that label can in principle be measured by an outside observer (the ‘user’) and the machine is said to ‘compute’the functionf . Two classical deterministic computing machines are ‘computationally equivalent’ under given labellings of their input and output states if they compute the same function under those labellings. But quantum computing machines, and indeed classical stochastic computing machines, do not ‘compute functions’ in the above sense: the output state of a stochastic machine is random with only the probability distribution function for the possible outputs depending on the input state. The output state of a quantum machine, although fully determined by the input state is not an observable and so the user cannot in general discover its label. Nevertheless, the notion of computational equivalence can be generalized to apply to such machines also. Again we define computational equivalence under given labellings,but it is now necessary to specify more precisely what is to be labelled. As far as the input is concerned, labels must be given for each of the possible ways of preparing the machine, which correspond, by definition, to all the possible input states. This is identical with the classical deterministic case. However, there is an asymmetry between input and output because there is an asymmetry between preparation and measurement: whereas a quantum system can be prepared in any desired permitted input state, measurement cannot in general determine its output state; instead one must measure the value of some observable. (Throughout this paper I shall be using the Schr ̈ odinger picture, in which the quantum state is a function of time but observables are constant operators.) Thus what must be labelled is the set of ordered pairs consisting of an output observable and a possible measured value of that observable (in quantum theory, a Hermitian operator and one of its eigenvalues). Such an ordered pair contains, in effect, the specification of a possible experiment that could be made on the output, together with a possible result of that experiment. Two computing machines are computationally equivalent under given labellings if in any possible experiment or sequence of experiments in which their inputs were prepared equivalently under the input labellings, and observables corresponding to each other under the output labellings were measured, the measured values of these observables for the two machines would be statistically indistinguishable. That is, the probability distribution functions for the outputs of the two machines would be identical. In the sense just described, a given computing machine M computes at most one function. However, there ought to be no fundamental difference between altering the input state in which M is prepared, and altering systematically the constitution of M so that it becomes a different machine M0 computing a different function. To formalize such operations, it is often useful to consider machines with two inputs, the preparation of one constituting a ‘program’ determining which function of the other is to be computed. To each such machine M there corresponds a set C (M) of ‘M-computable functions’. A functionf isM-computable ifM can computef when prepared with some program. The set C(M) can be enlarged by enlarging the set of changes in the constitution of M that are labelled as possible M-programs. Given two machines M andM0 it is possible to construct a composite machine whose set of computable functions contains the union of C (M) and C(M0). There is no purely logical reason why one could not go on ad infinitumbuilding more powerful

The Minimum Bayes Risk (MBR) framework has been a successful strategy for the training of hidden Markov models for large vocabulary speech recognition. Practical implementations of MBR must select an appropriate hypothesis space and loss function. The set of word sequences and a word-based Levenshtein distance may be assumed to be the optimal choice but use of phoneme-based criteria appears to be more successful. This paper compares the use of different hypothesis spaces and loss functions defined using the system constituents of word, phone, physical triphone, physical state and physical mixture component. For practical reasons the competing hypotheses are constrained by sampling. The impact of the sampling technique on the performance of MBR training is also examined.

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through “cheap learning” with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine-learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various “no-flattening theorems” showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss; for example, we show that n variables cannot be multiplied using fewer than 2 neurons in a single hidden layer.

We say a transfer model is a mapping between the questions in an intelligent tutoring system and the knowledge components (i.e., skills, strategies, declarative knowledge, etc) needed to answer a question correctly. [JKT00] showed how you could take advantage of 1) the Power Law Of Learning, 2) an existing transfer model, and 3) set of tutorial log files, to learn a function (using logistic regression) that will predict when a student will get a question correct. In the main conference proceeding [CHK2004] give an example of using this technique for transfer model selection. Koedinger and Junker [KJ99] also conceptualized a search space where each state is a new transfer model. The operators in this search space split, add or merge knowledge components based upon factors that are tagged to questions. Koedinger and Junker called this method learning factors analysis emphasizing that this method can be used to study learning. Unfortunately, the search space is huge and searching for good fitting transfer models is exponential. The main goal of this paper is show a technique that will make searching for transfer models more efficient. Our procedure implements a search method using association rules as a means of guiding the search. The association rules are mined from a dataset derived from student-tutor interaction logs. The association rules found in the mining process determine what operations to perform on the current transfer model. We report on the speed up achieved. Being able to find good transfer models quicker will help intelligent tutor system builders as well as cognitive science researchers better assess what makes certain problems hard and other problems easy for students.

Probability theory is mathematically the best understood paradigm for modeling and manipulating uncertain information. Probabilities of complex events can be computed from those of basic events on which they depend, using any of a number of strategies. Which strategy is appropriate depends very much on the known interdependencies among the events involved. Previous work on probabilistic databases has assumed a fixed and restrictivecombination strategy (e.g., assuming all events are pairwise independent). In this article, we characterize, using postulates, whole classes of strategies for conjunction, disjunction, and negation, meaningful from the viewpoint of probability theory. (1) We propose a probabilistic relational data model and a genericprobabilistic relational algebra that neatly captures various strategiessatisfying the postulates, within a single unified framework. (2) We show that as long as the chosen strategies can be computed in polynomial time, queries in the positive fragment of the probabilistic relational algebra have essentially the same data complexity as classical relational algebra. (3) We establish various containments and equivalences between algebraic expressions, similar in spirit to those in classical algebra. (4) We develop algorithms for maintaining materialized probabilistic views. (5) Based on these ideas, we have developed a prototype probabilistic database system called ProbView on top of Dbase V.0. We validate our complexity results with experiments and show that rewriting certain types of queries to other equivalent forms often yields substantial savings.

We introduce a novel distributional clustering algorithm that explicitly maximizes the mutual information per cluster between the data and given categories. This algorithm can be considered as a bottom up hard version of the recently introduced “Information Bottleneck Method”. We relate the mutual information between clusters and categories to the Bayesian classification error, which provides another motivation for using the obtained clusters as features. The algorithm is compared with the top-down soft version of the information bottleneck method and a relationship between the hard and soft results is established. We demonstrate the algorithm on the 20 Newsgroups data set. For a subset of two news-groups we achieve compression by 3 orders of magnitudes loosing only 10% of the original mutual information.

A new algorithm, called Hamming Clustering (HC), is proposed to extract a set of rules underlying a given classification problem. It is able to reconstruct the and-or expression associated with any Boolean function from a training set of

We propose a new scoring function for learning Bayesian networks from data using score+search algorithms. This is based on the concept of mutual information and exploits some well-known properties of this measure in a novel way. Essentially, a statistical independence test based on the chi-square distribution, associated with the mutual information measure, together with a property of additive decomposition of this measure, are combined in order to measure the degree of interaction between each variable and its parent variables in the network. The result is a non-Bayesian scoring function called MIT (mutual information tests) which belongs to the family of scores based on information theory. The MIT score also represents a penalization of the Kullback-Leibler divergence between the joint probability distributions associated with a candidate network and with the available data set. Detailed results of a complete experimental evaluation of the proposed scoring function and its comparison with the well-known K2, BDeu and BIC/MDL scores are also presented.

Multiplicative weight-updating algorithms such as Winnow have been studied extensively in the COLT literature, but only recently have people started to use them in applications. In this paper, we apply a Winnow-based algorithm to a task in natural language: contextsensitive spelling correction. This is the task of xing spelling errors that happen to result in valid words, such as substituting to for too, casual for causal, and so on. Previous approaches to this problem have been statistics-based; we compare Winnow to one of the more successful such approaches, which uses Bayesian classi ers. We nd that: (1) When the standard (heavily-pruned) set of features is used to describe problem instances, Winnow performs comparably to the Bayesian method; (2) When the full (unpruned) set of features is used, Winnow is able to exploit the new features and convincingly outperform Bayes; and (3) When a test set is encountered that is dissimilar to the training set, Winnow is better than Bayes at adapting to the unfamiliar test set, using a strategy we will present for combining learning on the training set with unsupervised learning on the (noisy) test set. In Machine Learning: Proceedings of the 13th International Conference, Lorenza Saitta, ed., Morgan Kaufmann, San Francisco, CA, 1996, pages 182{190. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonpro t educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Information Technology Center America; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Information Technology Center America. All rights reserved. Copyright c Mitsubishi Electric Information Technology Center America, 1996 201 Broadway, Cambridge, Massachusetts 02139 Dept. of Appl. Math. & CS, Weizmann Institute of Science, Rehovot 76100, Israel. 1. First printing, TR96-07, April 1996


Question: Minimum description length inference, Bayesian methods, and Kolmogorov complexity.

This paper delves into a theory of program size that is formally equivalent to information theory, exploring the intricate connections between the two. The authors highlight the significance of this relationship, which has implications for the understanding of complexity in computational systems. The paper also mentions the distribution of the Proceedings of the Symposium on Theory of Computing to all members, an effort aimed at broadening the reach of the latest research findings. In acknowledgment of potential errors in the proceedings, the authors invite feedback from the community, particularly errata submissions, to ensure the accuracy and integrity of the published work. An erratum from Cook and Reckhow regarding their paper is included in this issue, reflecting the commitment to scholarly rigor. Additionally, the paper discusses the inclusion of a longer-than-usual article as a continuation of a previously published piece. This decision is justified by the article's contribution to an area that warrants further exploration and discussion. The editorial policy, which typically favors contributions under five pages, is briefly mentioned, emphasizing the journal's commitment to concise and impactful publications while recognizing the need for flexibility in exceptional cases.

Widespread use of medical information systems and explosive growth of medical databases require traditional manual data analysis to be coupled with methods for efficient computer-assisted analysis. This paper presents selected data mining techniques that can be applied in medicine, and in particular some machine learning techniques including the mechanisms that make them better suited for the analysis of medical databases (derivation of symbolic rules, use of background knowledge, sensitivity and specificity of induced descriptions). The importance of the interpretability of results of data analysis is discussed and illustrated on selected medical applications.

This paper describes a novel Bayesian approach to unsupervised topic segmentation. Unsupervised systems for this task are driven by lexical cohesion: the tendency of wellformed segments to induce a compact and consistent lexical distribution. We show that lexical cohesion can be placed in a Bayesian context by modeling the words in each topic segment as draws from a multinomial language model associated with the segment; maximizing the observation likelihood in such a model yields a lexically-cohesive segmentation. This contrasts with previous approaches, which relied on hand-crafted cohesion metrics. The Bayesian framework provides a principled way to incorporate additional features such as cue phrases, a powerful indicator of discourse structure that has not been previously used in unsupervised segmentation systems. Our model yields consistent improvements over an array of state-of-the-art systems on both text and speech datasets. We also show that both an entropy-based analysis and a well-known previous technique can be derived as special cases of the Bayesian framework.1

In this paper an algorithm based on the concepts of genetic algorithms that uses an estimation of a probability distribution of promising solutions in order to generate new candidate solutions is proposed To esti mate the distribution techniques for model ing multivariate data by Bayesian networks are used The proposed algorithm identi es reproduces and mixes building blocks up to a speci ed order It is independent of the ordering of the variables in the strings rep resenting the solutions Moreover prior in formation about the problem can be incor porated into the algorithm However prior information is not essential Preliminary ex periments show that the BOA outperforms the simple genetic algorithm even on decom posable functions with tight building blocks as a problem size grows

We describe a generalization of the PAC learning model that is based on statistical decision theory. In this model the learner receives randomly drawn examples, each example consisting of an instance x E X and an outcome J E Y, and tries to find a decision rule h: X + A, where h E X, that specifies the appropriate action a E A to take for each instance x in order to minimize the expectation of a loss I( y, a). Here X, Y, and A are arbitrary sets. I is a real-valued function, and examples are generated according to an arbitrary joint distribution on Xx Y. Special cases include the problem of learning a function from X into Y, the problem of learning the conditional probability distribution on Y given X (regression), and the problem of learning a distribution on X (density estimation). We give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain decision rule spaces 2, and show how this implies learnability with bounded sample size, disregarding computational complexity. As an application, we give distribution-independent upper bounds on the sample size needed for learning with feedforward neural networks, Our theorems use a generalized notion of VC dimension that applies to classes of real-valued functions, adapted from Vapnik and Pollard’s work, and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space. ‘(‘1 1992 Academic Press, Inc.

Recent studies of probabilistic reasoning have postulated general-purpose inference algorithms that can be used to answer arbitrary queries. These algorithms are memoryless, in the sense that each query is processed independently, without reuse of earlier computation. We argue that the brain operates in the setting of amortized inference, where numerous related queries must be answered (e.g., recognizing a scene from multiple viewpoints); in this setting, memoryless algorithms can be computationally wasteful. We propose a simple form of flexible reuse, according to which shared inferences are cached and composed together to answer new queries. We present experimental evidence that humans exploit this form of reuse: the answer to a complex query can be systematically predicted from a person’s response to a simpler query if the simpler query was presented first and entails a sub-inference (i.e., a sub-component of the more complex query). People are also faster at answering a complex query when it is preceded by a sub-inference. Our results suggest that the astonishing efficiency of human probabilistic reasoning may be supported by interactions between inference and memory.

A universal algorithm for sequential data compression is presented. Its performance is investigated with respect to a nonprobabilistic model of constrained sources. The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable-to-block codes designed to match a completely specified source.

enter the circuit several times, possibly via different input processors. More elaborate combinatorial considerations than for standard VLSI circuits are needed to show how communication complexity can be used to get lower bounds on multilective VLSI computations. Near the end of the book, Hromkovic presents relations between communication complexity and some complexity measures of sequential computations. The core section introduces a uniform model of one-way communication protocols and shows that the corresponding uniform one-way communication complexity is strongly related to the size of deterministic finite automata. Then he relates communication complexity to the time and space complexity of Turing machines. Finally, he shows how communication complexity can be used to obtain lower bounds on the size and the depth of decisions nees and branching programs.

Decision trees and ensembles of decision trees are very popular in machine learning and often achieve state-of-the-art performance on black-box prediction tasks. However, popular variants such as C4.5, CART, boosted trees and random forests lack a probabilistic interpretation since they usually just specify an algorithm for training a model. We take a probabilistic approach where we cast the decision tree structures and the parameters associated with the nodes of a decision tree as a probabilistic model; given labeled examples, we can train the probabilistic model using a variety of approaches (Bayesian learning, maximum likelihood, etc). The probabilistic approach allows us to encode prior assumptions about tree structures and share statistical strength between node parameters; furthermore, it offers a principled mechanism to obtain probabilistic predictions which is crucial for applications where uncertainty quantification is important. Existing work on Bayesian decision trees relies on Markov chain Monte Carlo which can be computationally slow and suffer from poor mixing. We propose a novel sequential Monte Carlo algorithm that computes a particle approximation to the posterior over trees in a top-down fashion. We also propose a novel sampler for Bayesian additive regression trees by combining the above top-down particle filtering algorithm with the Particle Gibbs (Andrieu et al., 2010) framework. Finally, we propose Mondrian forests (MFs), a computationally efficient hybrid solution that is competitive with non-probabilistic counterparts in terms of speed and accuracy, but additionally produces well-calibrated uncertainty estimates. MFs use the Mondrian process (Roy and Teh, 2009) as the randomization mechanism and hierarchically smooth the node parameters within each tree (using a hierarchical probabilistic model and approximate Bayesian updates), but combine the trees in a non-Bayesian fashion. MFs can be grown in an incremental/online fashion and remarkably, the distribution of online MFs is the same as that of batch MFs.

Variational inference methods, including mean field methods and loopy belief propagation, have been widely used for approximate probabilistic inference in graphical models. While often less accurate than MCMC, variational methods provide a fast deterministic approximation to marginal and conditional probabilities. Such approximations can be particularly useful in high dimensional problems where sampling methods are too slow to be effective. A limitation of current methods, however, is that they are restricted to parametric probabilistic models. MCMC does not have such a limitation; indeed, MCMC samplers have been developed for the Dirichlet process (DP), a nonparametric distribution on distributions (Ferguson, 1973) that is the cornerstone of Bayesian nonparametric statistics (Escobar & West, 1995; Neal, 2000). In this paper, we develop a mean-field variational approach to approximate inference for the Dirichlet process, where the approximate posterior is based on the truncated stick-breaking construction (Ishwaran & James, 2001). We compare our approach to DP samplers for Gaussian DP mixture models.

We present an implementation of a probabilistic first-order logic called TensorLog, in which classes of logical queries are compiled into differentiable functions in a neuralnetwork infrastructure such as Tensorflow or Theano. This leads to a close integration of probabilistic logical reasoning with deep-learning infrastructure: in particular, it enables high-performance deep learning frameworks to be used for tuning the parameters of a probabilistic logic. Experimental results show that TensorLog scales to problems involving hundreds of thousands of knowledge-base triples and tens of thousands of examples.


Question: The relationship between minimum description length induction, Bayesian inference, and Kolmogorov complexity.

This paper explores the concept of modeling through the shortest data description, which emerges as a powerful and comprehensive principle that can supersede traditional methods such as least squares, minimum prediction error, and maximum likelihood. These classical principles, while historically respected, have limitations in estimating structure parameters, particularly in determining the order of models. The suggestion here is that their success is often coincidental, aligning with aspects of a deeper, more encompassing principle. Drawing inspiration from Kolmogorov's work on the shortest description length of individual, recursively definable objects, the paper delves into the algorithmic entropy concept. However, in the realm of statistical estimation, where data inherently contains a stochastic component, the focus shifts to the entropy of random variables, a notion that is more fitting for this context. Additionally, the paper briefly touches upon a combinatorial entropy concept, which, despite its simplicity, does not cause any significant confusion or overlap with the main entropy discussion. The essence of this work lies in the recognition of the shortest data description as a fundamental principle that can guide the estimation of model parameters, especially in the presence of randomness, offering a more refined approach to statistical modeling and estimation.

We propose a novel mixtures of Gaussian processes model in which the gating function is interconnected with a probabilistic logical model, in our case Markov logic networks. In this way, the resulting mixed graphical model, called Markov logic mixtures of Gaussian processes (MLxGP), solves joint Bayesian non-parametric regression and probabilistic relational inference tasks. In turn, MLxGP facilitates novel, interesting tasks such as regression based on logical constraints or drawing probabilistic logical conclusions about regression data, thus putting “machines reading regression data” in reach.

Recent approaches to text classification have used two different first-order probabilistic models for classification, both of which make the naive Bayes assumption. Some use a multi-variate Bernoulli model, that is, a Bayesian Network with no dependencies between words and binary word features (e.g. Larkey and Croft 1996; Koller and Sahami 1997). Others use a multinomial model, that is, a uni-gram language model with integer word counts (e.g. Lewis and Gale 1994; Mitchell 1997). This paper aims to clarify the confusion by describing the differences and details of these two models, and by empirically comparing their classification performance on five text corpora. We find that the multi-variate Bernoulli performs well with small vocabulary sizes, but that the multinomial performs usually performs even better at larger vocabulary sizes—providing on average a 27% reduction in error over the multi-variate Bernoulli model at any vocabulary size.

A universal algorithm for sequential data compression is presented. Its performance is investigated with respect to a nonprobabilistic model of constrained sources. The compression ratio achieved by the proposed universal code uniformly approaches the lower bounds on the compression ratios attainable by block-to-variable codes and variable-to-block codes designed to match a completely specified source.

We present a framework for information retrieval that combines document models and query models using a probabilistic ranking function based on Bayesian decision theory. The framework suggests an operational retrieval model that extends recent developments in the language modeling approach to information retrieval. A language model for each document is estimated, as well as a language model for each query, and the retrieval problem is cast in terms of risk minimization. The query language model can be exploited to model user preferences, the context of a query, synonomy and word senses. While recent work has incorporated word translation models for this purpose, we introduce a new method using Markov chains defined on a set of documents to estimate the query models. The Markov chain method has connections to algorithms from link analysis and social networks. The new approach is evaluated on TREC collections and compared to the basic language modeling approach and vector space models together with query expansion using Rocchio. Significant improvements are obtained over standard query expansion methods for strong baseline TF-IDF systems, with the greatest improvements attained for short queries on Web data.

Mappings to structured output spaces (strings, trees, partitions, etc.) are typically learned using extensions of classification algorithms to simple graphical structures (eg., linear chains) in which search and parameter estimation can be performed exactly. Unfortunately, in many complex problems, it is rare that exact search or parameter estimation is tractable. Instead of learning exact models and searching via heuristic means, we embrace this difficulty and treat the structured output problem in terms of approximate search. We present a framework for learning as search optimization, and two parameter updates with convergence the-orems and bounds. Empirical evidence shows that our integrated approach to learning and decoding can outperform exact models at smaller computational cost.

Several standard text-categorization techniques were applied to the problem of automated essay grading. Bayesian independence classifiers and knearest-neighbor classifiers were trained to assign scores to manually-graded essays. These scores were combined with several other summary text measures using linear regression. The classifiers and regression equations were then applied to a new set of essays. The classifiers worked very well. The agreement between the automated grader and the final manual grade was as good as the agreement between human graders.

Tony Robinson Cambridge University Engineering Department Cambridge CB2 1PZ England. Tel: [+44] 1223 332815 ajr@eng.cam.ac.uk We present a Bayesian framework for inferring the parameters of a mixture of experts model based on ensemble learning by variational free energy minimisation. The Bayesian approach avoids the over-fitting and noise level under-estimation problems of traditional maximum likelihood inference. We demonstrate these methods on artificial problems and sunspot time series prediction.

We have suggested that the mirror-neuron system might be usefully understood as implementing Bayes-optimal perception of actions emitted by oneself or others. To substantiate this claim, we present neuronal simulations that show the same representations can prescribe motor behavior and encode motor intentions during action–observation. These simulations are based on the free-energy formulation of active inference, which is formally related to predictive coding. In this scheme, (generalised) states of the world are represented as trajectories. When these states include motor trajectories they implicitly entail intentions (future motor states). Optimizing the representation of these intentions enables predictive coding in a prospective sense. Crucially, the same generative models used to make predictions can be deployed to predict the actions of self or others by simply changing the bias or precision (i.e. attention) afforded to proprioceptive signals. We illustrate these points using simulations of handwriting to illustrate neuronally plausible generation and recognition of itinerant (wandering) motor trajectories. We then use the same simulations to produce synthetic electrophysiological responses to violations of intentional expectations. Our results affirm that a Bayes-optimal approach provides a principled framework, which accommodates current thinking about the mirror-neuron system. Furthermore, it endorses the general formulation of action as active inference.

In the context of parameter estimation and model selection, it is only quite recently that a direct link between the Fisher information and information-theoretic quantities has been exhibited. We give an interpretation of this link within the standard framework of information theory. We show that in the context of population coding, the mutual information between the activity of a large array of neurons and a stimulus to which the neurons are tuned is naturally related to the Fisher information. In the light of this result, we consider the optimization of the tuning curves parameters in the case of neurons responding to a stimulus represented by an angular variable.

Since the experiments of Saffran et al. [Saffran, J., Aslin, R., & Newport, E. (1996). Statistical learning in 8-month-old infants. Science, 274, 1926-1928], there has been a great deal of interest in the question of how statistical regularities in the speech stream might be used by infants to begin to identify individual words. In this work, we use computational modeling to explore the effects of different assumptions the learner might make regarding the nature of words--in particular, how these assumptions affect the kinds of words that are segmented from a corpus of transcribed child-directed speech. We develop several models within a Bayesian ideal observer framework, and use them to examine the consequences of assuming either that words are independent units, or units that help to predict other units. We show through empirical and theoretical results that the assumption of independence causes the learner to undersegment the corpus, with many two- and three-word sequences (e.g. what's that, do you, in the house) misidentified as individual words. In contrast, when the learner assumes that words are predictive, the resulting segmentation is far more accurate. These results indicate that taking context into account is important for a statistical word segmentation strategy to be successful, and raise the possibility that even young infants may be able to exploit more subtle statistical patterns than have usually been considered.


Question: The relationship between Minimum Description Length induction, Bayesian methods, and Kolmogorov complexity.

This work explores the application of information-theoretic computational complexity to the realm of meta-mathematics, focusing on the evaluation of formal systems through the lens of instruction bit count and computational time. The study delves into the requisite bit length of instructions for a computer to execute both finite and infinite tasks, alongside the duration for their completion. This framework is then applied to gauge the intricacy of theorem proof, measured by the quantity of axiom bits presumed and the scale of the proofs required to derive the theorems from these axioms. The investigation aims to elucidate the information-theoretic constraints inherent in formal systems, providing insights into the computational resources necessary for proving mathematical assertions within such systems.

The problem of Most Probable Explanation (MPE) arises in the scenario of probabilistic inference: finding an assignment to all variables that has the maximum likelihood given some evidence. We consider the more general CNF-based MPE problem, where each literal in a CNF-formula is associated with a weight. We describe reductions between MPE and weighted MAX-SAT, and show that both can be solved by a variant of weighted model counting. The MPE-SAT algorithm is quite competitive with the state-of-the-art MAX-SAT, WCSP, and MPE solvers on a variety of problems.

Tony Robinson Cambridge University Engineering Department Cambridge CB2 1PZ England. Tel: [+44] 1223 332815 ajr@eng.cam.ac.uk We present a Bayesian framework for inferring the parameters of a mixture of experts model based on ensemble learning by variational free energy minimisation. The Bayesian approach avoids the over-fitting and noise level under-estimation problems of traditional maximum likelihood inference. We demonstrate these methods on artificial problems and sunspot time series prediction.

An adaptive statistical language model is described, which successfully integrates long distance linguistic information with other knowledge sources. Most existing statistical language models exploit only the immediate history of a text. To extract information from further back in the document’s history, we propose and use trigger pairs as the basic information bearing elements. This allows the model to adapt its expectations to the topic of discourse. Next, statistical evidence from multiple sources must be combined. Traditionally, linear interpolation and its variants have been used, but these are shown here to be seriously deficient. Instead, we apply the principle of Maximum Entropy (ME). Each information source gives rise to a set of constraints, to be imposed on the combined estimate. The intersection of these constraints is the set of probability functions which are consistent with all the information sources. The function with the highest entropy within that set is the ME solution. Given consistent statistical evidence, a unique ME solution is guaranteed to exist, and an iterative algorithm exists which is guaranteed to converge to it. The ME framework is extremely general: any phenomenon that can be described in terms of statistics of the text can be readily incorporated. An adaptive language model based on the ME approach was trained on the Wall Street Journal corpus, and showed a 32–39% perplexity reduction over the baseline. When interfaced to SPHINX-II, Carnegie Mellon’s speech recognizer, it reduced its error rate by 10–14%. This thus illustrates the feasibility of incorporating many diverse knowledge sources in a single, unified statistical framework.  1996 Academic Press Limited

Extreme learning machine (ELM) has become a popular topic in machine learning in recent years. ELM is a new kind of single-hidden layer feedforward neural network with an extremely low computational cost. ELM, however, has two evident drawbacks: 1) the output weights solved by Moore-Penrose generalized inverse is a least squares minimization issue, which easily suffers from overfitting and 2) the accuracy of ELM is drastically sensitive to the number of hidden neurons so that a large model is usually generated. This brief presents a sparse Bayesian approach for learning the output weights of ELM in classification. The new model, called Sparse Bayesian ELM (SBELM), can resolve these two drawbacks by estimating the marginal likelihood of network outputs and automatically pruning most of the redundant hidden neurons during learning phase, which results in an accurate and compact model. The proposed SBELM is evaluated on wide types of benchmark classification problems, which verifies that the accuracy of SBELM model is relatively insensitive to the number of hidden neurons; and hence a much more compact model is always produced as compared with other state-of-the-art neural network classifiers.

Markov chain Monte Carlo (e. g., the Metropolis algorithm and Gibbs sampler) is a general tool for simulation of complex stochastic processes useful in many types of statistical inference. The basics of Markov chain Monte Carlo are reviewed, including choice of algorithms and variance estimation, and some new methods are introduced. The use of Markov chain Monte Carlo for maximum likelihood estimation is explained, and its performance is compared with maximum pseudo likelihood estimation.

In many multivariate domains, we are interested in analyzing the dependency structure of the underlying distribution, e.g., whether two variables are in direct interaction. We can represent dependency structures using Bayesian network models. To analyze a given data set, Bayesian model selection attempts to find the most likely (MAP) model, and uses its structure to answer these questions. However, when the amount of available data is modest, there might be many models that have non-negligible posterior. Thus, we want compute the Bayesian posterior of a feature, i.e., the total posterior probability of all models that contain it. In this paper, we propose a new approach for this task. We first show how to efficiently compute a sum over the exponential number of networks that are consistent with a fixed order over network variables. This allows us to compute, for a given order, both the marginal probability of the data and the posterior of a feature. We then use this result as the basis for an algorithm that approximates the Bayesian posterior of a feature. Our approach uses a Markov Chain Monte Carlo (MCMC) method, but over orders rather than over network structures. The space of orders is smaller and more regular than the space of structures, and has much a smoother posterior “landscape”. We present empirical results on synthetic and real-life datasets that compare our approach to full model averaging (when possible), to MCMC over network structures, and to a non-Bayesian bootstrap approach.

Most real-world data is heterogeneous and richly interconnected. Examples include the Web, hypertext, bibliometric data and social networks. In contrast, most statistical learning methods work with “flat” data representations, forcing us to convert our data into a form that loses much of the link structure. The recently introduced framework of probabilistic relational models(PRMs) embraces the object-relational nature of structured data by capturing probabilistic interactions between attributes of related entities. In this paper, we extend this framework by modeling interactions between the attributes and the link structure itself. An advantage of our approach is a unified generative model for both content and relational structure. We propose two mechanisms for representing a probabilistic distribution over link structures: reference uncertaintyandexistence uncertainty . We describe the appropriate conditions for using each model and present learning algorithms for each. We present experimental results showing that the learned models can be used to predict link structure and, moreover, the observed link structure can be used to provide better predictions for the attributes in the model.

We propose a new unsupervised learning technique for extracting information from large text collections. We model documents as if they were generated by a two-stage stochastic process. Each author is represented by a probability distribution over topics, and each topic is represented as a probability distribution over words for that topic. The words in a multi-author paper are assumed to be the result of a mixture of each authors' topic mixture. The topic-word and author-topic distributions are learned from data in an unsupervised manner using a Markov chain Monte Carlo algorithm. We apply the methodology to a large corpus of 160,000 abstracts and 85,000 authors from the well-known CiteSeer digital library, and learn a model with 300 topics. We discuss in detail the interpretation of the results discovered by the system including specific topic and author models, ranking of authors by topic and topics by author, significant trends in the computer science literature between 1990 and 2002, parsing of abstracts by topics and authors and detection of unusual papers by specific authors. An online query interface to the model is also discussed that allows interactive exploration of author-topic models for corpora such as CiteSeer.

Probabilistic logic programs are logic programs in which some of the facts are annotated with probabilities. Several classical probabilistic inference tasks (such as MAP and computing marginals) have not yet received a lot of attention for this formalism. The contribution of this paper is that we develop efficient inference algorithms for these tasks. This is based on a conversion of the probabilistic logic program and the query and evidence to a weighted CNF formula. This allows us to reduce the inference tasks to wellstudied tasks such as weighted model counting. To solve such tasks, we employ state-ofthe-art methods. We consider multiple methods for the conversion of the programs as well as for inference on the weighted CNF. The resulting approach is evaluated experimentally and shown to improve upon the state-of-theart in probabilistic logic programming.

The problem of formulating general concepts from specific training examples has long been a major focus of machine learning research. While most previous research has focused on empirical methods for generalizing from a large number of training examples using no domain-specific knowledge, in the past few years new methods have been developed for applying domain-specific knowledge to formulate valid generalizations from single training examples. The characteristic common to these methods is that their ability to generalize from a single example follows from their ability to explain why the training example is a member of the concept being learned. This paper proposes a general, domain-independent mechanism, called EBG, that unifies previous approaches to explanation-based generalization. The EBG method is illustrated in the context of several example problems, and used to contrast several existing systems for explanation-based generalization. The perspective on explanation-based generalization afforded by this general method is also used to identify open research problems in this area.


Question: Minimum description length inference, Bayesian methods, and Kolmogorov complexity.

This paper explores the concept of logical reversibility in computation, a property absent in conventional computing automata such as Turing machines, which are inherently logically irreversible due to non-invertible transition functions. The study reveals a method to construct logically reversible machines that maintain their simplicity and computational capability, a finding with significant physical implications. Specifically, it suggests the potential for thermodynamically reversible computers capable of performing complex calculations at optimal speeds while dissipating energy far below the kT threshold per logical step. The proposed logically reversible automaton mirrors the operation of its irreversible counterpart in the initial stage of computation, with the critical difference of preserving all intermediate computational results, thereby circumventing the irreversible process of data erasure. Subsequent stages involve the generation of the desired output and a reversible process of eliminating all unwanted intermediate data by retracing the first stage's steps in reverse order. This approach is feasible due to the reversible nature of the initial computation, ensuring the machine's restoration to its original state, save for the output. The final configuration of the machine contains both the desired output and a reconstructed copy of the input, with no additional unwanted data. The paper illustrates these concepts using a three-tape Turing machine model and discusses the biosynthesis of messenger RNA as a real-world instance of reversible computation.

MAXIMUM ENTROPY MODELS FOR NATURAL LANGUAGE AMBIGUITY RESOLUTION Adwait Ratnaparkhi Supervisor Professor Mitch Marcus This thesis demonstrates that several important kinds of natural language ambiguities can be resolved to state of the art accuracies using a single statistical modeling technique based on the principle of maximum entropy We discuss the problems of sentence boundary detection part of speech tagging prepo sitional phrase attachment natural language parsing and text categorization under the maximum entropy framework In practice we have found that maximum entropy models o er the following advantages State of the art Accuracy The probability models for all of the tasks discussed per form at or near state of the art accuracies or outperform competing learning algo rithms when trained and tested under similar conditions Methods which outperform those presented here require much more supervision in the form of additional human involvement or additional supporting resources Knowledge Poor Features The facts used to model the data or features are linguis tically very simple or knowledge poor but yet succeed in approximating complex linguistic relationships Reusable Software Technology The mathematics of the maximum entropy framework

Hierarchical structure is ubiquitous in data across many domains. There are many hierarchical clustering methods, frequently used by domain experts, which strive to discover this structure. However, most of these methods limit discoverable hierarchies to those with binary branching structure. This limitation, while computationally convenient, is often undesirable. In this paper we explore a Bayesian hierarchical clustering algorithm that can produce trees with arbitrary branching structure at each node, known as rose trees. We interpret these trees as mixtures over partitions of a data set, and use a computationally efficient, greedy agglomerative algorithm to find the rose trees which have high marginal likelihood given the data. Lastly, we perform experiments which demonstrate that rose trees are better models of data than the typical binary trees returned by other hierarchical clustering algorithms.

The problem of Most Probable Explanation (MPE) arises in the scenario of probabilistic inference: finding an assignment to all variables that has the maximum likelihood given some evidence. We consider the more general CNF-based MPE problem, where each literal in a CNF-formula is associated with a weight. We describe reductions between MPE and weighted MAX-SAT, and show that both can be solved by a variant of weighted model counting. The MPE-SAT algorithm is quite competitive with the state-of-the-art MAX-SAT, WCSP, and MPE solvers on a variety of problems.

Graphical models bring together graph theory and probability theory in a powerful formalism for multivariate statistical modeling. In statistical signal processing— as well as in related fields such as communication theory, control theory and bioinformatics—statistical models have long been formulated in terms of graphs, and algorithms for computing basic statistical quantities such as likelihoods and marginal probabilities have often been expressed in terms of recursions operating on these graphs. Examples include hidden Markov models, Markov random fields, the forward-backward algorithm and Kalman filtering [ Rabiner and Juang (1993); Pearl (1988); Kailath et al. (2000)]. These ideas can be understood, unified and generalized within the formalism of graphical models. Indeed, graphical models provide a natural framework for formulating variations on these classical architectures, and for exploring entirely new families of statistical models. The recursive algorithms cited above are all instances of a general recursive algorithm known as the junction tree algorithm [ Lauritzen and Spiegelhalter, 1988]. The junction tree algorithm takes advantage of factorization properties of the joint probability distribution that are encoded by the pattern of missing edges in a graphical model. For suitably sparse graphs, the junction tree algorithm provides a systematic and practical solution to the general problem of computing likelihoods and other statistical quantities associated with a graphical model. Unfortunately, many graphical models of practical interest are not " suitably sparse, " so that the junction tree algorithm no longer provides a viable computational solution to the problem of computing marginal probabilities and other expectations. One popular source of methods for attempting to cope with such cases is the Markov chain Monte Carlo (MCMC) framework, and indeed there is a significant literature on

The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. I dedicate this book to my grandchildren. Preface The intent of this book is to describe some recent data mining tools that have proven effective in dealing with data sets which often involve uncertain description or other complexities that cause difficulty for the conventional approaches of logistic regression, neural network models, and decision trees. Among these traditional algorithms, neural network models often have a relative advantage when data is complex. We will discuss methods with simple examples, review applications, and evaluate relative advantages of several contemporary methods. Our intent is to cover the fundamental concepts of data mining, to demonstrate the potential of gathering large sets of data, and analyzing these data sets to gain useful business understanding. We have organized the material into three parts. Part I introduces concepts. Part II contains chapters on a number of different techniques often used in data mining. Part III focuses on business applications of data mining. Not all of these chapters need to be covered, and their sequence could be varied at instructor design. The book will include short vignettes of how specific concepts have been applied in real practice. A series of representative data sets will be generated to demonstrate specific methods and concepts. References to data mining software and sites such as www.kdnuggets.com will be provided. Chapter 1 gives an overview of data mining, and provides a description of the data mining process. An overview of useful business applications is provided. Chapter 2 presents the data mining process in more detail. It demonstrates this process with a typical set of data. Visualization of data through data mining software is addressed. Chapter 3 presents memory-based reasoning methods of data mining. Major real applications are described. Algorithms are demonstrated with prototypical data based on real applications. Chapter 4 discusses association rule methods. Application in the form of market basket analysis is discussed. A real data set is described, and a simplified version used to demonstrate association rule methods. Chapter 5 presents fuzzy data mining approaches. Fuzzy decision tree approaches are described, as well as fuzzy association rule applications. Real data mining applications are described and demonstrated Chapter 6 presents Rough …

We present a novel approach to anomaly detection in Bayesian networks, enabling both the detection and explanation of anomalous cases in a dataset. By exploiting the structure of a Bayesian network, our algorithm is able to efficiently search for local maxima of data conflict between closely related variables. Benchmark tests using data simulated from complex Bayesian networks show that our approach provides a significant improvement over techniques that search for anomalies using the entire network, rather than its subsets. We conclude with demonstrations of the unique explanatory power of our approach in determining the observation(s) responsible for an anomaly. APPROVED FOR PUBLIC RELEASE

T HE recent development of various methods of modulation such as PCM and PPM which exchange bandwidth for signal-to-noise ratio has intensified the interest in a general theory of communication. A basis for such a theory is contained in the important papers of Nyquist 1 and Hartley2 on this subject. In the present paper we will extend the theory to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information. The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. If the number of messages in the set is finite then this number or any monotonic function of this number can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely. As was pointed out by Hartley the most natural choice is the logarithmic function. Although this definition must be generalized considerably when we consider the influence of the statistics of the message and when we have a continuous range of messages, we will in all cases use an essentially logarithmic measure. The logarithmic measure is more convenient for various reasons:

The naive Bayes classiier, currently experiencing a renaissance in machine learning, has long been a core technique in information retrieval. We review some of the variations of naive Bayes models used for text retrieval and classiication, focusing on the distributional assumptions made about word occurrences in documents.

Many statistical models can be simulated forwards but have intractable likelihoods. Approximate Bayesian Computation (ABC) methods are used to infer properties of these models from data. Traditionally these methods approximate the posterior over parameters by conditioning on data being inside an -ball around the observed data, which is only correct in the limit →0. Monte Carlo methods can then draw samples from the approximate posterior to approximate predictions or error bars on parameters. These algorithms critically slow down as →0, and in practice draw samples from a broader distribution than the posterior. We propose a new approach to likelihood-free inference based on Bayesian conditional density estimation. Preliminary inferences based on limited simulation data are used to guide later simulations. In some cases, learning an accurate parametric representation of the entire true posterior distribution requires fewer model simulations than Monte Carlo ABC methods need to produce a single sample from an approximate posterior.

Language is used to communicate ideas. Ideas are mental tools for coping with a complex and uncertain world. Thus human conceptual structures should be key to language meaning, and probability—the mathematics of uncertainty— should be indispensable for describing both language and thought. Indeed, probabilistic models are enormously useful in modeling human cognition (Tenenbaum et al., 2011) and aspects of natural language (Bod et al., 2003; Chater et al., 2006). With a few early exceptions (e.g. Adams, 1975; Cohen, 1999b), probabilistic tools have only recently been used in natural language semantics and pragmatics. In this chapter we synthesize several of these modeling advances, exploring a formal model of interpretation grounded, via lexical semantics and pragmatic inference, in conceptual structure. Flexible human cognition is derived in large part from our ability to imagine possibilities (or possible worlds). A rich set of concepts, intuitive theories, and other mental representations support imagining and reasoning about possible worlds—together we will call these the conceptual lexicon. We posit that this collection of concepts also forms the set of primitive elements available for lexical semantics: word meanings can be built from the pieces of conceptual structure. Larger semantic structures are then built from word meanings by composition, ultimately resulting in a sentence meaning which is a phrase in the “language of thought” provided by the conceptual lexicon. This expression is truth-functional in that it takes on a Boolean value for each imagined world, and it can thus be used as the basis for belief updating. However, the connection between cognition, semantics, and belief is not direct: because language must flexibly adapt to the context of communication, the connection between lexical representation and interpreted meaning is mediated by pragmatic inference.


Question: Recipient revocable identity-based broadcast encryption: how to exclude certain recipients in IBBE without plaintext knowledge.

This paper presents an innovative Key-Policy Attribute-Based Encryption (KP-ABE) system designed to facilitate fine-grained access control over encrypted data. With the increasing reliance on third-party internet services for storing sensitive information, there arises a critical need for sophisticated encryption techniques. A significant limitation of conventional encryption methods is their coarse-grained sharing capability, typically involving the transfer of private keys to other parties. To address this, we introduce a novel cryptosystem that enables more nuanced data sharing. In KP-ABE, ciphertexts are tagged with attribute sets, and private keys are linked to access structures that dictate the user's decryption capabilities. The paper showcases the system's practicality in managing audit-log information and broadcast encryption scenarios. A key feature is the support for private key delegation, effectively incorporating Hierarchical Identity-Based Encryption (HIBE) functionality. This advancement paves the way for more secure and flexible data sharing mechanisms online.

Freenet is a popular peer to peer anonymous content-sharing network, with the objective to provide the anonymity of both content publishers and retrievers. Despite more than a decade of active development and deployment and the adoption of well-established cryptographic algorithms in Freenet, it remains unanswered how well the anonymity objective of the initial Freenet design has been met. In this paper we develop a traceback attack on Freenet, and show that the originating machine of a content request message in Freenet can be identified; that is, the anonymity of a content retriever can be broken, even if a single request message has been issued by the retriever. We present the design of the traceback attack, and perform both experimental and simulation studies to confirm the feasibility and effectiveness of the attack. For example, with randomly chosen content requesters (and random contents stored in the Freenet testbed), Emulab-based experiments show that, for <inline-formula><tex-math notation="LaTeX">$24$</tex-math> <alternatives><inline-graphic xlink:href="duan-ieq1-2453983.gif"/></alternatives></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$43$</tex-math><alternatives><inline-graphic xlink:href="duan-ieq2-2453983.gif"/> </alternatives></inline-formula> percent of the content request messages, we can identify their originating machines. We also briefly discuss potential solutions to address the developed traceback attack. Despite being developed specifically on Freenet, the basic principles of the traceback attack and solutions have important security implications for similar anonymous content-sharing systems.

Logistic regression is a powerful machine learning tool to classify data. When dealing with sensitive data such as private or medical information, cares are necessary. In this paper, we propose a secure system for protecting the training data in logistic regression via homomorphic encryption. Perhaps surprisingly, despite the non-polynomial tasks of training in logistic regression, we show that only additively homomorphic encryption is needed to build our system. Our system is secure and scalable with the dataset size.

Automatic dependent surveillance-broadcast (ADS-B) has become a crucial part of next generation air traffic surveillance technology and will be mandatorily deployed for most of the airspaces worldwide by 2020. Each aircraft equipped with an ADS-B device keeps broadcasting plaintext messages to other aircraft and the ground station controllers once or twice per second. The lack of security measures in ADS-B systems makes it susceptible to different attacks. Among the various security issues, we investigate the integrity and authenticity of ADS-B messages. We propose a new framework for providing ADS-B with authentication based on three-level hierarchical identity-based signature (HIBS) with batch verification. Previous signature-based ADS-B authentication protocols focused on how to generate signatures efficiently, while our schemes can also significantly reduce the verification cost, which is critical to ADS-B systems, since at any time an ADS-B receiver may receive lots of signatures. We design two concrete schemes. The basic scheme supports partial batch verification and the extended scheme provides full batch verification. We give a formal security proof for the extended scheme. Experiment results show that our schemes with batch verification are tremendously more efficient in batch verifying <inline-formula><tex-math notation="LaTeX">$n$ </tex-math><alternatives><inline-graphic xlink:href="yang-ieq1-2459709.gif"/></alternatives></inline-formula> signatures than verifying <inline-formula><tex-math notation="LaTeX">$n$</tex-math><alternatives> <inline-graphic xlink:href="yang-ieq2-2459709.gif"/></alternatives></inline-formula> signatures independently. For example, the running time of verifying 100 signatures is 502 and 484 ms for the basic scheme and the extended scheme respectively, while the time is 2500 ms if verifying the signatures independently.

This research addresses the implementation of encryption and digital signature technique for electronic health record to prevent cybercrime problem such as robbery, modification and unauthorized access. In this research, RSA 2048-bit algorithm, AES 256-bit and SHA 256 will be implemented in Java programming. Secure Electronic Health Record Information (SEHR) application design is intended to combine given services, such as confidentiality, integrity, authentication, and non-repudiation. Cryptography is used to ensure the file records and electronic documents for detailed information on the medical past, present and future forecasts that have been given only for the patients. The document will be encrypted using an encryption algorithm based on NIST Standard. In the application, there are two schemes, namely the protection and verification scheme. This research uses black-box testing and white-box testing to test the software input, output, and code without testing the process and design that occurs in the system. We demonstrated the implementation of cryptography in Secure Electronic Health Record Information (SEHR). The implementation of encryption and digital signature in this research can prevent archive thievery which is shown on implementation and is proven on the test.

Many P2P applications require security services such as privacy, anonymity, authentication, and non-repudiation. Such services could be provided through a hierarchical Public Key Infrastructure. However, P2P networks are usually Internet-scale distributed systems comprised of nodes with undetermined trust level, thus making hierarchical solutions unrealistic. In this paper, we propose Chord-PKI, a distributed PKI architecture which is build upon the Chord overlay network, in order to provide security services for P2P applications. Our solution distributes the functionality of a PKI across the peers, by using threshold cryptography and proactive updating. We analyze the security of the proposed infrastructure and through simulations, we evaluate its performance for various scenarios of untrusted node distributions.

Visual Cryptography and biometrics have been identified as the two most important aspects of digital security. In this paper, we propose a method for the preparation and secure transmission of fingerprint images using visual cryptography scheme. Visual cryptography (VC) is a kind of secret image sharing scheme that uses the human visual system to perform the decryption computations. A visual cryptography scheme (VCS) allows confidential messages to be encrypted into k-out-of-n secret sharing schemes. Whenever the number of participants from the group (n) is larger than or equal to the predetermined threshold value (k), the confidential message can be obtained by these participants. VCS is interesting because decryption can be done with no prior knowledge of cryptography and can be performed without any cryptographic computations. In this paper, the fingerprint image is broken up into n pieces, which individually yield no information about the image. These pieces of the images, called shares or shadows, may then be distributed among a group of n participants/dealers. By combining any k (k ≤ n) of these shares, the original fingerprint image can be recovered, but combining less than k of them will not reveal any information about the image. The scheme is perfectly secure and very easy to implement. To decode the encrypted information, i.e., to get the original information back, the shares are stacked and the secret image pops out. The only drawback of the VCS is the loss in contrast of reconstructed image. The proposed method for the reconstruction of fingerprint image is based on XNOR operation. This will enable one to obtain back perfect fingerprint image.

http://dx.doi.org/10.1016/j.infsof.2015.01.012 0950-5849/ 2015 Elsevier B.V. All rights reserved. ⇑ Corresponding author. E-mail addresses: drbabak2@live.utm.my (B.D. Rouhani), mdnazrim@utm.my (M.N. Mahrin), fa.nikpay@siswa.um.edu.my (F. Nikpay), rodina@um.edu.my (R.B. npourya2@live.utm.my (P. Nikfard). Babak Darvish Rouhani a,⇑, Mohd Naz’ri Mahrin , Fatemeh Nikpay , Rodina Binti Ahmad , Pourya Nikfard c

The concept of smart grid has emerged as a convergence of traditional power system engineering and information and communication technology. It is vital to the success of next generation of power grid, which is expected to be featuring reliable, efficient, flexible, clean, friendly, and secure characteristics. In this paper, we propose an efficient and privacy-preserving aggregation scheme, named EPPA, for smart grid communications. EPPA uses a superincreasing sequence to structure multidimensional data and encrypt the structured data by the homomorphic Paillier cryptosystem technique. For data communications from user to smart grid operation center, data aggregation is performed directly on ciphertext at local gateways without decryption, and the aggregation result of the original data can be obtained at the operation center. EPPA also adopts the batch verification technique to reduce authentication cost. Through extensive analysis, we demonstrate that EPPA resists various security threats and preserve user privacy, and has significantly less computation and communication overhead than existing competing approaches.

Public blockchains provide a decentralized method for storing transaction data and have many applications in different sectors. In order for a user to track transactions, a simple method is that every user keeps a local copy of the entire public ledger. Since the size of a ledger keeps growing, this method becomes increasingly less practical, especially for lightweight users such as IoT devices and smartphones. In order to deal with this problem, there have been some proposals. However, existing solutions either achieve a limited storage reduction (e.g., simple payment verification), or rely on some strong security assumption (e.g., the use of trusted server). We propose EPBC, a novel and efficient transaction verification scheme for public ledgers, which only requires lightweight users to store a small amount of data that is independent of the size of the blockchain. We analyze EPBC's performance and security, and discuss its integration with existing public ledger systems. Experimental results confirm that EPBC is practical for lightweight users.

Encrypted databases, a popular approach to protecting data from compromised database management systems (DBMS's), use abstract threat models that capture neither realistic databases, nor realistic attack scenarios. In particular, the "snapshot attacker" model used to support the security claims for many encrypted databases does not reflect the information about past queries available in any snapshot attack on an actual DBMS.
 We demonstrate how this gap between theory and reality causes encrypted databases to fail to achieve their "provable security" guarantees.