Understanding the Constraints in Maximum Entropy Methods for Modeling and Inference.

The principle of maximum entropy, developed more than six decades ago, provides a systematic approach to modeling inference, and data analysis grounded in the principles of information theory, Bayesian probability and constrained optimization. Since its formulation, criticisms about the consistency of that method and the role of constraints have been raised. Among these, the chief criticism is that maximum entropy does not satisfy the principle of causation, or similarly, that maximum entropy updating is inconsistent due to an inadequate representation of causal information. We show that these criticisms rest on misunderstanding and misapplication of the way constraints have to be specified within the maximum entropy method. Correction of these problems eliminates the seeming paradoxes and inconsistencies critics claim to have detected. We demonstrate that properly formulated maximum entropy models satisfy the principle of causation.

Information theory: A foundation for complexity science.

Modeling and inference are central to most areas of science and especially to evolving and complex systems. Critically, the information we have is often uncertain and insufficient, resulting in an underdetermined inference problem; multiple inferences, models, and theories are consistent with available information. Information theory (in particular, the maximum information entropy formalism) provides a way to deal with such complexity. It has been applied to numerous problems, within and across many disciplines, over the last few decades. In this perspective, we review the historical development of this procedure, provide an overview of the many applications of maximum entropy and its extensions to complex systems, and discuss in more detail some recent advances in constructing comprehensive theory based on this inference procedure. We also discuss efforts at the frontier of information-theoretic inference: application to complex dynamic systems with time-varying constraints, such as highly disturbed ecosystems or rapidly changing economies.

Does the U.S. Navy's reliance on objective standards prevent discrimination in promotions and retentions?

To prevent discrimination, the U.S. Navy enlisted-personnel promotion process relies primarily on objective measures. However, it also uses the subjective opinion of a sailor's superior. The Navy's promotion and retention process involves two successive decisions: The Navy decides whether to promote an individual, and conditional on that decision, the sailor decides whether to stay. Using estimates of these correlated decision-making processes, we find that during 1997-2008, Blacks and Hispanics were less likely to be promoted than Whites, especially during wartime. The Navy's decision-making affects Blacks' differential promotion rates by twice as much as differences in the groups' characteristics. However, Nonwhite retention probabilities, even when not promoted, are higher than for Whites, in part because they have fewer opportunities in the civilian market. Females have lower promotion rates than males and slightly lower retention rates during wartime.

Interpreting Social Accounting Matrix (SAM) as an Information Channel.

Information theory, and the concept of information channel, allows us to calculate the mutual information between the source (input) and the receiver (output), both represented by probability distributions over their possible states. In this paper, we use the theory behind the information channel to provide an enhanced interpretation to a Social Accounting Matrix (SAM), a square matrix whose columns and rows present the expenditure and receipt accounts of economic actors. Under our interpretation, the SAM's coefficients, which, conceptually, can be viewed as a Markov chain, can be interpreted as an information channel, allowing us to optimize the desired level of aggregation within the SAM. In addition, the developed information measures can describe accurately the evolution of a SAM over time. Interpreting the SAM matrix as an ergodic chain could show the effect of a shock on the economy after several periods or economic cycles. Under our new framework, finding the power limit of the matrix allows one to check (and confirm) whether the matrix is well-constructed (irreducible and aperiodic), and obtain new optimization functions to balance the SAM matrix. In addition to the theory, we also provide two empirical examples that support our channel concept and help to understand the associated measures.

What May Visualization Processes Optimize?

In this paper, we present an abstract model of visualization and inference processes, and describe an information-theoretic measure for optimizing such processes. In order to obtain such an abstraction, we first examined six classes of workflows in data analysis and visualization, and identified four levels of typical visualization components, namely disseminative, observational, analytical and model-developmental visualization. We noticed a common phenomenon at different levels of visualization, that is, the transformation of data spaces (referred to as alphabets) usually corresponds to the reduction of maximal entropy along a workflow. Based on this observation, we establish an information-theoretic measure of cost-benefit ratio that may be used as a cost function for optimizing a data visualization process. To demonstrate the validity of this measure, we examined a number of successful visualization processes in the literature, and showed that the information-theoretic measure can mathematically explain the advantages of such processes over possible alternatives.