Part 13 (1/2)

A transmission affected by thermodynamic perturbations that make it less than perfectly reliable will introduce an additional level of uncertainty to contend with, but one that decreases information capacity. An increase in the Boltzmann entropy of the physical medium that const.i.tutes the signal carrier corresponds to a decrease in the correlation between sent and received signals. Although this does not decrease the signal entropy, it reduces the amount of uncertainty that can be removed by a given signal, and thus reduces the information capacity.

This identifies two contributors to the entropy of a signal-one a.s.sociated with the probability of a given signal being sent and the other a.s.sociated with a given signal being corrupted. This complementary relations.h.i.+p is a hint that the physical and informational uses of the concept of entropy are more than merely a.n.a.logous. By exploring the relations.h.i.+p between Shannon entropy and Boltzmann entropy, we can shed light on the reason why change in Shannon entropy is critical to information. But the connection is subtle, and its relations.h.i.+p to the way that a signal conveys its information ”content” is even subtler.

INFORMATION AND REFERENCE.

Warren Weaver, who wrote a commentary article that appeared in a book presentation of Shannon's original paper, commented that using the term information to describe the measure of the unpredictability reduced by a given signal is an atypical use of the term.6 This is because Shannon's notion of information is agnostic with respect to what a signal is or could be about. This agnosticism has led to considerable confusion outside of the technical literature, because it is almost ant.i.thetical to the standard colloquial use of the term. Shannon was interested in measuring information for engineering purposes. So he concentrated exclusively on the properties of transmission processes and communication media, and ignored what we normally take to be information, that is, what something tells us about something else that is not present in the signal medium itself.

This was not merely an arbitrary simplification, however. It was necessary because the same sign or signal can be given any number of different interpretations. Dirt on a boot can provide information on anything from personal hygiene to evidence about the sort of geological terrain a person recently visited. The properties of some medium that give it the potential to convey information don't determine what it is about, they merely make reference possible. So, in order to provide a finite measure of the information potential of a given signal or channel, Shannon had to ignore any particular interpretation process, and stop the a.n.a.lysis prior to including any consideration of what a sign or signal might be about. What is conveyed is not merely a function of this reduction of Shannon entropy.

This is where the relation between Shannon and Boltzmann entropy turns out to be more than merely a.n.a.logical. A fuller conception of information requires that these properties be considered with respect to two different levels of a.n.a.lysis of the same phenomenon: the formal characteristics of the signal and the material-energetic characteristics of the signal. Consider what we know about Boltzmann entropy. If, within the boundaries of a physical system such as a chamber filled with a gas, a reduction of entropy is observed, one can be pretty certain that something not in that chamber is causing this reduction of entropy. Being in an improbable state or observing a non-spontaneous change toward such a state is evidence of extrinsic perturbation-work imposed from outside the system.

Despite their abstract character, information transmission and interpretation are physical processes involving material or energetic substrates that const.i.tute the transmission channel, storage medium, sign vehicles, and so on. But physical processes are subject to the laws of thermodynamics. So, in the case of Shannon entropy, no information is provided if there is no reduction in the uncertainty of a signal. But reduction of the Shannon entropy of a given physical medium is necessarily also a reduction of its Boltzmann entropy. This can only occur due to the imposition of outside constraints on the sign/signal medium because a reduction of Boltzmann entropy does not tend to occur spontaneously. When it does occur, it is evidence of an external influence.

Openness to external modification is obvious in the case of a person selecting a signal to transmit, but it is also the case in more subtle conditions. Consider, for example, a random hiss of radio signals received by a radio antenna pointed toward the heavens. A normally distributed radio signal represents high informational entropy, the expected tendency in an unconstrained context, which for example might be the result of random circuit noise. If this tendency were to be altered away from this distribution in any way, it would indicate that some extrinsic non-random factor was affecting the signal. The change could be due to an astronomical object emitting a specific signal. But what if instead of a specific identifiable signal, there is just noise when there shouldn't be any?

Just such an event did occur in 1965 when two Bell Labs scientists, Arno Penzias and Robert Wilson, aimed a sensitive microwave antenna skyward, away from any local radio source, and discovered that they were still recording the ”hiss” of microwave noise no matter which way they pointed the antenna. They were receiving a more or less random microwave signal from everywhere at once! The obvious initial a.s.sumption was that it probably indicated a problem with the signal detection circuit, not any specific signal. Indeed, they even suspected that it might be the result of pigeons roosting in the antenna. Because it exhibited high Shannon entropy and its locus of origin also had high Shannon entropy (i.e., equal probability from any location), they a.s.sumed that it couldn't be a signal originating from any object in s.p.a.ce.

Only after they eliminated all potential local sources of noise did they consider the more exotic possibility: that it did not originate in the receiving system itself, but rather from outside, everywhere in the cosmos at once. If the signal had consisted only of a very narrow band of frequencies, had exhibited a specific oscillatory pattern, or was received only from certain directions, that would have provided Shannon information, because then compared with signals that could have been recorded, this would have stood out. They only considered it to be information about something external, and not mere noise, when they had compared it to many more conceivable options, thus effectively increasing the entropy (uncertainty) of potential sources that could be eliminated. As they eliminated each local factor as a possible source of noise, they eliminated this uncertainty. In the end, what was considered the least reasonable explanation was the correct one: it was emanating from empty s.p.a.ce, the cosmic background. The signal was eventually interpreted as the deeply red-s.h.i.+fted heat of the Big Bang.

In simple terms, the brute fact of these deviations from expectation, both initially, then with respect to possible sources of local noise, and much later with respect to alternative cosmological theories, was what made this hiss information about something. Moreover, the form of this deviation from expectation-its statistical uniformity and lack of intrinsic asymmetries-provided the basis for the two scientists' interpretation. Heat is random motion. The form of this deviation from what they expected-that the signal would be irregularly distributed and correlated with specific objects-ultimately provided the clue to its interpretation. In other words, this became relevant for a second kind of entropy reduction: a.n.a.lytic reduction in the variety of possible sources. By comparing the form of the received signal with what could have been its form, they were able to eliminate many possible causes of both intrinsic and extrinsic physical influence until only one seemed plausible.

This demonstrates that something other than merely the reduction of the Shannon entropy of a signal is relevant to understanding how that signal conveys evidence of another absent phenomenon. Information is made available when the state of some physical system differs from what would be expected. And what this information can be about depends on this expectation. If there is no deviation from expectation, there is no information. This would be the case for complete physical isolation of the signal medium from outside influence. The difference is that an isolated signal source cannot be about anything. What information can be about depends on the nature of the reduction process and the constraints exhibited by the received signal. How a given medium can be modified by interaction with extrinsic factors, or how it can be manipulated, is what is most relevant for determining what information can be conveyed by it.

Not only is the Shannon entropy of an information-bearing process important to its capacity; its physical dynamics with respect to the physical context in which it is embedded is important to the determination of what it can be about. But in comparison to the Shannon entropy of the signal, the physical constraints on the forms that a change in signal entropy can take contribute another independent potential for entropy reduction: a reduction of the entropy of possible referents. In other words, once we begin considering the potential entropy of the cla.s.s of things or events that a given medium can convey, the physical characteristics of the medium, not merely its range of potential states, become important. This reduces the Shannon entropy of the range of possible phenomena that a given change in that medium can be about. In other words, what we might now call referential information is a second order form of information, over and above Shannon information (Figures 12.1, 12.2).

A first hint of the relations.h.i.+p between information as form and as a sign of something else is exemplified by the role that pattern plays in the a.n.a.lysis. In Shannon's terms, pattern is redundancy. From the sender's point of view, any redundancy (defined as predictability) of the signal has the effect of reducing the amount of information that can be sent. In other words, redundancy introduces a constraint on channel capacity. Consider how much less information could be packed into this page if I could only use two characters (like the 0s and 1s in a computer). Having twenty-six letters, cases, punctuation marks, and different-length character strings (words), separated by s.p.a.ces (also characters), decreases the repet.i.tion, increases the possibility of what character can follow what, and thereby decreases redundancy. Nevertheless, there is sufficient redundancy in the possible letter combinations to make it possible to fairly easily discover typos (using redundancy for error correction will be discussed below).

FIGURE 12.1: Depiction of the logic that Claude Shannon used to define and measure potential information-conveying capacity (”Shannon information”).

FIGURE 12.2: Depiction of the way that Shannon information depends on the susceptibility of a medium to modification by work. It derives its potential to convey information about something extrinsic to that medium by virtue of the constraints imposed on that medium by whatever is responsible for this work.

Less information can get transmitted if some transmissions are predictable from previous ones, or if there are simply fewer alternatives to choose from. But the story is slightly more complicated when we add reference. From the receiver's point of view, there must be some redundancy with what is already known for the information conveyed by a signal to even be a.s.sessed. In other words, the context of the communication must already be redundantly structured. Both sender and receiver must share the set of options that const.i.tute information.

Shannon realized that the introduction of redundancy is also necessary to compensate for any unreliability of a given communication medium. If the reliability of a given sign or signal is questionable, this introduces an additional source of unpredictability that does not contribute to the intended information: noise. But just as redundancy reduces the unpredictability of signals, it can also reduce the unreliability of the medium conveying information. Introducing expected redundancy into a message being transmitted makes it possible to distinguish these two sources of Shannon entropy (the variety of possible signals that could have been generated versus the possible errors that could have arisen in the process). In the simplest case, this is accomplished by resending a signal multiple times. Because noise is, by definition, not constrained by the same factors as is the selection of the signal, each insertion of a noise-derived signal error will be uncorrelated with any other, but independent transmissions of multiple identical signals will be correlated with one another by definition. In this way, noisy components of a signal or a received message can be detected and replaced.

Error-reducing redundancy can be introduced by means other than by signal retransmission. English utilizes only a fraction of possible letter combinations, with very asymmetric probabilities and combinatorial options. Grammar and syntax further limit what is an appropriate and inappropriate word string. Last but not least, the distinction between sense and nonsense limits what words and phrases are likely to occur in the same context. This internal redundancy of written English makes typos relatively easy to identify and correct.

Redundancy as a means of information rectification is also relevant to a.s.sessing the reliability of second-order (i.e., referential) information. For example, when one hears multiple independent reports by observers of the same event, the redundancy in the content of these various accounts can serve an a.n.a.logous function. Even though there will be uncorrelated details from one account to another, the redundancies between independent reports make these more redundant aspects more reliable (so long as they are truly independent, i.e., don't reflect the influence of common biases). So, whereas redundancy decreases information capacity, it is also what makes it possible to distinguish information from noise, both in terms of the signal and in terms of what it conveys.

IT TAKES WORK.

What a signal medium can indicate is dependent on the possibility of physical relation with some relevant features of its physical context, and the possibility that this can result in a change of its Shannon entropy. Since reduction in the Shannon entropy of a physical medium is also often a reduction of its physical entropy, such a change is evidence that work was done to modify the signal medium. Recall that work in some form is required to change something from its spontaneous state or tendency to a non-spontaneous state. So, for any physical medium, a contragrade change in its state is an indication that work has been performed. Moreover, a contragrade change can only be produced by extrinsic perturbation; the medium must be an open system in some respect. The reference conveyed by a reduction in Shannon entropy is therefore a function of the ways that the medium is susceptible to outside interference.

Though the external factors that alter a system's entropy (variety of possible states) are not intrinsic features of that medium, the signal constraint is an intrinsic feature. Referential information is in this sense inferred from the form of the constraints embodied in the relations.h.i.+p between unconstrained possibility and received signal. In this way, Shannon information, which is a.s.sessed in terms of this constraint, embodies a trace of the work that produced it.

But openness to the possibility of an extrinsic influence involves specific physical susceptibilities, which also constrain what kind of work is able to modify it, thus constraining the domain of potential extrinsic phenomena that can be thereby indicated. Because of this explicit physical constraint, even the absence of any change of signal entropy can provide referential information. No change in entropy is one possible state. This means that even unconstrained fluctuations may still be able to convey referential information. This is information about the fact that of the possible influences on signal constraint, none were present. It is the mere possibility of exhibiting constraints due to extrinsic influence that is the basis of a given medium's informative power.

This possibility demonstrates that reference is more than just the consequence of physical work to change a signal medium. Although such a relation to work is fundamental, referential information can be conveyed both by the effect of work and by evidence that no work has been done.7 This is why no news can be news that something antic.i.p.ated has not yet occurred, as in the examples of messages conveyed by absence discussed above. The informative power of absence is one of the clearest indications that Shannon information and referential information are not equivalent. This is again because the constraint is both intrinsic and yet not located in the signal medium; it is rather a relations.h.i.+p between what is and what could have been its state at any given moment. A constraint is not an intrinsic property but a relational property, even if just in relation to what is possible. Only when a physical system exhibits a reduction of entropy compared to some prior state, or more probable state, is extrinsic influence indicated. This ubiquitous tendency can, conversely, be the background against which an unaltered or unconstrained feature can provide information about what hasn't occurred. If the sign medium exhibits no constraint, or hasn't diverged from some stable state, it can be inferred that there has been no extrinsic influence even though one could have been present. The relations.h.i.+p of present to absent forms of a sign medium embodies the openness of that medium to extrinsic intervention, whether or not any interaction has occurred. Importantly, this also means that the possibility of change due to work, not its actual effect, is the feature upon which reference depends. It is what allows absence itself, absence of change, or being in a highly probable state, to be informative.

Consider a typo in a ma.n.u.script. It can be thought of as a reduction of referential information because it reflects a lapse in the constraint imposed by the language that is necessary to convey the intended message. Yet it is also information about the proficiency of the typist, information that might be useful to a prospective employer. Or consider a technician diagnosing the nature of a video hardware problem by observing the way the image has become distorted. What is signal and what is noise is not intrinsic to the sign medium, because this is a determination with respect to reference. In both, the deviation from a predicted or expected state is taken to refer to an otherwise un.o.bserved cause. Similarly, a sign that doesn't exhibit the effects of extrinsic influence-for example, setting a burglar alarm to detect motion-can equally well provide information that a possible event (a break-in) did not occur.

In all these cases, the referential capacity of the informational vehicle is dependent on physical work that has, or could have, altered the state of some medium open to extrinsic modification. This tells us that the link between Shannon entropy and Boltzmann entropy is not mere a.n.a.logy or formal parallelism. More important, it demonstrates a precise link to the concept of work. Gregory Bateson's description of information as ”a difference that makes a difference” is actually a quite elegant description of work. Now we can see why this must be so. The capacity to reflect the effect of work is the basis of reference.

TAMING THE DEMON.

This should not surprise us. The existence of an intimate relations.h.i.+p between information and work has been recognized since almost the beginning of the science of thermodynamics. In 1867, James Clerk Maxwell initially explored the relations.h.i.+p in terms of a thought experiment. He imagined a microscopic observer (described as a demon), who could a.s.sess the velocity of individual gas molecules on either side of a divided container and could control a door between them to allow only faster- (or slower-) than-average molecules through one way or the other. Using this information about each molecule, the demon would thus be able to decrease the entropy of the system in contradiction to the second law of thermodynamics. This would normally only be possible by doing thermodynamic work to drive the entire system away from thermodynamic equilibrium; and once in this far-from-equilibrium state, this thermodynamic gradient itself could be harnessed to do further work. So it appears on the face of things that the demon's information about molecular velocity is allowing the system to progressively reverse the second law, with only the small amount of work that determines the state of the door. This seems consistent with our intuition that information and entropy have opposite signs, in the sense that a decrease in entropy of the system increases the predictability of molecular velocities on either side of this divide. For this reason, information is sometimes described as ”negentropy,”8 and has been equated with the orderliness of a system.

Maxwell's demon does not have to be a tiny homunculus. The same process could conceivably be embodied in a mechanical device able to link differences of detected molecular velocity and the correlated operation of the door. In the century that followed Maxwell's presentation of this thought experiment, many sophisticated a.n.a.lyses probed the question of whether the information gleaned by such an apparatus would in fact be able to cheat the second law of thermodynamics. As many a.n.a.lyses were subsequently to show, the mechanisms able to gather such information and use it to open the pa.s.s-through would inevitably require more work than the potential gained by creating this increase of heat gradient, and would therefore produce a net increase in total entropy of the system. The increase in entropy would inevitably exceed the reduction of entropy produced by the demon's efforts.

Although purely theoretical, this a.n.a.lysis validated the a.s.sumptions of thermodynamics, and also made it possible to measure the amount of Shannon information required to produce a given amount of Boltzmann entropy decrease. So, although the demon's activity in transforming differences of molecular velocity into differences of local entropy doesn't ultimately violate the second law, it provides a model system for exploring the relations.h.i.+p between Shannon information and work.

But if we replace the demon with an equivalent mechanical apparatus, does it make sense to say that it is using information about velocity to effect this change in entropy, or is it merely a mechanistic linkage between some physical change that registers velocity and whatever physical process opens the door? Although as external observers we can interpret a signal whose changes correlate with molecular velocity as representing information about that property, there is nothing about the mechanism linking this signal state to the state of the door that makes it more than just a physical consequence of interacting with the signal.

What enables the observer to interpret that the signal is about velocity is the independent availability of a means for relating differences of molecular velocity to corresponding differences of signal state. The differential activation of the door mechanism is merely a function of the physical linkage of the signal-detection and door-operation mechanisms. Molecular velocity is otherwise irrelevant, and a correlation between signal value and molecular velocity is not in any way necessary to the structure or operation of the door-opening mechanism. A correlation with molecular velocity is, however, critical to how one might design such a mechanism with this Maxwellian outcome in mind. And it is cryptically implied in the conception of an observing demon. Unlike its mechanical subst.i.tute, the demon must respond because of this correlation in order to interpret the signal to be about molecular velocity. The designer must ensure that this correlation exists, while the demon must a.s.sume that it exists, or at least be acting with respect to this correlation and not merely with respect to the signal. Thus the demon, like an outside observer, must already have information about this habit of correlation in order to interpret the signal as indicating this missing correlate. In other words, an independent source of information about this correlation is a precondition for the signal to be about velocity, but the signal-contingent door-opening mechanism has no independent access to this additional information.

More important, correlation is not a singular physical interaction, but rather a regularity of physical interactions. A mechanism that opens the door in response to a given signal value is not responding to this regularity but only to a singular physical influence. In this respect, there is an additional criterion besides being susceptible to extrinsic modification that const.i.tutes the referential value of an informing medium: this modifiability must have a general character.

The a.n.a.lysis so far has exposed a common feature of both the logic of information theory (Shannon) and the logic of thermodynamic theory (Boltzmann). This not only helps explain the a.n.a.logical use of the entropy concept in each, it also explains why it is necessary to link these approaches into a common theory to begin to define the referential function of information. Both of these formal commonalities and the basis for their unification into a theory of reference depend on physical openness. In the case of cla.s.sic information theory, the improbability of receiving a given sign or signal with respect to the background expectation of its receipt compared to other options defines the measure of potential information. In the case of cla.s.sic thermodynamics, the improbability of being in some far-from-equilibrium state is a measure of its potential to do work, and also a measure of work that was necessarily performed to s.h.i.+ft it into this state. Inversely, being in a most probable state provides no information about any extrinsic influence, and indeed suggests that to the extent that this medium is sensitive to external perturbation, none was present that could have left a trace.

The linkage between these two theories hinges on the materiality of communication (e.g., the const.i.tution of its sign and/or signal medium). So, in a paradoxical sense, the absent content that is the hallmark of information is a function of the necessary physicality of information processes.

13.

SIGNIFICANCE.