Uncertainty and Bayesian probability

bayesian_processThis post is to address the relation between the uncertainty on the state of a system with the information we have of it. Said in those terms is kind of obvious, the more information the less uncertainty. More information can take the form of knowing some other aspect of the system or being more precise information on previously acknowledge aspects.

In mathematics the way to deal with this kind of problem is probability and the way probabilities change when additional data is taken into account is the law of Bayes, hence Bayesian probabilities.

To illustrate how this works I’m going to go through an example. Lets imagine we have a tank of worm water, like a bath tub, and we want to know what temperature the water in the tank has. We could stick a thermometer in the tank and measure its temperature, assuming the temperature of the water in the tank is well mixed then that would yield the temperature of the tank with the uncertainty characteristic of our measuring device.

simple_measurement
Fig. (1) The uncertainty distribution of a simple measurement of the water in the tank

If we are using a mercury thermometer the uncertainty would be around \pm0.2\textrm{ }\textdegree\textrm{C}. Measuring devices’ uncertainties are usually assumed to be normally distributed unless specific evidence is available. Usually the uncertainty level used is 95% or 2\sigma (two standard deviations). Meaning that when I say that the water in the tank is at 36.9\pm0.2\textrm{ }\textdegree\textrm{C}, it means I’m 95% certain that the temperature is between 36.7 and 37.1. The probability distribution of Fig. (1) shows the exact meaning, we are more confident the closer we get to the mean value.

To see how our knowledge of the system varies when further information is taken into account we are going to consider that this tank is not alone in the universe but it interacts with it.

Lets make the tank be in thermal equilibrium by adding a hot water inlet and a outlet, configured in such a way that the level of water is constant. Lets assume the tank is well insulated on all of its lateral walls and floor but open to the room temperature air on its top surface. For this system to be in equilibrium the mass flows of the inlet and outlet must coincide and the incoming heat through the inlet must be equal to the heat losses to the ambient.

This system is easily described with a simple equation relating the variables in play:

mc_p(T_{in}-T)=Ah(T-T_{amb})                                   (1)

Where:

  • m is the mass flow.
  • c_p is the specific heat of the water, which we are going to take as a constant known with absolute certainty to be c_p=4.187\quad\frac{\textrm{kJ}}{\textrm{kg}\textdegree\textrm{C}}   .
  • T_{in} is the temperature of the water comming into the tank.
  • T is the temperature of the tank.
  • A is the area of the surface water of the tank, which we also will assume is a perfectly known quantity T_{in}=0.64\quad\textrm{m}^2
  • h is the convection film coeficient, which we’ll assume is h=10 \quad\frac{\textrm{W}}{\textrm{m}^2\textdegree\textrm{C}}.
  • T_{amb} is the ambient temperature of the air in contact with the water surface.

Lets assume we are measuring the inlet temperature, the ambient temperature and the mass flow, as follows:

  1. T_{in}= 58 \pm 0.2\quad \textdegree\textrm{C}.
  2. T_{amb}= 17 \pm 0.2\quad \textdegree\textrm{C}
  3. m= 0.0015 \pm 3.7E-5\quad \frac{\textrm{kg}}{\textrm{s}}

Having these measurements gives us a pretty good idea of the temperature of the tank even before explicitly measuring it. Fig. (2) shows this probability, it turns out we already know the temperature of the tank is 37.257 \pm 0.52\textrm{ }\textdegree\textrm{C}. It is less precise that the direct measurement but the monitoring of the interactions with the outside does provide an estimate of the temperature of the system.

pdf_T_prior
Fig. (2) Prior probability of the state of the tank, before measuring it explicitly.

Now lets consider what happens when we measure the temperature and the measurement device shows like before 36.9\pm0.2\textrm{ }\textdegree\textrm{C}. Since our prior knowledge of the state of the system is different, in the previous case we didn’t know anything of the system before hand, now we believe the temperature is 37.257 \pm 0.52\textrm{ }\textdegree\textrm{C}. How this influences the ultimate state of our knowledge, after the measurement?

Bayes’ theorem provides the method that allows us to to update beliefs when new evidence arrives (more on that on wikipedia). Applying the theorem to the example at hand we find that the prior beliefs modify the end result bringing the mean a little towards the mean of our prior beliefs and reduces a little the uncertainty of the measurement. Figure (3) shows how our prior probabilities (blue) bring the measurement (green) slightly towards the right to, transforming our prior believes (blue) to to our later, more precise, believes (red). The final state of knowledge of the system is 36.946 \pm 0.187\textrm{ }\textdegree\textrm{C}, the uncertainty has gone down from 0.2 to 0.187 because of our prior knowledge, moving the mean from 36.9 to 36.946.

bayesian_process
Fig. (3) The Bayesian process. Our prior believes (blue) influence our final state of knowledge (red) after measuring the system (green). It doesn’t only modify the most likely value but also reduces the uncertainty of the measurement.

Joseph’s tiling the court yard problem

An example of a problem involving the resolution of Pell’s 2nd order diophantine equation

Problem statement

Joseph is planning to tile his square court yard with square tiles. He has requested an offer from his two friends Frank and Peter. Frank proposes to use tiles with an area 17 times bigger than Peter. Knowing that Peter’s tiles cover exactly the whole floor of the court and that Frank’s on the other hand fall short for the area a one of Peter’s tiles, how many tiles are the minimum that Peter and Frank, each have proposed in their respective offers?

Solution:

Let and be the number of tiles used by Peter and Frank respectively (we know they must be perfect squares).

Let A be the area of Peter’s tiles and 17A the area of Frank’s, then:

  • The total area of Peter’s tiles is Ax².
  • The total area of Frank’s tiles is on the other hand 17Ay².

Since the difference in Frank’s and Peter’s total tiles’ areas is the area of one of Peter’s tiles, we have:

Ax² – 17Ay² = A

, which when dividing each side of the equation by A, yields:

x² – 17y² = 1                    (1)

The problem then consists is finding the positive integer roots of equation (1).

This is a particular case of Pell’s equation: x² – Dy² = 1 where D is a positive integer. It is a particular type of 2nd order diophantine equation. In particular it has infinite solutions over the integers for any value of D. The proof of this amazing fact can be found here

With modern computing, an equation like this is easily solved by brute force, just by trying all possible integers starting from 1. None the less if D was some big number it would still be of interest to implement some better algorithm. So I am going to solve the problem by hand to illustrate the procedure.

First we rearrange equation (1):

\frac{x^2}{y^2} = \frac{1}{y^2} + 17

where we notice that on the left side of the equation as y increases even slightly  1 /y² << 17, meaning x² / y² is roughly 17 or x / y is roughly the square root of 17. This insight suggests that we could try and find rational approximations to the irrational quantity \sqrt{17} and search for the solution among the numerator and denumerators of such fractions.

To find rational approximations we use the continued fractions technique. It has been proven that the solution is always within the set of continued fractions, or convergents of the \sqrt{D} in the general Pell’s equation (also in the previous link)

To calculate the continuous fractions of \sqrt{17} we proceed:

  1.                                                  17 = (4 + z)² = 16 + 8z +z²
  2.                                                    1 = 8z + z² = z (8 + z)
  3.                                                             z=1 / (8 + z)

This last expression can be used recursively to calculate the value of z, replacing z once and again by  \frac{1}{8+z}. These are the continued fractions, so:

\sqrt{17}=4+\cfrac{1}{8+\cfrac{1}{8+\cfrac{1}{8+\dots}}}

as we continue expanding the terms we get closer and closer to \sqrt{17}, each of the approximations is called a convergent. So the first convergent would be 4, the second 4+\frac{1}{8}=\frac{33}{8}, …

Lets try these convergents as a solution of (1)

  • first convergent: 4=\frac{4}{1} so  x = 4,  y = 1 which gives  x² – 17y² = -1
  • second convergent: \frac{33}{8} so x = 33, y = 8 and x² – 17y² = 1

Eureka!! we have found the solution and we only had to try to numbers. Petter will use 33²   = 1069 tiles and Frank 8² = 64.

I hope it was interesting for you, it surely was for me. It is almost magic how the integer solutions of the equation form a rational approximation of an irrational number… who would have thought.