On testing and causal statements

All changes result in unintended consequences.

Whenever we start going down a path of implementing a policy, we need to think about the consequences — both the seen and the unseen.

It is perhaps most importantly the unseen we need to look at. We cannot say that an intervention has achieved the best outcome and that is better than something when we have not actually compared it. So when we look at a risk-reduction process in information technologies or implement a new economic policy or for that matter any other intervention, we need to investigate it fully.

In business, good project managers and, more importantly, portfolio managers will investigate the various results and compare them against what is in effect a null hypothesis. That is, they will compare a sample project against the status quo. When doing so, we need to also contemplate all the alternatives. It is where many people fail.

More importantly, when we are doing correlational studies and investigations that do not allow for experimentally controlled trials (the gold standard of double-blind testing), it is critical that we investigate causation in a rigorous process.

Austin Bradford Hill (1897–1991) was a British medical statistician whose great contribution to science was to leave us with a set of minimal conditions that are required to establish a causal relationship between two events. Hill’s criteria has become the basis of modern epidemiological research. It is one of the mainstay methodologies I use in my research both in evaluating economic effects and when investigating malicious software and other security controls. In particular, it is useful in evaluating the human aspects of information security and risk. It is not just epidemiology but also other fields such as economics that can benefit from such an approach.

Hill’s criteria is the basis of good scientific research where we are seeking to establish a causal relationship amongst social phenomena and in particular ones where we cannot engage in controlled trials. In some instances, it is in fact better than a controlled trial, as the process of creating a controlled trial changes the environment and creates a bias in many of the results. Although it is true that a controlled trial may provide the best answer to a particular problem, it is not always true that we are investigating the same problem. One example would be looking at studies of irrationality. The university-controlled trials testing the reactions of students generally biased the results. In selecting risk trials, for instance, we take selective forms of risk that bias the results towards male or female risk takers in the study. Later studies have now shown that the original studies into irrationality have been the result of poor methodology with both women and men exhibiting similar levels of risk. What was demonstrated is that the forms of risk taking differ between men and women but overall the levels of risk are similar.

If we are to make a claim that population growth results in poverty or that capitalist governments cause poverty in developing nations or even that Keynesian spending results in long-term growth, then to be scientific in our approach, we need to demonstrate a causal relationship. Hill’s criteria provide one of the ways in which we can do so.

For example, as an economic investigation, we can formulate a strategy and hypothesis based on welfare-based systems such as a guaranteed minimum wage.

It seems like a good process, and we have created a safety net. Again, an issue is that we are not thinking of the unseen events, that is, what will occur if there was no welfare? In starting such a welfare system, the differential incentive to work decreases. It is not hard to explain.

Basically, it is not poverty but a differential between income that people see as the greatest disparity. Even in Western nations where there is no need to be poor and in fact where most of the poor are wealthier than the middle class 100 years ago, what we see is a desire for more against the others in society.

So, the end result of such a policy is that more people enter welfare. The immediate result is that those earning need to pay more for the increased welfare state; there are less incentives for productive work. The differential decreases further, making welfare more attractive and creating increased incentives for state support. The result is more welfare… A degrading cycle of creating more and more welfare.

It takes us back to Hill’s criteria. Here we have nine criteria to measure any correlation effects against. The criteria are as follows:

  • A temporal relationship where the cause always precedes the outcome. If there is some factor that is believed to cause an event, then it must always necessarily precede the event. The first criteria is the most critical and essential of all of Hill’s criteria. If the first criteria is not true, then we have a correlation alone and no causal effect.
  • Next, we need to consider the strength of the relationship. It is a statistical measure of the strength where the factors are highly related. We can look at the Pearson number for correlation as a means of testing the value.
  • Next, there is an effect-response relationship. It is a measure of input. As we increase the amount of one factor, the other must also increase. For instance, if we put more time into training people in security awareness, then naturally, for it to be causal in the relationship we would have to have improved security. The improvement is not required to be linear, and we may find that each incremental expense returns less, but it must return something more than it would’ve if it wasn’t there.
  • The fourth relationship is consistency. The results need to be replicable and repeatable. They should apply in different population groups and samples.
  • Next, we look at plausibility. The association that we are purporting exists needs to be supported by a valid theoretical basis. There needs to be some phenomena that can act in a manner that causes the result or event.
  • The sixth criteria is that we consider alternative explanations. Many so-called scientists fail here. They merely assume a relationship matches with their understanding. It may be true that we can dismiss many arguments out of hand as they have already been investigated and shown to be false, but it does not mean that we do not consider alternative explanations. We must always consider multiple hypotheses prior to making any conclusion about a causal relationship between events we seek to explain and investigate.
  • Experimental evidence is also important. Even though we cannot expect to completely re-create an event, we should be able to implement an appropriate experimental regime that supports our causal argument.
  • Next, there is a requirement that the causal effect is specific. It is one of the weaker criteria, and we can demonstrate causal effects without it. The absence of specificity does not negate a causal relationship, but the existence of specificity between associations does add additional support to the existence of a causal relationship. Here it is important to always examine specific causal relationships within a larger systemic environment.
  • Lastly, we have coherence. Ideally, any association we are purporting exists should fit within the body of existing theory and knowledge. There are ways, of course, to introduce new theory, and Thomas Kuhn referred to the changes to the accepted theoretical basis of science as a “paradigm shift.” To reject the existing theoretical basis of science, we need to have particularly good and strong proof and evidence supporting our new claim of causality.

The third of Hill’s criteria, the effect-response relationship, is one that seems to be missing in much of the so-called science we see. For instance, in carbon studies we should see a related increase in atmospheric CO2 leading to a corresponding increase in global temperatures (all other things being equal). We should also see a corresponding and commensurate decrease in global temperatures as atmospheric CO2 levels decline. The matter has been something that is rarely investigated and hence still relegates much of the climate study to pseudoscience.

In a couple of my publications, for instance, we looked at the effects of economic sanctions on criminal groups involved in cyber crime. Two of such papers are:

Criminal Specialisation as a Corollary of Rational Choice” (2012)


Territorial Behaviour and the Economics of Botnets” (2012).

In demonstrating the economic effects of a policy designed to reduce cyber crime, we need to investigate all of Hill’s criteria. In such an instance, what we find is that cyber criminals are rational actors. Like most other people in society when acting individually, they act in their own rational interest. When they are offered opportunities that provide better returns for low risk, they will take such opportunities over the antecedent of a poor return or one with high risk.

Of course, the current way to answer the matter and the mainstay of many Keynesian economists that propagate government circles is to argue that what is rational for the individual may not be rational for society as a whole. They argue that irrationality comes of collective rationality.

Of course, what they are saying is not that the collected actions of the many are in fact irrational but that people choose things that they did not desire. It is a typical political fallacy that is designed to appear scientific. Many economists make very poor scientists.

Putting forward such an argument is of course rational in itself. For when we see government favouring Keynesian-aligned big-government economics and rewarding those who support the idea of big government, we also see incentives given by government to those who promote big government. So the individual behaviour of such rational economists is to support an irrational policy itself. It is of course rational to support a biased policy when you gain from it at the expense of others.

What could be termed irrational is a libertarian policy that exists with little support and certainly none from government. As a libertarian, one has to fight harder and do more. But here, it again comes to what is a subjective value. When comparing rationality, we need to look at the values of the individual. In my case, my greatest value is freedom, and it cannot be given through big government and subjugation. In this case, economic constraints and an acceptance of lower benefits come as the cost of upholding one’s values.
One further part of this rant is that we need to start accepting that it is not irrational to hold one’s values. In fact, the only rational choice is to uphold one’s values. As such, we start to see where one’s values lie.

Never miss a story from Craig Wright (Bitcoin SV is the original Bitcoin)