Co-Developing Causal Graphs with Domain Experts Guided by Weighted FDR-Adjusted p-values
Eli Y. Kling

 
 
 
   
    









1]Eli Y. Kling
Email: eli.kling@avanade.com
Alternate Email: eli_kling@hotmail.com
LinkedIn: https://www.linkedin.com/in/elikling/https://www.linkedin.com/in/elikling/
[1]Avanade, London, UK




indentpara
    2em
    


plainnat



Co-Developing Causal Graphs with Domain Experts Guided by Weighted FDR-Adjusted p-values
    [
    September 2024
========================================================================================



§ ABSTRACT 
This paper proposes an approach facilitating co-design of causal graphs between subject matter experts and statistical modellers. Modern causal analysis starting with formulation of causal graphs provides benefits for robust analysis and well-grounded decision support. Moreover, this process can enrich the discovery and planning phase of data science projects.

The key premise is that plotting relevant statistical information on a causal graph structure can facilitate an intuitive discussion between domain experts and modellers. Furthermore, Hand-crafting causality graphs, integrating human expertise with robust statistical methodology, enables ensuring responsible AI practices. 

The paper focuses on using multiplicity-adjusted p-values,  controlling for the false discovery rate (FDR), as an aid for co-designing the graph. A family of hypotheses relevant to causal graph construction is identified, including assessing correlation strengths, directions of causal effects, and how well an estimated structural causal model induces the observed covariance structure.

An iterative flow is described where an initial causal graph is drafted based on expert beliefs about likely causal relationships. The subject matter expert's beliefs, communicated as ranked scores could be incorporated into the control of the measure proposed by Benjamini and Kling, the FDCR (False Discovery Cost Rate). The FDCR-adjusted p-values then provide feedback on which parts of the graph are supported or contradicted by the data. This co-design process continues, adding, removing, or revising arcs in the graph, until the expert and modeller converge on a satisfactory causal structure grounded in both domain knowledge and data evidence.





§ INTRODUCTION


The current common practice for conducting data science development projects is to kick off with a "Discovery" phase, where the problem is defined. This phase is often conducted in a workshop setting, where the data scientist and the subject matter expert work together to specify the target, postulate potential drivers or features and identify the data sources. Many frameworks specify a 'hypothesis workshop' as part of the kick-off and shaping steps. These sessions are usually conducted before the statistical modeller sees the data and are typically meant as a vehicle for identifying and prioritising relevant data sources. This practice encourages blowing up the list of features to "through" at a model. Automatic feature selection and modelling tools have been developed to address the tsunami of features.

The increasing reliance on complex models and automated data-mining systems raises important questions about interpretability, inference and correct methodology implementation [<cit.>]. For example, <cit.> voices a paradigm common to statistical modelling and data mining: perform an automatic variable selection and then allow the expert to overlay the business or physical context. This practice results in models that are not easily explained. Thus also posing a challenge to implementing responsible AI guidelines.

The issues and concerns above are exacerbated when the models are used to drive decisions and actions. The nature of many challenges in the business and manufacturing worlds is such that it is often impossible to design experiments to cleanly assess the impact of decisions and actions on outcomes. For instance, the Market Mix Methodology (MMM) attempts to attribute success to marketing activities in a world of confounding co-activities and poor data [<cit.>]. In cases where experimentation is not possible, there is a risk that misspecification and misuse of confounders will not only result in wrong action-to-outcome impact estimates but could also indicate the wrong direction of effect, as demonstrated by Simpson's paradox. As <cit.> state, “The Simpson's paradox is not so much of a paradox but rather a warning of how sensitive causal reasoning can be with respect to model misspecifications.” A robust approach to address these situations is to deploy techniques developed in the field of statistical causality analysis [<cit.>].

Modern statistical causal analysis, exemplified by Pearl and Mackenzie <cit.>, begins with the formulation of a causal graph. This approach offers several benefits to the quality and robustness of analysis and decision support. The causal graph serves as an intuitive tool that bridges the knowledge and beliefs of a subject matter expert (SME) with the statistical modeller's data-driven insights. More importantly, Pearl <cit.> emphasises that mapping causal mediating factors illuminates 'intrinsic properties of reality that have tangible policy implications'.

<cit.> discusses the tension between the two analysis goals explainability and productiveness.Each has a different starting point and nuances of the methodology used. Specifying correctly a causal graph has the benefit of expalianbility and bringing to the for potential introduction of bias that is not compliant with responsible AI guidelines. Once an explainable model is crafted the statistical modeller may proceed to fit a prediction oriented model guided by the insights of the discovery step. This is in contrast to <cit.> "Using complex predictors may be unpleasant, but the soundest path is to go for predictive accuracy first, then try to understand why"

A growing body of research explores graphical methods to visualise and communicate the results of causality analysis. For instance, <cit.> discuss supporting lay users with no specific expertise in machine learning, promoting an interactive approach aimed at "emotionally connecting" the subject matter expert.

While the field of automatically discovering causal graphs is active [<cit.>], this paper focuses on an approach to aid discussion between the statistical modeler and the subject matter expert while constructing a causality graph. The discussion explores the formation of a causality graph as a tool to be used during the discovery and exploratory data analysis phase.

This paper posits that plotting relevant information on a causality graph facilitates discussion between the statistical modeller and the subject matter expert. This approach does not argue for setting aside rigour for the sake of simplifying the discussion. The False Discovery Rate (FDR) or False Discovery Cost Rate (FDCR) is an appropriate measure for introducing multiplicity considerations in an intuitive way. Figure <ref> demonstrates that this type of presentation is self-explanatory and suitable for both systematic-level discussions with subject matter experts and modelling-level discussions with trained statisticians.



This paper proposes an approach to guide the construction of a causality graph that will underpin inference, estimation, and predictions. For simplicity but without loss of generality, the approach describes stepwise crafting of a Structural Equation Model (SEM) or Structural Causal Model (SCM) where the adjustment of the False Discovery Rate (FDR) is deployed as an aid and guide [<cit.>].

The paper starts with discussing which hypotheses are relevant to the co-design of a causality graph (Section <ref>). Once a family of hypotheses is defined, Section <ref> describes the control of the False Discovery Rate (FDR) and its generalisation to FDCR using costs of false alarms. Section <ref> demonstrates the use of such adjustment in the construction of a Structural Causal Model (SCM) using a toy example. The paper concludes with Section <ref>, touching on thoughts not explored here.



§ IDENTIFYING THE FAMILY OF HYPOTHESES



This paper follows a paradigm of collaboratively working with subject matter experts (SMEs) to translate their expertise and experience into a hypothesised causality graph. It is natural to transform this process into a set of hypotheses that reflect the SME's beliefs. The next logical step is applying some multiplicity control. The False Discovery Rate (FDR) is a natural multiplicity measure for this setting.

Before delving into multiplicity error measures and their control, it is important to define the family of hypotheses and understand the co-dependence between the hypotheses, the statistics, and the resulting p-values. [<cit.>] discuss using Pearl's theorems on the properties of conditional independence relations [<cit.>] and avoiding the execution of some statistical tests to reduce the computational load. A note of caution: avoiding calculating the test does not exclude the hypotheses from the family if it is pertinent to structuring the causality graph unless the correlation to another hypothesis to another hypothesis is 1.

In the context of assessing of a causality graph, hypotheses are postulated based on SME's experience, their prior beliefs, and suggestions derived from explanatory analysis. As <cit.> points out, the hypotheses focus on the discovery of edges in the Directed Acyclic Graph (DAG). 

The identification of the relevant hypothesis is somewhat derived from the typical co-design process. Note how the choice of modelling is incorporated here:


    
  * Identify the outcomes of interest and decisions that are likely to affect the outcomes
    
  * Use the Six Sigma fish–bone diagram to list potential drivers of the outcomes
    
  * Obtain, clean, and prepare data for the information listed in the fish–bone diagram
    
  * Calculate the correlations and their corresponding p-values (The null hypothesis for each pair is that there is no correlation or co-dependence)
    
  * Assign a causal direction to each pair (could also be 'no causal relationship')
    
  * Assign a belief score to each postulated causal relationship. For instance:
    
        0 = Causal relation is not possible
        
        1 = Could be a causal relationship
        
        2 = It is likely this is a causal relationship
        
        3 = Known causal relationship
        
    
    
    
  * Using the causality graph defined by (4) & (5), fit a Structural Causal Model (SCM). This naturally defines a sub-family of hypotheses:
    
        
  * The model does not explain the data (tested with a set of fit statistics)
        
  * The model coefficients are zero
        
  * The mean of the residuals is zero (could also hypothesise on the variance of the residuals)
    
    
  * A further set of hypotheses are based on the assertion that the correlations induced by the SCM in (7) are the same as seen in (4). This is in line with the field of structure learning. For example, <cit.> advise that a DAG should be as close as possible to the global dependence structure


Step (8) is linked notionally to the definition of faithfulness. A graph is faithful to some distribution if the graph connectivity represents exactly the dependencies and independences dictated by the distribution (see for instance, <cit.>.)

Moreover, rather than examining the correlations, it makes sense to work with covariances so that the direction is also considered.

The hypotheses in (8) are an important feedback to the co-design process as 'unattended' correlations would be highlighted. However, the way (8) is formulated above results in counter-intuitive p-values (big is good) and does not really prove the SCM is reflecting the covariances and variances. Rather, it only shows that there is no evidence to refute that it does. (8) should be couched similarly to equivalence testing, where the null hypotheses are that the covariances and variances induced by the SCM are different from the observed correlations:


    H_0^i: |ρ_scm^i - ρ_data^i| ≥δ


    H_1^i: |ρ_scm^i - ρ_data^i| < δ


Where, for n variables, ρ_data^i (i = 1, …, n(n+1)/2) are the observed correlations in the data. ρ_scm^i (i = 1, …, n(n+1)/2) are the matching correlations induced by the SCM and δ is a threshold parameter. It is customary to form the above as two one-sided hypothesis tests with two p-values that should be small if the two measures are similar.

Examining the p-values in (7) and (8) informs suggested changes to the causality graph. Steps (7) and (8) are repeated until the Modeller and the SME are satisfied. Thus, multiplicity of hypothesis control is called for. As the process could be viewed as a screening process where the proportion of true null hypotheses is small, the control of the False Discovery Rate (FDR) is appropriate. I.e., the p-values reviewed during the process should be FDR adjusted. We propose that due to the consistency property of the FDR adjustment, it is sufficient to consider the adjustment for each iteration separately. Moreover, the FDR adjustment provides a mechanism for inclusion of the belief scores where they can define the weights for a weighted FDR adjustment.

Another source of hypotheses comes from the field of causality structure learning where conditional independence tests underpin a stepwise (forward or backward) causality graph construction. This paper is concerned with "manual" co-design of the causality graph rather than an automated, data-driven algorithm. Thus, the hypotheses in (8) are sufficient for driving and informing the discussion between the SME and the Statistical Modeller, although not as nuanced as the conditional independence tests.

It is important to understand the correlation structure among all the p-values used, as some of the procedures for the control of the FDR (e.g. the Bejamini-Hochberg procedure [<cit.>]), control the FDR only under independence or under Positive Regression Dependence (PRDS). In cases where complex correlations are suspected, it is possible to use the Benjamini-Yekutieli algorithm [<cit.>]  or bootstrap as demonstrated by <cit.> using the foundations laid by <cit.>.

A nuance of the above requirement for independence or at least PRDS is that it is sufficient to show it for the true null hypothesis. As the hypotheses pertain to the existence of a causal effect manifested by conditional dependence, the true null hypotheses that could be interdependent are those considering the same edge. Thus, there might be a correlation for the p-values assessing the coefficient for modelling an edge and the p-value for the correlation induced by the SCM for that edge. Arguably, they are measuring overlapping constructs and thus are PRDS. However, the hypotheses in (8) are formulated as an equivalence between the correlation structure induced by the SCM and the observed correlation. For a world where A does not directly cause B, the true causality graph will not have an edge linking nodes A and B (edge AB). For an SCM containing an edge AB, the hypothesis that the coefficient for the regression of B on A is zero is true. However, the hypothesis that the induced correlation between A and B is equivalent to the observed could be true or false as the correlation may be induced. Simple cases where a null coefficient is fully informative of the existence of the correlation can be constructed. Arguably these are simple cases where a model crafting session is not required. Therefore, the p-values considered here could be treated as independent or PRDS. 

For illustration purpose, consider a simple causality Graph with one mediator T → M; T → O; M → O. Assuming T, M & O are continuous and the functional form linking them is linear, the Structural Equation Model (SEM) could be:


    M̂ = β_0 + β_1 T + ε_1


    Ŷ = β_2 + β_3 T + β_4 M + ε_2


Where ε_j ∼ N(0, σ_j); j = 1, 2

The hypotheses of interest are:


    
  * H_0,i^Parms: β_i = 0;
    
    H_1,i^Parms: β_i ≠ 0;
    
    ∀ i ∈{0, 1, 2, 3, 4}
    
  * H_0,j^Noise: ε_j ∼ N(0, .);
    
    H_1,j^Noise: ε_j ≁N(0, .);
    
    ∀ j ∈{1, 2}
    
    
  * H_0,x,y^Cov: |SEM induced Cov(x,y) - observed  Cov(x,y)| > δ; 
    
    H_1,x,y^Cov: |SEM  induced  Cov(x,y) - observed  Cov(x,y)| ≤δ; 
    
    ∀ x, y ∈{T, M, Y} – upper triangle & diagonal
    


(i) asserts that the coefficients are not zero. For simple linear modelling, the p-values could be theoretically derived; (ii) tests the assumption that the noise is Gaussian distributed. This could be tested using a Chi-square test. It could be argued that the roles of H_0 and H_1 should be flipped, posing a challenge of testing for "non-normality".

The process we describe is iterative. At each step, edges could be added or removed and the coefficients refitted (either backwards or forwards construction). It could be argued that the hypotheses of all the steps should be considered as one family. The consistency property of the FDR (when Number of True null hypotheses: n_0 ≪ Number of false null hypotheses: n_1) allows for working within iteration. Thus circumventing confusion due to having several p-values generated for the same hypothesis. Therefore, It is sufficient to control the FDR within each iteration.

Before providing an example for the process, the FDR and the False Discovery Control Rate (FDCR) and their control are discussed in the next section.




§ FALSE DISCOVERY [COST] RATE



This section is based on technical papers by <cit.> and <cit.>.

When testing several hypotheses simultaneously to reach an overall decision, there is a trade-off between controlling the type I error for per-hypothesis considerations and the overall hypothesis, which is usually their union. This issue of balancing error rates is known as the Multiplicity Problem. This dilemma is encountered almost universally where statistics are applied. Some of the most prominent statisticians have addressed it. However, there is neither a common nor global approach. For instance, <cit.> discussed a special case of the problem, pairwise comparisons. He continued discussing it as late as 1991 [<cit.>]. An extensive overview of the research of the Multiplicity problem may be found in <cit.>, <cit.>, <cit.>, <cit.>, and <cit.>.

<cit.> demonstrated this: "For example, a particular survey may identify a small p-value, say p = .005, and claim that the associated effect is 'statistically significant.' This p-value is interpretable as follows: when there is no causal basis for the effect, there is only a 0.5% probability of observing a result as extreme as the observed result. On the other hand, it is possible that the multiplicity adjusted p-value is .15 (Adj-p = .15), which is not statistically significant. This adjusted p-value incorporates the multiple tests performed and can be interpreted as follows: when there is no causal basis for any effect tested, there is a 15% chance that somewhere in the experiment a result as extreme as the observed result of .005 will appear."

Classical procedures aim to control the probability of committing at least one type-I error when considering a family of hypotheses simultaneously to control the multiplicity effect. The control of this Familywise Error Rate (FWE, see Equations <ref> & <ref>) is usually required in the strong sense (Equation <ref>), i.e., under all configurations of true and false hypotheses [<cit.>]. The main problem with classical methods is that they tend to have low power. Consequently, it has been argued that no special control is needed (e.g. [<cit.>]).

An alternative, more powerful measure was introduced by <cit.>: the False Discovery Rate (FDR, Equation <ref>). It is an appropriate error rate to control in many problems where the (strong) control of the FWE is not needed. The FDR is the expected ratio of the number of erroneous rejections to the number of rejections (discoveries) and is equal to or less than the FWE. The two error rates are equal when the number of true null hypotheses (m_0) equals the number of hypotheses under test (m). When m_0 < m, the FDR may be substantially lower than the Familywise Error Rate, so an FDR controlling procedure at conventional levels can be more powerful. <cit.> provided a linear step-up procedure (BH) that controls the FDR for independent test statistics. <cit.> showed that when the test statistics are PRDS correlated, the BH procedure controls the FDR. Furthermore, they introduced a resampling-based procedure that controls the FDR [<cit.>].

A useful aspect of the BH procedure is that it provides quite consistent discoveries when applied to subsets of the family of hypotheses. For example, when evaluating individual hypotheses pertaining to enumeration districts, the family could be the whole of the United States or just a specific state. FWE control procedures usually will provide conflicting individual decisions for the different hypotheses families, whereas the FDR control will generate consistent discoveries when m_0 ≪ m.

<cit.>, <cit.>, <cit.>, and <cit.> frame the control of the FDR using a Bayesian paradigm. This approach allows the discussion of a prior belief on the probability that the null hypothesis is correct. This is somewhat related to the discussion in this paper of assisting experts in mapping out their beliefs on causality.

Following the notation used by <cit.> and <cit.>: For a composition of m sub-hypotheses (H_0i; i=1,2,…,m) let the intersection hypothesis be H_00 = ⋂_i=1^m H_0i. Let R_i (i=0,1,2,…,m) be 1 if H_0i is rejected and zero otherwise; and let V_i (i=0,1,2,…,m) be 1 if H_0i is erroneously rejected and zero otherwise. Note, if H_0i is true then V_i=R_i. Furthermore, let the number of rejections ("Discoveries") be R = ∑_i=1^m R_i, and the number of erroneous rejections be V = ∑_i=1^m V_i. Note that R and V do not include R_0 and V_0. Let I_R=1 when R>0 otherwise I_R=0. Similarly, I_V=1 when V>0 otherwise I_V=0. To complete the notation used in this paper, define I_0 as the set of indices of the true null sub-hypotheses (I_0={j;H_0j is true, 1 ≤ j ≤ m}); and the number of true null sub-hypotheses is m_0=||I_0||.

The family error measures could be defined using the above terms:


    Strong-FWE   = P(V>0) = E[max V_j; 1≤ j≤ m] 
    Weak-FWE   = E[V_0] = E[V_0/R_0] 
    FDR   = E[V/R] 
    Weighted FDR   = WFDR = E[∑_i=1^m w_i V_i/∑_i=1^m w_i R_i]


where V/R and V_0/R_0 are defined as zero when V=R=0 and V_0=R_0=0 respectively.

The FWE is appropriate when even one erroneous discovery is not desired. Procedures such as the Bonferroni procedure, Holm's procedure, Hochberg's Procedure [<cit.>], and Tukey's  T-method for pairwise comparisons [<cit.>], all control the FWE in the strong sense (Equation <ref>). This type of control is relevant to situations where any erroneous discovery implies a very high cost. For example, such conservativeness is required when examining the primary end-points during Phase III clinical trials.



Control of the FWE in the weak sense (Equation <ref>) is achieved by testing directly the intersection null hypothesis (and not controlling the individual hypothesis). For instance, by using the multivariate Hotelling T^2 statistic. Thus, the overall type I error rate is controlled only when all the sub-hypotheses are true (m_0 = m). This situation is very common in Statistical Process Control (SPC); where once an out-of-control signal is given (the intersection hypothesis is rejected) it is assumed that it is no longer necessary to protect from erroneous sub-discoveries, and on the other hand increased power is desired. 

Generally, the use of the FDR (Equations <ref> & <ref>) is appropriate in situations where high power is imperative and a pre-specified percent of the wrongly rejected (individual) hypotheses is tolerable and does not affect the quality of the overall decision. This is usually characterised by the belief that m_0 ≪ m. The FDR may be used in pilot or screening studies, grouping analysis, and situations where many variables are considered such as data-mining and Bioinformatics. In such situations, the error from a single erroneous rejection is not always as crucial for drawing conclusions for the family hypothesis. Thus, we are ready to bear more errors when many hypotheses are rejected, but with less when fewer are rejected. The last notion is reflected by the control of the FDR - for which one must specify the acceptable expected proportion of wrong discoveries. For an example, see <cit.> for a review of papers that proposed adopting the FDR for multiple CUSUM charts.

Though widely researched, there are no clear-cut rules for which error measure to use and what multiplicity control procedure to use. <cit.> and <cit.> argued that this decision is part of the modelling process. They show that in situations where costs or weights can be attributed to erroneous discoveries, the weak-FWE and the FDR are special cases of a generic cost-based error measure, the False Discovery Cost Ratio (FDCR, Equations <ref>). They show that the W-BH procedure [<cit.>] keeps its control also when the test statistics are PRDS ON I_0 and propose a procedure for the control of the FDCR when the test statistics are PRDS.

Assigning variable cost of C_i (i=0,1,2,…,m) of an individual erroneous discovery (e.g., rejecting H_0i results in stopping machine i for maintenance) and a fixed cost C_0 for the overall discovery (rejecting H_00 results in calling in an engineer), the cost of false discoveries is C_0 I_V_0 + ∑_i=1^m C_i V_i. In the spirit of the FDR and using the above notation, the expected proportion of the cost of false discoveries is:


    E[C_0 I_V_0 + ∑_i=1^m C_i V_i/C_0 I_R_0 + ∑_i=1^m C_i R_i] ≤  E[C_0 I_V + ∑_i=1^m C_i V_i/C_0 I_R + ∑_i=1^m C_i R_i],


where the proportion is zero when the denominator is zero and I_R=I_{R>0} and I_V =I_{V>0}. 

<cit.> define the False Discovery Cost Rate as the expected ratio of the cost wasted due to erroneous discoveries to the total cost related to the discoveries:


    FDCR = E[C_0 I_V_0 + ∑_i=1^m C_i V_i/C_0 I_R_0 + ∑_i=1^m C_i R_i] = E[∑_i=0^m C_i V_i/∑_i=0^m C_i R_i],

where the proportion is to be defined as zero when the denominator is zero.

The Weak FWE, the FDR, and the W-FDR are special cases of this measure obtained by specific structures of the costs. When C_0=0 the FDCR is the W-FDR. The FDR is obtained by further assigning equal costs to the m hypotheses. On the other hand, when the cost consists only of C_0 the FDCR is the FWE in the weak sense. 

It is interesting to examine the meaning of controlling the FWE in the strong sense from the viewpoint of the FDCR. Seeking to control the probability of making any false rejection suggests that every erroneous discovery is very costly and perceived as infinite. Thus, the Strong-FWE can be approximated by the FDCR, but it cannot be expressed in the context of additive costs. The Strong-FWE may be expressed in terms of "relative cost", E[max(v_i)/max(R_i], which does not render itself easily to economic interpretation.

<cit.> propose testing simultaneously H_00 and the rest of the sub-hypotheses, using the FDCR to correct for multiplicity. In particular, testing H_00 through Hotelling's T^2, weighted Simes' statistic (Equation <ref>), and Fisher's statistic (Equation <ref>)


    P^ws = min_j [∑_i=1^m c_i/∑_i=1^j C_(i) P_(j)].



    P^F = -2∑_i=1^m ln p_i.


They showed that an FDCR controlling procedure could be constructed by applying the W-BH procedure to the m+1 p-values P_0, P_1, …, P_m, where the weighted Simes adjusted p-value for P_0 (corresponding to H_00) is used. Thus, the control of the FDCR could be achieved when the p-values associated with the m_0 true hypotheses are independent or PRDS.



§ CONSTRUCTING A CAUSALITY GRAPH WHILE CONTROLLING THE FDR



The process of constructing a causality graph and a statistical model should involve a statistically trained data scientist and a subject matter expert (SME) working through a process similar in spirit to variable selection. They identify the target variables, potential explanatory variables, and assign beliefs to the direction of causality. These beliefs could be expressed as a score. Then the statistician prepares the data for analysis and iteratively constructs the causality graph and models with the SME.

<cit.> propose an approach for variable selection in linear models that controls the False Discovery Rate (FDR) where the discoveries are the selected variables to be included in the model. They point out that in many practical situations, there are only a few relevant variables among the many recorded. It makes sense to view the process of constructing the causality graph as a systematic screening of variable selection hypotheses pertaining to strengths of relationships and direct and indirect causality effects.

To illustrate such a process, a synthetic example was generated. The demonstration will iteratively create a Structural Causality Model where feedback to the SME is a graphical representation overlaid with the False Discovery Cost Rate (FDCR) adjusted p-values. The toy example describes an offshore wind farm operator who decides how fast to allow the turbine to turn. The key causality relationships of interest are "Rotational_RPM" → "Energy_Yield" and "Rotational_RPM" → "Perceived_Noise". Appendix A describes the construction of the toy example. Table <ref> lists the variables the SME suggested could have bearing on the outcome. 20,000 observations were generated.



The univariate exploratory analysis kicks off the co-design process, facilitating conversations between the statistician and the SME. The next step is to evaluate the pairwise relationships. Table <ref> lists the Pearson pairwise correlations, and Table <ref> lists the FDR adjusted (BH adjusted) p-values obtained through bootstrapping the Pearson R^2 [<cit.>].






Reviewing the correlations and the adjusted p-values is a good starting point for designing the first draft causality graph and attributing the SME's belief. For this example, the belief is scored simply:





    
  * As an SME, I do not believe there is a causal relationship between the two variables – this is implicit by not drawing an arc in the causality graph
    
  * As an SME, I am not sure whether there is a causal relationship between the two variables
    
  * As an SME, I believe there is a causal relationship between the two variables
    
  * As an SME, I am sure there is a causal relationship between the two variables




The C_i weights for the FDCR are set to 1/(belief+0.0001). All unprovided weights, such as for the tests for the intercept and C_00 (Weighted Simes), are set to 1.

Assume the SME provides the following beliefs:



Figure <ref> sets the legend for the graphical representation of the results. The estimates and FDCR adjusted p-values are presented alongside the edges. The non-significant adjusted p-values are highlighted as those mark where the SME and statistician should focus their discussion.



Figure <ref> shows a possible outcome where the significant correlations are accounted for, and the direction of the causation is informed by the SME's experience. The covariances and variances induced by the SCM model and their corresponding FDCR adjusted p-values are presented in Figure <ref>.






The next iteration takes into account that:


  * The estimate for Strength_degradation → Rotational_RPM is practically zero, albeit statistically significant.

  * The covariance between sea_temperature and Perceived_noise generated by the model is not equivalent to the measured relationship.

  * The covariance between wind_speed and Perceived_noise generated by the model is not equivalent to the measured relationship.



After several iterations, the model could look as presented in Figures <ref> and <ref>.







§ CONCLUSION




This paper demonstrated how the FDR and FDCR could be put to use in the context of co-designing a causality graph in the discovery step of an analytical project. 

This approach favours hand crafting the causality graph but does not preclude using concepts and tools developed for automatic causality discovery. The use of FDR control during model discovery has been proposed by several researchers. For example, algorithms such as FDR-IAMB [<cit.>] and FDR-IAPC [<cit.>] & [<cit.>] apply techniques for controlling the multiplicity of hypotheses. These algorithms aim to learn the structure of the causality graph from the data and use FDR control as a weighting for a greedy algorithm.

The primary focus lies in facilitating communication between statistical modellers and subject matter experts (SMEs) by balancing statistical rigour with clear and intuitive presentation of results. One potential improvement would be to replace p-values with e-values as suggested by <cit.>] and <cit.>.

While the example used employs linear regression for modelling causal relationships, the discussion is not limited to this specific functional form. Statistical modellers have the flexibility to choose the modelling approach for conditional probabilities and select appropriate measures of variable (parent node) contribution. For instance, a random forest might be used as the functional representation, with variable importance as the contribution measure.

The hypothesis testing framework described could be generalised. Instead of focusing on a specific parameter value, the null hypothesis could be formulated as: "the parent variable does not contribute to modelling the child." In this case, a nested bootstrap procedure is recommended by <cit.> and <cit.>:

1. [Outer] Bootstrap a distribution of p-values and performs FDR adjustment.

2. [Inner] Bootstrap a distribution of contribution measures or regression parameters for deriving p-values.

When using Bootstrap to obtain the distribution of the p-values. Careful thought should be given to ensuring the correlation structures under the appropriate null hypotheses is maintained.



§ ACKNOWLEDGEMENTS


I would like to thank Professor Yoav Benjamini for his valuable contributions to the original research and for granting permission to use material from our technical paper.






§ TOY EXAMPLE GENERATION



The data example is deliberately simple, using mainly Gaussian relationships and a variable that is not really participating in the system.

The example simulates the decision of how fast to turn a wind turbine, where the outcomes are energy produced and perceived noise. It is loosely based on research by <cit.>, who state:


“By accounting for individual site conditions we confirm that load factors do decline with age, at a similar rate to other rotating machinery. Wind turbines are found to lose 1.6 ± 0.2% of their output per year, with average load factors declining from 28.5% when new to 21% at age 19. This trend is consistent for different generations of turbine design and individual wind farms. This level of degradation reduces a wind farm's output by 12% over a twenty year lifetime, increasing the levelised cost of electricity by 9%.”


The variables in the toy example are generated as follows:


    Winter_Ind   ∼Binomial(n=1, p=0.7) 
    Sea_Temperature   ∼𝒩(μ = 20 - Winter_Ind· 10, σ = 2) 
    Wind_Speed   ∼𝒩(μ = 40 + Winter_Ind· 20, σ = 10) 
    Strength_Degradation   ∼𝒩(μ = 1.5, σ = 0.1) 
    Rotational_RPM   ∼𝒩(μ = 1.2 + Wind_Speed/10, σ = 0.2) 
    Energy_Yield   ∼𝒩(μ = 10 + Rotational_RPM/1.5 - Strength_Degradation/10, σ = 2) 
    Perceived_Noise   ∼𝒩(μ = 20 + Rotational_RPM/1.5 - Wind_Speed/4, σ = 1)


Where 𝒩(μ, σ) denotes a normal distribution with mean μ and standard deviation σ.

Note:

    
  * Rotational_RPM is measured in thousands (i.e., the actual RPM is 1000 times the value used in the model).
    
  * Energy_Yield is measured in Cent/kWh.
    
  * Perceived_Noise is an arbitrary scale.


The toy example was developed using Python and used capabilities provided by the DoWhy project [https://github.com/py-why/dowhy].



