Statistical analysis in study plans of pre-clinical safety studies

Safety evaluation studies are essential for regulatory approval before a new pharmaceutical can enter clinical phases. Conducted in GLP (Good Laboratory Practice)-certified facilities, these studies adhere to stringent guidelines to ensure the integrity and reliability of the data.

While study plans typically cover design, dosing, duration, and safety endpoints in detail, the methods for statistical analysis are often described in more general terms. Study plans with a vague statement on statistical analysis like “data will be analysed using appropriate statistical tools” can lead to bias in selecting a statistical tool which will tremendously impact the interpretation of the study findings.

Statistical analysis methods outlined in study plans often receive less scrutiny during GLP audits. A comprehensive audit of statistical analysis in GLP pre-clinical safety studies is essential for maintaining data quality and regulatory compliance. This ensures that findings are robust and can be reliably used for further research or regulatory submission.

OECD (Organization for Economic Co-operation and Development) Test Guidelines (TGs) for toxicity studies state that the statistical tools to analyze data should be explained in the study plan. For example, TGs for repeated dose 90-day oral toxicity study in rodents (TG No.408) and non-rodents (TG No.409), and for carcinogenicity studies (TG No. 451) state that ‘The statistical methods and the data to be analysed should be selected during the design of the study’.

The Redbook of the FDA (Food and Drug Administration, USA) emphasizes a description of the experimental design, including the methods for control of bias and a statement of the proposed statistical methods to be used in the Study Plan.

Detailed information on the statistical analysis is given in OECD Guidance Document No.116 on the conduct and design of chronic toxicity and carcinogenicity studies, supporting TGs 451 (carcinogenicity studies), 452 (chronic toxicity studies) and 453 (combined chronic toxicity/carcinogenicity studies), and Document No 35 (Guidance Notes for Analysis and Evaluation of Chronic Toxicity and Carcinogenicity Studies) also provide information on various statistical tools that can be used for analysing toxicology data obtained from repeated dose administration studies.

Both Guidance Documents 116 and 35 have provided a statistical decision tree with statistical tools that can be used for analysing data obtained from toxicology studies.

Statistics in clinical trials
ICH (International Council for Harmonisation) Guideline prescribes for clinical trials that ‘all important details of its design and conduct and the principal features of its proposed statistical analysis should be clearly specified in a protocol written before the trial begins’.

The Guideline recommends to develop a statistical analysis plan (SAP) as a separate document after finalising the protocol. SAP should cover a more detailed information on the statistical analysis. ICH recommends that the analysis should be done according to the SAP.

Adhering to the SAP for the analysis of data increases the credibility of the results. If any deviations from the SAP are made, they should be properly justified, which is technically not so easy, and difficult to convince regulators.

Statistical analysis of toxicology data
The Guidance Notes (ENV/JM/MONO (2002)) emphasize that the primary aim of a toxicology study is to identify and characterize biologically significant responses. This means that the study should not only detect any toxic effects but also evaluate the relevance of these effects in terms of their impact on health, environmental safety, and regulatory considerations.

The goal is to provide data that can inform risk assessments and regulatory decisions, ensuring that potential hazards are clearly understood in the context of exposure scenarios. The Guidance Notes also emphazize that ‘Where statistical analyses are used to reach a judgment, an awareness of the validity of the tests employed and the degree of certainty (i.e. confidence) pertaining within the context of the study should be demonstrated’.

This means that the Study Directors must not only apply appropriate statistical tests but also understand their validity and limitations. It is also expected that the Study Director should be aware of the underlying principles of statistical tools. When safety evaluation studies for new pharmaceuticals fail to clearly explain statistical analysis in their study plans, it can lead to significant misunderstandings and potential consequences for drug development. Let us work out an example.

The absolute liver weight of mice in a 13-week repeated dose administration study is given in Table 1.

n- Number of observations; SD-Standard Deviation

The data were analyzed following the decision tree given in OECD Guidance Document No 116. Initially, the data were subjected to normality and homogeneity tests, and the data passed both these tests.   

Since the number of groups is more than two, one-way analysis of variance (ANOVA) is the recommended statistical tool for finding differences of means among the groups, as per the decision tree. One-way ANOVA showed insignificance (P=0.0804), indicating no difference in means of absolute weights among the groups. As per the decision tree, no further statistical analysis needs to be carried out.

On closely observing, it can be found that there is a dose-dependent increase in the absolute liver weight. When the data was analyzed by Dunnett’s Multiple Comparison Test (DMCT), it was found the absolute liver weight of High Dose group was different from the control group (P= 0.0399). Dunnett never recommended ANOVA for applying DMCT.

Guidance Document No. 116 indicates that one should be aware of the distinction between statistical significance and biological importance. Statistical analysis involves more than the reporting of the statistical significance of a hypothesis test.

According to this Guidance Document, statistical analysis is a part of the interpretation of the biological importance, not an alternative. This Guidance Document says, ‘when reporting the results of significance tests, precise P-values (e.g., P=0.051) should be reported rather than referring to specific critical values. Similarly, declaring a result non-significant (often designated as P>0.05, again a nomenclature not favoured by statisticians) should not be interpreted as meaning the effect is not biologically important or that the null hypothesis is correct. Rather it means that there is not sufficient evidence to reject the null hypothesis.

The Guidance Document No. 116 also emphasizes determining the size of the difference rather than making a ‘significant’ or ‘no significant difference’ based on a P value. The insignificance of a significant P has been discussed in several recent publications.

Purpose of statistical analysis
According to ICH S2 R1: Guidance on Genotoxicity, ‘the application of statistical methods can aid in data interpretation; however, adequate biological interpretation is of critical importance’. The primary goal of conducting a toxicology study is to identify biological responses to substances, and statistics play a supportive role in analyzing the data to uncover these responses.

It is very important to study the data before subjecting them to statistical analysis.  It helps, sometimes, to choose the appropriate statistical methods, as demonstrated in Table 1.  As per OECD Concensus Document (Consensus Document the Role and Responsibilities of the Study Director in GLP Studies), the Study Director is responsible for drawing the final overall conclusions from the study. The study Director can draw a conclusion based on biological relevance with or without statistical significance.

Role of statisticians in analysing data
In GLP facilities, while statistical analyses are important, there isn’t a strict requirement for a dedicated statistician to oversee all data analysis. I However, the requirement is not so for clinical trials, where a trial statistician is responsible for the statistical aspects of the trial.

Conclusion
Explaining statistical analysis in detail in GLP pre-clinical safety studies is crucial to ensure the integrity and reliability of data. While explaining ensure that the statistical methods used comply with GLP regulations and relevant guidelines, ensure that the planned analyses align with the objectives of the study. It is a requirement that the Study Directors understand the underlying principles of statistical tools that are used.

There are limitations of statistics in toxicology as elucidated by Gad and Weil in Statistics in ‘Experimental Design for Toxicologists’ published in 1986 and by Katsumi Kobayashi and K. Sadasivan Pillai in ‘A Handbook of Applied Statistics in Pharmacology’, published in 2012.

Some limitations are that statistics cannot improve poor data, statistical significance may not imply biological significance, an effect that may have biological significance may not be statistically significant, and the lack of statistical significance does not prove the safety of the test item.