The Sound of Silence ( S.O.S) campaign which was her brainchild- was launched by celebrity and Actress Gul Panag at The Techfest 2017 held at IIT - Mumbai midst thousands of IIT'ians from across India to bring to light the un -heard cries and silent suffering / death of millions of beagles within the closed and cold mortar walls of testing laboratories in India and across the world. Project D.O.N.T was borne out of this campaign to find a AI tool / model to replace dogs in testing.

This project was prepared and presented on July 11th 2018 to Ms. Harriet Green OBE, CEO - IBM Asia Pacific by Shiranee Pereira- Founder of Project D.O.N.T. Input received from Dr. Gargi Dasgupta, Senior Manager, IBM Research, Bangalore is gratefully acknowledged. It was taken up in 2019 by the ‘Science for Social Good’ group at IBM Research Laboratory, New York. A team of five research scientists headed by Dr. Payel Das -worked on the project to bring out a brilliant machine learning model which can predict human toxicity with far more precision than animals. The findings of which have been published in the paper ‘Accurate clinical toxicity prediction using multi-task deep neural nets and contrastive molecular explanations has been published in NATURE Scientific Reportshttps://www.nature.com/articles/s41598-023-31169-8

1.Toxicities predicted in vitro or in vivo are not necessarily in concordance with each other nor to humans, thus reducing their ability to predict clinical human toxicity. Hence toxicity remains a major driver in drug candidate failure in drug development, resulting in the high cost of drugs that make it into the market.

The granularity of toxicity tests varies across the in vitro (in cells), in vivo (in animals), and clinical platforms (in humans). In vitro testing is the most granular and captures the ability of a chemical to disrupt biological pathways at the cellular level. In contrast, in vivo /clinical testing are coarse-grained and captures the interactions of chemicals at multiple levels in animals /humans, but at the organ and tissue levels. Thus, ML models trained on in vitro and in vivo data might not reliably capture clinical toxicity.

Despite toxicity being a multi-task problem, ML (Machine Learning) models have predicted toxicity in each platform separately with single-task models. A single molecule can demonstrate simultaneously a multitude of responses in different assays and different living organisms.Various solutions for modeling multiple toxic endpoints have been reported, by creating separate binary classification models for each endpoint or by using multiple classification models that define classes. Yet, thus far, the multiplicity problem has been modeled by predicting multiple endpoints within the same testing platform: in vitro, in vivo, or in humans, separately.

Hence, we realized the need to develop an Explainable Machine learning (XML) model that simultaneously models- in vivo, invitro and clinical toxicity data. Thus a Multi Task Deep Neural Network (MTDNN) model was developed with an ability to predict clinical human toxicity accurately.

2.Morgan finger prints and pretrained SMILES embeddings (SE) were used. SMILES as an input performed better. SE were created in-house using a neural network-based translation model that translates from non-canonical SMILES to canonical SMILES, there by encoding for the relationships between the chemicals. To obtain a base model for the relationships between chemicals, the translation model was trained on 103 million chemicals in Pubchem58 and 35 million chemicals in ZINC-1259. The trained translation model was then applied to the chemicals present in the in vitro, in vivo, and clinical datasets to obtain their SE.

3.The datasets used were TOX21 (for in vitro- 1491 drugs), RTECS (for in vivo- 42,639 chemicals – (Acute oral toxicity - LD50) and ClinTOX (for human clinical toxicity- 8014 molecules).

4.We found that this MTDNN model can predict accurately toxicity for all endpoints – in vivo, invitro and clinical as a different task within one model while comparing to singe task and transfer learning counter parts.

5.We evaluated the performance of our framework by metrics of AUC-ROC (Area under Receiver Operating Characteristic Curve) and Balanced Accuracy. We compared the performance of STDNN to MTDNN with either SMILES embeddings or Morgan fingerprints as input. Performance of different platform combinations in the MTDNN was contrasted by combining all three in vivo, in vitro and clinical platforms, clinical and in vitro platforms, clinical and in vivo platforms and in vitro and in vivo platforms.

6. Skewed datasets are a prevalent problem in predictive toxicology. Regardless of the platform, the distribution of toxic and nontoxic examples is often imbalanced. Within our datasets, the imbalance is biased towards the "nontoxic" class in ClinTox and Tox21, and the "toxic" class in RTECS. This biases the AUC-ROC values towards a small fraction of true toxic or true nontoxic predictions. Hence another metric, Balanced Accuracy, takes into account this imbalance and has been used as a more representative metric for predictive toxicology models. Balanced accuracy averages the ‘sensitivity’ and the ‘specificity’.

7.To increase trustworthiness and provide confidence to explain the model’s prediction a post hoc CEM (Contrastive Explanations Method) was adapted that returns Pertinent Positives (PP) and Pertinent Negatives (PN) which in turn correspond to known toxicophores.

The CEM more comprehensively explains DNN predications by identifying present (PP) and absent ( PN) substructures within the molecules that corelate to a prediction.

E.g. For a molecule predicted to be toxic the PP sub structures are the minimum and necessary substructures within the molecule that corelates to the ‘toxic’ prediction. Conversely the PN substructures --missing from the molecule – are those that when added converts the ‘toxic’ prediction to ‘non- toxic’.

We obtained all the PP and PN sub structures for all the molecules used from RTECS, ClinTOX and TOX21. For each molecule, ten PP and ten PN substructures were collected, totaling 94,240 PP and PN substructures.

8.To discern whether the CEM correctly pinpoints PP and PNs substructures correlating to toxicity, we further matched all collected PP and PNs to known toxicophores. We expected to see a larger number of matches with PP and PN substructures correlating to toxicity (toxicophores), than to the converse (nontoxic substructures). Indeed, for ClinTox and Tox21, but not for RTECS, there are a larger number of matches to known toxicophores with CEM-derived toxicophores than with CEM-derived nontoxic substructures.

The literature contains a vast and diverse array of known toxicophores. Mutagenic toxicophores, in particular, have been widely used to verify results of computationally predicted toxicophores8. Here, we matched the toxicophores obtained from the CEM to known mutagenic toxicophores collected in vitro experimentally, or computationally, and known reactive substructures commonly used to filter molecules. The CEM was able to identify toxicophores across all these types of known toxicophores, both from PP substructures of correctly predicted toxic molecules and PN substructures of correctly predicted nontoxic molecules.

Interestingly and importantly (and not surprisingly) using our multi-task model and its transfer learning counterpart, we demonstrated the minimal relative importance of in vivo data to make accurate predictions of clinical toxicity The addition of in vivo data in the MTDNN or its transfer learning counterpart did not improve clinical toxicity predictions. Instead, the addition of in vitro data to clinical data was sufficient in improving the predictions of clinical toxicity by AUC-ROC.

10.The relative importance for in vivo data for predictions of clinical toxicity was assessed by training different combinations of platforms in the multi-task model, with or without in vivo data, and its transfer learning counterpart. To investigate whether the type of chemicals within the in vitro, in vivo, and clinical datasets affected the ability of in vivo data to predict clinical toxicity, we visualized the relationships between the chemicals using t-distributed stochastic neighbor embeddings (t-SNE). t-SNE, is a method that maps high-dimensional data to lower dimensions while preserving local similarities (i.e., distances between datapoints). We applied t-SNE mapping to SE of the chemicals in the Tox21, RTECS, and ClinTox datasets, with each dot representing a chemical and distance representing similarity. The map ( given below) is dominated by RTECS chemicals (green) due to the larger number of chemicals present in RTECS than both the Tox21 and ClinTox datasets. However, when examining overlap of the chemicals, the majority of the overlap is among ClinTox (purple) and Tox21 (red) chemicals, with some overlap between ClinTox (purple) and RTECS (green) chemicals. Thus, chemicals present in the clinical dataset (ClinTox) are more related to the chemicals present in the in vitro dataset (Tox21) rather than with the in vivo (RTECS) dataset.

Fig 1.t-SNE of SMILES embeddings of chemicals in the Tox21, RTECS, and ClinTox datasets. Distances correlate to similarities of the chemicals across these datasets; shorter the linear distance, the more similar are the chemicals. ClinTox chemicals overlap more with Tox21 chemicals than with RTECS.

11.Hence these results strongly suggest that there is a minimal relative importance of in vivo data for predicting clinical toxicity in particular when unsupervised pre-trained SMILES embeddings were used as an input to multi-task models; thus, providing possible guidance on what aspects of animal data cannot be considered in predicting clinical toxicity We further provided a more complete and consistent molecular explanation of the predicted toxicities across different platforms by analyzing the contrastive substructures present within a molecule.