Bioinformatics Retrieval Augmentation Data (BRAD) Digital Assistant
Joshua Pickard Marc Andrew Choi Natalie Oliven Cooper Stansbury Jillian Cwycyshyn Nicholas Galioto Alex Gorodetsky Alvaro Velasquez Indika Rajapakse



The Impact of Balancing Real and Synthetic Data on Accuracy and Fairness in Face Recognition
    Andrea Atzori0000-0002-6910-206X Pietro Cosseddu0009-0006-4998-5164 Gianni Fenu0000-0003-4668-2476 Mirko Marras0000-0003-1989-6057
    Accepted Sep 3 2024 to ApJ Letters

======================================================================================================================================



§ ABSTRACT 
    We present a prototype for a Bioinformatics Retrieval Augmentation Data (BRAD) digital assistant.
    
    BRAD integrates a suite of tools to handle a wide range of bioinformatics tasks, from code execution to online search. We demonstrate BRAD's capabilities through (1) improved question-and-answering with retrieval augmented generation (RAG), (2) BRAD's ability to run and write complex software pipelines, and (3) BRAD's ability to organize and distribute tasks across individual and teams of agents. We use BRAD for automation of bioinformatics workflows, performing tasks ranging from gene enrichment and searching the archive to automatic code generation and running biomarker identification pipelines.
    
    
    
    BRAD is a step toward the ultimate goal to develop a digital twin of laboratories driven by self-contained loops for hypothesis generation and testing of digital biology experiments.




§ INTRODUCTION


Large language models (s) and scientific foundation models (s), have expanded the potential applications of artificial intelligence <cit.>.
Originally designed for machine translation, s have now found applications in areas such as question-and-answering, code generation, and data analysis. These models are driving innovation across disciplines, advancing research in aerospace <cit.>, chemical engineering <cit.>, and bioinformatics <cit.>.
Allowing models to function as independent agents capable of analyzing data and making decisions, or as collaborative teams where agents share instructions and results, offers a promising avenue for enhancing human-in-the-loop supervision in AI-driven discovery <cit.>.

By enhancing scientific understanding and accelerating discovery, these technologies offer unprecedented opportunities for research. 



Although a single  can accomplish a myriad of tasks, Multi-Agent  systems are increasing in popularity <cit.>.  By delegating tasks to an array of individual -powered agents, these agents can collaborate to solve various tasks, such as software development <cit.> and biomedical discovery <cit.>. In bioinformatics, Multi-Agents can power autonomous labs equipped with data analysis, enhancing scientific knowledge and accelerate discovery <cit.>.




Our ultimate goal is the design of a self-driving, digital biology laboratory. Self-driving labs integrate both experimental and analytical processes for a lab and vastly accelerate scientific progress. Although they are still in their infancy <cit.>, closed loop systems have already been implemented in other domains such as chemistry <cit.> and synthetic biology <cit.>. Bioinformatics Retrieval Augmentation Data (BRAD) 
digital assistant integrates recent  and  technologies with traditional bioinformatics tools, creating an advanced -based agent, capable to be chained together as a Multi-Agent System that can conduct complex decision-making and execute digital biology experiments. 






The prototype BRAD research assistant utilizes a  to operate an array of modules. The  Module integrates documentation, protocols, and literature using Retrieval Augmented Generation (RAG) <cit.>, while the  Module enables searches across online data and literature repositories. Additionally, the  Module automates code generation and execution. Together, these main features—supported by a robust software framework—allow BRAD to provide in depth responses to research grade user queries by performing its own independent research.



We demonstrate BRAD's ability to execute standard research workflows efficiently. Individually, a BRAD bot can handle tasks like literature searches, gene set enrichment from databases such as Enrichr or Gene Ontology, and generation and execution of Python and MATLAB software. 
When organized effectively, BRAD bots can work in tandem to solve more complex challenges such as biomarker identification, network analysis, or exploratory data analysis. This collaboration between BRADs allows each agent to tackle tasks of greater complexity, producing results that surpass the sum of their individual efforts (see <ref>). 
The remainder of this article first presents the anatomy (or software architecture) of BRAD and then presents several results of BRAD's ability to dynamically learn new topics and use cases of this code, such as biomarker identification and enrichment.







































§ ANATOMY OF BRAD

At the heart of this architecture is an (s) that orchestrates the usage of BRAD's different modules. These modules enable BRAD to interface directly with documentation, published research, the internet, software, and user data. This modular architecture grants BRAD the flexibility to integrate diverse tools while offering users a framework to customize and expand BRAD's capabilities. 




When responding to a user query, BRAD's modules adheres to a standard template 
(see <ref>, left):
(1) a semantic router <cit.> selects the most appropriate module to handle the query; (2) within the selected module, a series of  prompts are used to retrieve literature, search online, execute code, or other module specific tasks; (3) an  summarizes BRAD's findings and continues the conversation. 









For queries demanding multiple modules, BRAD's  acts similar to a ReAct agent <cit.> by breaking down the query into a series of tasks to be handled by individual modules. 
These tasks are executed sequentially, with BRAD compiling the results from each module to form a cohesive response. The  can create 
linear workflows or more advanced pipelines that include feedback loops, branching, and other control flow mechanisms. 


The backbone of BRAD's design is code 

that supports its integration with other software and maintain its ease of use. Each BRAD chatbot or chat session outputs all figures, downloaded data, and other relevant information to a centralized output directory. This directory also maintains a comprehensive log that tracks BRAD's interactions with the user, databases, and other tools. Built on the LangChain framework, BRAD can utilize s and APIs either locally or from providers like https://build.nvidia.com/meta/llama3-70bNVIDIA and https://openai.com/OpenAI. It can also be integrated into any LangChain or LangGraph architecture, ensuring broad compatibility with standard frameworks <cit.>. In the remainder of this section, we outline the capabilities of BRAD's main modules, with further implementation details available in SI.<ref>.



















 §.§ Lab Notebook Module

The  Module connects BRAD to internally stored documents, including literature, protocols, textbooks, and any other  files. It employs RAG to tailor the 's responses based on curated user-provided information.

BRAD can choose between a previously constructed document database or create one dynamically. For further detail on database creation, see SI <ref>.


RAG functions in two phases: retrieval and generation <cit.>.  During retrieval,  the aforementioned database is searched to find chunks of text relevant to the user’s query. In the generation phase, the retrieved documents are combined with the user query to create a prompt for the . This approach enhances response quality and reduces hallucinations <cit.>.

The  module allows users to control database construction on their own literature and customize the retrieval and generation processes (see <ref>). BRAD supports various techniques, including similarity-based search, maximum marginal relevance (MMR) <cit.>, and multi-query retrieval during the retrieval phase, as well as compression <cit.> and reranking <cit.> in the generation phase.












  
Example Use Case.
As an example of the output, see <ref> (Q1): How can we view Pore-C as a hypergraph? This question requires the synthesis of information from different fields. For instance, Pore-C is a recently developed experimental genomics technique that characterizes three-dimensional chromatin structure by identifying multi-way contacts in the chromatin polymer. On the other hand, hypergraphs are a higher-order generalization of graphs that allows edges to contain multiple nodes. Since Pore-C data captures multi-way contacts, hypergraphs are a natural way to view Pore-C data. 


Using OpenAI's GPT-4.o mini as the underlying  for response generation, we equipped BRAD with three different configurations to test BRAD's question and answering capabilities: vanilla , a standard RAG with no enhancements (i.e. only MMR retrieval and generation), and an enhanced RAG with additional features, including MMR, multi-query retrieval, reranking and contextual compression. When BRAD employs a RAG to improve its response, it is supplemented with ten chunks, generated by recursively chunking the papers our database[See <https://rajapakse.lab.medicine.umich.edu/papers/digital-library>]. In our view, the enhanced RAG, where BRAD makes full use of its , best answers this question.






Despite the fact that this question has a clear cut meaning, it is still quite open ended, allowing the BRAD chatbots freedom to dive into the specifics as much as possible. Consequently, we deem vanilla 's response to be the worst. In particular, its discussion of Higher-Order Interactions parrots its own previously generated text. Additionally, although it claims to give perspectives into how to analysis and visualize the data, it completely forgets to discuss visualization techniques. 


Comparing this to standard RAG, standard RAG is also overly repetitive, as seen through repeating information in the first paragraph and the enumerated list of concepts. However, its discussion about hyperedges is more insightful. Even though it also fails to give pointed recommendations about visualization tools, it comments about the capability rather than omit the discussion altogether. 



As a result, we believe the enhanced RAG best answers the question. It fixes many of the issues that the standard RAG fixes without the overly repetitive nature of the standard RAG. Specifically, its discussion on Higher-Order Interactions and Analysis and Algorithms adds new insights compared to both vanilla  and the standard RAG. 








0.95Q1: How can we view Pore-C as a hypergraph?
[raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = Enhanced RAG,colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 8cm]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly in the context of chromatin interactions. To view Pore-C as a hypergraph, we can consider the following aspects:


⋮

In summary, conceptualizing Pore-C data as a hypergraph allows for a more nuanced understanding of genomic interactions, accommodating the complexity of higher-order relationships that are inherent in chromatin organization.








[title = Standard RAG,colback=green!5!white,colframe=green!75!black,fonttitle=, width = 8cm]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly in the context of chromatin interactions. Viewing Pore-C data as a hypergraph can provide a more nuanced understanding of the complex relationships between genomic elements. 


In a hypergraph, nodes represent individual genomic features (such as genes, regulatory elements, or chromatin domains), and hyperedges represent interactions or relationships that can connect multiple nodes simultaneously. This is particularly relevant for Pore-C data, where interactions are not limited to pairs of genomic elements but can involve multiple elements at once.

⋮

In summary, viewing Pore-C data as a hypergraph allows for a richer representation of the complex interactions within the genome, facilitating deeper analysis and understanding of chromatin architecture and its functional implications.


[title = Vanilla ,colback=red!5!white,colframe=red!75!black,fonttitle=,  width = 8cm,before=
]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly focusing on chromatin interactions. To view Pore-C as a hypergraph, we can consider the following aspects:


⋮

    
    
    


In summary, conceptualizing Pore-C data as a hypergraph allows for a more nuanced understanding of the complex interactions within the genome, enabling researchers to explore the functional implications of these interactions in a more comprehensive manner.



    See SI<ref> for the full vanilla  output.








  
Quantitative Assessment.
Beyond the qualitative assesment of the  module, we employed the RAG Assessment (RAGAs) framework to quantify the advantage of BRAD's RAG system <cit.>.

We generated a set of research questions and evaluated BRAD’s ability to answer them using various features of the . These questions were manually classified into three different topics: Mathematics, Biology, and Computer Science and Data.


This framework employs an evolution-based question generation approach to produce a diverse range of questions <cit.>. This process begins by invoking an  to generate a basic set of questions from the document database. Then, it takes those same questions as an input and asks the  to "evolve" the questions by making them more complex or simple. This prompt engineering is preformed iteratively until the final set of questions are generated.

BRAD then answers the generated questions, and its performance is scored based on the expected answer for each question. 
A detailed explanation of how each metric is calculated is included in SI <ref>.



Using the same document database, we generated three types of questions: (1) simple questions where answers are found directly in the database, (2) reasoning questions that require deeper understanding of a single piece of text, and (3) multicontext questions that require understanding multiple documents to answer successfully. We asked each of the same three BRAD chatbots as before the generated set of questions and compared their responses using RAGAs's evaluation metrics. 
















Compared to its counterparts, the standard RAG overall has the greatest mean answer correctness, indicating it is the best performer. However, the enhanced RAG has a higher mean answer correctness best performance on questions related to computer science and data. We believe that during contextual compression, these mathematical symbols' meaning is lost in its summary, leading to enhanced RAG having a lower mean answer correctness than standard RAG. 









The faithfulness, a proxy for reliability, of all three models is relatively low.
This demonstrates that the BRAD chatbots do not regurgitate what was presented in the papers. 
Interestingly, RAG consistently has substantially higher faithfulness. Although this may be perceived as a positive, there can be adverse effects as shown in Q1, where the standard RAG parrots text found in the retrieved documents but does not provide synthesis or further analysis. 

Context precision is fairly high for all three models, indicating that chunks relevant to the ground truth are being retrieved for the RAG systems and that the vanilla , which does have the lowest context precision, provides on-topic answers. We see significantly less variability in the enhanced RAG's context precision with multi-context questions, validating reranking's effectiveness. This is further validated with higher answer correctness scores for questions that span multiple topics as BRAD needs to identify the most important chunks from at least two distinct subtopics. Finally, both the standard and enhanced RAG significantly outperform the vanilla  with respect to answer relevancy, demonstrating considerable mitigation of hallucinations. Note that context recall, which is not shown, was consistently above 95% for all three versions.






























 §.§ Software Module

BRAD's  Module features a multistage pipeline that utilizes s to write and execute code in response to user queries. It supports running Python, MATLAB, and Bash scripts. Although BRAD is primarily designed for workflows that involve saving data to files at each stage, as is common in bioinformatics, this module also supports  generated code that avoids writing files between stages.














BRAD performs a series of steps to safely execute a piece of software (see <ref> and SI.<ref>). The process starts with BRAD reviewing the documentation of all available scripts and leveraging the  to select the code that best addresses the task. Based on user input and selected code, BRAD uses the software documentation, few shot learning examples, and the  to to generate the appropriate code. Refer to SI.<ref> for details on the implementation.

To ensure safe execution of  generated code, BRAD is restricted to running whitelisted software. This allows BRAD to verify that the correct paths, inputs, outputs, and other arguments are provided for each script. Additional checks ensure the closure of all delimiters and other issues identified while testing -generated code.



Errors identified during code validation are sent back to the  along with the original query for refinement. This iterative process of code generation and validation may continue until the code executes without error. Once all issues are resolved, the code is run.




We designed scripts to execute  generated code without running whitelisted scripts. While this approach allows BRAD to fully utilize  generated code, this process introduces several challenges: (1) ensuring the safety of -generated code, and (2) generating bug-free, free-form code with s. To demonstrate this feature, we designed a script that enables the safe execution of the full  library with BRAD. However, ensuring  generated code is error free remains a challenge.



































 §.§ Digital Library Module

The  module provides BRAD access to an array of literature, archives, and online databases (see <ref>). Building on a standard RAG, BRAD can retrieve and share information from these sources, either directly with the user or through other modules. Below, we outline the integration of several databases, and we'd like to note the relative ease of integrating new databases.  



















  
Enrichr.
Enrichr is a comprehensive gene set enrichment tool commonly used in many bioinformatics pipelines <cit.>.

We integrated Enrichr into BRAD to leverage gene set annotations, curated pathways, and ontology knowledge to enhance BRAD's responses to user queries.










  
Gene Ontology.
The Gene Ontology (GO) database is an initiative to provide a uniform approach and vocabulary to studying genes and gene products across different species <cit.>.


When gene names, GO terms, or other keywords are identified, BRAD has the ability to define these GO terms and retrieve charts that diagram their relationships with other genes.

Moreover, the GO terms are associated with papers that may be integrated via other databases and modules of BRAD.





  
Literature Repositories.
BRAD integrates the arXiv, bioRxiv, and PubMed literature databases. BRAD can both display search results to the user, link articles, summarize recent abstracts, download and display papers, and embed the new papers into its database.



  
Integration with Other Modules.
Online retrieval works symbiotically with the other modules. For instance, the Lab Notebook RAG and the Literature database search complement each other. These modules can be used in tandem application for performing literature reviews and constructing RAG databases (see <ref> and SI<ref>). When the Lab Notebook does not contain literature relevant to a user query, new literature can be obtained to improve the RAG database. 

Conversely, when new literature is obtained, the RAG is used to summarize the documents and respond to the user.

Similar iterations can occur between the software engine and the database searches.












 §.§ Additional BRAD Modules

BRAD also contains modules responsible for documentation generation, planning and executing multistage queries, documentation and code generation, and more. Refer to SI.<ref>-<ref> for additional implementation details.







§ DIGITAL LABORATORY

This section presents various examples and results achieved by BRADs. We demonstrate the ability to (1) perform a comprehensive search about a gene, using network analysis, multiple databases, and RAG, (2) automatically build a RAG database targeting a specific question from scratch, (3) run biomarker identification pipelines, and (4) perform automatic exploratory data analysis. The first three examples are performed using single BRAD chatbot, and performing exploratory search of data is done with multiple BRAD agents that pass information between one another.

Each use case demonstrates how a single BRAD or a team of BRADs tackles or automates complex research tasks under human supervision.









 §.§ Enhanced Search

Integrating the  and  modules improves quality of the RAG responses generated by the . Consider, for instance, Q2: What role does PCNA play in the cell cycle and imaging? BRAD orchestrates the following process: (1) identifies which genes interact with PCNA; (2) determines the pathways related to PCNA; and (3) uses the related genes and pathways to search the literature and formulate a comprehensive answer. In addition to providing a detailed text response, this process generates supplementary materials

such as images, spreadsheets, and other documentation to offer further context to the user (see <ref> and SI.<ref>).





Q2: What role does PCNA play in the cell cycle and imaging?
[raster equal height,raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = System Output,colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 12cm]
PCNA’s Role in the Cell Cycle: PCNA (Proliferating Cell Nuclear Antigen) plays a crucial role in the cell cycle, specifically in the S phase. It is loaded onto DNA during S phase, and its interaction with the minimal human Cdt1 PIP degron triggers the degradation of a cell cycle reporter, providing a direct physical readout of DNA replication.



PCNA’s Role in Cell Cycle Imaging: PCNA is a DNA replication sensor that can be used to track the cell cycle. It appears as a diffuse nuclear pattern in G1 phase, punctate in S phase, diffuse again in G2 phase, and diffuses throughout the whole cell in M phase. It is a high-contrast S/G2 transition marker, but requires high-resolution images and regular nuclear morphology and monolayer culture.



...



See <ref> for the full output and generated documents.







 §.§ Building a RAG Database

We evaluated BRAD's capacity to respond to questions without a pre-existing database. BRAD was tasked with constructing a database from scratch to address research questions. Through alternating between the  and  modules (see fig.<ref>, bottom
), BRAD successfully built a local literature database and retrieved relevant information online to answer the questions.

Consider the question Q3: What are single cell foundation models? As these are a recent technology, BRAD has no information regarding these models. Recognizing the insufficiency of available information from the empty database,

BRAD gathers research papers and articles related to single-cell analysis and foundation models. With this new information, BRAD enhances its response. This process was repeated twice until
the  deemed the response to be satisfactory. The final BRAD response provided a fairy detailed overview of single-cell foundation models, including several notorious examples and their use cases. For a comprehensive summary of how BRAD executed this process, see SI.<ref>.




Q3: What is a single cell foundation model?
[raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = RAG: Iteration 1 
(empty database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Based on the provided source and content, the initial database is empty, and I couldn't find any information regarding “single cell foundation models." I am confident that I cannot provide an accurate answer to this question with the current information available.




Search Terms

    
  * Single cell foundation models
    
  * Machine learning for single cell data analysis
    
  * Single cell machine learning
    
  * Deep learning for single cell analysis
























[title = RAG: Iteration 2 
(populated database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Single cell foundation models are artificial intelligence models designed to analyze and understand data from individual cells. They are constructed using single-cell transcriptomic data, which measures the expression levels of genes in individual cells. These models utilize machine learning algorithms, such as transformers, to learn gene-level and cell-level representation. They can accurately categorize known cell types and have the potential to uncover new biological insights in single-cell RNA sequencing data. Single cell foundation models are used in genomics and multi-omics research, showing promise in various downstream tasks, such as identifying cell types and states, discovering novel cell populations, inferring gene regulation networks, and more. Examples of single cell foundation models include scGPT, Genefomer, and scFoundation. They are a growing field of study, with ongoing advancements expected as single-cell technologies and omics sciences continue to develop.







 §.§ Biomarker Identification and Enrichment

Biomarker identification – finding genes that indicate a biological state – remains a challenge <cit.>. A typical framework for sensor selection involves two steps: (1) modeling and (2) observability analysis. Equipped with modeling and observability software from <cit.>, BRAD was provided with given several datasets <cit.> and tasked with uncovering potential biomarkers.




After normalizing data with , BRAD built several models of dynamics for each dataset using Dynamic Mode Decomposition and Data Guided Control frameworks. BRAD then passed the models into observability software to identify and rank the contribution of each gene as a sensor. Finally, BRAD queried the top-ranked genes in Enrichr to determine the relevant biological pathways and processes associated with the data.

BRAD built models of the dynamics in both the data space and based upon Geneformer

embeddings <cit.>, which were also generated by the  module. While both processes model the dynamics, BRAD correctly identified, based on the  and documentation found in the observability code, that the observability codes should not be applied in Geneformer embedding space. This constraint results from the inability to interpret observability results in embedding space.

A challenge for scs will be to make these results interpretable in the original space of the data, perhaps using a decoder or alternative dynamics perspective.

Despite reporting that the observability codes did not make sense to execute, BRAD still ran these codes in the embedded space.







 §.§ Automated Cellular Reprogramming Analysis


A team of BRADs was employed to study in silico perturbations of Fibroblasts toward hematopoietic stem cells (HSCs). Several hundred reprogramming regimes were identified with Data Guided Control <cit.>, each consisting of using a subset of ten transcription factors, and after embedding each recipe with Geneformer <cit.>, the BRADs were tasked with performing exploratory analysis to identify the best recipe.






  
Coordination.
A central BRAD generated high-level instructions for a team of five BRAD workers based on detailed documentation of the experiment. Each worker used the  module to translate these objectives into tasks for the  module. Expert review led to adjustments, including switching to cosine similarity, removing the literature review task, and adapting file paths for our lab's system. This human-in-the-loop (HITL) approach ensures the BRAD team's analysis aligns with our research goals and lab environment.





  
Execution.
After revising the plans, the BRAD team executed its analyses, generating figures, code, and new data files. 
In minutes, the BRADs produced outputs that would typically take humans hours or even days. However, some duplication of files suggested better data structuring was needed initially. While the figures were unpolished, they enabled rapid exploratory analysis of cell clusters from different transcription factor reprogramming, ensuring reproducibility and allowing for human feedback. All outputs from the BRADs require thorough human review, shifting researchers' roles from writing and debugging code to validating software, documenting analysis pipelines, and reviewing results. This shift enhances research efficiency and aligns with the software development life cycle, where validation and testing are essential in ensuring accuracy.







§ DISCUSSION

This work demonstrates the potential of leveraging s to power semi-autonomous bioinformatic assistants, such as BRAD. Our prototype BRADs were deployed on an array of tasks, ranging from standard research projects such as searching literature or gene enrichment, to more complex procedures such as executing software pipelines and performing exploratory data analysis. Through automation, we achieved improved speed, reproducibility, and documentation of our workflows, albeit on well-known, standard tasks in a prototyped environment.


The major strengths of the BRAD system lie in its ability to reliably perform common research tasks. The breakdown and distribution of problems, such as the multiple tools used to answer Q2, the automated execution of the biomarker pipeline, and the distributed workflow on Geneformer, split seemingly complex tasks into small, manageable problems that can be understood and executed by an . This modular approach offers both a blueprint for scaling up the number of agents that collaborate while still allowing humans to have fine control over the different aspects of the pipeline.

There are two central challenges in our current system. First, providing instructions that allow an  to split and distribute a complex task in a thoughtful way is a nontrivial problem for the user. Inconsistencies between the organization and capabilities of the user's instructions, the , and the available  cause issues. Furthermore, the detection of an error or corruption in a multi-stage or multi-BRAD workflow is often unclear to the system, particularly without user intervention.

Maintaining a HITL is both necessary to effectively use BRAD and an important stopgap for preventing errors from propagating in the system. For instance, in the Geneformer example, the human was used to modify parameters or instructions to the individual agents prior to their execution. Allowing users to interact with the  ensures that the initial human instructions are appropriately interpreted prior to distributing the instructions to the team of agents. To augment this, each human-approved plan is saved for potential future use in order to maximize the HITL contributions.

The second challenge of this work is effectively passing information between different modules and agents. For instance, the structure, purpose, and organization of new data files created by one agent in the Geneformer experiment must be communicated between agents that touch the same file. Passing information between modules of a single BRAD is managed with  memory, and we plan to develop new solutions for passing information between multiple agents.

In summary, this work highlights the potential use cases of BRADs working as a Multi Agent system to act as research assistants. Despite the persisting challenges, there currently are opportunities to use s for well-defined and repetitive tasks, such as running software pipelines or performing enrichment. BRAD, as a tool, successfully demonstrates these capabilities. Moving forward, we aim to enhance the planning and coordination between modules and multiple BRAD agents, with the goal of advancing towards a fully autonomous digital laboratory.



§ ACKNOWLEDGMENTS

We thank the members of the Rajapakse Lab, Adam Lord, and Santosh Srivastava for helpful and inspiring discussions. We also thank Vivan Nyati, a student of Greenhills, Ann Arbor, for his valuable contributions over the summer. This work was supported by the Defense Advanced Research Projects Agency award number HR00112490472 (IR), the Air Force Office of Scientific Research (AFOSR) award number FA9550-22-1- 0215 (IR), support from NVIDIA (IR), and  NIGMS GM150581 (JP).








    Supporting Information














§ ANATOMY OF BRAD EXTENDED


This section provides an in-depth overview of BRAD's software architecture, intended for readers who are either interested in implementing a similar system or seeking to understand the technical specifications of BRAD. For a more comprehensive guide, please refer to the https://drive.google.com/file/d/1X8hSXvoGOEv4bR4B3mf3EYRF-bbIzRsd/view?usp=sharingsoftware manual here.





 §.§ Base Architecture

Behind each module of BRAD is a framework that standardizes information flow through BRAD, memory, and external resources. The configuration parameters for each module are specified in . These default settings can be overridden by supplemental configuration files when instantiating individual BRAD instances. Module specific parameters are further detailed in their respective sections.



  
Language Model Integration.
BRAD's integration is built atop the LangChain interface for querying s. This supports a wide array of s that could be integrated into a BRAD chatbot. For ease of use, BRAD provides a simplified interface to run s hosted by https://build.nvidia.com/explore/discoverNVIDIA and https://platform.openai.com/docs/overviewOpenAI, and also supports locally hosted https://github.com/ggerganov/llama.cpp.



  
Output Directory.
An output directory is created for each BRAD chatbot. These output directories are the location where any downloaded database results, new literature, images, or other main results will be placed. Within , the  parameter allows users to specify where these directories should be created. New directories are placed in the appropriate location and time stamped. After exiting a chatbot, the session state will be saved within the output directory and can be used to turn on BRAD from the same session or starting point in the future.



  
Logging. A detailed log is provided within each output directory. The log is a  formatted file that tracks the state of BRAD as it responds to each query. It records user inputs, outputs, and the specific actions taken, such as  calls, file operations, or the use of other module-specific tools. There are three primary uses for these log files:

    
  (1) Clarity: The log files record every database query, executed software, referenced literature, and other resources used. Users can refer to the log file to understand how BRAD processes an input.
    
  (2) Documentation: BRAD's documentation generation features, including both code and PDF generation, utilize the log files to detail the process after completion.
    
  (3) Debugging: These logs are useful for identifying errors.

Separate from the log files but also useful for understanding how BRAD responds to a user query, a  flag can be set in  to have BRAD display each of the steps taken along the way.



  
Utilities. A large array of utilities have been developed to standardize the interface between the  and each module. This includes parsing responses, reading data from files, managing the output directory, and more.



 §.§ Lab Notebook Module

RAG employs a two step process: retrieval and generation. During the retrieval phase, a database is searched to identify documents and information similar to the user query. During the generation phase, the retrieved documents and user query are combined to form a single prompt for the . Based upon the additional context provided in the retrieved documents, this RAG can provide improve responses by (1) reducing hallucinations, (2) improving the accuracy of responses, and (3) providing verifiable sources for each response. BRAD contains several features to augment the traditional RAG system.




  
Database Construction and Chunking.
Chunking breaks documents into smaller pieces or chunks of texts that can later be used to help an  respond to a user query. It is important to divide documents to create more focused text that RAG models can use to generate better responses. Although there are various ways to split up texts, BRAD employs two methods:

    
  * Recursive chunking divides text into smaller chunks based on size and a set of separators, such as periods and line breaks.
    
  * Semantic chunking encodes sentences into an embedding space and then applies similarity metrics to determine which sentences should be grouped into the same chunk

Although semantic chunking creates more interpretable chunks, recursive chunking captures for more complex linguistic structures. By identifying sections with a high density of separators, we can filter out chunks that belong to reference sections of a paper. The distribution of these separators is bimodal, 
with little data in the middle range, helping to prevent the RAG from retrieving chunks filled with buzzwords but lacking substantive content. 



  
Similarity Search v. MMR.
Given a user query, BRAD has two mechanisms for retrieving the initial text chunks: Similarity Search and Maximum Marginal Relevance (MMR). In both methods, the user query is embedded as a vector so that similarity metrics, primarily cosine distance, quantify the relevance of chunks of text to the user query. Similarity search compares chunks to the prompt and determines which chunks are most similar to the prompt. This is a straight forward approach to chunk retrieval but suffers from possible redundancies in the chunks which may hamper BRAD's response. MMR retrieves chunks by finding chunks that are maximize similarities to the prompt and to other chunks based on a parameter λ. The MMR based retrieval selects a more diverse set of chunks, mitigating the redundancy issue.




  
Multi-query Retrieval.
A challenge of retrieval is questions or queries with similar semantic meaning can be worded differently. To mitigate this, multi-query retrieval takes the original user input and uses an  to create several different ways of expressing the same query. For each new expression as well as the original user query, the aforementioned retrieval methods are applied to find chunks relevant to the user query.





  
Compression.
Since the retrieved chunks can be unorganized, we use contextual compression to summarize the contexts of each of the selected chunks. This method passes each chunk one at a time, along with the user query, to an  with instructions to summarize the retrieved text, highlight the pieces of text relevant to the user's query, and to remove information not relevant to the user's query. Since the summaries are often shorter than the chunks themselves, this method can be effective when the context of an  is a limiting factor of the generation stage of RAG. While this is the most computationally expensive feature of BRAD's RAG system, since an  call is used for each retrieved chunk, this feature, produced the greatest improvements in the quality of BRAD's responses.



  
Reranking. Reranking is a common feature in RAG systems and provides a secondary validation of the relevance of retrieved chunks of text, prior to the generation stage. 

To maximize relevance and diversity of chunks given to the  during generation, we implemented a PageRank based reranker. From the retrieved chunks, we create a weighted graph where the nodes are the chunks of text and the edges are the similarity between the chunks. Then, the PageRank algorithm is used to select the most relevant chunks to be used in the generation phase. By having this secondary filter, an increased number of text chunks can be retrieved from the RAG database without necessarily passing duplicated or irrelevant information to the .




















  
Scalability of Database and Runtime.
Prior to runtime of the RAG, creating the vectorized text database can take several minutes to hours, depending on its size. For instance, our database consisting of more than 500 papers and books we created a database that consist of roughly 18000 pages. Although creating the vectorized database takes a few minutes, once the database is created, BRAD runs considerably fast, despite the size of the database. Furthermore, based on the , this can also heavily influence computational cost.

While the enhancements mentioned above improve the performance of the RAG, they also introduce additional computational costs.



































 §.§ Software Module

BRAD's coding module provides a multistage pipeline that uses s in conjunction with custom documentation to both analyze data as well as improve the response to a user's query. BRAD contains implementations to execute python, , and bash scripts or pipelines.





  
Multistage Execution Pipeline.
A series of steps are required for BRAD to run a piece of code. The process begins with a user's query, where BRAD first reads the documentation of all available code to determine the appropriate script to execute. With the users input and selected code, BRAD will reread, in greater detail, the documentation of the code to execute. Using this information, along with a set of examples based on the principle of few shot learning, an  then writes a snippit of code to execute the appropriate script.

Once an initial block of code is generated, a series of validations are performed prior to execution. To address issues found in the testing issues of this pipeline with llamas 2 and 3, among other models, the proposed code is verified to ensure closure of all delimiters and correctly pointing to an output directory for all read or write operations, among other common syntax errors. After these validations, the code is compiled to ensure it is bug-free before execution.

Any errors identified during the validation stages are passed, along with the code and original query, back to the   for refinement. This iterative process of code generation and validation may be repeated several times until the code executes without error. Once all issues are resolved, the code is run.






  
Software Specification.

In order for BRAD to run code, the following requirements are enforced:

    
  * Location: All code that a user wants BRAD to run must be included in a BRAD specific path.
    
  * Documentation: Any code to be run must be well documented, adhering to standard python or conventions, enabling BRAD to understand the software's purpose and execution process.
    
  * Output Arguments: Any code that writes information to disk should accept at least one argument where BRAD can indicate the correct output directory for new files.

These minimal requirements ensure that BRAD can effectively manage, understand, and execute the code while maintaining a consistent and organized workflow. Additionally, these three requirements are intended to allow for ease of adoption of previously generated code.







  
Software Limitations and Considerations.
While useful and effective, this framework for  powered execution of code has several limitations and vulnerabilities. It expects bug-free software from the user, which can be challenging to ensure. It can be susceptible to malicious code, if the scripts executed by Brad are not curated.









 §.§ Digital Library

Beyond traditional text-based retrieval, there is a growing collection of online databases common to bioinformatics workflows that can be integrated into the RAG pipeline.

Here, we outline the integration of several databases and highlight that our architecture allows for the seamless addition of new databases into BRAD; see <ref>.





  
Enrichr.
Enrichr is a comprehensive gene set enrichment tool commonly used in many bioinformatics pipelines <cit.>.

We integrated Enrichr into BRAD to leverage gene set annotations, curated pathways, and ontology knowledge to enhance BRAD's responses to user queries.

This allows for enrichment of literature mined gene sets as well as those found in users data directly within the chatbot's interface. 











  
Gene Ontology.
The Gene Ontology (GO) database is a bioinformatics initiative to provide a uniform approach and vocabulary to studying genes and gene products across different species <cit.>.

This database aims to provide a consistent, curated vocabulary and annotations for genes across different species, making it an excellent resource to integrate into BRAD.

When gene names, GO terms, or other keywords are identified in BRADs responses to the user, BRAD has the ability to define these GO terms and download charts that diagram their relationships with other genes.

Moreover, the GO database provides papers associated with different terms, and these papers can be integrated via the webscraping codes.



  
PubMed, arXiv, and bioRxiv.
While RAG allows users to supplement s with additional information from a text database, the response quality is constrained by the relevance of the literature in the initial database.

However, databases may sometimes fail to provide satisfactory answers due to containing irrelevant information or lacking the latest research.

To address this, BRAD incorporates the ability to update its literature database.

We implemented searching of arXiv, bioRxiv, and PubMed, but could broaden this process to include additional databases.

BRAD can both display search results to the user, link articles, summarize recent abstracts, download and display papers, and embed the new papers into its database.

This enables BRAD to form a closes, human in the loop system.

User feedback – either in the form of detected poor responses or in the form of explicitly telling BRAD to update – allow BRAD to improve its retrieval knowledge base over time.



When retrieving new documents on PubMed, BRAD only gathers publicly accessible papers. 


This ultimately limits the documents BRAD has access to, excluding key information and notable findings.

To combat this problem, users may directly add their own documents to databases to ensure their favorite findings are accounted for in BRAD's responses.














 §.§ Planner Module




Unlike other modules of BRAD that respond to individual user queries, the  module can handle complex queries that require multiple tools. Given a user input, the scheduler determines a series of smaller tasks that can be executed using one module at a time whose collective output will address the user's query. These tasks are shown to and edited by a human, after which point the set of tasks are automatically executed; see <ref> for a high level schematic of a workflow involving the . 






 §.§ Module Selection and Control Flow


A series of mechanisms are utilized by Brad to determine the appropriate module to execute specific tasks or orchestrate complex processes involving multiple modules.



  
Module Selection with Semantic Routing.
BRAD's direction of a query to the RAG or any other module is a standard control flow problem.

To route a users query to the appropriate module or function, BRAD uses a series of semantic routers.

To manage the control flow and determine which databases or modules to use in response to a user query, we employ semantic routers to make fast, real time decisions.

Semantic routers are prediction or classification models that process user queries to determine the appropriate path for the prompt to flow through the chatbot.

At a high level, BRAD uses semantic routing to determine if a query requires fetching data online, access to private data or custom software, or a standard RAG and  response.

At a lower level, these routers can be used to identify specific function calls and parameters from queries to BRAD.



  
Adaptive Routing through Reinforcement Learning.
Routers are initialized with a series of prompts to guide BRAD's control flow, and these prompts are refined over time via a simple reinforcement learning procedure.

When interacting with BRAD, semantic routers direct user prompts to use different modules, perform online retrieval, or query the text database.

Users can force a query down a path or indicate if a prior query was misrouted.

As users interact with BRAD, the set of prompts to direct down different paths is recorded.

This feedback is incorporated into the router, enabling similar queries to follow the appropriate route in the future.



  
Control Flow.
The router can manage more intricate tasks schedules beyond simple linear execution. For example, the scheduler might first execute RAG to gather initial information and then, based on the quality of the answer, proceed to search the web for additional literature. The scheduler's combination of an instruction pointer and an instruction list allows for sophisticated control flow patterns, including loops, if/else statements, and other logic.

For instance, consider the scenario where BRAD receives a query addressed at the RAG but has no text database for retrieval. In this case, the scheduler might construct a three stage loop consisting of: (1) download relevant literature to the user's query and include it in the database; (2) propose an answer based on RAG to the user's query; and (3) determine if the answer in stage (2) is satisfactory, in which case the chain ends, or if the answer is unsatisfactory, the instruction pointer will return to step (1). In Step (3), Brad uses the  at the core of BRAD to evaluate complex control flow decisions and determine if previous outputs in the chain satisfy particular conditions set by the scheduler.








 §.§ Prompt Templates

A core component of BRAD is a prompt template library designed to interface with various modules and generate responses for the users. This library of templates contains two distinct classes and is based on recent  prompting techniques.




  
Input v. Output Prompts.
We developed two classes of prompts corresponding to the primary uses of an  within BRAD. The first class of prompts is designed to transform a user's query into actions, leveraging the  to generate or execute code. This includes tasks such as data or software selection, generating or editing code to execute, or selecting databases and online search terms, among other tasks. In these prompts, a series of techniques for improving the responses via structured output, few shot prompting, and other methods are used.

The second class of prompts is used to generate output responses for the user. These include the generation stage of RAG, where pieces of text are formatted into a prompt template to improve user responses, prompts used to summarize data resources and code utilized, and prompts for generating output PDF documentation. These prompts also convey specific instructions and often are templated to insert information and data retrieved while executing various modules.



  
Few Show Prompting.
Few-shot prompting is a technique to provide  responses by including a set example outputs as input to the model.
For instance, to execute code with BRAD a set of prompts have been constructed that contain examples on to construct an appropriate piece of code in either or python. These few examples are both built into BRAD's prompt library and can additionally be provided in the documentation of software being run. By providing examples, the variability in response structure as well as the quality of the 's output are improved.



  
Chain of Thought Prompting.
Chain of thought prompting is a technique that uses reasoned through examples as input to an  to demonstrate to how to reason through complex or multistep queries. When asking an  to solve a problem, the user could specify in the prompt to solve the problem using a series of steps. In BRAD's code execution program, a series of such prompts are provided. When formulating a function call to execute a script, the  is asked to generate an appropriate piece of code based on the users input, software documentation, and few shot examples of running the code. The output of the prompt is structured to require the  to explain is chain of thought and reasoning behind the constructed code.



§ RESULTS AND EXAMPLES

This section presents extended results and implementation details related to the use cases of BRAD discussed in the paper.


 §.§ Lab Notebook Evaluation



  §.§.§ Qualitative Example

Since all the BRAD responses to Q1 were too long, the complete output is provided here.








0.95Q1: How can we view Pore-C as a hypergraph?
[raster equal height,raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = Enhanced RAG,colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 8cm]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly in the context of chromatin interactions. To view Pore-C as a hypergraph, we can consider the following aspects:

    
  * Vertices and Edges: In a hypergraph, vertices represent entities, and hyperedges connect multiple vertices. In the context of Pore-C, the vertices can represent genomic loci (e.g., genes, regulatory elements, or specific regions of the genome). The hyperedges can represent the interactions between these loci, where a single hyperedge can connect multiple loci that interact with each other within a specific spatial context.
    
  * Higher-Order Interactions: Unlike traditional graphs where edges connect pairs of vertices, hypergraphs allow for the representation of higher-order interactions. In Pore-C, certain genomic regions may interact with several other regions simultaneously, which can be effectively captured by hyperedges. This is particularly relevant in understanding complex regulatory networks and chromatin architecture.
    
  * Data Representation: When analyzing Pore-C data, one can construct a hypergraph where each hyperedge corresponds to a set of loci that are in close spatial proximity or show significant interaction frequencies. This representation can help in identifying clusters of interacting regions and understanding the functional implications of these interactions.
    
  * Analysis and Algorithms: Viewing Pore-C data as a hypergraph opens up various analytical approaches, such as community detection, clustering, and network analysis, which can provide insights into the organization and function of the genome. Algorithms designed for hypergraphs can be applied to identify key regulatory regions, interaction hubs, and other biologically relevant structures.
    
  * Visualization: Hypergraphs can be visualized using specialized software that can handle higher-order relationships, allowing researchers to explore the complex interactions revealed by Pore-C data in a more intuitive manner.

In summary, conceptualizing Pore-C data as a hypergraph allows for a more nuanced understanding of genomic interactions, accommodating the complexity of higher-order relationships that are inherent in chromatin organization.



0.95Q1: How can we view Pore-C as a hypergraph?
[raster equal height,raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = Standard RAG,colback=green!5!white,colframe=green!75!black,fonttitle=, width = 8cm]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly in the context of chromatin interactions. Viewing Pore-C data as a hypergraph can provide a more nuanced understanding of the complex relationships between genomic elements.

In a hypergraph, nodes represent individual genomic features (such as genes, regulatory elements, or chromatin domains), and hyperedges represent interactions or relationships that can connect multiple nodes simultaneously. This is particularly relevant for Pore-C data, where interactions are not limited to pairs of genomic elements but can involve multiple elements at once.

Here’s how you can conceptualize Pore-C as a hypergraph:

    
  * Nodes: Each node in the hypergraph can represent a specific genomic feature, such as a gene, enhancer, or other regulatory elements.
    
  * Hyperedges: A hyperedge connects multiple nodes that interact with each other. In the context of Pore-C, a hyperedge could represent a chromatin interaction that involves several genomic regions simultaneously. For example, if a gene interacts with multiple enhancers, this relationship can be captured as a single hyperedge connecting the gene and all relevant enhancers.
    
  * Multiplicity of Interactions: Pore-C data often reveals that genomic regions can interact with many others, which is a key characteristic of hypergraphs. This allows for a more comprehensive representation of the complex interactions that occur within the nucleus.
    
  * Analysis: By treating Pore-C data as a hypergraph, researchers can apply various graph-theoretical approaches to analyze the structure and dynamics of chromatin interactions. This can include identifying clusters of interacting regions, understanding the role of specific genomic features in the overall architecture, and exploring how these interactions change under different conditions.
    
  * Visualization: Hypergraphs can be visualized using specialized software that allows for the representation of multi-way interactions, making it easier to interpret the complex relationships inherent in Pore-C data.

In summary, viewing Pore-C as a hypergraph allows for a richer representation of genomic interactions, capturing the complexity of chromatin organization and facilitating advanced analyses of genomic architecture.



0.95Q1: How can we view Pore-C as a hypergraph?
[raster equal height,raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = C. GPT-4.o,colback=red!5!white,colframe=red!75!black,fonttitle=,  width = 8cm,before=
]
Pore-C is a technique used to study the three-dimensional organization of the genome, particularly focusing on chromatin interactions. To view Pore-C as a hypergraph, we can consider the following aspects:

    
  * Vertices and Edges: In a hypergraph, vertices represent entities, and hyperedges connect multiple vertices. In the context of Pore-C, the vertices can represent genomic loci (e.g., genes, regulatory elements, or specific regions of the genome). The hyperedges can represent the interactions between these loci, where a single hyperedge can connect multiple loci that interact with each other.
    
  * Higher-Order Interactions: Unlike traditional graphs where edges connect pairs of vertices, hypergraphs allow for the representation of higher-order interactions. In Pore-C data, a single interaction event may involve multiple genomic regions simultaneously, which can be effectively captured by hyperedges.
    
  * Complexity of Genomic Interactions: The genome is not organized in a linear fashion; rather, it exhibits complex interactions that can involve multiple regions. By modeling these interactions as a hypergraph, we can better represent the multifaceted nature of chromatin architecture and the relationships between different genomic elements.
    
  * Analysis and Visualization: Viewing Pore-C data as a hypergraph can facilitate advanced analytical techniques, such as community detection, clustering, and network analysis, which can help identify functional modules or regulatory networks within the genome.
    
  * Applications: This hypergraph representation can be useful for understanding how different genomic regions collaborate in processes such as gene regulation, transcription, and the overall spatial organization of the genome.

In summary, conceptualizing Pore-C data as a hypergraph allows for a more nuanced understanding of the complex interactions within the genome, enabling researchers to explore the functional implications of these interactions in a more comprehensive manner.






  §.§.§ RAG Assessment (RAGAs) evaluation metrics
















Five metrics were used to evaluate the utility of BRAD's RAG system:

    
  * Faithfulness: This quantifies the consistency between BRAD's answer and the literature.
    
  * Answer Relevance: This evaluates if the response is applicable to the particular question.
    
  * Context Precision: This quantifies the ranking of the chunks BRAD uses to formulate its response relative to the ground truth.
    
  * Context Recall: This measures the use of context in BRAD's response.
    
  * Answer Correctness: This is the F1 score of claims that can be verified from the ground truth.

All metrics are from 0 to 1 with higher scores indicating better performance.

In order to determine these five metrics, a combination of the following four components were used: question, response, context, and ground truth. Since the questions are based on cutting edge research topics, their ground truth is based in research papers in our aforementioned database. The context is the chunks of text RAG uses to generate a response.

To develop quantifiable metrics from  responses, (<cit.>) proposed the following scheme. For an individual question, the context used to construct the question and the answer produced by BRAD's RAG are split into small, verifiable claims. Then, statements from BRAD's answer can be compared in a one for one fashion to find claims indicated by BRAD substantiated or not substantiated in context of the literature. This approach allows us to construct a standard confusion matrix where:

    
  * TP: True Positives are statements that are present in both the ground truth and the generated response.
    
  * FP: False Positives are statements that are present in the generated response but not the ground truth.
    
  * TN: True Negatives are statements that are present in the ground truth but not in the generated response.
    
  * FN: False Negatives are statements that are present in the ground truth but not in the generated response.





  
Faithfulness.
Faithfulness refers to the factual consistency of the generated response to a question against the given context. Since this metric is entirely based on the context given, if the context given is faulty, then a factually incorrect answer can still be considered faithful.

Each of the generated claims is then checked against the given context to determine if the claim can be inferred from context.

    Faithfulness = Number of Claims from Response inferred from Context
    /Number of Claims from Response.




  
Answer Relevance.
Answer Relevance is a measure of how closely the generated response and given prompt are related. Answers that are incomplete or contain redundant information are typically assigned lower scores. We use the following formula to calculate answer relevance

    Answer Relevance = 1/N(∑_k=1^N e_g_k· e_o/e_g_ke_o)

where N is the number of generated questions, e_g_k is the vectorized embedding of the kth generated question from a generated response, and e_o is the vectorized embedding of the original question.



  
Context Precision.
Context Precision measures how highly ranked ground truth relevant chunks are compared to other chunks in the context of the response. We use the following formula to calculate context precision

    Context Precision = 1/Number of Relevant Chunks(∑_k=1^NNumber of Relevant Chunks up to position k·δ_k /k).

where δ_k is 1 is the chunk is relevant and 0 otherwise. Although this does not directly prove anything inherently about the generated response, it does serve as a metric to demonstrate the effectiveness of page rank reranking.



  
Context Recall.
Context Recall measures how related the retrieved context compares to the ground truth. It is calculated similar to faithfulness.

    Context Recall = Number of Ground Truth claims that can be linked to the context/Number of Ground Truth claims.




  
Answer Correctness.
Answer Correctness measures how the generated response compares to the ground truth. It is calculated by averaging the semantic similarity between the generated response and the ground truth and factual similarity. The semantic similarity is calculated by embedding both the ground truth and the generated response into the same embedding space and calculating the cosine similarity between the two vectors (similar to answer relevance calculations). The factual similarity is calculated through the F1 score

    F1 = TP/TP + 1/2(FP + FN).




 §.§ Questions for Quantitative Experiment

Provided is a list of questions used for the Quantitative experiment.



    
  * How do B cells undergo affinity maturation during the germinal center reaction? 
  * How is the timely clearance of neutrophils critical for the resolution of inflammation? 
  * How did Levinson and Smith use the method of proving the divergence integrated along any limit cycle to establish uniqueness in their study of limit cycles? 
  * What steps were involved in the PacBio library preparation for the American Gut Project samples? 
  * How does Morse theory relate to the number of critical points in a Morse function on a manifold? 
  * How do ectopically expressed transcription factors interact with endogenous components of recipient cells' transcriptional network in induced lineage reprogramming? 
  * How are the kinetic parameters for each gene estimated using nonlinear least-squares methods in single-cell kinetics data? 
  * How has in vivo reprogramming with viral vectors been used as a strategy for regeneration of the central nervous system in mice? 
  * How does Batch Normalization apply a linear transformation to the output mean in the context of OrthDNNs? 
  * How is protein identification confirmed in the antibody characterization process? 
  * How does the concept of the unit interval relate to the Kraft inequality in coding theory? 
  * How is the mean radius of gyration (Rg2) of chromosome 12 TADs calculated and presented in the data analysis? 
  * How many critical points does the height function on the n-sphere have? 
  * How do B cells gain access to the GC reaction and what role does inter-clonal competition play in this process? 
  * What are some challenges in studying cell-cycle dynamics, particularly in terms of protein levels and post-translational modifications? 
  * How do the highly non-linear dynamics of cooperative and competitive regulation of transcription factors impact the differentiation efficiency in cell reprogramming experiments? 
  * What is the significance of periodic network connectivity in the analysis of the discrete-time consensus algorithm? 
  * How did the recognition of the power of computational techniques contribute to the growth of network analysis and network science? 
  * How can physicians claim CME credits for reading content from the Journal of Investigative Dermatology? 
  * How does the inclusion of temporal information in the system impact the complexity and effectiveness of the detector-generation process in immune-inspired network IDS? 
  * How is β-catenin involved in the promitogenic effect of TGF-β? 
  * How is IRF2 editing associated with keratinocyte differentiation and skin barrier formation? 
  * How did the use of a "physiological" buffer impact the activity of RNA polymerases in cells during the experiment? 
  * How does the spatial organization of human chromosomes change during the cell cycle? 
  * How does the substitution of serines with glutamate in cyclin B1 affect its rate of nuclear import in Xenopus oocytes? 
  * How is nuclei isolation performed for skeletal muscle tissue samples? 
  * How can Hodge Decomposition be used to identify tie strength with high accuracy? 
  * How does CTCF binding factor play a role in creating metastable states related to DNA methylation? 
  * How did Lou Pecora's research interests shift from solid-state physics to chaos theory while working at the U.S. Naval Research Laboratory in Washington? 
  * How are nucleoids released from Escherichia coli and what is their structure? 
  * How does Cyclin E control S phase progression during Drosophila embryogenesis? 
  * How does the experimental efficiency impact the quality of in silico contact maps in Hi-C, SPRITE, and GAM experiments? 
  * What are some of the mathematical problems discussed by David Hilbert in his lecture at the Second International Congress of Mathematics in 1900? 
  * How can sequence count data be used in differential expression analysis? 
  * How were peaks called in the ChIP-seq data analysis, and what role did input controls play in this process? 
  * How is the level of nucleosome occupancy at the TSS related to accessibility during mitosis? 
  * How can the parameters of an affine function be obtained by evaluating the function at specific vectors? 
  * How does over-fitting affect the RMS prediction error in data fitting models? 
  * How does DEC205 targeting affect the interactions between GC BCs and TFH cells in the context of antigen presentation? 
  * How do nearby trajectories behave close to the chaotic attractor in the Lorenz system? 
  * How do deep neural networks contribute to the mapping of gene expression levels from WSIs in the study of tumor heterogeneity and survival outcomes in breast and lung cancer? 
  * How does the spectral dimension of a simplicial complex affect the possibility of frustrated synchronization in a network? 
  * What is the significance of transforming a matrix A into real canonical form B in the context of solving differential equations? 
  * What is the significance of matrix inverses in solving linear equations and least squares problems? 
  * How does supervised learning typically approach binary classification tasks? 
  * How can data-driven algorithmic approaches help in predicting transcription factors for directed cell conversion? 
  * Why were researchers shocked to find that the longest REM episodes occurred near the beginning of sleep, rather than near the end? 
  * What was explored in the study regarding the temporal program of gene expression in human fibroblasts in response to serum? 
  * How are homomorphism numbers calculated in the context of randomly weighted graphs? 
  * How does the CalCB method correct for known Hi-C biases and CNV in the estimation of Hi-C contact matrices? 
  * How can weights be optimized to minimize downside risk for a given target annualized return? 
  * How can a higher-order tensor be regarded as a formal representation of a linear mapping between different spaces, such as matrix and vector spaces, based on the concept of an isomorphic link? 
  * How is haplotype imputation utilized in single-cell chromatin conformation capture? 
  * How are vector fields defined and approximated in the context of integrability? 
  * How are local connectivity patterns represented in hypergraphs? 
  * How does Edge PageRank differentiate itself from traditional centrality measures in networks? 
  * How does T2T-CHM13 improve variant calls compared to GRCh38 in the context of the 1KGP datasets? 
  * How has the direct conversion of various types of cells been successfully achieved and what are the potential applications of this method in medical treatment and research? 
  * How does the local track-correction module improve tracking performance in EllipTrack? 
  * How do binary relationships affect complex topologies in network science? 
  * What happens to most cells in Drosophila melanogaster embryos after entering G1 phase? 
  * What causes two Hopf bifurcation points to merge in degenerate cases? 
  * How does a monotone subsequence in (xn) in R imply a convergent subsequence for every bounded sequence in R? 
  * How does a spiral scaffold affect chromosome loop formation during cell division, especially in relation to inner and outer loop sizes in various models? 
  * How are eigenvalues of a square matrix affected by adding a matrix with all entries as 1? 
  * How do dominant-negative mutations contribute to genetic dominance in cancer development despite other initiation mechanisms? 
  * How do germinal centers aid immune response coordination and high-affinity antibody generation, considering TFH cell movement, GC reuse, and GC B cell clonal restriction? 
  * How do Gata4, Mef2c, and Tbx5 affect induced cardiac transcriptional reprogramming? 
  * What concept explains cascades of period doubling bifurcations in mathematical ecology models like the logistic map and Ricker model, discovered by Feigenbaum in discrete-time dynamical systems? 
  * How does a monotone subsequence in a bounded sequence in R relate to the Bolzano-Weierstrass Theorem? 
  * How do manifolds contribute to proving saddle equilibrium states in dynamical systems? 
  * What controls cell cycles in Drosophila embryogenesis, coordinating phases with growth and G1 arrest until larva hatches? 
  * What role does Myc activity play in apoptosis and oncogenic risk in Burkitt’s lymphoma? 
  * How does a magnetic field affect the supercurrent behavior observed by experimentalists studying tunneling supercurrents? 
  * How has math influenced immunology, especially with help from physicists? 
  * How does the requirement for adjacent edges to share two nodes in the random walk process affect its representation and Markovian nature, changing the use of traditional graph and matrix techniques for analysis? 
  * How do plasma cells differ from effector cells in humoral immunity roles, given the complex regulation mechanisms at cellular and molecular levels? 
  * How can feature mappings enhance regression models with dimension reduction and standardized features? 
  * How can protein expression in cells be controlled through growth factors availability or production in other cell types? 
  * How do post-translational modifications affect E2F1 stability and function in response to DNA damage like cisplatin? 
  * How do positive eigenvalues in a matrix connect to Sylvester's Law of Inertia and the span of eigenvectors for positive eigenvalues in the transformed matrix? 
  * How did Gemma Frisius propose longitude could be determined, as pursued by Huygens and others? 
  * How to calculate minimal state variables for a given transfer function in linear dynamical systems? 
  * How can campaign strategies be influenced by clustered voter representatives, and what is the role of k? 
  * How does finding a compatible similarity measure benefit workload and efficiency in stochastic consensus clustering? 
  * How does the spindle assembly checkpoint help ensure correct sister-chromatid alignment during mitosis? 
  * How does V(D)J recombination impact allelic exclusion in B-cell development? 
  * How do transcription factors affect cardiac reprogramming through co-occupied binding sites and synergistic effects of Tbx5 and Gata4? 
  * What is the initial mechanism that holds sister chromatids together by intertwining duplicated DNA molecules at adjacent replication forks? 
  * What technological innovation enabled sound and image transmission in the 20th century, leading to advancements in electronics? 
  * How does Ime2 stimulate Ndt80 in yeast for meiosis I and G1/S progression? 
  * Which locus in the (Re λ, Imλ)-plane indicates the Neimark-Sacker bifurcation with a stable closed invariant curve outside the circle? 
  * How does Mengerian property connect to hypergraphs in graph theory? 
  * How does AEP relate to entropy in information theory when dividing sequences into typical and nontypical sets based on sample entropy? 
  * What kind of trajectories in the restricted three-body problem return close to their initial state? 
  * How is Collagenase IV used in iPS cell dissociation for culture before transferring to a new dish on SNL feeder cells? 
  * How was the "global" category of smooth manifolds and maps created from the "local" category of open sets in Euclidean space and smooth maps? 
  * What makes t-SNE unique in visualizing high-dimensional data effectively? 
  * How does Ig gene rearrangement in B-cell development relate to Ig gene allelic exclusion regulation? 
  * How is Tucker decomposition connected to higher-order PCA using factor matrices and a core tensor? 
  * How does RIG-I activation by RN7SL1 impact cGAS-STING signaling in metastatic cells? 
  * How is the preselection productive HCDR3 length distribution approximated to calculate the percentage of initial productive IgH rearrangements removed during B cell development? 
  * What stress-response system in mammalian cells prevents excessive cell proliferation by activating p53 in response to overproduction of key mitogenic signaling proteins? 
  * How does RIG-I activation by RN7SL1 impact cGAS-STING signaling in cancer cells? 
  * How does serum stimulation affect DNA replication timing and gene expression in fibroblasts? 
  * How many common neighbors do (1,1) and (2,2) have in a SRG? 
  * What was discussed by Laemmli about metaphase chromosome bands? 
  * What spatial changes occur in Hi-C maps as cells progress from prometaphase to metaphase in mitosis? 
  * What's the condition for a map's derivative to be an immersion between smooth manifolds? 
  * What concept represents non-stationary, periodic solutions in dynamical systems and why are they significant in nonlinear dynamics? 
  * How does community modularity change with τ values in time-evolving analysis, shown by a decreasing trend from 0.70 to 0.36? 
  * How do high-order interactions affect diversity in ecological communities in simulations with different interaction strengths and species numbers? 
  * What was the AS change in FOXP1 gene at day 10 post neural induction and its impact on exon 18 inclusion in H9 hESCs? 
  * How does aligning subjective and objective probability distributions address model misspecification concerns in rational expectations models? 
  * How do miRNAs affect E2F1 protein levels in cells with 3'UTR reporter due to c-myc and E2F1, considering transcript expression in senescent populations? 
  * How do pigmented cells and chromatin affect skin patterns in zebrafish? 
  * How can vectors model cash flow movement in finance? 
  * How do p53 activators affect CDK2 activity before mitosis in relation to p21 expression? 
  * How do Fos and Jun proteins interact in wound healing, and what role do HIF factors play in the hypoxic response to tissue injury? 
  * How can genetic analysis help prioritize therapy for cancer patients without actionable oncogene mutations, considering gene signature overlap with specific cell types in different body regions? 
  * What roles do cdcgenes play in cell cycle control, especially in mitosis entry and S phase replication origin activation? 
  * What role do homoclinic orbits play in bifurcation diagrams in curve dynamics with limit cycles and saddle points? 
  * What components form the inner kinetochore in higher eukaryotic cells and how do they aid in chromosome attachment and tension during cell division? 
  * What are the implications of recent research on converting glial cells, like astrocytes, into neurons for brain repair therapies? 
  * What's the BSC capacity with crossover probability for a coding scheme with two 3-length codewords and an average error probability of 0.285? 
  * How does Morse theorem affect density of Morse functions in space of smooth functions on a manifold? 
  * What defines a true somatic stem cell in terms of self-renewal and progeny generation, considering differentiation and lineage commitment in Waddington's model? 
  * How does the cell cycle affect epigenetic marks for memory, considering mitosis and S-phase impact on gene expression and chromatin? 
  * How do Wnts affect somite patterning in myogenesis? 
  * How do Wnt proteins affect somite cell commitment? 
  * How do cells in an epithelium maintain adhesive contacts with neighbors during division, and how does this relate to junction formation and cell cleavage in cytokinesis? 
  * How is transversality defined in dynamical systems with periodic orbits and hypersurfaces? 
  * How do basic fibroblast growth factor-impregnated gelatin microspheres affect fibroblast migration in 3D wound healing? 
  * How is the adjacency matrix used in link prediction algorithms in different fields? 
  * How do Gata4, Mef2c, and Tbx5 affect induced cardiac myocyte reprogramming and heart repair? 
  * Which company provided the artificial dura for the primate BMI task? 
  * How to simplify integro-differential equations using real canonical form for an n x n matrix? 
  * Can you provide examples of research studies on dynamical processes in complex networks beyond pairwise interactions? 
  * What is Herman's theorem's role in analyzing Morse-Smale vector fields with the Poincare-Bendixson theorem? 
  * How do algorithms for integer multiplication use polynomial rings and tensors? 
  * How does LZ algorithm efficiency impact file compression in terms of bits for pointers and match lengths in LZ77? 
  * How do tangent vectors and constants characterize an expanding map on a compact Riemannian manifold, and how does this relate to stability in topological spaces? 
  * How are external circuit parameters chosen to place complex-conjugate poles slightly right of jω-axis in a multiterminal device circuit? 
  * How do V, D, and J segment rearrangements impact antibody and T cell receptor encoding in the immune system? 
  * How can a cell create lasting memory of a response using transcription and DNA replication? 
  * How does TF over-expression affect network dominance in the omnigenic model, considering simplified regulatory networks and TFs' role in cellular reprogramming? 
  * How does a graphon relate to the node set [0,1] in terms of structure and connectivity? 
  * What is the role of the Nbs1 subunit in activating ATM through the MRN complex in response to double-strand breaks? 
  * What are the key features and practical uses of the Watts model in social contagions on networks, considering its relationship with Granovetter's threshold models and the incorporation of simplicial complexes in modeling transmission dynamics? 
  * What happens to the fixed points of period-seven cycles as r changes in one or two-parameter bifurcations? 
  * How does vertex-transitivity demonstrate the relationship between a graph's structure and its automorphisms, especially in Cayley graphs and the lexicographic product? 
  * How does Cyclin B1-Cdk1 complexes affect CDK2 activity post-mitosis in proliferating cells? 
  * How is the two-parameter bifurcation diagram constructed for a system with codim 1 and codim 2 bifurcations of equilibria, limit cycles, and homoclinic orbits? 
  * How to reconstitute functional modules in vitro using gene expression and morphology from various cell types? 
  * How do centrosomes help with sister chromatid bi-orientation during spindle assembly? 
  * How does scRNA-seq compare to snRNA-seq in studying transcriptional reprogramming and chromatin interactions? 
  * How does removing ECB sequences affect Start synchronization in yeast cells, considering growth rate differences? 
  * Which fellowship organization supported the research at the Yale Stem Cell Center Genomics Core Facility? 
  * How do factors contribute to hiHeps induction for drug metabolism? 
  * How do aged HSCs compare to younger HSCs in terms of regenerative potential, proliferation, and lineage tracing data? 
  * How is the Edge PageRank equilibrium point represented in a dynamical system, considering node centrality and random walks? 
  * How can Sis interference in Vj-Jj interval prevent secondary recombination events on the same Ig allele? 
  * How can periodic solutions be shown under Hopf conditions with Lyapunov-Schmidt reduction and Poincaré normalization? 
  * How does lateral inhibition affect feather primordia spacing in chicks, considering growth and regeneration in pattern formation? 
  * How does the lasso penalty promote sparsity in regression for neuroimaging data? 
  * How is the Laplace operator shown in TT-format for tensors in linear operators and elliptic PDE solutions? 
  * What advantage does CD9 offer in isolating murine hematopoietic stem cells? 
  * How is a rewriting event in GNA models defined in relation to node mapping and Kronecker matrix sampling? 
  * How is fresh frozen tissue fixed with Formaldehyde for immunoﬂuorescence staining in mouse embryo tissue? 
  * What causes unexplained stock market fluctuations attributed to decentralized networks and herd behavior? 
  * What causes motor neuron degeneration in ALS and how do interactions with glia contribute? 







 §.§ Building a Database

Typically, we use our prior knowledge and publications of interest to construct a literature database for BRAD. However, the web scraping module makes it easy for BRAD to construct a new database aimed at a particular topic area or question.

To demonstrate the utility of the RAG and Web Scraping features integrated into BRAD, we tested BRAD's ability to respond to questions without a pre-existing database. BRAD was initially provided an empty database. Through alternating between RAG and web scraping, following <ref>, BRAD was able to adequately answer complex research questions by retrieving relevant information online.

For instance, consider the question: “What are single cell foundation models?" Initially, BRAD has no information on this topic. Responding with only the RAG module, BRAD will correctly identify a lack of information on this topic and can trigger a web search of this topic. Realizing the insufficiency of information, BRAD then proceeded to perform a web search to gather recent research papers and articles relevant to single cell analysis and foundation models. Equipped with the new information, BRAD's answer to the initial question is improved and is able to discuss this research area, providing the names of several notorious scs and a discussion of their use cases.

This process outlined in <ref> is designed by the planner module with clear instructions from the user. The  engine of BRAD is used at multiple stages throughout this process. Besides its use in the RAG and Web Search modules, the  also acts as an evaluator to determine the quality of the RAG's responses. Once the response meets sufficient quality, the  signals to exit the loop. Additionally, we have implemented safeguards to prevent the system from executing an infinite loop.





Question: What is a single cell foundation model?
[raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = RAG: Iteration 1 
(empty database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Based on the provided source and content, the initial database is empty, and I couldn't find any information regarding “single cell foundation models." I am confident that I cannot provide an accurate answer to this question with the current information available.


[title=Web Search,
                  colback=red!5!white,
                  colframe=red!75!black,
                  halign title=flush right
                  ]
arXiv search terms: “single cell foundation models", “machine learning for single cell data analysis"






[title = RAG: Iteration 1 
(populated database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Single cell foundation models are large-scale pretrained deep language models used for analyzing and annotating single-cell RNA sequencing (scRNA-seq) data. They can help improve the accuracy and speed of cell type annotation and identification in various research areas. Examples of these models include scBERT and a method for ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data. These models are used to facilitate queries and analysis of large cell atlas databases, such as the Tabular Muris and Tabular Sapiens, which contain single cell transcriptomic data for mice and humans, respectively.

Additionally, single cell foundation models can utilize transfer learning to enable predictions in network biology and gene regulation, making them valuable tools in understanding cell behavior and interactions in various biological processes. However, these models should be used to aid in basic scientific research and generating or validating mechanistic hypotheses, rather than providing definitive answers.


[title=Web Search,
                  colback=red!5!white,
                  colframe=red!75!black,
                  halign title=flush right
                  ]
arXiv search terms: “single cell machine learning", “deep learning for single cell analysis"


[title = RAG: Iteration 2 
(populated database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Single cell foundation models are artificial intelligence models designed to analyze and understand data from individual cells. They are constructed using single-cell transcriptomic data, which measures the expression levels of genes in individual cells. These models utilize machine learning algorithms, such as transformers, to learn gene-level and cell-level representation. They can accurately categorize known cell types and have the potential to uncover new biological insights in single-cell RNA sequencing data. Single cell foundation models are used in genomics and multi-omics research, showing promise in various downstream tasks, such as identifying cell types and states, discovering novel cell populations, inferring gene regulation networks, and more. Examples of single cell foundation models include scGPT, Genefomer, and scFoundation. They are a growing field of study, with ongoing advancements expected as single-cell technologies and omics sciences continue to develop.




Question: What is the role of MYOD in Weintraub's cellular reprogramming experiment?
[raster valign=top, raster halign=center,raster columns=1, raster rows=1,
colframe=white,colback=white,colbacktitle=blue!50!white,
]
[title = RAG: Iteration 1 
(empty database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
Based on the information provided in the initial database, I don't have any specific data about Weintraub's cellular reprogramming experiment or the role of MYOD in that experiment. After searching the literature and documents from a text database, I found that Weintraub's experiment aimed to reprogram fibroblasts into myoblasts using the MyoD gene. MyoD is a myogenic regulatory factor that plays a critical role in muscle cell differentiation and development. It reprograms fibroblasts to a muscle cell fate by binding to specific muscle-specific DNA elements and regulating the expression of muscle-specific genes. However, I couldn't find detailed information on the exact role of MYOD in this experiment. It's recommended to consult the original research article for more comprehensive insights.


[title=Web Search,
                  colback=red!5!white,
                  colframe=red!75!black,
                  halign title=flush right
                  ]
PubMed search terms: “MYOD", “Weintraub", “cellular reprogramming", “transcription factor", “myogenic differentiation", “muscle cell development", ...







[title = RAG: Iteration 2 
(populated database),colback=blue!5!white,colframe=blue!75!black,fonttitle=, width = 6cm]
MYOD is a transcription factor that plays a crucial role in Weintraub's cellular reprogramming experiment, where it successfully reprogrammed fibroblasts into skeletal muscle cells when expressed alone. MYOD initiates cell reprogramming by contacting E-boxes within closed chromatin and interacting with various regulatory layers such as nucleosome density, histone post-translational modifications, and chromatin folding. Additionally, MYOD can interact with other factors, such as the basal transcription machinery and the transcription elongation factor b, to co-regulate gene expression.

In the context of Weintraub's experiment, MYOD's co-activation involves interactions with the epigenetic machinery. MYOD's binding and activity can be modulated by other factors, including those indirectly recruited by MYOD to its target sites. Furthermore, MYoD makes use of long noncoding RNAs (lncRNAs) in cellular reprogramming. For instance, LncMyoD plays a role in the first step of MYOD-mediated trans-differentiation by interacting with MYOD across the genome and increasing chromatin accessibility at the co-occupied sites. LncMyoD is essential for the differentiation of muscle precursors and the myogenic conversion of fibroblast cells, but it is not needed for the differentiation of already committed myoblasts.

These interactions and regulatory roles of MYOD highlight the complexity of the mechanisms underlying cellular reprogramming and the importance of understanding the molecular dynamics of transcription factors and their interacting partners in this process.








 §.§ Advanced Search


Below, we show the full output produced by BRAD in the process of answering Q2: What role does PCNA play in the cell cycle and imaging? To answer this, BRAD executed a pipeline consisting of (1) running the  module to find first order interactions of PCNA in the STRING and HuRI databases <cit.>, (2) performing gene enrichment with the  module, and (3) using the RAG system in the  to find other relevant information. After completing these tasks, the  module was used to produce a report summarizing the entire process.

The  module, although not a focal point of the paper, is a module designed to process the chat log and  to produce a single,  document that describes BRAD's analysis. The goal is to ensure BRAD's work is transparent, well-documented, and clear, even for users unfamiliar with AI, so they can easily understand what occurred. The  is implemented by using s to generate latex code, according to a user specified  template, configured similarly to whitelisted software, to explain the results of each module.

In the below example, which begins on the following page, BRAD specifies an extended user prompt, characteristic of inputs that could initiate this pipeline. Then, a summary of the full pipeline is provided. The final main section is a summary of the outputs, in which BRAD lists the main information retrieved from the  module as well a a table and chart generated by the  module, as well as references to the used literature.

The report is generated by BRAD entirely in  and then automatically compiled. The formatting, writing, and style of the report is not always consistent or ideal, we find that the reports often reflect or summarize the pipelines well. That said, BRAD can also provide the  used to generate the report, from which a user could make edits.




[colback=red, width=, boxrule=0pt, sharp corners]
The remaining pages are a report automatically generated by BRAD in response to Q2.


[pages=-]BRAD-output-2.pdf















































