Upload 62 files

.dockerignore

@@ -0,0 +1 @@
+.git

.github/dependabot.yml

@@ -0,0 +1,15 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "pip" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "weekly"

.github/workflows/docker-bulid.yml

@@ -0,0 +1,24 @@
+name: GitHub Actions Workflow
+run-name: ${{ github.actor }} has started docker build workflow.
+on:
+  pull_request:
+    types: [opened, edited, ready_for_review]
+jobs:
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Git checkout
+        uses: actions/checkout@master
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          driver: docker
+      - name: Build Dockerfile
+        uses: docker/build-push-action@v4
+        with:
+          push: false
+          tags: assafelovic/gpt-researcher:latest
+          file: Dockerfile
+

.gitignore

@@ -0,0 +1,10 @@
+#Ignore env containing secrets
+.env
+.envrc
+#Ignore Virtual Env
+env/
+.venv/
+#Ignore generated outputs
+outputs/
+#Ignore pycache
+**/__pycache__/

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[email protected].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -0,0 +1,42 @@
+# Contributing to GPT Researcher
+First off, we'd like to welcome and thank you for your interest and effort in contributing to our open source project ❤️. Contributions of all forms are welcome, from new features and bug fixes, to documentation and more. 
+
+We are on a mission to build the #1 AI agent for comprehensive, unbiased, and factual research online. And we need your support to achieve this grand vision. 
+
+Please take a moment to review this document in order to make the contribution process easy and effective for everyone involved.
+
+## Reporting Issues
+
+If you come across any issue or have an idea for an improvement, don't hesitate to create an issue on GitHub. Describe your problem in sufficient detail, providing as much relevant information as possible. This way, we can reproduce the issue before attempting to fix it or respond appropriately.
+
+## Contributing Code
+
+1. **Fork the repository and create your branch from `master`.** 
+If it's not an urgent bug fix, you should branch from `master` and work on the feature or fix in there.
+
+2. **Conduct your changes.**
+Make your changes following best practices for coding in the project's language. 
+
+3. **Test your changes.**
+Make sure your changes pass all the tests if there are any. If the project doesn't have automated testing infrastructure, test your changes manually to confirm they behave as expected.
+
+4. **Follow the coding style.**
+Ensure your code adheres to the coding conventions used throughout the project, that includes indentation, accurate comments, etc.
+
+5. **Commit your changes.**
+Make your git commits informative and concise. This is very helpful for others when they look at the git log.
+
+6. **Push to your fork and submit a pull request.**
+When your work is ready and passes tests, push your branch to your fork of the repository and submit a pull request from there.
+
+7. **Pat your back and wait for the review.**
+Your work is done, congratulations! Now sit tight. The project maintainers will review your submission as soon as possible. They might suggest changes or ask for improvements. Both constructive conversation and patience are key to the collaboration process.
+
+
+## Documentation
+
+If you would like to contribute to the project's documentation, please follow the same steps: fork the repository, make your changes, test them, and submit a pull request. 
+
+Documentation is a vital part of any software. It's not just about having good code. Ensuring that the users and contributors understand what's going on, how to use the software or how to contribute, is crucial.
+
+We're grateful for all our contributors, and we look forward to building the world's leading AI research agent hand-in-hand with you. Let's harness the power of Open Source and AI to change the world together!

Dockerfile

@@ -0,0 +1,29 @@
+FROM python:3.11.4-slim-bullseye as install-browser
+
+RUN apt-get update \
+    && apt-get satisfy -y \
+    "chromium, chromium-driver (>= 115.0)" \
+    && chromium --version && chromedriver --version
+
+FROM install-browser as gpt-researcher-install
+
+ENV PIP_ROOT_USER_ACTION=ignore
+
+RUN mkdir /usr/src/app
+WORKDIR /usr/src/app
+
+COPY ./requirements.txt ./requirements.txt
+RUN pip install -r requirements.txt
+
+FROM gpt-researcher-install AS gpt-researcher
+
+RUN useradd -ms /bin/bash gpt-researcher \
+    && chown -R gpt-researcher:gpt-researcher /usr/src/app
+
+USER gpt-researcher
+
+COPY --chown=gpt-researcher:gpt-researcher ./ ./
+
+EXPOSE 8000
+CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Assaf Elovic
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

actions/web_scrape.py

@@ -0,0 +1,220 @@
+"""Selenium web scraping module."""
+from __future__ import annotations
+
+import logging
+import asyncio
+from pathlib import Path
+from sys import platform
+
+from bs4 import BeautifulSoup
+from webdriver_manager.chrome import ChromeDriverManager
+from webdriver_manager.firefox import GeckoDriverManager
+from selenium import webdriver
+from selenium.webdriver.chrome.service import Service
+from selenium.webdriver.chrome.options import Options as ChromeOptions
+from selenium.webdriver.common.by import By
+from selenium.webdriver.firefox.options import Options as FirefoxOptions
+from selenium.webdriver.remote.webdriver import WebDriver
+from selenium.webdriver.safari.options import Options as SafariOptions
+from selenium.webdriver.support import expected_conditions as EC
+from selenium.webdriver.support.wait import WebDriverWait
+from fastapi import WebSocket
+
+import processing.text as summary
+
+from config import Config
+from processing.html import extract_hyperlinks, format_hyperlinks
+
+from concurrent.futures import ThreadPoolExecutor
+
+executor = ThreadPoolExecutor()
+
+FILE_DIR = Path(__file__).parent.parent
+CFG = Config()
+
+
+async def async_browse(url: str, question: str, websocket: WebSocket) -> str:
+    """Browse a website and return the answer and links to the user
+
+    Args:
+        url (str): The url of the website to browse
+        question (str): The question asked by the user
+        websocket (WebSocketManager): The websocket manager
+
+    Returns:
+        str: The answer and links to the user
+    """
+    loop = asyncio.get_event_loop()
+    executor = ThreadPoolExecutor(max_workers=8)
+
+    print(f"Scraping url {url} with question {question}")
+    await websocket.send_json(
+        {"type": "logs", "output": f"🔎 Browsing the {url} for relevant about: {question}..."})
+
+    try:
+        driver, text = await loop.run_in_executor(executor, scrape_text_with_selenium, url)
+        await loop.run_in_executor(executor, add_header, driver)
+        summary_text = await loop.run_in_executor(executor, summary.summarize_text, url, text, question, driver)
+
+        await websocket.send_json(
+            {"type": "logs", "output": f"📝 Information gathered from url {url}: {summary_text}"})
+
+        return f"Information gathered from url {url}: {summary_text}"
+    except Exception as e:
+        print(f"An error occurred while processing the url {url}: {e}")
+        return f"Error processing the url {url}: {e}"
+
+
+
+def browse_website(url: str, question: str) -> tuple[str, WebDriver]:
+    """Browse a website and return the answer and links to the user
+
+    Args:
+        url (str): The url of the website to browse
+        question (str): The question asked by the user
+
+    Returns:
+        Tuple[str, WebDriver]: The answer and links to the user and the webdriver
+    """
+
+    if not url:
+        return "A URL was not specified, cancelling request to browse website.", None
+
+    driver, text = scrape_text_with_selenium(url)
+    add_header(driver)
+    summary_text = summary.summarize_text(url, text, question, driver)
+
+    links = scrape_links_with_selenium(driver, url)
+
+    # Limit links to 5
+    if len(links) > 5:
+        links = links[:5]
+
+    # write_to_file('research-{0}.txt'.format(url), summary_text + "\nSource Links: {0}\n\n".format(links))
+
+    close_browser(driver)
+    return f"Answer gathered from website: {summary_text} \n \n Links: {links}", driver
+
+
+def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
+    """Scrape text from a website using selenium
+
+    Args:
+        url (str): The url of the website to scrape
+
+    Returns:
+        Tuple[WebDriver, str]: The webdriver and the text scraped from the website
+    """
+    logging.getLogger("selenium").setLevel(logging.CRITICAL)
+
+    options_available = {
+        "chrome": ChromeOptions,
+        "safari": SafariOptions,
+        "firefox": FirefoxOptions,
+    }
+
+    options = options_available[CFG.selenium_web_browser]()
+    options.add_argument(f"user-agent={CFG.user_agent}")
+    options.add_argument('--headless')
+    options.add_argument("--enable-javascript")
+
+    if CFG.selenium_web_browser == "firefox":
+        service = Service(executable_path=GeckoDriverManager().install())
+        driver = webdriver.Firefox(
+            service=service, options=options
+        )
+    elif CFG.selenium_web_browser == "safari":
+        # Requires a bit more setup on the users end
+        # See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
+        driver = webdriver.Safari(options=options)
+    else:
+        if platform == "linux" or platform == "linux2":
+            options.add_argument("--disable-dev-shm-usage")
+            options.add_argument("--remote-debugging-port=9222")
+        options.add_argument("--no-sandbox")
+        options.add_experimental_option(
+            "prefs", {"download_restrictions": 3}
+        )
+        driver = webdriver.Chrome(options=options)
+
+    print(f"scraping url {url}...")
+    driver.get(url)
+
+    WebDriverWait(driver, 10).until(
+        EC.presence_of_element_located((By.TAG_NAME, "body"))
+    )
+
+    # Get the HTML content directly from the browser's DOM
+    page_source = driver.execute_script("return document.body.outerHTML;")
+    soup = BeautifulSoup(page_source, "html.parser")
+
+    for script in soup(["script", "style"]):
+        script.extract()
+
+    # text = soup.get_text()
+    text = get_text(soup)
+
+    lines = (line.strip() for line in text.splitlines())
+    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
+    text = "\n".join(chunk for chunk in chunks if chunk)
+    return driver, text
+
+
+def get_text(soup):
+    """Get the text from the soup
+
+    Args:
+        soup (BeautifulSoup): The soup to get the text from
+
+    Returns:
+        str: The text from the soup
+    """
+    text = ""
+    tags = ['h1', 'h2', 'h3', 'h4', 'h5', 'p']
+    for element in soup.find_all(tags):  # Find all the <p> elements
+        text += element.text + "\n\n"
+    return text
+
+
+def scrape_links_with_selenium(driver: WebDriver, url: str) -> list[str]:
+    """Scrape links from a website using selenium
+
+    Args:
+        driver (WebDriver): The webdriver to use to scrape the links
+
+    Returns:
+        List[str]: The links scraped from the website
+    """
+    page_source = driver.page_source
+    soup = BeautifulSoup(page_source, "html.parser")
+
+    for script in soup(["script", "style"]):
+        script.extract()
+
+    hyperlinks = extract_hyperlinks(soup, url)
+
+    return format_hyperlinks(hyperlinks)
+
+
+def close_browser(driver: WebDriver) -> None:
+    """Close the browser
+
+    Args:
+        driver (WebDriver): The webdriver to close
+
+    Returns:
+        None
+    """
+    driver.quit()
+
+
+def add_header(driver: WebDriver) -> None:
+    """Add a header to the website
+
+    Args:
+        driver (WebDriver): The webdriver to use to add the header
+
+    Returns:
+        None
+    """
+    driver.execute_script(open(f"{FILE_DIR}/js/overlay.js", "r").read())

actions/web_search.py

@@ -0,0 +1,25 @@
+from __future__ import annotations
+import json
+from duckduckgo_search import DDGS
+
+ddgs = DDGS()
+
+def web_search(query: str, num_results: int = 4) -> str:
+    """Useful for general internet search queries."""
+    print("Searching with query {0}...".format(query))
+    search_results = []
+    if not query:
+        return json.dumps(search_results)
+
+    results = ddgs.text(query)
+    if not results:
+        return json.dumps(search_results)
+
+    total_added = 0
+    for j in results:
+        search_results.append(j)
+        total_added += 1
+        if total_added >= num_results:
+            break
+
+    return json.dumps(search_results, ensure_ascii=False, indent=4)

agent/llm_utils.py

@@ -0,0 +1,124 @@
+from __future__ import annotations
+
+import json
+
+from fastapi import WebSocket
+import time
+
+import openai
+from langchain.adapters import openai as lc_openai
+from colorama import Fore, Style
+from openai.error import APIError, RateLimitError
+
+from agent.prompts import auto_agent_instructions
+from config import Config
+
+CFG = Config()
+
+openai.api_key = CFG.openai_api_key
+
+from typing import Optional
+import logging
+
+def create_chat_completion(
+    messages: list,  # type: ignore
+    model: Optional[str] = None,
+    temperature: float = CFG.temperature,
+    max_tokens: Optional[int] = None,
+    stream: Optional[bool] = False,
+    websocket: WebSocket | None = None,
+) -> str:
+    """Create a chat completion using the OpenAI API
+    Args:
+        messages (list[dict[str, str]]): The messages to send to the chat completion
+        model (str, optional): The model to use. Defaults to None.
+        temperature (float, optional): The temperature to use. Defaults to 0.9.
+        max_tokens (int, optional): The max tokens to use. Defaults to None.
+        stream (bool, optional): Whether to stream the response. Defaults to False.
+    Returns:
+        str: The response from the chat completion
+    """
+
+    # validate input
+    if model is None:
+        raise ValueError("Model cannot be None")
+    if max_tokens is not None and max_tokens > 8001:
+        raise ValueError(f"Max tokens cannot be more than 8001, but got {max_tokens}")
+    if stream and websocket is None:
+        raise ValueError("Websocket cannot be None when stream is True")
+
+    # create response
+    for attempt in range(10):  # maximum of 10 attempts
+        response = send_chat_completion_request(
+            messages, model, temperature, max_tokens, stream, websocket
+        )
+        return response
+
+    logging.error("Failed to get response from OpenAI API")
+    raise RuntimeError("Failed to get response from OpenAI API")
+
+
+def send_chat_completion_request(
+    messages, model, temperature, max_tokens, stream, websocket
+):
+    if not stream:
+        result = lc_openai.ChatCompletion.create(
+            model=model, # Change model here to use different models
+            messages=messages,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            provider=CFG.llm_provider, # Change provider here to use a different API
+        )
+        return result["choices"][0]["message"]["content"]
+    else:
+        return stream_response(model, messages, temperature, max_tokens, websocket)
+
+
+async def stream_response(model, messages, temperature, max_tokens, websocket):
+    paragraph = ""
+    response = ""
+    print(f"streaming response...")
+
+    for chunk in lc_openai.ChatCompletion.create(
+            model=model,
+            messages=messages,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            provider=CFG.llm_provider,
+            stream=True,
+    ):
+        content = chunk["choices"][0].get("delta", {}).get("content")
+        if content is not None:
+            response += content
+            paragraph += content
+            if "\n" in paragraph:
+                await websocket.send_json({"type": "report", "output": paragraph})
+                paragraph = ""
+    print(f"streaming response complete")
+    return response
+
+
+def choose_agent(task: str) -> dict:
+    """Determines what agent should be used
+    Args:
+        task (str): The research question the user asked
+    Returns:
+        agent - The agent that will be used
+        agent_role_prompt (str): The prompt for the agent
+    """
+    try:
+        response = create_chat_completion(
+            model=CFG.smart_llm_model,
+            messages=[
+                {"role": "system", "content": f"{auto_agent_instructions()}"},
+                {"role": "user", "content": f"task: {task}"}],
+            temperature=0,
+        )
+
+        return json.loads(response)
+    except Exception as e:
+        print(f"{Fore.RED}Error in choose_agent: {e}{Style.RESET_ALL}")
+        return {"agent": "Default Agent",
+                "agent_role_prompt": "You are an AI critical thinker research assistant. Your sole purpose is to write well written, critically acclaimed, objective and structured reports on given text."}
+
+

agent/prompts.py

@@ -0,0 +1,137 @@
+from datetime import datetime
+def generate_agent_role_prompt(agent):
+    """ Generates the agent role prompt.
+    Args: agent (str): The type of the agent.
+    Returns: str: The agent role prompt.
+    """
+    prompts = {
+        "Finance Agent": "You are a seasoned finance analyst AI assistant. Your primary goal is to compose comprehensive, astute, impartial, and methodically arranged financial reports based on provided data and trends.",
+        "Travel Agent": "You are a world-travelled AI tour guide assistant. Your main purpose is to draft engaging, insightful, unbiased, and well-structured travel reports on given locations, including history, attractions, and cultural insights.",
+        "Academic Research Agent": "You are an AI academic research assistant. Your primary responsibility is to create thorough, academically rigorous, unbiased, and systematically organized reports on a given research topic, following the standards of scholarly work.",
+        "Business Analyst": "You are an experienced AI business analyst assistant. Your main objective is to produce comprehensive, insightful, impartial, and systematically structured business reports based on provided business data, market trends, and strategic analysis.",
+        "Computer Security Analyst Agent": "You are an AI specializing in computer security analysis. Your principal duty is to generate comprehensive, meticulously detailed, impartial, and systematically structured reports on computer security topics. This includes Exploits, Techniques, Threat Actors, and Advanced Persistent Threat (APT) Groups. All produced reports should adhere to the highest standards of scholarly work and provide in-depth insights into the complexities of computer security.",
+        "Default Agent": "You are an AI critical thinker research assistant. Your sole purpose is to write well written, critically acclaimed, objective and structured reports on given text."
+    }
+
+    return prompts.get(agent, "No such agent")
+
+
+def generate_report_prompt(question, research_summary):
+    """ Generates the report prompt for the given question and research summary.
+    Args: question (str): The question to generate the report prompt for
+            research_summary (str): The research summary to generate the report prompt for
+    Returns: str: The report prompt for the given question and research summary
+    """
+
+    return f'"""{research_summary}""" Using the above information, answer the following'\
+           f' question or topic: "{question}" in a detailed report --'\
+           " The report should focus on the answer to the question, should be well structured, informative," \
+           " in depth, with facts and numbers if available, a minimum of 1,200 words and with markdown syntax and apa format.\n "\
+            "You MUST determine your own concrete and valid opinion based on the given information. Do NOT deter to general and meaningless conclusions.\n" \
+           f"Write all used source urls at the end of the report in apa format.\n " \
+           f"Assume that the current date is {datetime.now().strftime('%B %d, %Y')}"
+
+def generate_search_queries_prompt(question):
+    """ Generates the search queries prompt for the given question.
+    Args: question (str): The question to generate the search queries prompt for
+    Returns: str: The search queries prompt for the given question
+    """
+
+    return f'Write 3 google search queries to search online that form an objective opinion from the following: "{question}"'\
+           f'Use the current date if needed: {datetime.now().strftime("%B %d, %Y")}.\n' \
+           f'You must respond with a list of strings in the following format: ["query 1", "query 2", "query 3"].'
+
+
+def generate_resource_report_prompt(question, research_summary):
+    """Generates the resource report prompt for the given question and research summary.
+
+    Args:
+        question (str): The question to generate the resource report prompt for.
+        research_summary (str): The research summary to generate the resource report prompt for.
+
+    Returns:
+        str: The resource report prompt for the given question and research summary.
+    """
+    return f'"""{research_summary}""" Based on the above information, generate a bibliography recommendation report for the following' \
+           f' question or topic: "{question}". The report should provide a detailed analysis of each recommended resource,' \
+           ' explaining how each source can contribute to finding answers to the research question.' \
+           ' Focus on the relevance, reliability, and significance of each source.' \
+           ' Ensure that the report is well-structured, informative, in-depth, and follows Markdown syntax.' \
+           ' Include relevant facts, figures, and numbers whenever available.' \
+           ' The report should have a minimum length of 1,200 words.'
+
+
+def generate_outline_report_prompt(question, research_summary):
+    """ Generates the outline report prompt for the given question and research summary.
+    Args: question (str): The question to generate the outline report prompt for
+            research_summary (str): The research summary to generate the outline report prompt for
+    Returns: str: The outline report prompt for the given question and research summary
+    """
+
+    return f'"""{research_summary}""" Using the above information, generate an outline for a research report in Markdown syntax'\
+           f' for the following question or topic: "{question}". The outline should provide a well-structured framework'\
+           ' for the research report, including the main sections, subsections, and key points to be covered.' \
+           ' The research report should be detailed, informative, in-depth, and a minimum of 1,200 words.' \
+           ' Use appropriate Markdown syntax to format the outline and ensure readability.'
+
+def generate_concepts_prompt(question, research_summary):
+    """ Generates the concepts prompt for the given question.
+    Args: question (str): The question to generate the concepts prompt for
+            research_summary (str): The research summary to generate the concepts prompt for
+    Returns: str: The concepts prompt for the given question
+    """
+
+    return f'"""{research_summary}""" Using the above information, generate a list of 5 main concepts to learn for a research report'\
+           f' on the following question or topic: "{question}". The outline should provide a well-structured framework'\
+           'You must respond with a list of strings in the following format: ["concepts 1", "concepts 2", "concepts 3", "concepts 4, concepts 5"]'
+
+
+def generate_lesson_prompt(concept):
+    """
+    Generates the lesson prompt for the given question.
+    Args:
+        concept (str): The concept to generate the lesson prompt for.
+    Returns:
+        str: The lesson prompt for the given concept.
+    """
+
+    prompt = f'generate a comprehensive lesson about {concept} in Markdown syntax. This should include the definition'\
+    f'of {concept}, its historical background and development, its applications or uses in different'\
+    f'fields, and notable events or facts related to {concept}.'
+
+    return prompt
+
+def get_report_by_type(report_type):
+    report_type_mapping = {
+        'research_report': generate_report_prompt,
+        'resource_report': generate_resource_report_prompt,
+        'outline_report': generate_outline_report_prompt
+    }
+    return report_type_mapping[report_type]
+
+def auto_agent_instructions():
+    return """
+        This task involves researching a given topic, regardless of its complexity or the availability of a definitive answer. The research is conducted by a specific agent, defined by its type and role, with each agent requiring distinct instructions.
+        Agent
+        The agent is determined by the field of the topic and the specific name of the agent that could be utilized to research the topic provided. Agents are categorized by their area of expertise, and each agent type is associated with a corresponding emoji.
+
+        examples:
+        task: "should I invest in apple stocks?"
+        response: 
+        {
+            "agent": "💰 Finance Agent",
+            "agent_role_prompt: "You are a seasoned finance analyst AI assistant. Your primary goal is to compose comprehensive, astute, impartial, and methodically arranged financial reports based on provided data and trends."
+        }
+        task: "could reselling sneakers become profitable?"
+        response: 
+        { 
+            "agent":  "📈 Business Analyst Agent",
+            "agent_role_prompt": "You are an experienced AI business analyst assistant. Your main objective is to produce comprehensive, insightful, impartial, and systematically structured business reports based on provided business data, market trends, and strategic analysis."
+        }
+        task: "what are the most interesting sites in Tel Aviv?"
+        response:
+        {
+            "agent:  "🌍 Travel Agent",
+            "agent_role_prompt": "You are a world-travelled AI tour guide assistant. Your main purpose is to draft engaging, insightful, unbiased, and well-structured travel reports on given locations, including history, attractions, and cultural insights."
+        }
+    """

agent/research_agent.py

@@ -0,0 +1,188 @@
+# Description: Research assistant class that handles the research process for a given question.
+
+# libraries
+import asyncio
+import json
+import hashlib
+
+from actions.web_search import web_search
+from actions.web_scrape import async_browse
+from processing.text import \
+    write_to_file, \
+    create_message, \
+    create_chat_completion, \
+    read_txt_files, \
+    write_md_to_pdf
+from config import Config
+from agent import prompts
+import os
+import string
+
+
+CFG = Config()
+
+
+class ResearchAgent:
+    def __init__(self, question, agent, agent_role_prompt, websocket=None):
+        """ Initializes the research assistant with the given question.
+        Args: question (str): The question to research
+        Returns: None
+        """
+
+        self.question = question
+        self.agent = agent
+        self.agent_role_prompt = agent_role_prompt if agent_role_prompt else prompts.generate_agent_role_prompt(agent)
+        self.visited_urls = set()
+        self.research_summary = ""
+        self.dir_path = f"./outputs/{hashlib.sha1(question.encode()).hexdigest()}"
+        self.websocket = websocket
+
+    async def stream_output(self, output):
+        if not self.websocket:
+            return print(output)
+        await self.websocket.send_json({"type": "logs", "output": output})
+
+
+    async def summarize(self, text, topic):
+        """ Summarizes the given text for the given topic.
+        Args: text (str): The text to summarize
+                topic (str): The topic to summarize the text for
+        Returns: str: The summarized text
+        """
+
+        messages = [create_message(text, topic)]
+        await self.stream_output(f"📝 Summarizing text for query: {text}")
+
+        return create_chat_completion(
+            model=CFG.fast_llm_model,
+            messages=messages,
+        )
+
+    async def get_new_urls(self, url_set_input):
+        """ Gets the new urls from the given url set.
+        Args: url_set_input (set[str]): The url set to get the new urls from
+        Returns: list[str]: The new urls from the given url set
+        """
+
+        new_urls = []
+        for url in url_set_input:
+            if url not in self.visited_urls:
+                await self.stream_output(f"✅ Adding source url to research: {url}\n")
+
+                self.visited_urls.add(url)
+                new_urls.append(url)
+
+        return new_urls
+
+    async def call_agent(self, action, stream=False, websocket=None):
+        messages = [{
+            "role": "system",
+            "content": self.agent_role_prompt
+        }, {
+            "role": "user",
+            "content": action,
+        }]
+        answer = create_chat_completion(
+            model=CFG.smart_llm_model,
+            messages=messages,
+            stream=stream,
+            websocket=websocket,
+        )
+        return answer
+
+    async def create_search_queries(self):
+        """ Creates the search queries for the given question.
+        Args: None
+        Returns: list[str]: The search queries for the given question
+        """
+        result = await self.call_agent(prompts.generate_search_queries_prompt(self.question))
+        await self.stream_output(f"🧠 I will conduct my research based on the following queries: {result}...")
+        return json.loads(result)
+
+    async def async_search(self, query):
+        """ Runs the async search for the given query.
+        Args: query (str): The query to run the async search for
+        Returns: list[str]: The async search for the given query
+        """
+        search_results = json.loads(web_search(query))
+        new_search_urls = self.get_new_urls([url.get("href") for url in search_results])
+
+        await self.stream_output(f"🌐 Browsing the following sites for relevant information: {new_search_urls}...")
+
+        # Create a list to hold the coroutine objects
+        tasks = [async_browse(url, query, self.websocket) for url in await new_search_urls]
+
+        # Gather the results as they become available
+        responses = await asyncio.gather(*tasks, return_exceptions=True)
+
+        return responses
+
+    async def run_search_summary(self, query):
+        """ Runs the search summary for the given query.
+        Args: query (str): The query to run the search summary for
+        Returns: str: The search summary for the given query
+        """
+
+        await self.stream_output(f"🔎 Running research for '{query}'...")
+
+        responses = await self.async_search(query)
+
+        result = "\n".join(responses)
+        os.makedirs(os.path.dirname(f"{self.dir_path}/research-{query}.txt"), exist_ok=True)
+        write_to_file(f"{self.dir_path}/research-{query}.txt", result)
+        return result
+
+    async def conduct_research(self):
+        """ Conducts the research for the given question.
+        Args: None
+        Returns: str: The research for the given question
+        """
+        self.research_summary = read_txt_files(self.dir_path) if os.path.isdir(self.dir_path) else ""
+
+        if not self.research_summary:
+            search_queries = await self.create_search_queries()
+            for query in search_queries:
+                research_result = await self.run_search_summary(query)
+                self.research_summary += f"{research_result}\n\n"
+
+        await self.stream_output(f"Total research words: {len(self.research_summary.split(' '))}")
+
+        return self.research_summary
+
+
+    async def create_concepts(self):
+        """ Creates the concepts for the given question.
+        Args: None
+        Returns: list[str]: The concepts for the given question
+        """
+        result = self.call_agent(prompts.generate_concepts_prompt(self.question, self.research_summary))
+
+        await self.stream_output(f"I will research based on the following concepts: {result}\n")
+        return json.loads(result)
+
+    async def write_report(self, report_type, websocket=None):
+        """ Writes the report for the given question.
+        Args: None
+        Returns: str: The report for the given question
+        """
+        report_type_func = prompts.get_report_by_type(report_type)
+        await self.stream_output(f"✍️ Writing {report_type} for research task: {self.question}...")
+
+        answer = await self.call_agent(report_type_func(self.question, self.research_summary),
+                                       stream=websocket is not None, websocket=websocket)
+        # if websocket is True than we are streaming gpt response, so we need to wait for the final response
+        final_report = await answer if websocket else answer
+
+        path = await write_md_to_pdf(report_type, self.dir_path, final_report)
+
+        return answer, path
+
+    async def write_lessons(self):
+        """ Writes lessons on essential concepts of the research.
+        Args: None
+        Returns: None
+        """
+        concepts = await self.create_concepts()
+        for concept in concepts:
+            answer = await self.call_agent(prompts.generate_lesson_prompt(concept), stream=True)
+            await write_md_to_pdf("Lesson", self.dir_path, answer)

agent/run.py

@@ -0,0 +1,60 @@
+import asyncio
+import datetime
+
+from typing import List, Dict
+from fastapi import WebSocket
+from config import check_openai_api_key
+from agent.research_agent import ResearchAgent
+
+
+class WebSocketManager:
+    def __init__(self):
+        self.active_connections: List[WebSocket] = []
+        self.sender_tasks: Dict[WebSocket, asyncio.Task] = {}
+        self.message_queues: Dict[WebSocket, asyncio.Queue] = {}
+
+    async def start_sender(self, websocket: WebSocket):
+        queue = self.message_queues[websocket]
+        while True:
+            message = await queue.get()
+            if websocket in self.active_connections:
+                await websocket.send_text(message)
+            else:
+                break
+
+    async def connect(self, websocket: WebSocket):
+        await websocket.accept()
+        self.active_connections.append(websocket)
+        self.message_queues[websocket] = asyncio.Queue()
+        self.sender_tasks[websocket] = asyncio.create_task(self.start_sender(websocket))
+
+    async def disconnect(self, websocket: WebSocket):
+        self.active_connections.remove(websocket)
+        self.sender_tasks[websocket].cancel()
+        del self.sender_tasks[websocket]
+        del self.message_queues[websocket]
+
+    async def start_streaming(self, task, report_type, agent, agent_role_prompt, websocket):
+        report, path = await run_agent(task, report_type, agent, agent_role_prompt, websocket)
+        return report, path
+
+
+async def run_agent(task, report_type, agent, agent_role_prompt, websocket):
+    check_openai_api_key()
+
+    start_time = datetime.datetime.now()
+
+    # await websocket.send_json({"type": "logs", "output": f"Start time: {str(start_time)}\n\n"})
+
+    assistant = ResearchAgent(task, agent, agent_role_prompt, websocket)
+    await assistant.conduct_research()
+
+    report, path = await assistant.write_report(report_type, websocket)
+
+    await websocket.send_json({"type": "path", "output": path})
+
+    end_time = datetime.datetime.now()
+    await websocket.send_json({"type": "logs", "output": f"\nEnd time: {end_time}\n"})
+    await websocket.send_json({"type": "logs", "output": f"\nTotal run time: {end_time - start_time}\n"})
+
+    return report, path

client/index.html

@@ -0,0 +1,135 @@
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <title>GPT Researcher</title>
+    <meta name="description" content="A research assistant powered by GPT-4">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <link rel="icon" href="./static/favicon.ico">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <link href="https://fonts.googleapis.com/css2?family=Montserrat:wght@400;700&display=swap" rel="stylesheet">
+    <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css" rel="stylesheet">
+    <link rel="stylesheet" href="/site/styles.css"/>
+    <style>
+        .avatar {
+            width: 60px;
+            height: 60px;
+            border-radius: 50%;
+        }
+
+        .agent-name {
+            text-align: center;
+        }
+
+        .agent-item {
+            display: flex;
+            flex-direction: column;
+            align-items: center;
+        }
+
+        .agent-choices {
+            display: none;
+        }
+
+        .btn-show {
+            display: none;
+        }
+    </style>
+</head>
+
+<body>
+
+<section class="landing">
+    <div class="max-w-5xl mx-auto text-center">
+        <h1 class="text-4xl font-extrabold mx-auto lg:text-7xl">
+            Say Goodbye to <br>
+            <span
+                    style="background-image:linear-gradient(to right, #9867F0, #ED4E50); -webkit-background-clip: text; -webkit-text-fill-color: transparent;">Hours
+                    of Research</span>
+        </h1>
+        <p class="max-w-5xl mx-auto text-gray-600 mt-8" style="font-size:20px">
+            Say Hello to GPT Researcher, your AI mate for rapid insights and comprehensive research. GPT Researcher
+            takes care of everything from accurate source gathering to organization of research results - all in one
+            platform designed to make your research process a breeze.
+        </p>
+        <a href="#form" class="btn btn-primary">Get Started</a>
+    </div>
+</section>
+
+<main class="container" id="form">
+    <div class="agent-item"><img src="/static/defaultAgentAvatar.JPG" class="avatar"
+                                                alt="Auto Agent"></div>
+    <form method="POST" class="mt-3" onsubmit="GPTResearcher.startResearch(); return false;">
+        <div class="form-group">
+            <label for="task" class="agent-question">What would you like me to research next?</label>
+            <input type="text" id="task" name="task" class="form-control" required>
+            <input type="radio" name="agent" id="autoAgent" value="Auto Agent" checked hidden>
+        </div>
+        <div class="form-group">
+            <div class="row">
+
+
+            </div>
+            <button type="button" id="btnShowAuto" class="btn btn-secondary mt-3 btn-show">Auto Agent</button>
+        </div>
+        <div class="form-group">
+            <label for="report_type" class="agent-question">What type of report would you like me to generate?</label>
+            <select name="report_type" class="form-control" required>
+                <option value="research_report">Research Report</option>
+                <option value="resource_report">Resource Report</option>
+                <option value="outline_report">Outline Report</option>
+            </select>
+        </div>
+        <input type="submit" value="Research" class="btn btn-primary button-padding">
+    </form>
+
+    <div class="margin-div">
+        <h2>Agent Output</h2>
+        <p class="mt-2 text-left" style="font-size: 0.8rem;">An agent tailored specifically to your task
+                        will be generated to provide the most precise and relevant research results.</p>
+        <div id="output"></div>
+    </div>
+    <div class="margin-div">
+        <h2>Research Report</h2>
+        <div id="reportContainer"></div>
+        <div id="reportActions">
+            <div class="alert alert-info" role="alert" id="status"></div>
+            <a onclick="GPTResearcher.copyToClipboard()" class="btn btn-secondary mt-3">Copy to clipboard</button>
+            <a id="downloadLink" href="#" class="btn btn-secondary mt-3" target="_blank">Download as PDF</a>
+        </div>
+    </div>
+</main>
+
+<footer>
+    <p>GPT Researcher &copy; 2023 | <a target="_blank" href="https://github.com/assafelovic/gpt-researcher">GitHub
+        Page</a></p>
+</footer>
+
+<script src="https://cdnjs.cloudflare.com/ajax/libs/showdown/1.9.1/showdown.min.js"></script>
+<script src="/site/scripts.js"></script>
+<script>
+    // const btnChoose = document.getElementById('btnChoose');
+    const btnShowAuto = document.getElementById('btnShowAuto');
+    const autoAgentDiv = document.getElementById('autoAgentDiv');
+    const agentChoices = document.getElementsByClassName('agent-choices');
+
+    /**
+    btnChoose.addEventListener('click', function () {
+        btnShowAuto.style.display = 'inline-block';
+        btnChoose.style.display = 'none';
+        autoAgentDiv.style.display = 'none';
+        agentChoices[0].style.display = 'flex';
+    });
+    **/
+
+    btnShowAuto.addEventListener('click', function () {
+        btnShowAuto.style.display = 'none';
+        btnChoose.style.display = 'inline-block';
+        autoAgentDiv.style.display = 'flex';
+        agentChoices[0].style.display = 'none';
+    });
+</script>
+</body>
+
+</html>

client/scripts.js

@@ -0,0 +1,150 @@
+const GPTResearcher = (() => {
+    const init = () => {
+      // Not sure, but I think it would be better to add event handlers here instead of in the HTML
+      //document.getElementById("startResearch").addEventListener("click", startResearch);
+      //document.getElementById("copyToClipboard").addEventListener("click", copyToClipboard);
+
+      updateState("initial");
+    }
+
+    const startResearch = () => {
+      document.getElementById("output").innerHTML = "";
+      document.getElementById("reportContainer").innerHTML = "";
+      updateState("in_progress")
+  
+      addAgentResponse({ output: "🤔 Thinking about research questions for the task..." });
+  
+      listenToSockEvents();
+    };
+  
+    const listenToSockEvents = () => {
+      const { protocol, host, pathname } = window.location;
+      const ws_uri = `${protocol === 'https:' ? 'wss:' : 'ws:'}//${host}${pathname}ws`;
+      const converter = new showdown.Converter();
+      const socket = new WebSocket(ws_uri);
+  
+      socket.onmessage = (event) => {
+        const data = JSON.parse(event.data);
+        if (data.type === 'logs') {
+          addAgentResponse(data);
+        } else if (data.type === 'report') {
+          writeReport(data, converter);
+        } else if (data.type === 'path') {
+          updateState("finished")
+          updateDownloadLink(data);
+
+        }
+      };
+  
+      socket.onopen = (event) => {
+        const task = document.querySelector('input[name="task"]').value;
+        const report_type = document.querySelector('select[name="report_type"]').value;
+        const agent = document.querySelector('input[name="agent"]:checked').value;
+  
+        const requestData = {
+          task: task,
+          report_type: report_type,
+          agent: agent,
+        };
+  
+        socket.send(`start ${JSON.stringify(requestData)}`);
+      };
+    };
+  
+    const addAgentResponse = (data) => {
+      const output = document.getElementById("output");
+      output.innerHTML += '<div class="agent_response">' + data.output + '</div>';
+      output.scrollTop = output.scrollHeight;
+      output.style.display = "block";
+      updateScroll();
+    };
+  
+    const writeReport = (data, converter) => {
+      const reportContainer = document.getElementById("reportContainer");
+      const markdownOutput = converter.makeHtml(data.output);
+      reportContainer.innerHTML += markdownOutput;
+      updateScroll();
+    };
+  
+    const updateDownloadLink = (data) => {
+      const path = data.output;
+      document.getElementById("downloadLink").setAttribute("href", path);
+    };
+  
+    const updateScroll = () => {
+      window.scrollTo(0, document.body.scrollHeight);
+    };
+  
+    const copyToClipboard = () => {
+      const textarea = document.createElement('textarea');
+      textarea.id = 'temp_element';
+      textarea.style.height = 0;
+      document.body.appendChild(textarea);
+      textarea.value = document.getElementById('reportContainer').innerText;
+      const selector = document.querySelector('#temp_element');
+      selector.select();
+      document.execCommand('copy');
+      document.body.removeChild(textarea);
+    };
+
+    const updateState = (state) => {
+      var status = "";
+      switch (state) {
+        case "in_progress":
+          status = "Research in progress..."
+          setReportActionsStatus("disabled");
+          break;
+        case "finished":
+          status = "Research finished!"
+          setReportActionsStatus("enabled");
+          break;
+        case "error":
+          status = "Research failed!"
+          setReportActionsStatus("disabled");
+          break;
+        case "initial":
+          status = ""
+          setReportActionsStatus("hidden");
+          break;
+        default:
+          setReportActionsStatus("disabled");
+      }
+      document.getElementById("status").innerHTML = status;
+      if (document.getElementById("status").innerHTML == "") {
+        document.getElementById("status").style.display = "none";
+      } else {
+        document.getElementById("status").style.display = "block";
+      }
+    }
+
+    /**
+     * Shows or hides the download and copy buttons
+     * @param {str} status Kind of hacky. Takes "enabled", "disabled", or "hidden". "Hidden is same as disabled but also hides the div"
+     */
+    const setReportActionsStatus = (status) => {
+      const reportActions = document.getElementById("reportActions");
+      // Disable everything in reportActions until research is finished
+
+      if (status == "enabled") {
+        reportActions.querySelectorAll("a").forEach((link) => {
+          link.classList.remove("disabled");
+          link.removeAttribute('onclick');
+          reportActions.style.display = "block";
+        });
+      } else {
+        reportActions.querySelectorAll("a").forEach((link) => {
+          link.classList.add("disabled");
+          link.setAttribute('onclick', "return false;");
+        });
+        if (status == "hidden") {
+          reportActions.style.display = "none";
+        }
+      }
+    }
+
+    document.addEventListener("DOMContentLoaded", init);
+    return {
+      startResearch,
+      copyToClipboard,
+    };
+  })();

client/styles.css

@@ -0,0 +1,131 @@
+@keyframes gradientBG {
+    0% {background-position: 0% 50%;}
+    50% {background-position: 100% 50%;}
+    100% {background-position: 0% 50%;}
+}
+
+body {
+    font-family: 'Montserrat', sans-serif;
+    color: #fff;
+    line-height: 1.6;
+    background-size: 200% 200%;
+    background-image: linear-gradient(45deg, #151A2D, #2D284D, #151A2D);
+    animation: gradientBG 10s ease infinite;
+}
+
+.landing {
+    display: flex;
+    justify-content: center;
+    align-items: center;
+    height: 100vh;
+    text-align: center;
+}
+
+.landing h1 {
+    font-size: 3.5rem;
+    font-weight: 700;
+    margin-bottom: 2rem;
+}
+
+.landing p {
+    font-size: 1.5rem;
+    font-weight: 400;
+    max-width: 500px;
+    margin: auto;
+    margin-bottom: 2rem;
+}
+
+.container {
+    max-width: 900px;
+    margin: auto;
+    padding: 20px;
+    background-color: rgba(255, 255, 255, 0.1);
+    border-radius: 12px;
+    box-shadow: 0px 10px 25px rgba(0, 0, 0, 0.1);
+    transition: all .3s ease-in-out;
+    margin-bottom: 180px;
+}
+
+.container:hover {
+    transform: scale(1.01);
+    box-shadow: 0px 15px 30px rgba(0, 0, 0, 0.2);
+}
+
+input, select, #output, #reportContainer {
+    background-color: rgba(255,255,255,0.1);
+    border: none;
+    color: #fff;
+    transition: all .3s ease-in-out;
+}
+
+input:hover, input:focus, select:hover, select:focus {
+    background-color: #dfe4ea;
+    border: 1px solid rgba(255, 255, 255, 0.5);
+    box-shadow: 0px 4px 8px rgba(0, 0, 0, 0.1);
+    transition: all 0.3s ease-in-out;
+}
+
+.btn-primary {
+    background: linear-gradient(to right, #0062cc, #007bff);
+    border: none;
+    transition: all .3s ease-in-out;
+}
+
+.btn-secondary {
+    background: linear-gradient(to right, #6c757d, #6c757d);
+    border: none;
+    transition: all .3s ease-in-out;
+}
+
+.btn:hover {
+    opacity: 0.8;
+    transform: scale(1.1);
+    box-shadow: 0px 10px 20px rgba(0, 0, 0, 0.3);
+}
+
+.agent_question {
+    font-size: 1.2rem;
+    font-weight: 500;
+    margin-bottom: 0.5rem;
+}
+
+footer {
+    position: fixed;
+    left: 0;
+    bottom: 0;
+    width: 100%;
+    background: linear-gradient(to right, #151A2D, #111827);
+    color: white;
+    text-align: center;
+    padding: 10px 0;
+}
+
+.margin-div {
+    margin-top: 20px;
+    margin-bottom: 20px;
+    padding: 10px;
+}
+
+.agent_response {
+    background-color: #747d8c;
+    margin: 10px;
+    padding: 10px;
+    border-radius: 12px;
+}
+
+#output {
+    height: 300px;
+    overflow: auto;
+    padding: 10px;
+    margin-bottom: 10px;
+    margin-top: 10px;
+}
+
+#reportContainer {
+    background-color: rgba(255,255,255,0.1);
+    border: none;
+    color: #fff;
+    transition: all .3s ease-in-out;
+    padding: 10px;
+    border-radius: 12px;
+}

config/__init__.py

@@ -0,0 +1,9 @@
+from config.config import Config, check_openai_api_key
+from config.singleton import AbstractSingleton, Singleton
+
+__all__ = [
+    "check_openai_api_key",
+    "AbstractSingleton",
+    "Config",
+    "Singleton",
+]

config/config.py

@@ -0,0 +1,83 @@
+"""Configuration class to store the state of bools for different scripts access."""
+import os
+
+import openai
+from colorama import Fore
+from dotenv import load_dotenv
+
+from config.singleton import Singleton
+
+load_dotenv(verbose=True)
+
+
+class Config(metaclass=Singleton):
+    """
+    Configuration class to store the state of bools for different scripts access.
+    """
+
+    def __init__(self) -> None:
+        """Initialize the Config class"""
+        self.debug_mode = False
+        self.allow_downloads = False
+
+        self.selenium_web_browser = os.getenv("USE_WEB_BROWSER", "chrome")
+        self.llm_provider = os.getenv("LLM_PROVIDER", "ChatOpenAI")
+        self.fast_llm_model = os.getenv("FAST_LLM_MODEL", "gpt-3.5-turbo-16k")
+        self.smart_llm_model = os.getenv("SMART_LLM_MODEL", "gpt-4")
+        self.fast_token_limit = int(os.getenv("FAST_TOKEN_LIMIT", 2000))
+        self.smart_token_limit = int(os.getenv("SMART_TOKEN_LIMIT", 4000))
+        self.browse_chunk_max_length = int(os.getenv("BROWSE_CHUNK_MAX_LENGTH", 8192))
+        self.summary_token_limit = int(os.getenv("SUMMARY_TOKEN_LIMIT", 700))
+
+        self.openai_api_key = os.getenv("OPENAI_API_KEY")
+        self.temperature = float(os.getenv("TEMPERATURE", "1"))
+
+        self.user_agent = os.getenv(
+            "USER_AGENT",
+            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36"
+            " (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36",
+        )
+
+        self.memory_backend = os.getenv("MEMORY_BACKEND", "local")
+        # Initialize the OpenAI API client
+        openai.api_key = self.openai_api_key
+
+    def set_fast_llm_model(self, value: str) -> None:
+        """Set the fast LLM model value."""
+        self.fast_llm_model = value
+
+    def set_smart_llm_model(self, value: str) -> None:
+        """Set the smart LLM model value."""
+        self.smart_llm_model = value
+
+    def set_fast_token_limit(self, value: int) -> None:
+        """Set the fast token limit value."""
+        self.fast_token_limit = value
+
+    def set_smart_token_limit(self, value: int) -> None:
+        """Set the smart token limit value."""
+        self.smart_token_limit = value
+
+    def set_browse_chunk_max_length(self, value: int) -> None:
+        """Set the browse_website command chunk max length value."""
+        self.browse_chunk_max_length = value
+
+    def set_openai_api_key(self, value: str) -> None:
+        """Set the OpenAI API key value."""
+        self.openai_api_key = value
+
+    def set_debug_mode(self, value: bool) -> None:
+        """Set the debug mode value."""
+        self.debug_mode = value
+
+
+def check_openai_api_key() -> None:
+    """Check if the OpenAI API key is set in config.py or as an environment variable."""
+    cfg = Config()
+    if not cfg.openai_api_key:
+        print(
+            Fore.RED
+            + "Please set your OpenAI API key in .env or as an environment variable."
+        )
+        print("You can get your key from https://platform.openai.com/account/api-keys")
+        exit(1)

config/singleton.py

@@ -0,0 +1,24 @@
+"""The singleton metaclass for ensuring only one instance of a class."""
+import abc
+
+
+class Singleton(abc.ABCMeta, type):
+    """
+    Singleton metaclass for ensuring only one instance of a class.
+    """
+
+    _instances = {}
+
+    def __call__(cls, *args, **kwargs):
+        """Call method for the singleton metaclass."""
+        if cls not in cls._instances:
+            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
+        return cls._instances[cls]
+
+
+class AbstractSingleton(abc.ABC, metaclass=Singleton):
+    """
+    Abstract singleton class for ensuring only one instance of a class.
+    """
+
+    pass

docker-compose.yml

@@ -0,0 +1,9 @@
+version: '3'
+services:
+  gpt-researcher:
+    image: assafelovic/gpt-researcher
+    build: ./
+    environment: 
+      OPENAI_API_KEY: ${OPENAI_API_KEY}
+    ports:
+      - 8000:8000

js/overlay.js

@@ -0,0 +1,29 @@
+const overlay = document.createElement('div');
+Object.assign(overlay.style, {
+    position: 'fixed',
+    zIndex: 999999,
+    top: 0,
+    left: 0,
+    width: '100%',
+    height: '100%',
+    background: 'rgba(0, 0, 0, 0.7)',
+    color: '#fff',
+    fontSize: '24px',
+    fontWeight: 'bold',
+    display: 'flex',
+    justifyContent: 'center',
+    alignItems: 'center',
+});
+const textContent = document.createElement('div');
+Object.assign(textContent.style, {
+    textAlign: 'center',
+});
+textContent.textContent = 'Tavily AI: Analyzing Page';
+overlay.appendChild(textContent);
+document.body.append(overlay);
+document.body.style.overflow = 'hidden';
+let dotCount = 0;
+setInterval(() => {
+    textContent.textContent = 'Tavily AI: Analyzing Page' + '.'.repeat(dotCount);
+    dotCount = (dotCount + 1) % 4;
+}, 1000);

main.py

@@ -0,0 +1,71 @@
+from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
+from fastapi.staticfiles import StaticFiles
+from fastapi.templating import Jinja2Templates
+from pydantic import BaseModel
+import json
+import os
+
+from agent.llm_utils import choose_agent
+from agent.run import WebSocketManager
+
+
+class ResearchRequest(BaseModel):
+    task: str
+    report_type: str
+    agent: str
+
+
+
+app = FastAPI()
+app.mount("/site", StaticFiles(directory="client"), name="site")
+app.mount("/static", StaticFiles(directory="client/static"), name="static")
+# Dynamic directory for outputs once first research is run
+@app.on_event("startup")
+def startup_event():
+    if not os.path.isdir("outputs"):
+        os.makedirs("outputs")
+    app.mount("/outputs", StaticFiles(directory="outputs"), name="outputs")
+
+templates = Jinja2Templates(directory="client")
+
+manager = WebSocketManager()
+
+
+@app.get("/")
+async def read_root(request: Request):
+    return templates.TemplateResponse('index.html', {"request": request, "report": None})
+
+
+@app.websocket("/ws")
+async def websocket_endpoint(websocket: WebSocket):
+    await manager.connect(websocket)
+    try:
+        while True:
+            data = await websocket.receive_text()
+            if data.startswith("start"):
+                json_data = json.loads(data[6:])
+                task = json_data.get("task")
+                report_type = json_data.get("report_type")
+                agent = json_data.get("agent")
+                # temporary so "normal agents" can still be used and not just auto generated, will be removed when we move to auto generated
+                if agent == "Auto Agent":
+                    agent_dict = choose_agent(task)
+                    agent = agent_dict.get("agent")
+                    agent_role_prompt = agent_dict.get("agent_role_prompt")
+                else:
+                    agent_role_prompt = None
+
+                await websocket.send_json({"type": "logs", "output": f"Initiated an Agent: {agent}"})
+                if task and report_type and agent:
+                    await manager.start_streaming(task, report_type, agent, agent_role_prompt, websocket)
+                else:
+                    print("Error: not enough parameters provided.")
+
+    except WebSocketDisconnect:
+        await manager.disconnect(websocket)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=8000)

outputs/540826e12734403b01e368d80fefc9cae0571027/research-Papers on understanding learning mechanisms through chain of thought prompting.txt

@@ -0,0 +1,4 @@
+Information gathered from url https://arxiv.org/abs/2212.10001: The text does not provide specific information about papers on understanding learning mechanisms through chain of thought prompting. It primarily focuses on a paper titled "Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters" in the field of Computer Science.
+Information gathered from url https://arxiv.org/abs/2305.15408: The text does not provide any information regarding papers on understanding learning mechanisms through chain of thought prompting.
+Information gathered from url https://aclanthology.org/2023.acl-long.153/: The text does not provide any information regarding papers on understanding learning mechanisms through chain of thought prompting.
+Information gathered from url https://arxiv.org/pdf/2212.10001.pdf: Error: No text to summarize

outputs/540826e12734403b01e368d80fefc9cae0571027/research-Scientific articles on how thought promptings affect decision making - October 11, 2023.txt

@@ -0,0 +1,4 @@
+Information gathered from url https://medicalxpress.com/news/2023-08-brain-choices-decision-making.html: The provided text does not mention any scientific articles specifically on how thought promptings affect decision making on October 11, 2023. Therefore, there is no factual information, numbers, stats, or any other specific details to summarize.
+Information gathered from url https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1129835/full: The given text does not provide any information about scientific articles specifically on how thought promptings affect decision making on October 11, 2023. However, it does include citations of various scientific articles that discuss topics related to cognitive biases, decision making, and human behavior.
+Information gathered from url https://www.sciencedaily.com/releases/2022/10/221014135704.htm: The text does not provide any information about scientific articles on how thought promptings affect decision making on October 11, 2023.
+Information gathered from url https://news.stanford.edu/report/2021/09/15/mindsets-clearing-lens-life/: The text does not provide any information about scientific articles on how thought promptings affect decision making on October 11, 2023.

outputs/540826e12734403b01e368d80fefc9cae0571027/research-The cognitive process behind chain of thought prompting.txt

@@ -0,0 +1,4 @@
+Information gathered from url https://www.promptingguide.ai/techniques/cot: The cognitive process behind chain-of-thought (CoT) prompting involves enabling complex reasoning capabilities through intermediate reasoning steps. It allows for better results on complex tasks that require reasoning before responding. The authors claim that this is an emergent ability that arises with sufficiently large language models.
+Information gathered from url https://deepgram.com/learn/chain-of-thought-prompting-guide: The cognitive process behind chain of thought (CoT) prompting involves encouraging large language models (LLMs) to break down complex thoughts into intermediate steps. CoT prompts provide a series of examples or steps to guide LLMs in reasoning and reaching a final answer. It allows LLMs to decompose complex problems similar to how humans break down complicated math or logic questions into smaller steps. CoT prompting has been shown to boost LLMs' performance in complex arithmetic, commonsense, and symbolic reasoning tasks. It has been tested on various math word problem benchmarks and has shown minimal benefits until LLMs reach around 100 billion parameters. CoT prompting has also achieved state-of-the-art performance on certain benchmarks, surpassing models specifically finetuned on math word problems.
+Information gathered from url https://learnprompting.org/docs/intermediate/chain_of_thought: The cognitive process behind chain of thought prompting involves encouraging the language model to explain its reasoning. It improves the model's accuracy by showing it examples where the reasoning process is explained. It has been effective in tasks like arithmetic, commonsense, and symbolic reasoning. Models of around 100 billion parameters have achieved a 57% solve rate accuracy on the GSM8K task. Smaller models do not show the same performance gains.
+Information gathered from url https://www.linkedin.com/pulse/chain-of-thought-prompting-practical-guide-chatgpt-paul-zimeras: The cognitive process behind chain of thought prompting involves breaking down complex problems into manageable steps. It includes problem identification, understanding the problem, generating solutions, evaluating and selecting solutions, executing the solution, evaluating the solution, and acknowledging limitations and future directions. The process aims to address the original problem statement with a well-grounded decision. The Chain-of-Thought Prompting methodology provides a framework to guide problem-solving and encourages iterative and adaptive problem-solving.

outputs/540826e12734403b01e368d80fefc9cae0571027/research_report.md

@@ -0,0 +1,38 @@
+# Chain of Thought (CoT) Prompting: A Cognitive Process Unveiling Advanced Reasoning Capabilities in Large Language Models
+
+## Introduction
+
+The realm of artificial intelligence has revolutionized scientific research and discovery, with one such revolution being unveiled with the cognitive process behind chain-of-thought (CoT) prompting. This cognitive process seeks to advance reasoning capabilities in large language models (LLMs) by endorsing the practice of breaking down complex thoughts into intermediate steps, hence aiding in problem-solving. CoT prompting has been able to significantly improve the results of complex tasks demanding reasoning prior to generating a response.
+
+## Cognitive Process behind CoT Prompting
+
+CoT prompting operates by stimulating LLMs to dissect complex thoughts into smaller, manageable steps, thus fostering a more tailored approach to problem-solving (Promptingguide.ai, n.d.). It is implied that LLMs employ a strategy paralleling how humans resolve convoluted mathematical or logical inquiries by splitting them into lesser operations (Deepgram, n.d.). Simply put, CoT prompting is reminiscent of the way humans process information and reason. 
+
+## Impacts of CoT Prompting on LLMs' Performance
+
+Testing of CoT prompting on various mathematical word problem benchmarks has revealed the tangible benefits of this prompting methodology, especially when it comes to LLMs with around 100 billion parameters (Deepgram, n.d.). These benefits, however, were found to be insignificant for smaller models, underlining the crucial role that model size plays with respect to the benefits accrued from CoT prompting (Learnprompting.org, n.d.).
+
+Promisingly, CoT prompting managed to boost language models' performance in acute cognitive tasks such as arithmetic, commonsense, and symbolic reasoning tasks. Fostering advanced reasoning capabilities, one research study revealed that models encompassing around 100 billion parameters achieved a commendable solve rate accuracy of 57% on the GSM8K task (Learnprompting.org, n.d.). The leaps in performance attributed to CoT prompting are substantially supporting the proposition that LLMs with CoT prompting are becoming increasingly sophisticated in their problem-solving abilities and cognitive profoundness.
+
+## Advanced Characteristics of CoT Prompting
+
+CoT prompting is heralded for its unique methodology, which consists of several stages similar to a human problem-solving procedure. This includes the identification and understanding of the problem, the generation of solutions, evaluation and selection of the optimal response, execution and assessment of the solution, and finally acknowledging limitations and outlining future directions (Zimeras, n.d.). 
+
+Evincing humanlike problem-solving tactics, the systematic and iterative approach employed by the Chain-of-Thought Prompting methodology significantly augments the potential for LLMs to address complex problems with well-grounded decisions, therefore enhancing LLMs' ability to dissect intricate problems efficiently and systematically.
+
+## Conclusion
+
+Through examining and understanding the cognitive process behind CoT prompting, it is evident that it fosters LLMs' capacities to solve exceptionally complex problems that require a more sophisticated level of reasoning. Leveraging a quasi-human approach to problem-solving, CoT prompting exposes LLMs to a more systematic format of understanding and dissecting issues, thereby significantly boosting their potential for improved model performance.
+
+Then again, one cannot dismiss the primary limitation tied to CoT prompting application, being that their advantages significantly rely on the language model size. While the research surrounding CoT prompting remains relatively in its nascent phases, existing outcomes substantiate the tangible benefits that such a cognitive process delivers in unfolding higher cognitive abilities in large language models.
+
+## References
+
+Deepgram. (n.d.). Chain of thought prompting guide. https://deepgram.com/learn/chain-of-thought-prompting-guide.
+
+Learnprompting.org. (n.d.). Intermediate/chain of thought. https://learnprompting.org/docs/intermediate/chain_of_thought.
+
+Promptingguide.ai. (n.d.). Techniques/COT. https://www.promptingguide.ai/techniques/cot.
+
+Zimeras, P. (n.d.). Chain of thought prompting: A practical guide [LinkedIn].
+https://www.linkedin.com/pulse/chain-of-thought-prompting-practical-guide-chatgpt paul-zimeras.

permchain_example/README.md

@@ -0,0 +1,33 @@
+# Permchain x Researcher
+Sample use of Langchain's Autonomous agent framework Permchain with GPT Researcher.
+
+## Use case
+Permchain is a framework for building autonomous agents that can be used to automate tasks and communication between agents to complete complex tasks. This example uses Permchain to automate the process of finding and summarizing research reports on any given topic.
+
+## The Agent Team
+The research team is made up of 3 agents:
+- Researcher agent (gpt-researcher) - This agent is in charge of finding and summarizing relevant research papers.
+- Editor agent - This agent is in charge of validating the correctness of the report given a set of criteria.
+- Reviser agent - This agent is in charge of revising the report until it is satisfactory.
+
+## How it works
+The research agent (gpt-researcher) is in charge of finding and summarizing relevant research papers. It does this by using the following process:
+- Search for relevant research papers using a search engine
+- Extract the relevant information from the research papers
+- Summarize the information into a report
+- Send the report to the editor agent for validation
+- Send the report to the reviser agent for revision
+- Repeat until the report is satisfactory
+
+## How to run
+1. Install required packages:
+    ```bash
+    pip install -r requirements.txt
+    ```
+2. Run the application:
+    ```bash
+    python test.py
+    ```
+   
+## Usage
+To change the research topic, edit the `query` variable in `test.py` to the desired topic.

permchain_example/editor_actors/editor.py

@@ -0,0 +1,51 @@
+from langchain.chat_models import ChatOpenAI
+from langchain.prompts import SystemMessagePromptTemplate
+from config import Config
+
+CFG = Config()
+
+EDIT_TEMPLATE = """You are an editor. \
+You have been tasked with editing the following draft, which was written by a non-expert. \
+Please accept the draft if it is good enough to publish, or send it for revision, along with your notes to guide the revision. \
+Things you should be checking for:
+
+- This draft MUST fully answer the original question
+- This draft MUST be written in apa format
+
+If not all of the above criteria are met, you should send appropriate revision notes.
+"""
+
+
+class EditorActor:
+    def __init__(self):
+        self.model = ChatOpenAI(model=CFG.smart_llm_model)
+        self.prompt = SystemMessagePromptTemplate.from_template(EDIT_TEMPLATE) + "Draft:\n\n{draft}"
+        self.functions = [
+            {
+                "name": "revise",
+                "description": "Sends the draft for revision",
+                "parameters": {
+                    "type": "object",
+                    "properties": {
+                        "notes": {
+                            "type": "string",
+                            "description": "The editor's notes to guide the revision.",
+                        },
+                    },
+                },
+            },
+            {
+                "name": "accept",
+                "description": "Accepts the draft",
+                "parameters": {
+                    "type": "object",
+                    "properties": {"ready": {"const": True}},
+                },
+            },
+        ]
+
+    @property
+    def runnable(self):
+        return (
+            self.prompt | self.model.bind(functions=self.functions)
+        )

permchain_example/research_team.py

@@ -0,0 +1,75 @@
+from operator import itemgetter
+from langchain.runnables.openai_functions import OpenAIFunctionsRouter
+
+from permchain.connection_inmemory import InMemoryPubSubConnection
+from permchain.pubsub import PubSub
+from permchain.topic import Topic
+
+'''
+    This is the research team. 
+    It is a group of autonomous agents that work together to answer a given question
+    using a comprehensive research process that includes: 
+        - Searching for relevant information across multiple sources
+        - Extracting relevant information
+        - Writing a well structured report
+        - Validating the report
+        - Revising the report
+        - Repeat until the report is satisfactory
+'''
+class ResearchTeam:
+    def __init__(self, research_actor, editor_actor, reviser_actor):
+        self.research_actor_instance = research_actor
+        self.editor_actor_instance = editor_actor
+        self.revise_actor_instance = reviser_actor
+
+    def run(self, query):
+        # create topics
+        editor_inbox = Topic("editor_inbox")
+        reviser_inbox = Topic("reviser_inbox")
+
+        research_chain = (
+            # Listed in inputs
+            Topic.IN.subscribe()
+            | {"draft": lambda x: self.research_actor_instance.run(x["question"])}
+            # The draft always goes to the editor inbox
+            | editor_inbox.publish()
+        )
+
+        editor_chain = (
+            # Listen for events in the editor_inbox
+            editor_inbox.subscribe()
+            | self.editor_actor_instance.runnable
+            # Depending on the output, different things should happen
+            | OpenAIFunctionsRouter({
+                # If revise is chosen, we send a push to the critique_inbox
+                "revise": (
+                        {
+                            "notes": itemgetter("notes"),
+                            "draft": editor_inbox.current() | itemgetter("draft"),
+                            "question": Topic.IN.current() | itemgetter("question"),
+                        }
+                        | reviser_inbox.publish()
+                ),
+                # If accepted, then we return
+                "accept": editor_inbox.current() | Topic.OUT.publish(),
+            })
+        )
+
+        reviser_chain = (
+            # Listen for events in the reviser's inbox
+            reviser_inbox.subscribe()
+            | self.revise_actor_instance.runnable
+            # Publish to the editor inbox
+            | editor_inbox.publish()
+        )
+
+        web_researcher = PubSub(
+            research_chain,
+            editor_chain,
+            reviser_chain,
+            connection=InMemoryPubSubConnection(),
+        )
+
+        res = web_researcher.invoke({"question": query})
+        print(res)
+        return res["draft"]

permchain_example/researcher.py

@@ -0,0 +1,36 @@
+from permchain.connection_inmemory import InMemoryPubSubConnection
+from permchain.pubsub import PubSub
+from permchain.topic import Topic
+
+
+class Researcher:
+    def __init__(self, search_actor, writer_actor):
+        self.search_actor_instance = search_actor
+        self.writer_actor_instance = writer_actor
+
+    def run(self, query):
+        # The research inbox
+        research_inbox = Topic("research")
+        search_actor = (
+            Topic.IN.subscribe()
+            | {
+                "query": lambda x: x,
+                "results": self.search_actor_instance.runnable
+            }
+            | research_inbox.publish()
+        )
+
+        write_actor = (
+            research_inbox.subscribe()
+            | self.writer_actor_instance.runnable
+            | Topic.OUT.publish()
+        )
+
+        researcher = PubSub(
+            search_actor,
+            write_actor,
+            connection=InMemoryPubSubConnection(),
+        )
+
+        res = researcher.invoke(query)
+        return res["answer"]