langchain_community.llms.octoai_endpoint.OctoAIEndpoint¶
class langchain_community.llms.octoai_endpoint.OctoAIEndpoint[source]¶
Bases: LLM
OctoAI LLM Endpoints.
OctoAIEndpoint is a class to interact with OctoAICompute Service large language model endpoints.
To use, you should have the octoai python package installed, and the
environment variable OCTOAI_API_TOKEN set with your API token, or pass
it as a named parameter to the constructor.
Example
from langchain_community.llms.octoai_endpoint  import OctoAIEndpoint
OctoAIEndpoint(
    octoai_api_token="octoai-api-key",
    endpoint_url="https://mpt-7b-demo-f1kzsig6xes9.octoai.run/generate",
    model_kwargs={
        "max_new_tokens": 200,
        "temperature": 0.75,
        "top_p": 0.95,
        "repetition_penalty": 1,
        "seed": None,
        "stop": [],
    },
)
from langchain_community.llms.octoai_endpoint  import OctoAIEndpoint
OctoAIEndpoint(
    octoai_api_token="octoai-api-key",
    endpoint_url="https://llama-2-7b-chat-demo-kk0powt97tmb.octoai.run/v1/chat/completions",
    model_kwargs={
        "model": "llama-2-7b-chat",
        "messages": [
            {
                "role": "system",
                "content": "Below is an instruction that describes a task.
                    Write a response that completes the request."
            }
        ],
        "stream": False,