text: Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKIntegrationsCallbacksChat modelsDocument loadersDocument transformersLLMsAI21Aleph AlphaAmazon API GatewayAnyscaleAzure OpenAIAzureML Online EndpointBananaBasetenBeamBedrockCerebriumAIChatGLMClarifaiCohereC TransformersDatabricksDeepInfraForefrontAIGoogle Cloud Platform Vertex AI PaLMGooseAIGPT4AllHugging Face HubHugging Face Local PipelinesHuggingface TextGen InferenceJSONFormerKoboldAI APILlama-cppCaching integrationsManifestModalMosaicMLNLP CloudoctoaiOpenAIOpenLLMOpenLMPetalsPipelineAIPredibasePrediction GuardPromptLayer OpenAIRELLMReplicateRunhouseSageMakerEndpointStochasticAITextGenTongyi QwenWriterMemoryRetrieversText embedding modelsAgent toolkitsToolsVector storesGrouped by providerIntegrationsLLMsHuggingface TextGen InferenceOn this pageHuggingface TextGen InferenceText Generation Inference is a Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets.This notebooks goes over how to use a self hosted LLM using Text Generation Inference.To use, you should have the text_generation python package installed.# !pip3 install text_generationfrom langchain.llms import HuggingFaceTextGenInferencellm = HuggingFaceTextGenInference(    inference_server_url="http://localhost:8010/",    max_new_tokens=512,    top_k=10,    top_p=0.95,    typical_p=0.95,    temperature=0.01,