Phi-3-mini-128k-instruct not working with pro inference api
Using this endpoint:https://api-inference.huggingface.co/models/microsoft/Phi-3-mini-128k-instruct/v1/I get this error:Error code: 500 - {‘error’: ‘The repository for microsoft/Phi-3-mini-128k-instruct contains custom code which must be executed to correctly load the model. You can inspect the repository content athttps://hf.co/microsoft/Phi-3-mini-128k-instruct.\nPleasepass the argumenttrust_remote_code=Trueto allow custom code to be run.’}client.list_deployed_models() shows that the model is deployed. The 4k version works fine.Thanks!
2024-04-26T18:03:29Z
hi@sam-paech,Looking at the available models on TGI, currently, onlymicrosoft/phi-3-mini-4k-instructis available. I can check if we have plans to support the other models.
2024-04-26T18:11:50Z
Running this code:from huggingface_hub import InferenceClientclient = InferenceClient(token=MYTOKEN)client.list_deployed_models()Returns this list for deployed text-generation models, which includes the 128k model:‘text-generation’: [‘b3ck1/gpt-neo-125M-finetuned-beer-recipes’,‘bigcode/octocoder’,‘bigcode/santacoder’,‘bigcode/starcoder’,‘bigcode/starcoder2-15b’,‘bigcode/starcoder2-3b’,‘bigcode/starcoderplus’,‘bigscience/bloom’,‘bigscience/bloom-560m’,‘blockblockblock/smol_llama-220M-GQA-bpw2.5’,‘codellama/CodeLlama-13b-hf’,‘codellama/CodeLlama-34b-Instruct-hf’,‘codellama/CodeLlama-7b-hf’,‘CohereForAI/c4ai-command-r-plus’,‘dh-unibe/gpt2-larger-walser’,‘dh-unibe/luther-xl’,‘EleutherAI/pythia-14m’,‘flax-community/gpt-neo-125M-apps’,‘google/flan-t5-xxl’,‘google/gemma-1.1-2b-it’,‘google/gemma-1.1-7b-it’,‘google/gemma-2b’,‘google/gemma-2b-it’,‘google/gemma-7b’,‘google/gemma-7b-it’,‘gpt2-large’,‘Gustavosta/MagicPrompt-Stable-Diffusion’,‘h2oai/h2o-danube2-1.8b-chat’,‘hsramall/hsramall-70b-chat-placeholder’,‘HuggingFaceFW/ablation-model-fineweb-v1’,‘HuggingFaceH4/starchat-beta’,‘HuggingFaceH4/starchat2-15b-v0.1’,‘HuggingFaceH4/tiny-random-LlamaForCausalLM’,‘HuggingFaceH4/zephyr-7b-alpha’,‘HuggingFaceH4/zephyr-7b-beta’,‘HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1’,‘HuggingFaceM4/idefics-80b-instruct’,‘HuggingFaceM4/idefics-9b-instruct’,‘huggingtweets/angelinacho-stillconor-touchofray’,‘huggingtweets/itspublu’,‘huggingtweets/j_j_j_j_j_jones’,‘huggingtweets/jackieracc_’,‘huggingtweets/jaguarunlocked’,‘huggingtweets/lulaoficial-ptbrasil’,‘IlyaGusev/saiga_7b_lora’,‘ismaelfaro/gpt2-poems.en’,‘ismaelfaro/gpt2-poems.es’,‘jppaolim/homerGPT2’,‘kashif/stack-llama-2’,‘MBZUAI/LaMini-Neo-1.3B’,‘MBZUAI/LLaVA-Phi-3-mini-4k-instruct’,‘meta-llama/Llama-2-13b-chat-hf’,‘meta-llama/Llama-2-13b-hf’,‘meta-llama/Llama-2-70b-chat-hf’,‘meta-llama/Llama-2-7b-chat-hf’,‘meta-llama/Llama-2-7b-hf’,‘meta-llama/Meta-Llama-3-70B-Instruct’,‘meta-llama/Meta-Llama-3-8B-Instruct’,‘microsoft/biogpt’,‘microsoft/BioGPT-Large’,‘microsoft/BioGPT-Large-PubMedQA’,‘microsoft/DialoGPT-large’,‘microsoft/DialoGPT-medium’,‘microsoft/phi-1_5’,‘microsoft/Phi-3-mini-128k-instruct’,‘microsoft/Phi-3-mini-4k-instruct’,‘mistralai/Mistral-7B-Instruct-v0.1’,‘mistralai/Mistral-7B-Instruct-v0.2’,‘mistralai/Mistral-7B-v0.1’,‘mistralai/Mixtral-8x7B-Instruct-v0.1’,‘model-attribution-challenge/bloom-350m’,‘model-attribution-challenge/codegen-350M-multi’,‘model-attribution-challenge/distilgpt2’,‘model-attribution-challenge/gpt2’,‘model-attribution-challenge/gpt2-xl’,‘model-attribution-challenge/xlnet-base-cased’,‘mywateriswet/ShuanBot’,‘NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO’,‘openai-community/gpt2-medium’,‘openai-gpt’,‘OpenAssistant/oasst-sft-1-pythia-12b’,‘OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5’,‘Pi3141/DialoGPT-medium-elon-2’,‘rachiteagles/cover_letter’,‘stabilityai/stablelm-2-1_6b’,‘succinctly/text2image-prompt-generator’,‘tiiuae/falcon-7b’,‘timdettmers/guanaco-33b-merged’,‘TinyLlama/TinyLlama-1.1B-Chat-v0.6’,‘togethercomputer/RedPajama-INCITE-Chat-3B-v1’,‘xlnet-base-cased’,‘zrowt/test-deere’],
2024-04-26T18:24:26Z
yes, it’s available with transformers, however, still incomplete the inference doesn’t support custom remote code.image1362×614 49.9 KBAnd here is the list with all models that are “warm”https://api-inference.huggingface.co/framework/text-generation-inferenceIf you need to deployPhi-3-mini-128k-instructas inference endpoint you’ll need a custom handler to supporttrust_remote_code=Truehuggingface.coCreate custom Inference HandlerWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-04-26T18:42:21Z
radames:https://api-inference.huggingface.co/framework/text-generation-inferenceThis list doesn’t contain some models that are listed as deployed & are working with the api, e.g. command-r-plus.I only just signed up for the pro subscription so I’m not sure how it’s supposed to work. But I would have assumed the list returned byclient.list_deployed_models()would represent the models that are available for inference with the API. If that isn’t the case, is there a way to get an authoritative list of deployed + working models?
2024-04-26T18:55:28Z
Oops I was wrong, your list does contain command-r-plus.
2024-06-10T13:13:28Z
Thanks for the info! I was also trying to deploy phi3 models on a dedicated endpoint, and the custom handler seems to be the only current solution.Is there a similar list for which models are currently supported on dedicated inference endpoints (without requiring a custom handler)?Trying to deploymicrosoft/phi-3-mini-4k-instructon one gets me a similar error about trust_remote_code in the logs.Also, according to themodel page,Phi-3 has been integrated in the development version (4.40.0.dev) oftransformers.Is it then a reasonable assumption that once dedicated inference endpoints use a transformers version > 4.40.0, thenmicrosoft/phi-3-mini-4k-instructwill be deployable on dedicated inference endpoints through TGI? Is it possible to see which version is currently used?
2024-06-10T13:48:32Z
Hi,Yes, the list of default libraries can be found here:https://huggingface.co/docs/inference-endpoints/others/runtime(includes everything except TGI version, team is going to fix that).Phi-3 has indeed been integrated natively in the Transformers library:transformers/src/transformers/models/phi3/modeling_phi3.py at main · huggingface/transformers · GitHub, which means that you can now load it without having to specifytrust_remote_code=True.
2024-06-10T21:58:24Z
Hi,@nielsr,I’m a bit confused by your answer. I tried to create a dedicated inference point withmicrosoft/Phi-3-mini-4k-instruct, but it failed with an error saying I need to specifytrust_remote_code=True.
2024-06-11T06:57:10Z
Hi,That’s probably because the current Transformers version of Inference Endpoints is 4.38.2 as per the doc here:https://huggingface.co/docs/inference-endpoints/others/runtime. Hence it will only be possible once this updates to Transformers v4.40.
2024-06-15T18:58:14Z
Hey, can you share the code you used to create the custom handler? I haven’t been able to make it work
2024-06-19T15:13:31Z
Essentiallythis, just with these params on the model initialization (deployed on a T4 GPU):model = AutoModelForCausalLM.from_pretrained(
            path, 
            torch_dtype=torch.bfloat16,
            device_map="cuda",
            trust_remote_code=True
        )
2024-08-17T06:00:12Z
hi, any update about this issue ? same error while creating an endpoint from a phi-3 based modelerror-hf-endpoint2426×1382 353 KB
2024-08-20T10:35:03Z
plus 1. Got the same errorValueError: The repository for /repository contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//repository.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
2024-08-26T13:55:08Z
This will be resolved once Inference Endpoints updates its Transformers version, will ping the team.