withpi
/

glean_full_1b_11-14_4bit

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

Nam Nguyen commited on Dec 11, 2024

Commit

ecaf5c0

·

verified ·

1 Parent(s): 0753a4b

Update handler.py

Files changed (1) hide show

handler.py +2 -1

handler.py CHANGED Viewed

@@ -46,9 +46,10 @@ class EndpointHandler:
                 min_p=0.1,
             )
         response = extract_assistant_response_simple(self.tokenizer.decode(outputs[0]))
         end = time.perf_counter()
         elapsed = end - start
-        return {"response": response, "elapsed": elapsed}

                 min_p=0.1,
             )
+        response_length = len(outputs[0])
         response = extract_assistant_response_simple(self.tokenizer.decode(outputs[0]))
         end = time.perf_counter()
         elapsed = end - start
+        return {"response": response, "response_token_length": response_length, "elapsed": elapsed}