---
license: apache-2.0
datasets:
- NeelNanda/pile-10k
base_model:
- AIDC-AI/Marco-o1
---
## Model Details
This model is an int4 model with group_size 128 and symmetric quantization of [AIDC-AI/Marco-o1](https://huggingface.co/AIDC-AI/Marco-o1) generated by [intel/auto-round](https://github.com/intel/auto-round).
### INT4 Inference(CPU/HPU/CUDA)
```python
import torch
from typing import List, Dict, Tuple
from transformers import AutoModelForCausalLM, AutoTokenizer
def load_model_and_tokenizer(path):
tokenizer = AutoTokenizer.from_pretrained(path,
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(path,
device_map="auto", ##change device map
trust_remote_code=True,
)
model.eval()
return tokenizer, model
def generate_response(model, tokenizer,
input_ids, attention_mask,
max_new_tokens=4096):
generated_ids = input_ids
with torch.inference_mode():
for _ in range(max_new_tokens):
outputs = model(input_ids=generated_ids, attention_mask=attention_mask)
next_token_id = torch.argmax(outputs.logits[:, -1, :], dim=-1).unsqueeze(-1)
generated_ids = torch.cat([generated_ids, next_token_id], dim=-1)
attention_mask = torch.cat([attention_mask, torch.ones_like(next_token_id)], dim=-1)
new_token = tokenizer.decode(next_token_id.squeeze(), skip_special_tokens=True)
print(new_token, end='', flush=True)
if next_token_id.item() == tokenizer.eos_token_id:
break
return tokenizer.decode(generated_ids[0][input_ids.shape[-1]:], skip_special_tokens=True)
def chat(model, tokenizer):
history: List[Dict[str, str]] = []
print("Enter 'q' to quit, 'c' to clear chat history.")
while True:
user_input = input("User: ").strip().lower()
if user_input == 'q':
print("Exiting chat.")
break
if user_input == 'c':
print("Clearing chat history.")
history.clear()
continue
if not user_input:
print("Input cannot be empty.")
continue
history.append({"role": "user", "content": user_input})
text = tokenizer.apply_chat_template(history, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096).to('cuda:0')
print('Assistant:', end=' ', flush=True)
response = generate_response(model, tokenizer, model_inputs.input_ids, model_inputs.attention_mask)
print()
history.append({"role": "assistant", "content": response})
def main():
path = "OPEA/Marco-o1-int4-sym-awq-inc"
tokenizer, model = load_model_and_tokenizer(path)
print('Starting chat.')
chat(model, tokenizer)
main()
"""9.11和9.8哪个数字大"""
## INT4
"""
Alright, I need to determine which number is larger between 9.11 and 9.8. Let's break this down step by step to ensure accuracy.
First, I'll consider the structure of these numbers. Both numbers are decimal numbers, which means they have a whole number part and a fractional part separated by a decimal point. Specifically, 9.11 has a whole number part of 9 and a fractional part of 0.11. Similarly, 9.8 has a whole number part of 9 and a fractional part of 0.8.
To compare these two numbers, I'll start by comparing their whole number parts. Both numbers have the same whole number part, which is 9. Since the whole number parts are equal, the comparison will focus on the fractional parts.
Next, I'll look at the fractional parts. 9.11 has a fractional part of 0.11, while 9.8 has a fractional part of 0.8. To compare these, I'll convert them to fractions to make the comparison easier.
0.11 is equivalent to 11/100, and 0.8 is equivalent to 8/10. To compare these fractions, I'll find a common denominator. The least common denominator for 100 and 10 is 100.
Converting 8/10 to a denominator of 100:
8/10 = (8 * 10) / (10 * 10) = 80/100
Now, comparing 11/100 and 80/100:
11/100 < 80/100
This means that 0.11 < 0.8.
Since the fractional part of 9.11 is smaller than the fractional part of 9.8, and the whole number parts are equal, it follows that 9.11 is less than 9.8.
But wait, let me double-check to ensure I haven't made any calculation errors. Another way to compare these numbers is to subtract the smaller number from the larger one to see if the result is positive or negative.
9.8 - 9.11 = 0.69
Since the result is positive, this confirms that 9.8 is indeed larger than 9.11.
Additionally, I can visualize these numbers on a number line to further confirm. On a number line, 9.11 would be slightly to the right of 9, and 9.8 would be further to the right of 9.8. This spatial representation aligns with the numerical comparison I performed earlier.
Moreover, considering the context, if these numbers represent measurements or values in a real-world scenario, 9.8 would be the larger value, indicating a greater quantity or magnitude compared to 9.11.
In summary, after multiple methods of comparison—direct comparison of fractional parts, subtraction, and visualization on a number line—it's clear that 9.8 is larger than 9.11.
"""
##BF16
"""Alright, I need to determine which number is larger between 9.11 and 9.8. Let's break this down step by step to ensure accuracy.
First, I'll consider the structure of these numbers. Both are decimal numbers, which means they have a whole number part and a fractional part separated by a decimal point. Specifically, 9.11 has a whole number part of 9 and a fractional part of 0.11. Similarly, 9.8 has a whole number part of 9 and a fractional part of 0.8.
Since both numbers have the same whole number part (9), the comparison will focus on their fractional parts. The fractional part of 9.11 is 0.11, and the fractional part of 9.8 is 0.8.
To compare 0.11 and 0.8, I'll convert them to fractions to make the comparison easier.
0.11 is equivalent to 11/100, and 0.8 is equivalent to 80/100.
Now, comparing 11/100 and 80/100, it's clear that 80/100 is larger than 11/100 because 80 is greater than 11.
Therefore, 0.8 is greater than 0.11.
Since both numbers have the same whole number part and the fractional part of 9.8 is larger than that of 9.11, it follows that 9.8 is greater than 9.11.
But wait, let me double-check to ensure I haven't made any calculation errors.
Calculating 9.11 and 9.8:
- 9.11 + 0.89 = 10.00
- 9.8 + 0.21 = 10.01
Comparing these sums, 10.01 is greater than 10.00, which further confirms that 9.8 is larger than 9.11.
Additionally, considering the place values:
- The tenths place in 9.11 is 1, and in 9.8 it's 8.
- The hundredths place in 9.11 is 1, and in 9.8 it's 0.
Since 8 is greater than 1 in the tenths place, this also indicates that 9.8 is larger.
Another way to look at it is by converting both numbers to percentages:
- 9.11 is 911%
- 9.8 is 980%
Clearly, 980% is greater than 911%, reinforcing that 9.8 is larger.
Lastly, thinking about real-world applications, if these numbers represent quantities like lengths or weights, 9.8 would be a larger measurement compared to 9.11.
In conclusion, after multiple verification methods, it's evident that 9.8 is indeed larger than 9.11.