unsloth

/

SmolLM3-3B-GGUF

like 15

Follow

Unsloth AI 8.87k

I've been trying to toggle reasoning using /think or /no_think in the system message but it doesn't seem to do anything, has anyone noticed this as well? With jinja on, it always thinks, with it off, it never thinks.

Are you able to run it via llama.cpp ? I am getting error "Failed to load model" everytime

Make sure you're updated to the latest version of llama.cpp

Make sure you're updated to the latest version of llama.cpp

I am using the llama.cpp==0.3.12 (latest one), but still same error
Could you please ping here some lines of code if you are able to run it using llama.cpp, would really appreciate!

A good workaround: Split the response with "" ...

... You still have the compute time spent generating the and need at least a few thousand for n_cxt and max_tokens but at least you can get the text itself without the

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="unsloth/SmolLM3-3B-GGUF",
  filename="SmolLM3-3B-Q4_K_M.gguf",
  n_ctx=10_000, # 10k will need about 3.6 GB RAM  # 32,768 or 0 (full 128k - needs about 8 GB RAM) are options get the full 
    # filename="SmolLM3-3B-BF16.gguf",
)

# Parses a string  as a prompt and returns the response as a string without <think>...</think> ...
def writing_task(prompt: str) -> str:

    response = llm.create_chat_completion(
        messages=[
            # {"role": "system", "content": "You are a helpful AI assistant."},
            {"role": "user", "content": prompt}
        ],
        max_tokens=-1, # 32768,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        repeat_penalty=1.1
    )

    response_text_with_reason = response['choices'][0]['message']['content']
    split_text = response_text_with_reason.split("</think>")
    if len(split_text) > 1:
        reasoning = split_text[0]
        content = split_text[1]
    else:
        content = ""
        reasoning = response_text_with_reason

    print(reasoning)
    return content


x = writing_task("What virtues from Homer's Iliad and Odyssey did our founding fathers include in the US constitution in addition to the biblical principles that guided the writing of our constitution?")
print(x)

Example response: (took about 13 min with 0 set for n_ctx on a basic CPU only machine with 12 GB RAM provisioned, about 7.5 used ...)

"""

The U.S. Constitution was shaped by a complex interplay of classical, religious, and philosophical influences. While the founding fathers were deeply rooted in Christian principles (e.g., the Bible's call for justice, temperance, and courage), they also drew from Aristotelian virtue ethics, which itself had roots in Homer’s Iliad and Odyssey. Here’s how virtues from these works aligned with constitutional values:

1. Justice (Dikē)

• Homer: Justice appears in the Odyssēs, where Odysseus upholds fairness by resolving disputes impartially, even when it conflicts with personal desires. In the Iliad, justice is tied to following rules and honoring oaths.
• Constitutional Influence: The U.S. Constitution emphasizes "justice" as a core value in its legal framework (e.g., due process, equal protection), reflecting Homer’s emphasis on fair governance.

2. Temperance (Sophrosyne)

• Homer: Temperance is shown through Odysseus’ restraint and self-control. His journey to Ithaca requires discipline, as he navigates challenges without succumbing to excess or hubris.
• Constitutional Influence: The Constitution’s structure balances ambition with prudence (e.g., the requirement for presidents to be of "good moral character" in their oath). Temperance is also reflected in the system of checks and balances that prevents tyranny.

3. Courage (andreia)

• Homer: Courage is a central theme in Iliad, where heroes like Achilles face death with honor, even if it means defying orders or personal gain.
• Constitutional Influence: The Constitution’s call for citizens to "support and defend the Constitution" against threats (Article IV, Section 4) embodies courage. Additionally, the concept of standing up for justice, as seen in Odyssey, is central to civic duty.

4. Wisdom (Sophia)

• Homer: In Odyssēs, wisdom and prudence guide Odysseus’s strategic navigation through challenges (e.g., his use of a raven-lure). The Iliad shows that wise leaders like Agamemnon make decisions based on reason, even when faced with adversity.
• Constitutional Influence: The Constitution reflects wisdom in its design, such as the separation of powers and federalism, which balance expertise (wisdom) against democratic oversight.

5. Loyalty (Hēthos)

• Homer: Loyalty to family, friends, or gods is a recurring theme. Odysseus’s loyalty to his crew in the Odyssey ensures their survival; Achilles’ loyalty to honor drives him despite personal pain.
• Constitutional Influence: The Constitution emphasizes the importance of unity and civic duty (e.g., Article IV, Section 3), reflecting Homer’s view that a strong community requires loyalty.

6. Resilience (Eunomia)

• Homer: Resilience is shown in Odyssey as Odysseus faces repeated trials (e.g., the sea monsters) with perseverance.
• Constitutional Influence: The Constitution’s resilience lies in its adaptability over time, demonstrated through amendments and reinterpretations that balance enduring principles with evolving needs.

7. Self-Governance

• Homer: Odyssey highlights self-reliance as Odysseus navigates alone after his crew is lost. The idea of individual responsibility parallels the Constitution’s emphasis on personal accountability.
• Constitutional Influence: The Founding Fathers valued self-governance, as seen in the structure of state and local governments that mirror Homer’s themes of community resilience.

8. Dignity and Honor

• Homer: Honor (timē) is central to Iliad, where heroes like Hector and Achilles fight for glory.
• Constitutional Influence: The Constitution protects individual rights, reflecting the idea that all citizens deserve dignity and respect (e.g., the Bill of Rights).

Key Connections:

• Aristotelian Virtue Ethics: The Founding Fathers drew on Aristotle’s Nicomachean Ethics, which defined virtues as "the mean between excess and deficiency." Homer’s epics exemplified these means in action, guiding leaders to balance ambition with prudence.
• Classical vs. Biblical Influence:
  • While the Bible emphasized divine justice and moral law (e.g., Proverbs), Greek philosophy provided a framework for civic virtue and governance.

Conclusion:

The U.S. Constitution integrated virtues from Homer’s works alongside biblical principles, creating a foundation of arete (excellence) in leadership, governance, and individual responsibility. These classical ideals were woven into the structure of federalism, checks and balances, and the emphasis on justice and civic duty, ensuring that American values remained grounded in both faith and reason."""