Hugging Face's logo
Jim Lai's picture
45

Jim Lai

grimjim
Caniker's profile picture TiFoil's profile picture BATMAN2701's profile picture
·

AI & ML interests

Experimenting primarily with 7B-12B parameter text completion models. Not all models are intended for direct use, but aim for research and/or educational purposes.

Recent Activity

new activity about 12 hours ago
google/gemma-2-2b-it:SLERP merge example code?
published a model about 19 hours ago
grimjim/DeepSauerHuatuoSkywork-R1-o1-Llama-3.1-8B
updated a model about 20 hours ago
grimjim/DeepSauerHuatuoSkywork-R1-o1-Llama-3.1-8B
View all activity

Organizations

Social Post Explorers's profile picture Hugging Face Discord Community's profile picture Debased AI's profile picture Anthracite's profile picture Anthracite Core's profile picture

Posts 19

view post
Post
1689
A recent merge has provided another interesting result on the current Open LLM leaderboard.
open-llm-leaderboard/open_llm_leaderboard

Combining an o1 reasoning merge with VAGOsolutions's Llama-3.1 SauerkrautLM 8B Instruct model resulted in a lower IFEval, but a higher result in every other benchmark. This result is currently my best Llama 3.1 8B merge result to date.
grimjim/SauerHuatuoSkywork-o1-Llama-3.1-8B
The results suggest that defects in output format and/or output parsing may be limiting benchmark performance of various o1 models.
view post
Post
1584
I've arrived at an interesting result on the current Open LLM leaderboard.
open-llm-leaderboard/open_llm_leaderboard
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
grimjim/HuatuoSkywork-o1-Llama-3.1-8B

Unfortunately, I need more information to evaluate the parent models used in the merge.
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
View all posts

Collections 5