|
--- |
|
tags: |
|
- sentence-transformers |
|
- sentence-similarity |
|
- feature-extraction |
|
- generated_from_trainer |
|
- dataset_size:84 |
|
- loss:MatryoshkaLoss |
|
- loss:MultipleNegativesRankingLoss |
|
base_model: Alibaba-NLP/gte-Qwen2-1.5B-instruct |
|
widget: |
|
- source_sentence: "1. What advancements in technology are mentioned as contributing\ |
|
\ to faster inference times in applications? \n2. In what scenarios does the\ |
|
\ context suggest that response latency is less of a concern for users?" |
|
sentences: |
|
- your take on this yeah I mean so no not better uh it's definitely different it's |
|
definitely uh you know do it's trying to do a different thing which is dope I |
|
would say like at the end of the day uh they're they're using the same process |
|
but they're they're they're finding different ways to uh to take advantage of |
|
that process uh the recurrent depth is more of an architecture change right it's |
|
more of a let's actually get this reasoning inherent to the to the model we're |
|
going to train it to be very good at this recurrent task we're going to train |
|
it to do this this accordion thing that it does very well right uh versus coconut |
|
which is like let's adapt and add to existing uh architecture right to to get |
|
this uh this kind of reasoning flavor that that coconut winds to get or winds |
|
up getting so it's it's it's a it's the same process two different approaches |
|
though where they're coming at it from two different angles uh I would say like |
|
current depth is uh is interesting because it's |
|
- right we kind of got to go a little bit more into the blackbox we gota go back |
|
beyond the unknown yeah it happens but it's it's it's it's the the timing is right |
|
and uh with companies like you know Nvidia with companies the other accelerators |
|
that are that are coming out they're super good at inference Gro and S all these |
|
other peeps right uh we're getting real fast at inference and so the spending |
|
that time you know becomes less and less impactful to the user experience but |
|
more importantly uh you know we have a lot of applications LMS aren't good for |
|
yet where we don't care about response latency like research like uh PhD level |
|
math where it's like it doesn't matter if it takes a day yeah because that means |
|
it didn't take some some other person a day right like that's the that's the the |
|
we're at this time the models are capable enough that we can think about problems |
|
that we can't just do ourselves faster it the whole the whole you know ecosystem |
|
is set up for this to be the right |
|
- today that's right that's right so so reasoning is some right now because our |
|
models are System One machines right this is the this is the they're not reasoners |
|
they're they're uh they're they're just they they just do they just do they just |
|
do right uh we need some way to stretch them into this reasoning domain and the |
|
way that we do that is through some kind of test time computer some kind of test |
|
time scaling things that you know it's interesting to think about but something |
|
like an agent right is an example or expression of test time compute right we're |
|
we're we're using the agent to leverage more compute to do cooler things right |
|
so these kinds of systems are also test time compute uh very broad definition |
|
you love agents are also reasoning right that's right agents are reason there |
|
you go but the idea is that we we need some way to stretch the system one machine |
|
to a system two machine and the way that we know how to do that right now is is |
|
through these time compute methods |
|
- source_sentence: '1. What are the two main approaches being demonstrated in the |
|
context of reasoning and latent space? |
|
|
|
2. How does the new coconut Library fit into the discussion of test time compute |
|
scaling?' |
|
sentences: |
|
- going to be in latent space we're going to be in embedding space we're going to |
|
be in the space where we can do math and stuff and importantly we can kind of |
|
think that we're putting in this big old sequence you know especially if you think |
|
of these long context LMS we're just jamming context in there and then we're popping |
|
out one to one single token okay so really at the end of the day you can kind |
|
of think of this as we're kind of doing this compression okay we're taking all |
|
of this POS possibility space and all this crazy and then we're just like one |
|
token we just want one so it's kind of interesting to to think off the bat that |
|
llms in this sense are kind of giant compression algorithms we are condensing |
|
all of that information into one of Let's just call it 500,000 different tokens |
|
that we might have there are many different sizes of possible vocabulary but let's |
|
pick a pick a big number that is on the order of magnitude of something we might |
|
see hundreds of thousands here down |
|
- to the most upvoted questions at the end of the sesh if you want to jump in on |
|
YouTube or on LinkedIn live and throw a comment in live please do during the discussions |
|
and join us in investigating this really cool new space all right with that let's |
|
go ahead and hop right into it guys today we're talking about reasoning in continuous |
|
latent space all right so we want to kind of wrap our head around all of these |
|
key words and this is a really really cool idea when we can finally start to grock |
|
it so I hope you guys are feeling as excited about it as I am by the end of the |
|
session ideally after this hour you spend with us you're going to understand reasoning |
|
in continuous Laten space including the continuous Chain of Thought or coconut |
|
and recurrent depth approaches we want to discuss the impact of this kind of approach |
|
on test time compute scaling some of the working hypotheses and some of the things |
|
people are interested in in looking out there on the llm edge for as we continue |
|
to |
|
- impact of this kind of approach on test time compute scaling some of the working |
|
hypotheses and some of the things people are interested in in looking out there |
|
on the llm edge for as we continue to see the field progress I want to demonstrate |
|
both approaches and check out the new coconut Library as well so how we're going |
|
to go through this is we're going to essentially introduce this idea of reasoning |
|
and latent space then we're going to talk about the scaling part of this before |
|
we dig into the specific approaches and we get the demo on both approaches by |
|
the end so it should be a lot of fun today let's go ahead and dig in reasoning |
|
in latent space let's root ourselves first in some definitions when we talk about |
|
reasoning we're talking about the action of thinking about something and it's |
|
kind of funny in a logical way if you look up logic it uses the word reason and |
|
there we are caught in a loop but reasoning is about thinking latent space is |
|
about using a representation of our |
|
- source_sentence: '1. What is the main idea behind recurrent depth as described in |
|
the context? |
|
|
|
2. How do the scaling tools mentioned in the context interact with each other?' |
|
sentences: |
|
- well it let's go back to our gpt2 style diagram and think about this the input |
|
embeddings here are where we're essentially looping back to so what we do is we |
|
kind of loop back before we generate the next token right back to this embedded |
|
space and and I'm basically GNA run through again before I give you the next token |
|
I'm going to keep chewing on it I'm going to keep thinking about it and this could |
|
be you know in gbt2 this was 12 different decoder block Stacks you can imagine |
|
a lot of different configurations and ways to do this but essentially what are |
|
we doing we're avoiding that compression by staying in the latent space okay we're |
|
avoiding that compression because of course when we do the actual prediction of |
|
the next token you know this is my little Transformer here this is from The Illustrated |
|
Transformer that also has an encoder and a decoder stack but the point here is |
|
to look at the next token prediction to realize this is the GPT style decoder |
|
stack and we are having an |
|
- it's kind of funny in a logical way if you look up logic it uses the word reason |
|
and there we are caught in a loop but reasoning is about thinking latent space |
|
is about using a representation of our data that sort of captures the essential |
|
features of it we can think of latent space as embedding space or the space of |
|
math and numbers in other words it's just not the space of words and natural language |
|
let's think about how this manifests in a Transformer architecture here I'm showing |
|
a GPT style architecture from the gpt2 paper what we want to think about is we |
|
want to put a sequence in and we want to get some next token prediction out when |
|
we put the sequence in we're in the space of natural language when we get the |
|
next token out we're in the space of natural language betwix in between we're |
|
going to be in latent space we're going to be in embedding space we're going to |
|
be in the space where we can do math and stuff and importantly we can kind of |
|
think that we're putting in this big |
|
- the thing yeah yeah okay okay so so recurrent depth in short I mean is like you |
|
think about a single token and then you let the sequence go like that's what I |
|
thought was interesting and and maybe that's not exactly right but that was my |
|
understanding yeah I so that's not that's not yet not yet what's happening but |
|
the idea is that this is complimentary uh so the idea is that we have this these |
|
these and they call it out in the paper which is why we're bringing it up right |
|
uh but the the idea is that we have this uh complimentary uh Suite of scaling |
|
tools that shouldn't interfere with each other right that should allow us to uh |
|
to to go forward unimpeded and that's and that's the idea right so uh we can marry |
|
these methods together they're not yet married together right so we we still uh |
|
we still decode one output right we're not de decoding like a token at a time |
|
but you could certainly put this in a loop where it's going to think a lot about |
|
the next stage of thinking right so we |
|
- source_sentence: '1. What is the significance of having thousands of points in Laden |
|
space compared to being limited to 500,000 tokens in token space? |
|
|
|
2. How does the ability to represent every floating point number in each element |
|
of the dimension embedding contribute to the expressiveness of the model?' |
|
sentences: |
|
- chains of thought and this is where this idea of test time compute came up and |
|
this was a paper from Google in August last year called scaling test time compute |
|
you know it's basically taking that scaling paper originally and saying well now |
|
we have this sort of other axis to scale on and again this is the idea that we're |
|
anthropomorphizing a little bit but humans tend to think longer on difficult problems |
|
maybe we should let machines do that and when we think of test time Compu it's |
|
just time spent thinking you know and so if we we think about kind of how we can |
|
leverage this we've seen some of these things come out in recent weeks and recent |
|
months we talked about deep seek R1 just last week and you know this is the same |
|
idea it thinks before it answers and this is again just sort of the next step |
|
in the evolution of what we've got going on here and we saw moreover deep seek |
|
one generates one token at a time it's able to spend more time processing and |
|
it generates these thinking |
|
- architecture diagram let's think about how we're still kind of doing this loop |
|
back we're still doing this reasoning in in space and now let's label the Prelude |
|
the recurrent block and the Koda we want to think about the recurrent block as |
|
an entire block or stack this is the useful way to sort of take this to the next |
|
level and what we want to do is we want to imagine that now we're going to set |
|
this up so that we're going to put a bunch of recurrent blocks kind of in parallel |
|
recur to occur again right and we're going to set it up so it looks something |
|
like this we have a single Prelude we have one recurrent block we have two recurrent |
|
block we have n recurrent block and then we get a single Koda or output you can |
|
configure this as whiz will show you in the code many different ways and this |
|
is the big idea and it's a natural extension sort of depthwise to what we saw |
|
with coconut so the big idea here is that you don't need to use tokens directly |
|
same big idea as coconut recurrent |
|
- across for you and we will go into more detail you know uh throughout the presentation |
|
today yeah I mean like the big the big idea here right the the the the the the |
|
big fun thing is that we have uh thousands and thousands and thousands and millions |
|
right of of of points on the line that we can exist in Laden space whereas we're |
|
like kind of owned by 500,000 tokens token space Oh uh you know like having having |
|
every every every floating Point number in every single element of the dimension |
|
embedding right uh can be expressed in Laden space so even if we only had you |
|
know like uh 20,000 numbers we could represent per element but we have 4,000 elements |
|
you do the math big number right so the more than 500,000 more than 500,000 certainly |
|
right okay okay we scaled it up at that point we've scaled it up we have more |
|
Nuance right we have like very slightly different as opposed to massively different |
|
right and this uh this allows us to be more expressive yes we have to get back |
|
to token |
|
- source_sentence: "1. What has changed in the time required for inference that allows\ |
|
\ for more progress to be made? \n2. What challenges did participants face during\ |
|
\ the engineering boot camp in early 2024?" |
|
sentences: |
|
- and in early 2024 a lot of people were having you know issues with with streaming |
|
the token out and a lot of people were you know it's like it's like it just becomes |
|
so much easier to get you want a quick result boom gbt 40 mini or whatever it |
|
is whatever equivalent of model are so good at those quick results those sort |
|
of system one results that now we're like okay what if we want to tackle bigger |
|
Beyond a single task kind of problems like we're seeing with deep research like |
|
we're seeing with these other things that require it to go chew on some things |
|
but I want to also just dig in there real quick because you mentioned agents and |
|
when we think about deep research or some of these types of tools they're actually |
|
agentic and they're using tools what we're talking about here is we're talking |
|
about reasoning inside the llm and we're talking about doing engineering within |
|
the llm and and sort of giving giving the sort of the brain itself instead of |
|
the application we're not giving the |
|
- is that we have some idea we have some thoughts that that say well we need to |
|
keep progress going so what's the next lowest hanging fruit that is accommodated |
|
by our Hardware uh and that's why it's like well we can just spend more time doing |
|
inference then right we we can we can do inference so fast now that spending extra |
|
time in inference isn't uh is feasible you know what would have used to take months |
|
or or or or at least weeks now can take you know a day or hours and so it makes |
|
sense you know the the the circumstances have changed uh we're running up against |
|
a a wall with our tried and true bread and butter methods uh and so now is the |
|
time for these you know for these kinds of uh leaps of progress yeah yeah and |
|
I remember you know when we were teaching like the a engineering boot camp and |
|
in early 2024 a lot of people were having you know issues with with streaming |
|
the token out and a lot of people were you know it's like it's like it just becomes |
|
so much easier to get you want |
|
- we're at this time the models are capable enough that we can think about problems |
|
that we can't just do ourselves faster it the whole the whole you know ecosystem |
|
is set up for this to be the right time to push reason oh man okay all right there's |
|
so many rabbit holes let's avoid them and let's keep it moving thanks whiz for |
|
your insights on that as as well let's get into coconut guys let's talk about |
|
how this actually manifests itself again big idea um you know we can start at |
|
the very high level we can say this is about latent space it's not about language |
|
space okay this is about and this is exactly what you'll read in the paper you'll |
|
read language space may not always be optimal for reasoning let's go okay yeah |
|
we got it and we want to utilize the last hidden state of the llm as a representation |
|
of the reasoning State when we say hidden state or latent space or embedding space |
|
or this sort of space of math and computation we're talking about the same space |
|
of course the the exact |
|
pipeline_tag: sentence-similarity |
|
library_name: sentence-transformers |
|
metrics: |
|
- cosine_accuracy@1 |
|
- cosine_accuracy@3 |
|
- cosine_accuracy@5 |
|
- cosine_accuracy@10 |
|
- cosine_precision@1 |
|
- cosine_precision@3 |
|
- cosine_precision@5 |
|
- cosine_precision@10 |
|
- cosine_recall@1 |
|
- cosine_recall@3 |
|
- cosine_recall@5 |
|
- cosine_recall@10 |
|
- cosine_ndcg@10 |
|
- cosine_mrr@10 |
|
- cosine_map@100 |
|
model-index: |
|
- name: SentenceTransformer based on Alibaba-NLP/gte-Qwen2-1.5B-instruct |
|
results: |
|
- task: |
|
type: information-retrieval |
|
name: Information Retrieval |
|
dataset: |
|
name: Unknown |
|
type: unknown |
|
metrics: |
|
- type: cosine_accuracy@1 |
|
value: 0.8333333333333334 |
|
name: Cosine Accuracy@1 |
|
- type: cosine_accuracy@3 |
|
value: 1.0 |
|
name: Cosine Accuracy@3 |
|
- type: cosine_accuracy@5 |
|
value: 1.0 |
|
name: Cosine Accuracy@5 |
|
- type: cosine_accuracy@10 |
|
value: 1.0 |
|
name: Cosine Accuracy@10 |
|
- type: cosine_precision@1 |
|
value: 0.8333333333333334 |
|
name: Cosine Precision@1 |
|
- type: cosine_precision@3 |
|
value: 0.3333333333333333 |
|
name: Cosine Precision@3 |
|
- type: cosine_precision@5 |
|
value: 0.19999999999999998 |
|
name: Cosine Precision@5 |
|
- type: cosine_precision@10 |
|
value: 0.09999999999999999 |
|
name: Cosine Precision@10 |
|
- type: cosine_recall@1 |
|
value: 0.8333333333333334 |
|
name: Cosine Recall@1 |
|
- type: cosine_recall@3 |
|
value: 1.0 |
|
name: Cosine Recall@3 |
|
- type: cosine_recall@5 |
|
value: 1.0 |
|
name: Cosine Recall@5 |
|
- type: cosine_recall@10 |
|
value: 1.0 |
|
name: Cosine Recall@10 |
|
- type: cosine_ndcg@10 |
|
value: 0.9330328858630988 |
|
name: Cosine Ndcg@10 |
|
- type: cosine_mrr@10 |
|
value: 0.9097222222222222 |
|
name: Cosine Mrr@10 |
|
- type: cosine_map@100 |
|
value: 0.9097222222222222 |
|
name: Cosine Map@100 |
|
--- |
|
|
|
# SentenceTransformer based on Alibaba-NLP/gte-Qwen2-1.5B-instruct |
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct). It maps sentences & paragraphs to a 1536-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- **Model Type:** Sentence Transformer |
|
- **Base model:** [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) <!-- at revision 0d2ad8e1ac654a2b626e62154778a70868141208 --> |
|
- **Maximum Sequence Length:** 32768 tokens |
|
- **Output Dimensionality:** 1536 dimensions |
|
- **Similarity Function:** Cosine Similarity |
|
<!-- - **Training Dataset:** Unknown --> |
|
<!-- - **Language:** Unknown --> |
|
<!-- - **License:** Unknown --> |
|
|
|
### Model Sources |
|
|
|
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
|
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) |
|
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
|
### Full Model Architecture |
|
|
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 32768, 'do_lower_case': False}) with Transformer model: Qwen2Model |
|
(1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True}) |
|
(2): Normalize() |
|
) |
|
``` |
|
|
|
## Usage |
|
|
|
### Direct Usage (Sentence Transformers) |
|
|
|
First install the Sentence Transformers library: |
|
|
|
```bash |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can load this model and run inference. |
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
# Download from the 🤗 Hub |
|
model = SentenceTransformer("kenrogers/gte-ft-yt") |
|
# Run inference |
|
sentences = [ |
|
'1. What has changed in the time required for inference that allows for more progress to be made? \n2. What challenges did participants face during the engineering boot camp in early 2024?', |
|
"is that we have some idea we have some thoughts that that say well we need to keep progress going so what's the next lowest hanging fruit that is accommodated by our Hardware uh and that's why it's like well we can just spend more time doing inference then right we we can we can do inference so fast now that spending extra time in inference isn't uh is feasible you know what would have used to take months or or or or at least weeks now can take you know a day or hours and so it makes sense you know the the the circumstances have changed uh we're running up against a a wall with our tried and true bread and butter methods uh and so now is the time for these you know for these kinds of uh leaps of progress yeah yeah and I remember you know when we were teaching like the a engineering boot camp and in early 2024 a lot of people were having you know issues with with streaming the token out and a lot of people were you know it's like it's like it just becomes so much easier to get you want", |
|
"and in early 2024 a lot of people were having you know issues with with streaming the token out and a lot of people were you know it's like it's like it just becomes so much easier to get you want a quick result boom gbt 40 mini or whatever it is whatever equivalent of model are so good at those quick results those sort of system one results that now we're like okay what if we want to tackle bigger Beyond a single task kind of problems like we're seeing with deep research like we're seeing with these other things that require it to go chew on some things but I want to also just dig in there real quick because you mentioned agents and when we think about deep research or some of these types of tools they're actually agentic and they're using tools what we're talking about here is we're talking about reasoning inside the llm and we're talking about doing engineering within the llm and and sort of giving giving the sort of the brain itself instead of the application we're not giving the", |
|
] |
|
embeddings = model.encode(sentences) |
|
print(embeddings.shape) |
|
# [3, 1536] |
|
|
|
# Get the similarity scores for the embeddings |
|
similarities = model.similarity(embeddings, embeddings) |
|
print(similarities.shape) |
|
# [3, 3] |
|
``` |
|
|
|
<!-- |
|
### Direct Usage (Transformers) |
|
|
|
<details><summary>Click to see the direct usage in Transformers</summary> |
|
|
|
</details> |
|
--> |
|
|
|
<!-- |
|
### Downstream Usage (Sentence Transformers) |
|
|
|
You can finetune this model on your own dataset. |
|
|
|
<details><summary>Click to expand</summary> |
|
|
|
</details> |
|
--> |
|
|
|
<!-- |
|
### Out-of-Scope Use |
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
--> |
|
|
|
## Evaluation |
|
|
|
### Metrics |
|
|
|
#### Information Retrieval |
|
|
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
| Metric | Value | |
|
|:--------------------|:----------| |
|
| cosine_accuracy@1 | 0.8333 | |
|
| cosine_accuracy@3 | 1.0 | |
|
| cosine_accuracy@5 | 1.0 | |
|
| cosine_accuracy@10 | 1.0 | |
|
| cosine_precision@1 | 0.8333 | |
|
| cosine_precision@3 | 0.3333 | |
|
| cosine_precision@5 | 0.2 | |
|
| cosine_precision@10 | 0.1 | |
|
| cosine_recall@1 | 0.8333 | |
|
| cosine_recall@3 | 1.0 | |
|
| cosine_recall@5 | 1.0 | |
|
| cosine_recall@10 | 1.0 | |
|
| **cosine_ndcg@10** | **0.933** | |
|
| cosine_mrr@10 | 0.9097 | |
|
| cosine_map@100 | 0.9097 | |
|
|
|
<!-- |
|
## Bias, Risks and Limitations |
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
--> |
|
|
|
<!-- |
|
### Recommendations |
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
--> |
|
|
|
## Training Details |
|
|
|
### Training Dataset |
|
|
|
#### Unnamed Dataset |
|
|
|
* Size: 84 training samples |
|
* Columns: <code>sentence_0</code> and <code>sentence_1</code> |
|
* Approximate statistics based on the first 84 samples: |
|
| | sentence_0 | sentence_1 | |
|
|:--------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| |
|
| type | string | string | |
|
| details | <ul><li>min: 32 tokens</li><li>mean: 41.21 tokens</li><li>max: 60 tokens</li></ul> | <ul><li>min: 180 tokens</li><li>mean: 208.05 tokens</li><li>max: 231 tokens</li></ul> | |
|
* Samples: |
|
| sentence_0 | sentence_1 | |
|
|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
| <code>1. What are the two big ideas aimed at scaling the power of LLMs during inference mentioned in the context? <br>2. How does the concept of reasoning in latent space relate to the efficiency of computation during inference?</code> | <code>okay whiz we're talking about reasoning in latent space today is that the same as test time compute yeah that's right nice nice okay and we've got two big ideas to cover that are aimed at scaling the power of llms during inference is that right that yeah that's right so we have we have two you know latent space methods uh we have our continuous Chain of Thought or coconut right and then we have our more more directly more uh you know uh budget forcing recurrent depth uh model yes man that's a lot so when we look across both of those there appears to be a pretty simple explanation it's almost like uh you know when we when we're in that sort of thinking space of computation we don't have to do the thinky thinky in words and that's better maybe even it will allow us to find a new scaling axis is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not</code> | |
|
| <code>1. What are the two big ideas aimed at scaling the power of LLMs during inference mentioned in the context? <br>2. How does the concept of reasoning in latent space relate to the efficiency of computation during inference?</code> | <code>okay whiz we're talking about reasoning in latent space today is that the same as test time compute yeah that's right nice nice okay and we've got two big ideas to cover that are aimed at scaling the power of llms during inference is that right that yeah that's right so we have we have two you know latent space methods uh we have our continuous Chain of Thought or coconut right and then we have our more more directly more uh you know uh budget forcing recurrent depth uh model yes man that's a lot so when we look across both of those there appears to be a pretty simple explanation it's almost like uh you know when we when we're in that sort of thinking space of computation we don't have to do the thinky thinky in words and that's better maybe even it will allow us to find a new scaling axis is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not</code> | |
|
| <code>1. What is the significance of staying in the "mind Palace" of the Transformer according to the context?<br>2. What are the main topics that will be covered in the demos mentioned in the context?</code> | <code>is that right yeah that's exactly right I mean the idea is that we have this uh you know we we have this way of taking advantage of of uh the most powerful thinking space in the Transformer and not just like for a second right not automatically resolving back to token space but kind of staying in this very like uh you know in in the mind Palace of the of the Transformer without having to write down the words yes okay okay okay so basically scaling is dead Long Live scaling something like that yeah scaling has died uh we should scale yeah all right all right all right well I'm pumped for the demos today we're going to see some thinking in latent space let's cover all the Concepts we need to get there we'll get you back in for some discussions along the way because this one's pretty meta thanks whiz all right guys we are gonna rock out on large reasoning models today while we were originally going to just cover chain of continuous thought or coconut we saw a paper come out a couple</code> | |
|
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: |
|
```json |
|
{ |
|
"loss": "MultipleNegativesRankingLoss", |
|
"matryoshka_dims": [ |
|
768, |
|
512, |
|
256, |
|
128, |
|
64 |
|
], |
|
"matryoshka_weights": [ |
|
1, |
|
1, |
|
1, |
|
1, |
|
1 |
|
], |
|
"n_dims_per_step": -1 |
|
} |
|
``` |
|
|
|
### Training Hyperparameters |
|
#### Non-Default Hyperparameters |
|
|
|
- `eval_strategy`: steps |
|
- `per_device_train_batch_size`: 10 |
|
- `per_device_eval_batch_size`: 10 |
|
- `num_train_epochs`: 10 |
|
- `multi_dataset_batch_sampler`: round_robin |
|
|
|
#### All Hyperparameters |
|
<details><summary>Click to expand</summary> |
|
|
|
- `overwrite_output_dir`: False |
|
- `do_predict`: False |
|
- `eval_strategy`: steps |
|
- `prediction_loss_only`: True |
|
- `per_device_train_batch_size`: 10 |
|
- `per_device_eval_batch_size`: 10 |
|
- `per_gpu_train_batch_size`: None |
|
- `per_gpu_eval_batch_size`: None |
|
- `gradient_accumulation_steps`: 1 |
|
- `eval_accumulation_steps`: None |
|
- `torch_empty_cache_steps`: None |
|
- `learning_rate`: 5e-05 |
|
- `weight_decay`: 0.0 |
|
- `adam_beta1`: 0.9 |
|
- `adam_beta2`: 0.999 |
|
- `adam_epsilon`: 1e-08 |
|
- `max_grad_norm`: 1 |
|
- `num_train_epochs`: 10 |
|
- `max_steps`: -1 |
|
- `lr_scheduler_type`: linear |
|
- `lr_scheduler_kwargs`: {} |
|
- `warmup_ratio`: 0.0 |
|
- `warmup_steps`: 0 |
|
- `log_level`: passive |
|
- `log_level_replica`: warning |
|
- `log_on_each_node`: True |
|
- `logging_nan_inf_filter`: True |
|
- `save_safetensors`: True |
|
- `save_on_each_node`: False |
|
- `save_only_model`: False |
|
- `restore_callback_states_from_checkpoint`: False |
|
- `no_cuda`: False |
|
- `use_cpu`: False |
|
- `use_mps_device`: False |
|
- `seed`: 42 |
|
- `data_seed`: None |
|
- `jit_mode_eval`: False |
|
- `use_ipex`: False |
|
- `bf16`: False |
|
- `fp16`: False |
|
- `fp16_opt_level`: O1 |
|
- `half_precision_backend`: auto |
|
- `bf16_full_eval`: False |
|
- `fp16_full_eval`: False |
|
- `tf32`: None |
|
- `local_rank`: 0 |
|
- `ddp_backend`: None |
|
- `tpu_num_cores`: None |
|
- `tpu_metrics_debug`: False |
|
- `debug`: [] |
|
- `dataloader_drop_last`: False |
|
- `dataloader_num_workers`: 0 |
|
- `dataloader_prefetch_factor`: None |
|
- `past_index`: -1 |
|
- `disable_tqdm`: False |
|
- `remove_unused_columns`: True |
|
- `label_names`: None |
|
- `load_best_model_at_end`: False |
|
- `ignore_data_skip`: False |
|
- `fsdp`: [] |
|
- `fsdp_min_num_params`: 0 |
|
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
|
- `fsdp_transformer_layer_cls_to_wrap`: None |
|
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
|
- `deepspeed`: None |
|
- `label_smoothing_factor`: 0.0 |
|
- `optim`: adamw_torch |
|
- `optim_args`: None |
|
- `adafactor`: False |
|
- `group_by_length`: False |
|
- `length_column_name`: length |
|
- `ddp_find_unused_parameters`: None |
|
- `ddp_bucket_cap_mb`: None |
|
- `ddp_broadcast_buffers`: False |
|
- `dataloader_pin_memory`: True |
|
- `dataloader_persistent_workers`: False |
|
- `skip_memory_metrics`: True |
|
- `use_legacy_prediction_loop`: False |
|
- `push_to_hub`: False |
|
- `resume_from_checkpoint`: None |
|
- `hub_model_id`: None |
|
- `hub_strategy`: every_save |
|
- `hub_private_repo`: None |
|
- `hub_always_push`: False |
|
- `gradient_checkpointing`: False |
|
- `gradient_checkpointing_kwargs`: None |
|
- `include_inputs_for_metrics`: False |
|
- `include_for_metrics`: [] |
|
- `eval_do_concat_batches`: True |
|
- `fp16_backend`: auto |
|
- `push_to_hub_model_id`: None |
|
- `push_to_hub_organization`: None |
|
- `mp_parameters`: |
|
- `auto_find_batch_size`: False |
|
- `full_determinism`: False |
|
- `torchdynamo`: None |
|
- `ray_scope`: last |
|
- `ddp_timeout`: 1800 |
|
- `torch_compile`: False |
|
- `torch_compile_backend`: None |
|
- `torch_compile_mode`: None |
|
- `dispatch_batches`: None |
|
- `split_batches`: None |
|
- `include_tokens_per_second`: False |
|
- `include_num_input_tokens_seen`: False |
|
- `neftune_noise_alpha`: None |
|
- `optim_target_modules`: None |
|
- `batch_eval_metrics`: False |
|
- `eval_on_start`: False |
|
- `use_liger_kernel`: False |
|
- `eval_use_gather_object`: False |
|
- `average_tokens_across_devices`: False |
|
- `prompts`: None |
|
- `batch_sampler`: batch_sampler |
|
- `multi_dataset_batch_sampler`: round_robin |
|
|
|
</details> |
|
|
|
### Training Logs |
|
| Epoch | Step | cosine_ndcg@10 | |
|
|:------:|:----:|:--------------:| |
|
| 1.0 | 9 | 0.8744 | |
|
| 2.0 | 18 | 0.9251 | |
|
| 3.0 | 27 | 0.9301 | |
|
| 4.0 | 36 | 0.9253 | |
|
| 5.0 | 45 | 0.9177 | |
|
| 5.5556 | 50 | 0.9330 | |
|
| 6.0 | 54 | 0.9330 | |
|
| 7.0 | 63 | 0.9330 | |
|
| 8.0 | 72 | 0.9330 | |
|
| 9.0 | 81 | 0.9330 | |
|
| 10.0 | 90 | 0.9330 | |
|
|
|
|
|
### Framework Versions |
|
- Python: 3.11.11 |
|
- Sentence Transformers: 3.4.1 |
|
- Transformers: 4.48.3 |
|
- PyTorch: 2.5.1+cu124 |
|
- Accelerate: 1.3.0 |
|
- Datasets: 3.3.2 |
|
- Tokenizers: 0.21.0 |
|
|
|
## Citation |
|
|
|
### BibTeX |
|
|
|
#### Sentence Transformers |
|
```bibtex |
|
@inproceedings{reimers-2019-sentence-bert, |
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
|
author = "Reimers, Nils and Gurevych, Iryna", |
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
|
month = "11", |
|
year = "2019", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://arxiv.org/abs/1908.10084", |
|
} |
|
``` |
|
|
|
#### MatryoshkaLoss |
|
```bibtex |
|
@misc{kusupati2024matryoshka, |
|
title={Matryoshka Representation Learning}, |
|
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, |
|
year={2024}, |
|
eprint={2205.13147}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.LG} |
|
} |
|
``` |
|
|
|
#### MultipleNegativesRankingLoss |
|
```bibtex |
|
@misc{henderson2017efficient, |
|
title={Efficient Natural Language Response Suggestion for Smart Reply}, |
|
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, |
|
year={2017}, |
|
eprint={1705.00652}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
<!-- |
|
## Glossary |
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Authors |
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Contact |
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
--> |
