<p>First of all what is pretraining? The procedure helps the model to learn syntactic &lt;==&gt; semantic (this is a spectrum) features of the language using an enormous amount of raw text (40GB) and processing power. objective function: casual language model and mask language model</p>
<p>What about fine-tuning a pre-trained model? suppose there is a model which has knowledge about the general aspect of the English language (POS, dependency tree, subj ... a little of everything). fine-tuning help us to direct the focus of the model on the most important features in our dataset, let's say in your dataset some syntactic feature is the game-changer, and the model should be careful about it!
objective function: based on downstream task</p>
<p>Training from scratch isn't feasible for most of us, but there is an approach to continue the pre-training phase using your own corpus/corpora (task-specific) without damaging model pieces of knowledge (hopefully)!
objective function: casual language model and mask language model</p>
<p><a href="https://arxiv.org/pdf/1801.06146.pdf" rel="nofollow noreferrer">Here</a> is an article about this approach and its effectiveness and you can be inspired by <a href="https://github.com/allenai/scibert" rel="nofollow noreferrer">Scibert</a> and <a href="https://github.com/manueltonneau/covid-berts" rel="nofollow noreferrer">COVIDbert</a>. As you expect the use pre-trained bert as a starting point and continue pre-training using domain-specified corpus!</p>
Training a new dataset on bert
<p>I am new to BERT</p>
<p>I have a amazon review dataset, where I want to predict the star rating based on the review</p>
<p>I know I can use a pretrained bert model as shown <a href="https://github.com/nicknochnack/BERTSentiment/blob/main/Sentiment.ipynb" rel="nofollow noreferrer">here</a></p>
<p>But I want to train the bert model on my own dataset. Is that whats being done <a href="https://medium.com/analytics-vidhya/fine-tuning-bert-for-amazon-food-reviews-32e474de0e51" rel="nofollow noreferrer">here</a>? And can I apply this type of 'fine tuning' on a pretrained model with any dataset to get more accurate results or do I have to do something else to train the model from scratch</p>
<p>And if I do want to train a model from scratch, where would I start</p>