title: Splitting text DF into individual sentences DF: how to create a longer Pandas Data Frame with lambda and apply?
body: <p>This question might look long but I promise it is really not complicted.</p>
  
  <p>I have a DF with textblocks and some ID columns. I want to create a new DF that contains each sentence as its own row. </p>
  
  <pre class="lang-py prettyprint-override"><code>original_df = pd.DataFrame(data={"year":[2018,2019], "text_nr":[1,2], "text":["This is one sentence. This is another!","Please help me. I am lost. "]})
  original_df
  &gt;&gt;&gt;
         year  text_nr  text
      0  2018  1        "This is one sentence. This is another!"
      1  2019  2        "Please help me. I am lost."
  </code></pre>
  
  <p>I would like to split each textblock into individual sentences using spacy and create a new DF that looks like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>sentences_df
  &gt;&gt;&gt;
     year  text_nr  sent_nr sentence
  0  2018      1       1   "This is one sentence". 
  1  2018      1       2   "This is another!"
  2  2019      2       1   "Please help me."
  3  2019      2       2   "I am lost."
  </code></pre>
  
  <p>I have found a way to do it like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>import spacy
  nlp = spacy.load("en_core_news_sm")
  sentences_list = []
  
  for i, row in original_df.iterrows():
      doc = nlp(row["text"])
      sentences = [(row["year"],row["text_nr"],str(i+1),sent.string.replace('\n','').replace('\t','').strip()) for i, sent in enumerate(doc.sents)]
      sentences_list = sentences_list+sentences
  
  sentences_df = pd.DataFrame(sentences_list, columns = ["year",text_nr","sent_nr","sentence"])
  </code></pre>
  
  <p>But it is not very elegant and I read that <code>df.apply(lambda: ...)</code> method is much faster.
  However, when I try it, I never manage to get the correct result.I tried these two ways:</p>
  
  <ol>
  <li>First Attempt:</li>
  </ol>
  
  <pre class="lang-py prettyprint-override"><code>nlp = spacy.load("en_core_news_sm")
  def sentencizer (x, nlp_model):
      sentences = {}
      doc = nlp_model(x["text"])
      for i, sent in enumerate(doc.sents):
          sentences["year"]=x["year"]
          sentences["text_nr"]=x["text_nr"]
          sentences["sent_nr"] = str(i+1)
          sentences["sentence"] = sent.string.replace('\n','').replace('\t','').strip()
      return sentences
  sentences_df = original_df.head().apply(lambda x: pd.Series(sentencizer(x,nlp)),axis=1)
  
  </code></pre>
  
  <p>This only ever gets the last sentence</p>
  
  <pre class="lang-py prettyprint-override"><code>sentences_df
  &gt;&gt;&gt;
     year  text_nr sent_nr  sentence
  0  2018        1       2  "This is another!"
  1  2019        2       2  "I am lost!"
  </code></pre>
  
  <ol start="2">
  <li>Second Attempt</li>
  </ol>
  
  <pre class="lang-py prettyprint-override"><code>nlp = spacy.load("en_core_news_sm")
  def sentencizer (x, nlp_model):
      sentences = {"year":[],"text_nr":[],"sent_nr":[],"sentence":[]}
      doc = nlp_model(x["text"])
      for i, sent in enumerate(doc.sents):
          sentences["year"].append(x["year"])
          sentences["text_nr"].append(x["text_nr"])
          sentences["sent_nr"].append(str(i+1))
          sentences["sentence"].append(sent.string.replace('\n','').replace('\t','').strip())
      return sentences
  sentences_df = original_df.apply(lambda x: pd.Series(sentencizer(x,nlp)),axis=1)
  </code></pre>
  
  <p>This yields me a DF with lists as entrys:</p>
  
  <pre class="lang-py prettyprint-override"><code>sentences_df
  &gt;&gt;&gt;
     year          text_nr sent_nr    sentence
  0  [2018, 2018]  [1, 1]  [1, 2]  ["This is one sentence.", "This is another!"]
  1  [2019, 2019]  [2, 2]  [1, 2]  ["Please help me.", "I am lost."]
  </code></pre>
  
  <p>I could probably try to expand this last df, but I am sure there is a way to do this correctly in one go. I would like to use <code>spacy</code>for splitting the text as it has more advanced sentence boundry detection than just using regex/ string splitting. You do not need to download <code>spacy</code> to help me (-><code>string.split()</code> is fine for the dummy data here). I just need to find a logic that works along the same lines as the following so I can rewrite it to use it with <code>spacy</code>. </p>
  
  <pre class="lang-py prettyprint-override"><code>nlp = spacy.load("en_core_news_sm")
  doc = nlp("This is a sentence.\n This is another! ")
  sentences = [sent.string.strip() for sent in doc.sents] #doc.sents is a generator
  sentences
  &gt;&gt;&gt;
  ["This is a sentence", "This is another!"]
  </code></pre>
  
  <p>So something along the lines of this would be great:</p>
  
  <pre class="lang-py prettyprint-override"><code>text = "This is a sentence.\n This is another! "
  sentences = [sent.replace("\n","").strip() for sent in text.split(".")]
  sentences
  &gt;&gt;&gt;
  ["This is a sentence", "This is another!"]
  </code></pre>
  
  <p>Thanks a lot for any help. I am quite new to programming so please have mercy :)</p>