original_docstring: 
  	Applies the paragraph separation process (next function) on all _novel.json files to produce _preproc.json files.
  	:param overwrite: should already preprocessed files be re-preprocessed
  	:param files: optional list of files to work on
  	:param min_threshold: minimum length (in chars) a paragraph should be to be taken into account.
  						  Lower than this threshold often means it's a title or a chapter separator.
  	:param min_length: minimum length of final sub-paragraphs. It will not be strictly respected though (not used anymore).
  	:param max_length: maximum length of final sub-paragraphs. Strictly respected (not used anymore).
  	:param skip_begin: percent of the paragraphs to skip at the beginning of the books, to avoid non-relevant content
  	:param skip_end: percent of the paragraphs to skip at the end of the books, to avoid non-relevant content
  	:param verbose: 0 for silent execution, 1 for statistics and 2 for statistics and histogram of paragraphs sizes.
  	
docstring: Applies the paragraph separation process (next function) on all _novel.json files to produce _preproc.json files.
docstring_tokens: Applies the paragraph separation process ( next function ) on all _novel . json files to produce _preproc . json files .
code: def separate_paragraphs_all_files(overwrite: bool, files: List[str] = None, min_threshold: int = 20,
  								  min_length: int = 600, max_length: int = 900, skip_begin=0.0, skip_end=0.0,
  								  verbose: int = 1):
  	"""
  	Applies the paragraph separation process (next function) on all _novel.json files to produce _preproc.json files.
  	:param overwrite: should already preprocessed files be re-preprocessed
  	:param files: optional list of files to work on
  	:param min_threshold: minimum length (in chars) a paragraph should be to be taken into account.
  						  Lower than this threshold often means it's a title or a chapter separator.
  	:param min_length: minimum length of final sub-paragraphs. It will not be strictly respected though (not used anymore).
  	:param max_length: maximum length of final sub-paragraphs. Strictly respected (not used anymore).
  	:param skip_begin: percent of the paragraphs to skip at the beginning of the books, to avoid non-relevant content
  	:param skip_end: percent of the paragraphs to skip at the end of the books, to avoid non-relevant content
  	:param verbose: 0 for silent execution, 1 for statistics and 2 for statistics and histogram of paragraphs sizes.
  	"""
  	parser = ParagraphParser(min_threshold)
  
  	files = os.listdir(NOVEL_PATH) if files is None else [f + NOVEL_SUFFIX for f in files]
  	treated_files = os.listdir(PREPROC_PATH)
  	treated_ids = [f[:-len(PREPROC_SUFFIX)] for f in treated_files]
  
  	# Loop over all stored texts, already converted json format but not fully pre-processed
  	for f in files:
  		d_id = f[:-len(NOVEL_SUFFIX)]
  		if not os.path.exists(NOVEL_PATH + d_id + NOVEL_SUFFIX):
  			continue
  		if overwrite or d_id not in treated_ids:
  			separate_in_paragraphs(parser, d_id, skip_begin, skip_end, verbose)
code_tokens: def separate_paragraphs_all_files ( overwrite : bool , files : List [ str ] = None , min_threshold : int = 20 , min_length : int = 600 , max_length : int = 900 , skip_begin = 0.0 , skip_end = 0.0 , verbose : int = 1 ) : parser = ParagraphParser ( min_threshold ) files = os . listdir ( NOVEL_PATH ) if files is None else [ f + NOVEL_SUFFIX for f in files ] treated_files = os . listdir ( PREPROC_PATH ) treated_ids = [ f [ : - len ( PREPROC_SUFFIX ) ] for f in treated_files ] for f in files : d_id = f [ : - len ( NOVEL_SUFFIX ) ] if not os . path . exists ( NOVEL_PATH + d_id + NOVEL_SUFFIX ) : continue if overwrite or d_id not in treated_ids : separate_in_paragraphs ( parser , d_id , skip_begin , skip_end , verbose )
short_docstring: Applies the paragraph separation process (next function) on all _novel.json files to produce _preproc.json files.
short_docstring_tokens: Applies the paragraph separation process ( next function ) on all _novel . json files to produce _preproc . json files .
comment: """
  	Applies the paragraph separation process (next function) on all _novel.json files to produce _preproc.json files.
  	:param overwrite: should already preprocessed files be re-preprocessed
  	:param files: optional list of files to work on
  	:param min_threshold: minimum length (in chars) a paragraph should be to be taken into account.
  						  Lower than this threshold often means it's a title or a chapter separator.
  	:param min_length: minimum length of final sub-paragraphs. It will not be strictly respected though (not used anymore).
  	:param max_length: maximum length of final sub-paragraphs. Strictly respected (not used anymore).
  	:param skip_begin: percent of the paragraphs to skip at the beginning of the books, to avoid non-relevant content
  	:param skip_end: percent of the paragraphs to skip at the end of the books, to avoid non-relevant content
  	:param verbose: 0 for silent execution, 1 for statistics and 2 for statistics and histogram of paragraphs sizes.
  	"""$SEP$# Loop over all stored texts, already converted json format but not fully pre-processed