original_docstring: 
      Creates a dataframe of all 2D ACDC cmr filenames,
      splits the data in n folds with respect to the pathologies
      0. Load all 2D nrrd files in the given directory
      1. Create a dataframe with all original ACDC files, patient ids, pathology
      2. Create a list of all pathologies
      3. Create n (# of pathologies) subgroups of patients, filtered by pathology
      4. Create k splits (# of splits) of each patient subgroup
      5. For each pathology split: collect all 2D files
      6. Merge 2D files according to their split number and modality (train/val)
      return a df with all files for each split
      f_name, patient, split, train_modality, pathology
  
      :param img_pattern:
      :param path_to_data: path to 2D ACDC files
      :param kfolds:
      :return: dataframe with all splits
      
docstring: Creates a dataframe of all 2D ACDC cmr filenames,
  splits the data in n folds with respect to the pathologies
  0. Load all 2D nrrd files in the given directory
  1. Create a dataframe with all original ACDC files, patient ids, pathology
  2. Create a list of all pathologies
  3. Create n (# of pathologies) subgroups of patients, filtered by pathology
  4. Create k splits (# of splits) of each patient subgroup
  5. For each pathology split: collect all 2D files
  6. Merge 2D files according to their split number and modality (train/val)
  return a df with all files for each split
  f_name, patient, split, train_modality, pathology
docstring_tokens: Creates a dataframe of all 2D ACDC cmr filenames splits the data in n folds with respect to the pathologies 0 . Load all 2D nrrd files in the given directory 1 . Create a dataframe with all original ACDC files patient ids pathology 2 . Create a list of all pathologies 3 . Create n ( # of pathologies ) subgroups of patients filtered by pathology 4 . Create k splits ( # of splits ) of each patient subgroup 5 . For each pathology split : collect all 2D files 6 . Merge 2D files according to their split number and modality ( train / val ) return a df with all files for each split f_name patient split train_modality pathology
code: def create_acdc_dataframe_for_cv(path_to_data='data/raw/ACDC/2D/', kfolds=4,
                                   original_acdc_dir='data/raw/ACDC/original/all/', img_pattern='*img.nrrd'):
      """
      Creates a dataframe of all 2D ACDC cmr filenames,
      splits the data in n folds with respect to the pathologies
      0. Load all 2D nrrd files in the given directory
      1. Create a dataframe with all original ACDC files, patient ids, pathology
      2. Create a list of all pathologies
      3. Create n (# of pathologies) subgroups of patients, filtered by pathology
      4. Create k splits (# of splits) of each patient subgroup
      5. For each pathology split: collect all 2D files
      6. Merge 2D files according to their split number and modality (train/val)
      return a df with all files for each split
      f_name, patient, split, train_modality, pathology
  
      :param img_pattern:
      :param path_to_data: path to 2D ACDC files
      :param kfolds:
      :return: dataframe with all splits
      """
  
      from sklearn.model_selection import KFold
  
      seed = 42
      # img_pattern = '*img.nrrd'
      columns = ['fold', 'x_path', 'y_path', 'modality', 'patient', 'pathology']
      modality_train = 'train'
      modality_test = 'test'
  
      # list all nrrd image files within the subdirs of path_to_data
      acdc_x_files = sorted(glob.glob(os.path.join(path_to_data, '**/{}'.format(img_pattern))))
      logging.info('Found: {} files in {}'.format(len(acdc_x_files), path_to_data))
  
      # get all ACDC files + pathology as df
      # provide a seed to make shuffle idempotent
      df = get_acdc_dataset_as_df(original_acdc_dir)
      logging.info('Created a dataframe with shape: {}'.format(df.shape))
      pathologies = df['pathology'].unique()
      kfold = KFold(kfolds, shuffle=True, random_state=seed)
  
      # create a df to merge all splits into
      df_folds = pd.DataFrame(columns=columns)
  
      # for each pathology, create k folds of train/test splits
      for pathology in pathologies:
          # collect all patients with this pathology
          patients = df[df['pathology'] == pathology]['patient'].unique()
          logging.debug('{} Patients found for pathology: {}'.format(len(patients), pathology))
  
          # create k fold of train/test splits
          # split with the patient ids
          # to make sure that one patient occurs either as train or validation data
          # f = fold, idx = tuple with all indexes to split the patients in this fold
          # kfold.split returns the patient indexes for each fold
          for fold, idx in enumerate(kfold.split(patients)):
              train_idx, test_idx = idx
              # logging.debug("TRAIN:", train_idx, "TEST:", test_idx)
  
              # create one list for the train and test patient ids from the kfold split indexes
              patients_train, patients_test = [patients[i] for i in train_idx], [patients[i] for i in test_idx]
              logging.debug('Fold: {}, Pathology: {} train: {}'.format(fold, pathology, patients_train))
              logging.debug('Fold: {}, Pathology: {}, test: {}'.format(fold, pathology, patients_test))
  
              # filter the 2D filenames by the two patient id lists (train/test) for this fold
              # create one df for each split (train/test) with the corresponding 2D nrrd files
              df_train = filter_x_by_patient_ids(acdc_x_files, patients_train, modality_train, columns, fold, pathology)
              df_test = filter_x_by_patient_ids(acdc_x_files, patients_test, modality_test, columns, fold, pathology)
              logging.debug('Files x_train: {}'.format(len(df_train)))
              logging.debug('Files x_test: {}'.format(len(df_test)))
  
              # merge train and test files of this split
              df_fold = pd.concat([df_train, df_test])
              # merge this fold into folds dataset
              df_folds = pd.concat([df_fold, df_folds], sort=True)
  
      return df_folds
code_tokens: def create_acdc_dataframe_for_cv ( path_to_data = 'data/raw/ACDC/2D/' , kfolds = 4 , original_acdc_dir = 'data/raw/ACDC/original/all/' , img_pattern = '*img.nrrd' ) : from sklearn . model_selection import KFold seed = 42 columns = [ 'fold' , 'x_path' , 'y_path' , 'modality' , 'patient' , 'pathology' ] modality_train = 'train' modality_test = 'test' acdc_x_files = sorted ( glob . glob ( os . path . join ( path_to_data , '**/{}' . format ( img_pattern ) ) ) ) logging . info ( 'Found: {} files in {}' . format ( len ( acdc_x_files ) , path_to_data ) ) df = get_acdc_dataset_as_df ( original_acdc_dir ) logging . info ( 'Created a dataframe with shape: {}' . format ( df . shape ) ) pathologies = df [ 'pathology' ] . unique ( ) kfold = KFold ( kfolds , shuffle = True , random_state = seed ) df_folds = pd . DataFrame ( columns = columns ) for pathology in pathologies : patients = df [ df [ 'pathology' ] == pathology ] [ 'patient' ] . unique ( ) logging . debug ( '{} Patients found for pathology: {}' . format ( len ( patients ) , pathology ) ) for fold , idx in enumerate ( kfold . split ( patients ) ) : train_idx , test_idx = idx patients_train , patients_test = [ patients [ i ] for i in train_idx ] , [ patients [ i ] for i in test_idx ] logging . debug ( 'Fold: {}, Pathology: {} train: {}' . format ( fold , pathology , patients_train ) ) logging . debug ( 'Fold: {}, Pathology: {}, test: {}' . format ( fold , pathology , patients_test ) ) df_train = filter_x_by_patient_ids ( acdc_x_files , patients_train , modality_train , columns , fold , pathology ) df_test = filter_x_by_patient_ids ( acdc_x_files , patients_test , modality_test , columns , fold , pathology ) logging . debug ( 'Files x_train: {}' . format ( len ( df_train ) ) ) logging . debug ( 'Files x_test: {}' . format ( len ( df_test ) ) ) df_fold = pd . concat ( [ df_train , df_test ] ) df_folds = pd . concat ( [ df_fold , df_folds ] , sort = True ) return df_folds
short_docstring: Creates a dataframe of all 2D ACDC cmr filenames,
  splits the data in n folds with respect to the pathologies
  0.
short_docstring_tokens: Creates a dataframe of all 2D ACDC cmr filenames splits the data in n folds with respect to the pathologies 0 .
comment: """
      Creates a dataframe of all 2D ACDC cmr filenames,
      splits the data in n folds with respect to the pathologies
      0. Load all 2D nrrd files in the given directory
      1. Create a dataframe with all original ACDC files, patient ids, pathology
      2. Create a list of all pathologies
      3. Create n (# of pathologies) subgroups of patients, filtered by pathology
      4. Create k splits (# of splits) of each patient subgroup
      5. For each pathology split: collect all 2D files
      6. Merge 2D files according to their split number and modality (train/val)
      return a df with all files for each split
      f_name, patient, split, train_modality, pathology
  
      :param img_pattern:
      :param path_to_data: path to 2D ACDC files
      :param kfolds:
      :return: dataframe with all splits
      """$SEP$# img_pattern = '*img.nrrd'$SEP$# list all nrrd image files within the subdirs of path_to_data$SEP$# get all ACDC files + pathology as df$SEP$# provide a seed to make shuffle idempotent$SEP$# create a df to merge all splits into$SEP$# for each pathology, create k folds of train/test splits$SEP$# collect all patients with this pathology$SEP$# create k fold of train/test splits$SEP$# split with the patient ids$SEP$# to make sure that one patient occurs either as train or validation data$SEP$# f = fold, idx = tuple with all indexes to split the patients in this fold$SEP$# kfold.split returns the patient indexes for each fold$SEP$# logging.debug("TRAIN:", train_idx, "TEST:", test_idx)$SEP$# create one list for the train and test patient ids from the kfold split indexes$SEP$# filter the 2D filenames by the two patient id lists (train/test) for this fold$SEP$# create one df for each split (train/test) with the corresponding 2D nrrd files$SEP$# merge train and test files of this split$SEP$# merge this fold into folds dataset