original_docstring: 
      Note from Dan:
      All subjects should have channels 15, 16, 17 and 18 removed [...]
      If you want to make them truly identical, you could remove channel 19 from
      the subjects with more channels, although this should be 'good' data.
      
docstring: Note from Dan:
  All subjects should have channels 15, 16, 17 and 18 removed [...]
  If you want to make them truly identical, you could remove channel 19 from
  the subjects with more channels, although this should be 'good' data.
docstring_tokens: Note from Dan : All subjects should have channels 15 16 17 and 18 removed [ ... ] If you want to make them truly identical you could remove channel 19 from the subjects with more channels although this should be ' good ' data .
code: def preprocess(config):
      
  #     config = load_config(default_config='../train_sda.cfg');
      
      DATA_ROOT = config.eeg.get('dataset_root', './');
      SAMPLE_RATE = 400; # in Hz
      TRIAL_LENGTH = 32; # in sec
      
      TRIAL_LENGTH += 4; # add 4s after end of presentation
      
      TRIAL_SAMPLE_LENGTH = SAMPLE_RATE * TRIAL_LENGTH;    
      
      log.info('using dataset at {}'.format(DATA_ROOT));
      
      '''
      Note from Dan:
      All subjects should have channels 15, 16, 17 and 18 removed [...]
      If you want to make them truly identical, you could remove channel 19 from
      the subjects with more channels, although this should be 'good' data.
      '''    
      bad_channels = {};
      bad_channels[1]  = [5, 6,                   15, 16, 17, 18,  20, 21];
      bad_channels[2]  = [      7, 8,             15, 16, 17, 18,  20, 21];
      bad_channels[3]  = [5, 6,                   15, 16, 17, 18,  20, 21];
      bad_channels[4]  = [      7, 8,             15, 16, 17, 18,  20, 21];
      bad_channels[5]  = [      7, 8,             15, 16, 17, 18,  20, 21];
      bad_channels[6]  = [      7, 8, 9,  12,     15, 16, 17, 18         ];
      bad_channels[7]  = [5, 6,           12,     15, 16, 17, 18,  20    ];
      bad_channels[8]  = [      7, 8,             15, 16, 17, 18,  20, 21];
      bad_channels[9]  = [5, 6,           12,     15, 16, 17, 18,  20    ];
      bad_channels[10] = [5, 6,                   15, 16, 17, 18,  20, 21];
      bad_channels[11] = [5, 6,                   15, 16, 17, 18,  20, 21];
      bad_channels[12] = [5, 6,                   15, 16, 17, 18,  20, 21];
      bad_channels[13] = [5, 6,           12,     15, 16, 17, 18,  20    ];
      
      with log_timing(log, 'generating datasets'):
          for subject_id in xrange(1,14):
              search_path = os.path.join(DATA_ROOT, 'Sub{0:03d}*'.format(subject_id));
              path = glob.glob(search_path);
              
              if path is None or len(path) == 0:
                  log.warn('nothing found at {}'.format(search_path));
                  continue;
              else:
                  path = path[0];
              
              trials_filename = os.path.join(path, 'trials.pklz');        
              
              trials = None;        
              if not os.path.isfile(trials_filename):
                  log.debug('{} not found. running split_trial()'.format(trials_filename));
                  trials = split_trial(path, TRIAL_SAMPLE_LENGTH);
              else:
                  with log_timing(log, 'loading data from {}'.format(trials_filename)):    
                      trials = load(trials_filename);
                      
              assert trials;
               
              dataset_filename = os.path.join(path, 'dataset_13goodchannels_plus4s.pklz');
              dataset = generate_cases(subject_id, trials, bad_channels[subject_id]); # = data, labels
              with log_timing(log, 'saving dataset to {}'.format(dataset_filename)):
                  save(dataset_filename, dataset);
code_tokens: def preprocess ( config ) : DATA_ROOT = config . eeg . get ( 'dataset_root' , './' ) ; SAMPLE_RATE = 400 ; TRIAL_LENGTH = 32 ; TRIAL_LENGTH += 4 ; TRIAL_SAMPLE_LENGTH = SAMPLE_RATE * TRIAL_LENGTH ; log . info ( 'using dataset at {}' . format ( DATA_ROOT ) ) ; bad_channels = { } ; bad_channels [ 1 ] = [ 5 , 6 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 2 ] = [ 7 , 8 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 3 ] = [ 5 , 6 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 4 ] = [ 7 , 8 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 5 ] = [ 7 , 8 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 6 ] = [ 7 , 8 , 9 , 12 , 15 , 16 , 17 , 18 ] ; bad_channels [ 7 ] = [ 5 , 6 , 12 , 15 , 16 , 17 , 18 , 20 ] ; bad_channels [ 8 ] = [ 7 , 8 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 9 ] = [ 5 , 6 , 12 , 15 , 16 , 17 , 18 , 20 ] ; bad_channels [ 10 ] = [ 5 , 6 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 11 ] = [ 5 , 6 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 12 ] = [ 5 , 6 , 15 , 16 , 17 , 18 , 20 , 21 ] ; bad_channels [ 13 ] = [ 5 , 6 , 12 , 15 , 16 , 17 , 18 , 20 ] ; with log_timing ( log , 'generating datasets' ) : for subject_id in xrange ( 1 , 14 ) : search_path = os . path . join ( DATA_ROOT , 'Sub{0:03d}*' . format ( subject_id ) ) ; path = glob . glob ( search_path ) ; if path is None or len ( path ) == 0 : log . warn ( 'nothing found at {}' . format ( search_path ) ) ; continue ; else : path = path [ 0 ] ; trials_filename = os . path . join ( path , 'trials.pklz' ) ; trials = None ; if not os . path . isfile ( trials_filename ) : log . debug ( '{} not found. running split_trial()' . format ( trials_filename ) ) ; trials = split_trial ( path , TRIAL_SAMPLE_LENGTH ) ; else : with log_timing ( log , 'loading data from {}' . format ( trials_filename ) ) : trials = load ( trials_filename ) ; assert trials ; dataset_filename = os . path . join ( path , 'dataset_13goodchannels_plus4s.pklz' ) ; dataset = generate_cases ( subject_id , trials , bad_channels [ subject_id ] ) ; with log_timing ( log , 'saving dataset to {}' . format ( dataset_filename ) ) : save ( dataset_filename , dataset ) ;
short_docstring: Note from Dan:
  All subjects should have channels 15, 16, 17 and 18 removed [...]
  If you want to make them truly identical, you could remove channel 19 from
  the subjects with more channels, although this should be 'good' data.
short_docstring_tokens: Note from Dan : All subjects should have channels 15 16 17 and 18 removed [ ... ] If you want to make them truly identical you could remove channel 19 from the subjects with more channels although this should be ' good ' data .
comment: #     config = load_config(default_config='../train_sda.cfg');$SEP$# in Hz$SEP$# in sec$SEP$# add 4s after end of presentation$SEP$'''
      Note from Dan:
      All subjects should have channels 15, 16, 17 and 18 removed [...]
      If you want to make them truly identical, you could remove channel 19 from
      the subjects with more channels, although this should be 'good' data.
      '''$SEP$# = data, labels