original_docstring: 
      Expecting data in the PIVOTED format from the Stack Overflow query editor. 
      Here the csv file has an index of userIds, and the columns are the date from 
      start to end. The values are the counts of edits that that user performed on that
      day. There is a separate file for the userId.
      
docstring: Expecting data in the PIVOTED format from the Stack Overflow query editor.
  Here the csv file has an index of userIds, and the columns are the date from
  start to end. The values are the counts of edits that that user performed on that
  day. There is a separate file for the userId.
docstring_tokens: Expecting data in the PIVOTED format from the Stack Overflow query editor . Here the csv file has an index of userIds and the columns are the date from start to end . The values are the counts of edits that that user performed on that day . There is a separate file for the userId .
code: def transform_editing_data_to_file_folder_structure(path_to_csv_actions, path_to_csv_badges, path_to_data_dir):
      '''
      Expecting data in the PIVOTED format from the Stack Overflow query editor. 
      Here the csv file has an index of userIds, and the columns are the date from 
      start to end. The values are the counts of edits that that user performed on that
      day. There is a separate file for the userId.
      '''
      import tqdm
  
      data_actions = pd.read_csv(path_to_csv_actions)
      badge_achievements = pd.read_csv(path_to_csv_badges)
  
      data_actions = data_actions[data_actions.UserId.isin(badge_achievements.UserId)]
      badge_achievements = badge_achievements[badge_achievements.UserId.isin(data_actions.UserId)]
  
      start_date = pd.datetime(year=2009, month=1, day=1)
  
      badge_achievements.Date = pd.to_datetime(badge_achievements.Date)
      badge_achievements['day'] = (badge_achievements.Date - start_date).dt.days
  
      user_ids = badge_achievements.UserId.unique()
      size_data = len(user_ids)
  
      np.random.seed(11)
  
      train = np.random.choice(user_ids, size=int(np.floor(0.6 * size_data)), replace=False)
      user_ids = user_ids[~np.in1d(user_ids, train)]
      validate = np.random.choice(user_ids, size=int(np.floor(0.2 * size_data)), replace=False)
      user_ids = user_ids[~np.in1d(user_ids, validate)]
      test = np.random.choice(user_ids, size=int(np.floor(0.2 * size_data)), replace=False)
  
      data_actions.set_index('UserId', inplace=True)
      badge_achievements.set_index('UserId', inplace=True)
  
      num_days = (badge_achievements.Date.max() - start_date).days
  
      for dset in [train, validate, test]:
          for user in tqdm.tqdm(dset):
  
              trajectory = data_actions.loc[user]
              trajectory = trajectory.reset_index()
              trajectory['index'] = pd.to_datetime(trajectory['index'])
              trajectory['day'] = (trajectory['index'] - start_date).dt.days
              trajectory.rename(columns={'index': 'date', user: 'num_actions'}, inplace=True)
              trajectory.sort_values('day', inplace=True)
              trajectory.set_index('day', inplace=True)
              trajectory = trajectory.reindex(range(num_days+1), fill_value=0)
  
              action_trajectory = torch.tensor(trajectory[['num_actions']].values, dtype=torch.long)
              torch.save(action_trajectory, '{}/user_{}.pt'.format(path_to_data_dir, user))
  
      with open('{}/badge_achievements.json'.format(path_to_data_dir), 'w') as f:
          badge_dict = badge_achievements['day'].to_dict()
          badge_dict = {k: {'strunk_white': [int(v)]} for k,v in badge_dict.items()}
          json.dump(badge_dict, f)
  
      with open('{}/data_indexes.json'.format(path_to_data_dir), 'w') as f:
          obj = {}
          obj['train'] = [int(u) for u in train]
          obj['test'] = [int(u) for u in test]
          obj['validate'] = [int(u) for u in validate]
          json.dump(obj, f)
code_tokens: def transform_editing_data_to_file_folder_structure ( path_to_csv_actions , path_to_csv_badges , path_to_data_dir ) : import tqdm data_actions = pd . read_csv ( path_to_csv_actions ) badge_achievements = pd . read_csv ( path_to_csv_badges ) data_actions = data_actions [ data_actions . UserId . isin ( badge_achievements . UserId ) ] badge_achievements = badge_achievements [ badge_achievements . UserId . isin ( data_actions . UserId ) ] start_date = pd . datetime ( year = 2009 , month = 1 , day = 1 ) badge_achievements . Date = pd . to_datetime ( badge_achievements . Date ) badge_achievements [ 'day' ] = ( badge_achievements . Date - start_date ) . dt . days user_ids = badge_achievements . UserId . unique ( ) size_data = len ( user_ids ) np . random . seed ( 11 ) train = np . random . choice ( user_ids , size = int ( np . floor ( 0.6 * size_data ) ) , replace = False ) user_ids = user_ids [ ~ np . in1d ( user_ids , train ) ] validate = np . random . choice ( user_ids , size = int ( np . floor ( 0.2 * size_data ) ) , replace = False ) user_ids = user_ids [ ~ np . in1d ( user_ids , validate ) ] test = np . random . choice ( user_ids , size = int ( np . floor ( 0.2 * size_data ) ) , replace = False ) data_actions . set_index ( 'UserId' , inplace = True ) badge_achievements . set_index ( 'UserId' , inplace = True ) num_days = ( badge_achievements . Date . max ( ) - start_date ) . days for dset in [ train , validate , test ] : for user in tqdm . tqdm ( dset ) : trajectory = data_actions . loc [ user ] trajectory = trajectory . reset_index ( ) trajectory [ 'index' ] = pd . to_datetime ( trajectory [ 'index' ] ) trajectory [ 'day' ] = ( trajectory [ 'index' ] - start_date ) . dt . days trajectory . rename ( columns = { 'index' : 'date' , user : 'num_actions' } , inplace = True ) trajectory . sort_values ( 'day' , inplace = True ) trajectory . set_index ( 'day' , inplace = True ) trajectory = trajectory . reindex ( range ( num_days + 1 ) , fill_value = 0 ) action_trajectory = torch . tensor ( trajectory [ [ 'num_actions' ] ] . values , dtype = torch . long ) torch . save ( action_trajectory , '{}/user_{}.pt' . format ( path_to_data_dir , user ) ) with open ( '{}/badge_achievements.json' . format ( path_to_data_dir ) , 'w' ) as f : badge_dict = badge_achievements [ 'day' ] . to_dict ( ) badge_dict = { k : { 'strunk_white' : [ int ( v ) ] } for k , v in badge_dict . items ( ) } json . dump ( badge_dict , f ) with open ( '{}/data_indexes.json' . format ( path_to_data_dir ) , 'w' ) as f : obj = { } obj [ 'train' ] = [ int ( u ) for u in train ] obj [ 'test' ] = [ int ( u ) for u in test ] obj [ 'validate' ] = [ int ( u ) for u in validate ] json . dump ( obj , f )
short_docstring: Expecting data in the PIVOTED format from the Stack Overflow query editor.
short_docstring_tokens: Expecting data in the PIVOTED format from the Stack Overflow query editor .
comment: '''
      Expecting data in the PIVOTED format from the Stack Overflow query editor. 
      Here the csv file has an index of userIds, and the columns are the date from 
      start to end. The values are the counts of edits that that user performed on that
      day. There is a separate file for the userId.
      '''