title: How to solve data type error using pickle and pandas udf for XGBoost model deployment python?
body: <p>I have created a pandas udf() which splits dataset, fit XGBoost model, save it using pickle and returns a df with the saved model as a string column. The problem is when I call pandas udf(). It gives 'unsupported data type' error. But, when I run code without pandas udf() framework it runs successfully. Does anyone have any ideas on this?</p>
  <p>@pandas_udf(schema,PandasUDFType.GROUPED_MAP)
  def pickle_model(df1):</p>
  <pre><code>X = df1.iloc[:,1:50] 
  Y= df1.iloc[:,50] 
  seed = 7 
  test_size = 0.30 
  
  X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=test_size, random_state=seed)
  
  model = XGBClassifier(max_depth=3,...) 
  
  eval_set = [(X_train, y_train), (X_test, y_test)] 
  
  model.fit(X_train, y_train, early_stopping_rounds=8,...) 
  
  model_str = pickle.dumps(model)
  
  model_saved = pd.DataFrame([model_str],columns = ['model_str'])
  return model_saved 
  </code></pre>
  <p>pickled_model = df2.groupby('id').apply(pickle_model)
  pickled_model.collect()</p>
  <p>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 691.0 failed 4 times, most recent failure: Lost task 0.3 in stage 691.0 (TID 38146, 10.139.64.9, executor 40): java.lang.UnsupportedOperationException: Unsupported data type: struct&lt;type:tinyint,size:int,indices:array&lt;int&gt;,values:array&lt;double&gt;&gt;</p>