revoscalepy.RxSparkDataFrame(data=None, column_info=None,
    write_factors_as_indexes: bool = False)


Main generator for class RxSparkDataFrame, which extends RxSparkData.



Spark data frame obj from pyspark.sql.DataFrame.



Supported types are:
    ”bool” (stored as uchar),
    “integer” (stored as int32),
    “float32” (stored as FloatType),
    “numeric” (stored as float64),
    “character” (stored as string),
    “factor” (stored as uint32),



Bool value, if True, when writing to an RxSparkDataFrame data source, underlying factor indexes will be written instead of the string representations.


Object of class RxSparkDataFrame.


import os
from revoscalepy import rx_data_step, RxSparkDataFrame
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([('Monday',3012),('Friday',1354),('Sunday',5452)], ["DayOfWeek", "Sale"])
col_info = {"DayOfWeek": {"type":"factor"}}
ds = RxSparkDataFrame(df, column_info=col_info)
out_ds = RxSparkDataFrame(write_factors_as_indexes=True)
rx_data_step(ds, out_ds)
out_df = out_ds.spark_data_frame
# [Row(DayOfWeek=0, Sale=3012), Row(DayOfWeek=2, Sale=1354)]