pyspark/bigdl/models/lenet/utils.py
def get_mnist(sc, data_type="train", location="/tmp/mnist"):
    """
    Get mnist dataset and parallelize into RDDs.
    Data would be downloaded automatically if it doesn't present at the specific location.

    :param sc: SparkContext.
    :param data_type: "train" for training data and "test" for testing data.
    :param location: Location to store mnist dataset.
    :return: RDD of (features: ndarray, label: ndarray).
    """
    (images, labels) = mnist.read_data_sets(location, data_type)
    images = sc.parallelize(images)
    labels = sc.parallelize(labels + 1)  # Target start from 1 in BigDL
    record = images.zip(labels)
    return record
def get_mnist(sc, data_type="train", location="/tmp/mnist"):
    (images, labels) = mnist.read_data_sets(location, data_type)
    images = sc.parallelize(images)
    labels = sc.parallelize(labels + 1)  # Target start from 1 in BigDL
    record = images.zip(labels)
    return record
def get_mnist ( sc , data_type = "train" , location = "/tmp/mnist" ) : ( images , labels ) = mnist . read_data_sets ( location , data_type ) images = sc . parallelize ( images ) labels = sc . parallelize ( labels + 1 ) # Target start from 1 in BigDL record = images . zip ( labels ) return record
Get mnist dataset and parallelize into RDDs.
    Data would be downloaded automatically if it doesn't present at the specific location.

    :param sc: SparkContext.
    :param data_type: "train" for training data and "test" for testing data.
    :param location: Location to store mnist dataset.
    :return: RDD of (features: ndarray, label: ndarray).
Get mnist dataset and parallelize into RDDs . Data would be downloaded automatically if it doesn t present at the specific location .