{"action":"created","issue":{"url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/100","repository_url":"https://api.github.com/repos/aws/sagemaker-python-sdk","labels_url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/100/labels{/name}","comments_url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/100/comments","events_url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/100/events","html_url":"https://github.com/aws/sagemaker-python-sdk/issues/100","id":305949578,"number":100,"title":"Where and how may I preprocess input data before making predictions?","user":{"login":"mihi-r","id":37024793,"avatar_url":"https://avatars1.githubusercontent.com/u/37024793?v=4","gravatar_id":"","url":"https://api.github.com/users/mihi-r","html_url":"https://github.com/mihi-r","followers_url":"https://api.github.com/users/mihi-r/followers","following_url":"https://api.github.com/users/mihi-r/following{/other_user}","gists_url":"https://api.github.com/users/mihi-r/gists{/gist_id}","starred_url":"https://api.github.com/users/mihi-r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mihi-r/subscriptions","organizations_url":"https://api.github.com/users/mihi-r/orgs","repos_url":"https://api.github.com/users/mihi-r/repos","events_url":"https://api.github.com/users/mihi-r/events{/privacy}","received_events_url":"https://api.github.com/users/mihi-r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-16T14:30:17Z","updated_at":"2018-03-16T18:59:54Z","closed_at":null,"author_association":"NONE","body":"I'd like to preprocess my data by sending in a string input through `predictor.predict(data)` and turning it into numerical embeddings just as my  `train_input_fn` is doing with `vocab_processor.fit_transform` before going though my `model_fn` :\r\n\r\n```\r\ndef train_input_fn(training_dir, hyperparmeters):\r\n    return _input_fn(training_dir, 'meta_data_train.csv')\r\n\r\ndef _input_fn(training_dir, training_filename):\r\n    training_set = pd.read_csv(os.path.join(training_dir, training_filename), dtype={'Classification class name': object}, encoding='cp1252')\r\n    global n_words\r\n  # Prepare training and testing data\r\n    data = training_set['Features']\r\n    target = pd.Series(training_set['Labels'])\r\n\r\n    if training_filename == 'meta_data_test.csv':\r\n        vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor.restore(\"s3://sagemaker-blah/vocab\")\r\n        data = np.array(list(vocab_processor.transform(data)))\r\n        return tf.estimator.inputs.numpy_input_fn(\r\n            x={INPUT_TENSOR_NAME: data},\r\n            y=target,\r\n            num_epochs=100,\r\n            shuffle=False)()\r\n    elif training_filename == 'meta_data_train.csv':\r\n        vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor(MAX_DOCUMENT_LENGTH)\r\n        data = np.array(list(vocab_processor.fit_transform(data)))\r\n        vocab_processor.save(\"s3://sagemaker-blah/vocab\")\r\n        n_words = len(vocab_processor.vocabulary_)\r\n        return tf.estimator.inputs.numpy_input_fn(\r\n            x={INPUT_TENSOR_NAME: data},\r\n            y=target,\r\n            batch_size=len(data),\r\n            num_epochs=None,\r\n            shuffle=True)()\r\n```\r\n\r\nThe documentation says to do it through serving_input_fn but I'm not sure how I can access and manipulate the data from my tensor using `vocab_processor.transform`. Here's my serving_input_fn for context:\r\n\r\n```\r\ndef serving_input_fn(hyperparmeters):\r\n    tensor = tf.placeholder(tf.int64, shape=[None, MAX_DOCUMENT_LENGTH])\r\n    return build_raw_serving_input_receiver_fn({INPUT_TENSOR_NAME: tensor})()\r\n```\r\n\r\nI tried doing so through a input_fn instead:\r\n\r\n```\r\ndef input_fn(serialized_input, content_type):\r\n    vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor(MAX_DOCUMENT_LENGTH)\r\n    deserialized_input = pickle.loads(serialized_input)\r\n    deserialized_input = np.array(list(vocab_processor.fit_transform(deserialized_input)))\r\n    return deserialized_input\r\n```\r\nHere, I had an error deserializing:\r\n\r\n>  KeyError: '['\r\n\r\nWhat would be the best method to preprocess the data? "},"comment":{"url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/comments/373813344","html_url":"https://github.com/aws/sagemaker-python-sdk/issues/100#issuecomment-373813344","issue_url":"https://api.github.com/repos/aws/sagemaker-python-sdk/issues/100","id":373813344,"user":{"login":"mihi-r","id":37024793,"avatar_url":"https://avatars1.githubusercontent.com/u/37024793?v=4","gravatar_id":"","url":"https://api.github.com/users/mihi-r","html_url":"https://github.com/mihi-r","followers_url":"https://api.github.com/users/mihi-r/followers","following_url":"https://api.github.com/users/mihi-r/following{/other_user}","gists_url":"https://api.github.com/users/mihi-r/gists{/gist_id}","starred_url":"https://api.github.com/users/mihi-r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mihi-r/subscriptions","organizations_url":"https://api.github.com/users/mihi-r/orgs","repos_url":"https://api.github.com/users/mihi-r/repos","events_url":"https://api.github.com/users/mihi-r/events{/privacy}","received_events_url":"https://api.github.com/users/mihi-r/received_events","type":"User","site_admin":false},"created_at":"2018-03-16T18:59:54Z","updated_at":"2018-03-16T18:59:54Z","author_association":"NONE","body":"After further debugging, I found out I wasn't pickling correct. Once I get that resolved, this method should work. Thanks for your help!"}}
{"actor":{"display_login":"mihi-r"}}