{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/625","repository_url":"https://api.github.com/repos/Microsoft/CNTK","labels_url":"https://api.github.com/repos/Microsoft/CNTK/issues/625/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/CNTK/issues/625/comments","events_url":"https://api.github.com/repos/Microsoft/CNTK/issues/625/events","html_url":"https://github.com/Microsoft/CNTK/issues/625","id":161873013,"number":625,"title":"Memory allocation problem while training","user":{"login":"thanasio","id":6214090,"avatar_url":"https://avatars.githubusercontent.com/u/6214090?v=3","gravatar_id":"","url":"https://api.github.com/users/thanasio","html_url":"https://github.com/thanasio","followers_url":"https://api.github.com/users/thanasio/followers","following_url":"https://api.github.com/users/thanasio/following{/other_user}","gists_url":"https://api.github.com/users/thanasio/gists{/gist_id}","starred_url":"https://api.github.com/users/thanasio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thanasio/subscriptions","organizations_url":"https://api.github.com/users/thanasio/orgs","repos_url":"https://api.github.com/users/thanasio/repos","events_url":"https://api.github.com/users/thanasio/events{/privacy}","received_events_url":"https://api.github.com/users/thanasio/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-06-23T09:01:40Z","updated_at":"2016-08-17T14:06:09Z","closed_at":"2016-08-17T14:06:09Z","body":"Hi,\r\n\r\ni am trying to train a dataset from a 27 GB file of 115 features x 29,097,256 lines on a workstation with a geforce GTX 750 ti gpu and 16 GB of RAM.\r\n\r\nHere is my simple .cntk configuration file:\r\n\r\nRootDir = \"..\"\r\n\r\nConfigDir = \"$RootDir$/Config\"\r\nDataDir   = \"$RootDir$/Data\"\r\nOutputDir = \"$RootDir$/Output\"\r\nModelDir  = \"$OutputDir$/Models\"\r\n\r\n\r\ndeviceId = 0\r\n\r\ncommand = Simple_Demo_Train\r\n\r\nprecision = \"float\"\r\ntraceLevel = 0\r\nmodelPath = \"$ModelDir$/simple.dnn\"\r\noutputNodeNames = ScaledLogLikelihood\r\nnumFeatures = 115\r\noutput = 4\r\n\r\n\r\n#######################################\r\n#  TRAINING CONFIG                    #\r\n#######################################\r\n\r\nSimple_Demo_Train = [\r\n    action = \"train\"\r\n\r\n\r\n    SimpleNetworkBuilder = [\r\n        layerSizes = $numFeatures$:80*3:$output$\r\n        trainingCriterion = \"CrossEntropyWithSoftmax\"\r\n        evalCriterion = \"ErrorPrediction\"\r\n        layerTypes = \"Tanh\"\r\n        initValueScale = 1.0\r\n        applyMeanVarNorm = true\r\n        uniformInit = true\r\n        needPrior = true\r\n    ]\r\n\r\n    SGD = [\r\n        epochSize = 0 \r\n        minibatchSize = 128*2\t\r\n        learningRatesPerMB = 0.05*2\r\n\tautoAdjustMiniBatch = true\r\n\tautoAdjustLR=SearchBeforeEpoch\t\r\n\t\t\r\n        momentumPerMB = 0.9\r\n        dropoutRate = 0.0\r\n        maxEpochs = 300\r\n    ]\r\n       \r\n\r\n    reader = [\r\n        readerType = \"UCIFastReader\"\r\n\tcustomDelimiter=\",\"\r\n        file = \"$DataDir$/features_train.csv\"\r\n        miniBatchMode = \"Full\"\r\n        verbosity = 1  \r\n\r\n        features = [\r\n            dim = $numFeatures$       \r\n            start = 1    \r\n        ]\r\n\r\n        labels = [\r\n            start = 0    \r\n            dim = 1      \r\n            labelDim = $output$  \r\n            labelMappingFile = \"$DataDir$/labelsmap.txt\"\r\n        ]\r\n    ]\r\n]\r\n\r\nThe problem i get at the end is:\r\n=====================================================================\r\nconfigparameters: infusSimpleNet.cntk:traceLevel=0\r\n<<<<<<<<<<<<<<<<<<<< PROCESSED CONFIG WITH ALL VARIABLES RESOLVED <<<<<<<<<<<<<<<<<<<<\r\nCommands: Simple_Demo_Train\r\nPrecision = \"float\"\r\nCNTKModelPath: ../Output/Models/simple.dnn\r\nCNTKCommandTrainInfo: Simple_Demo_Train : 300\r\nCNTKCommandTrainInfo: CNTKNoMoreCommands_Total : 300\r\n##############################################################################\r\n#                                                                            #\r\n# Action \"train\"                                                             #\r\n#                                                                            #\r\n##############################################################################\r\nCNTKCommandTrainBegin: Simple_Demo_Train\r\nSimpleNetworkBuilder Using GPU 0\r\nCreating virgin network.\r\nMicrosoft::MSR::CNTK::GPUMatrix<ElemType>::SetUniformRandomValue (GPU): creating curand object with seed 1, sizeof(ElemType)==4\r\nPost-processing network...\r\n7 roots:\r\n    CrossEntropyWithSoftmax = CrossEntropyWithSoftmax()\r\n    EvalErrorPrediction = ErrorPrediction()\r\n    InvStdOfFeatures = InvStdDev()\r\n    MeanOfFeatures = Mean()\r\n    PosteriorProb = Softmax()\r\n    Prior = Mean()\r\n    ScaledLogLikelihood = Minus()\r\nValidating network. 30 nodes to process in pass 1.\r\nValidating --> labels = InputValue() :  -> [4 x *]\r\nValidating --> W3 = LearnableParameter() :  -> [4 x 80]\r\nValidating --> W2 = LearnableParameter() :  -> [80 x 80]\r\nValidating --> W1 = LearnableParameter() :  -> [80 x 80]\r\nValidating --> W0 = LearnableParameter() :  -> [80 x 115]\r\nValidating --> features = InputValue() :  -> [115 x *]\r\nValidating --> MeanOfFeatures = Mean (features) : [115 x *] -> [115]\r\nValidating --> InvStdOfFeatures = InvStdDev (features) : [115 x *] -> [115]\r\nValidating --> MVNormalizedFeatures = PerDimMeanVarNormalization (features, MeanOfFeatures, InvStdOfFeatures) : [115 x *], [115], [115] -> [115 x *]\r\nValidating --> W0*features = Times (W0, MVNormalizedFeatures) : [80 x 115], [115 x *] -> [80 x *]\r\nValidating --> B0 = LearnableParameter() :  -> [80 x 1]\r\nValidating --> W0*features+B0 = Plus (W0*features, B0) : [80 x *], [80 x 1] -> [80 x 1 x *]\r\nValidating --> H1 = Tanh (W0*features+B0) : [80 x 1 x *] -> [80 x 1 x *]\r\nValidating --> W1*H1 = Times (W1, H1) : [80 x 80], [80 x 1 x *] -> [80 x 1 x *]\r\nValidating --> B1 = LearnableParameter() :  -> [80 x 1]\r\nValidating --> W1*H1+B1 = Plus (W1*H1, B1) : [80 x 1 x *], [80 x 1] -> [80 x 1 x *]\r\nValidating --> H2 = Tanh (W1*H1+B1) : [80 x 1 x *] -> [80 x 1 x *]\r\nValidating --> W2*H2 = Times (W2, H2) : [80 x 80], [80 x 1 x *] -> [80 x 1 x *]\r\nValidating --> B2 = LearnableParameter() :  -> [80 x 1]\r\nValidating --> W2*H2+B2 = Plus (W2*H2, B2) : [80 x 1 x *], [80 x 1] -> [80 x 1 x *]\r\nValidating --> H3 = Tanh (W2*H2+B2) : [80 x 1 x *] -> [80 x \r\n1 x *]\r\nValidating --> W3*H2 = Times (W3, H3) : [4 x 80], [80 x 1 x *] -> [4 x 1 x *]\r\nValidating --> B3 = L\r\nearnableParameter() :  -> [4 x 1]\r\nValidating --> HLast = Plus (W3*H2, B3) : [4 x 1 x *], [4 x 1] -> [4 x 1 x *]\r\nValidating --> CrossEntropyWithSoftmax = CrossEntropyWithSoftmax (labels, HLast) : [4 x *], [4 x 1 x *] -> [1]\r\nValidating --> EvalErrorPrediction = ErrorPrediction (labels, HLast) : [4 x *], [4 x 1 x *] -> [1]\r\nValidating --> PosteriorProb = Softmax (HLast) : [4 x 1 x *] -> [4 x 1 x *]\r\nValidating --> Prior = Mean (labels) : [4 x *] -> [4]\r\nValidating --> LogOfPrior = Log (Prior) : [4] -> [4]\r\nValidating --> ScaledLogLikelihood = Minus (HLast, LogOfPrior) : [4 x 1 x *], [4] -> [4 x 1 x *]\r\nValidating network. 20 nodes to process in pass 2.\r\nValidating network, final pass.\r\n14 out of 30 nodes do not share the minibatch layout with the input data.\r\nPost-processing network complete.\r\nCreated model with 30 nodes on GPU 0.\r\nTraining criterion node(s):\r\n    CrossEntropyWithSoftmax = CrossEntropyWithSoftmax\r\nEvaluation criterion node(s):\r\n    EvalErrorPrediction = ErrorPrediction\r\nAllocating matrices for forward and/or backward propagation.\r\nMemory Sharing Structure:\r\n0000000000000000: {[EvalErrorPrediction Gradient[1]] [InvStdOfFeatures Gradient[115]] [LogOfPrior Gradient[4]] [MVNormalizedFeatures Gradient[115 x *]] [MeanOfFeatures \r\nGradient[115]] [PosteriorProb Gradient[4 x 1 x *]] [PosteriorProb Value[4 x 1 x *]] [Prior Gradient[4]] [ScaledLogLikelihood Gradient[4 x 1 x *]] [features Gradient[115 \r\nx *]] [labels Gradient[4 x *]] }\r\n000000000621A2C0: {[features Value[115 x *]] }\r\n00000000236B9890: {[MeanOfFeatures Value[115]] }\r\n00000000236B99D0: {[InvStdOfFeatures Value[115]] }\r\n00000000236B9D90: {[W0 Value[80 x 115]] }\r\n00000000236BA5B0: {[B0 Value[80 x 1]] }\r\n00000000268876B0: {[W1 Value[80 x 80]] }\r\n00000000268877F0: {[B1 Value[80 x 1]] }\r\n0000000026887B10: {[W2 Value[80 x 80]] }\r\n0000000026887C50: {[B2 Value[80 x 1]] }\r\n0000000026887F70: {[W3 Value[4 x 80]] }\r\n00000000268880B0: {[B3 Value[4 x 1]] }\r\n0000000026888330: {[labels Value[4 x *]] }\r\n00000000268885B0: {[Prior Value[4]] }\r\n0000000026888D30: {[EvalErrorPrediction Value[1]] }\r\n0000000026888DD0: {[ScaledLogLikelihood Value[4 x 1 x *]] }\r\n0000000026888E70: {[CrossEntropyWithSoftmax Value[1]] }\r\n0000000026888F10: {[LogOfPrior Value[4]] }\r\n0000000026888FB0: {[MVNormalizedFeatures Value[115 x *]] }\r\n0000000026889050: {[W0*features Value[80 x *]] }\r\n00000000268890F0: {[W0 Gradient[80 x 115]] [W0*features+B0 Value[80 x 1 x *]] }\r\n0000000026889190: {[H1 Value[80 x 1 x *]] [W0*features Gradient[80 x *]] }\r\n0000000026889230: {[W0*features+B0 Gradient[80 x 1 x *]] [W1*H1 Value[80 x 1 x *]] }\r\n00000000268892D0: {[W1 Gradient[80 x 80]] [W1*H1+B1 Value[80 x 1 x *]] }\r\n0000000026889370: {[H2 Value[80 x 1 x *]] [W1*H1 Gradient[80 x 1 x *]] }\r\n0000000026D4BF50: {[B0 Gradient[80 x 1]] [H1 Gradient[80 x 1 x *]] [W1*H1+B1 Gradient[80 x 1 x *]] [W2*H2 Value[80 x 1 x *]] }\r\n0000000026D4BFF0: {[W2 Gradient[80 x 80]] [W2*H2+B2 Value[80 x 1 x *]] }\r\n0000000026D4C090: {[H3 Value[80 x 1 x *]] [W2*H2 Gradient[80 x 1 x *]] }\r\n0000000026D4C130: {[B1 Gradient[80 x 1]] [H2 Gradient[80 x 1 x *]] [W2*H2+B2 Gradient[80 x 1 x *]] [W3*H2 Value[4 x 1 x *]] }\r\n0000000026D4C1D0: {[HLast Value[4 x 1 x *]] [W3 Gradient[4 x 80]] }\r\n0000000026D4C590: {[CrossEntropyWithSoftmax Gradient[1]] }\r\n0000000026D4C630: {[B2 Gradient[80 x 1]] [H3 Gradient[80 x 1 x *]] [HLast Gradient[4 x 1 x *]] }\r\n0000000026D4C6D0: {[W3*H2 Gradient[4 x 1 x *]] }\r\n0000000026D4C770: {[B3 Gradient[4 x 1]] }\r\nPrecomputing --> 3 PreCompute nodes found.\r\n    MeanOfFeatures = Mean()\r\n    InvStdOfFeatures = InvStdDev()\r\n    Prior = Mean()\r\nstarting epoch 1 at record count 0, and file position 0\r\nalready there from last epoch\r\n[CALL STACK]\r\n    > \r\nstd::locale::global\r\n    - callnewh\r\n    - operator new\r\n    - \r\nstd::vector<unsigned int,std::allocator<bool> >::_Reallocate\r\n    - \r\nstd::vector<float,std::allocator<float> >::push_back\r\n    - \r\nUCIParser<float,std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::DoneWithValue\r\n    - \r\nUCIParser<float,std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::Parse\r\n    - \r\nMicrosoft::MSR::CNTK::UCIFastReader<float>::EnsureDataAvailable\r\n    - \r\nMicrosoft::MSR::CNTK::UCIFastReader<float>::GetMinibatchImpl\r\n    - \r\nMicrosoft::MSR::CNTK::UCIFastReader<float>::TryGetMinibatch\r\n    - \r\nMicrosoft::MSR::CNTK::DataReaderBase::GetMinibatch\r\n    - \r\nMicrosoft::MSR::CNTK::DataReader::GetMinibatch\r\n    - \r\nMicrosoft::MSR::CNTK::DataReaderHelpers::GetMinibatchIntoNetwork<float>\r\n    - \r\nMicrosoft::MSR::CNTK::SGD<float>::PreCompute\r\n    - \r\nMicrosoft::MSR::CNTK::SGD<float>::TrainOrAdaptModel\r\n    - \r\nMicrosoft::MSR::CNTK::SGD<float>::Train\r\n    - \r\nDoTrain<Microsoft::MSR::CNTK::ConfigParameters,float>\r\n    - \r\nDoCommands<float>\r\nEXCEPTION occurred: bad allocation\r\n=====================================================================\r\n\r\nWhen i run under the same configurations with a smaller file 2/3 of its size it works well.\r\n\r\nTwo questions here:\r\n\r\nQ1. Is there a file size size limit versus physical memory available that we need to be aware of?\r\nQ2. If not, is there any configuration setting i am missing in order to process this file?\r\n\r\nMany thanks,\r\nThanasis"},"comment":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/comments/240421667","html_url":"https://github.com/Microsoft/CNTK/issues/625#issuecomment-240421667","issue_url":"https://api.github.com/repos/Microsoft/CNTK/issues/625","id":240421667,"user":{"login":"eldakms","id":12430061,"avatar_url":"https://avatars.githubusercontent.com/u/12430061?v=3","gravatar_id":"","url":"https://api.github.com/users/eldakms","html_url":"https://github.com/eldakms","followers_url":"https://api.github.com/users/eldakms/followers","following_url":"https://api.github.com/users/eldakms/following{/other_user}","gists_url":"https://api.github.com/users/eldakms/gists{/gist_id}","starred_url":"https://api.github.com/users/eldakms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eldakms/subscriptions","organizations_url":"https://api.github.com/users/eldakms/orgs","repos_url":"https://api.github.com/users/eldakms/repos","events_url":"https://api.github.com/users/eldakms/events{/privacy}","received_events_url":"https://api.github.com/users/eldakms/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T14:06:09Z","updated_at":"2016-08-17T14:06:09Z","body":"UCIFastReader fetches the complete file in memory. With this reader it is not possible to process files that exceed available memory. This reader is deprecated, we encourage people to use CNTKTextFormatReader instead. You can find more information about it here https://github.com/Microsoft/CNTK/wiki/CNTKTextFormat-Reader\r\n\r\nThank you!"}}
{"actor":{"display_login":"eldakms"}}