{"action":"opened","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5755","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5755/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5755/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5755/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/5755","id":337054305,"node_id":"MDU6SXNzdWUzMzcwNTQzMDU=","number":5755,"title":"ParallelWrapper with 2 GPUs has a strange behaviour","user":{"login":"ivan4github","id":7681513,"node_id":"MDQ6VXNlcjc2ODE1MTM=","avatar_url":"https://avatars3.githubusercontent.com/u/7681513?v=4","gravatar_id":"","url":"https://api.github.com/users/ivan4github","html_url":"https://github.com/ivan4github","followers_url":"https://api.github.com/users/ivan4github/followers","following_url":"https://api.github.com/users/ivan4github/following{/other_user}","gists_url":"https://api.github.com/users/ivan4github/gists{/gist_id}","starred_url":"https://api.github.com/users/ivan4github/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivan4github/subscriptions","organizations_url":"https://api.github.com/users/ivan4github/orgs","repos_url":"https://api.github.com/users/ivan4github/repos","events_url":"https://api.github.com/users/ivan4github/events{/privacy}","received_events_url":"https://api.github.com/users/ivan4github/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-29T16:02:56Z","updated_at":"2018-06-29T16:02:56Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nFollowing your advices explained in this issue ...\r\n\r\nhttps://github.com/deeplearning4j/deeplearning4j/issues/5644\r\n\r\nWe managed to setup a CUDnn **on a single GTX 1080** in a Linux box which is now processing **~900 examples/sec** as seen in http://ourhost:2020/train/overview on dl4j-1.0.0-beta. That's great !\r\n\r\nFully enthusiastic with this result, we felt wings growing on our back and bought **a second GTX 1080** aside the first one with a NVidia SLI bridge plugged between both.\r\n\r\nWe modified our code to add a **ParallelWrapper**......\r\n\r\n... and nothing is behaviouring as expected:\r\n\r\n\r\n\r\n\r\nwith **ParallelWrapper.workspaceMode(WorkspaceMode.ENABLED)** \r\n\r\nEverything seems to be zeroed somewhere : \r\n\r\nneval:9\r\nngpu:2\r\nnavg:3\r\nishalf:false\r\n18:00:31,493 INFO  ~ Loaded [JCublasBackend] backend\r\n18:00:35,094 INFO  ~ Number of threads used for NativeOps: 32\r\n18:00:36,028 INFO  ~ Number of threads used for BLAS: 0\r\n18:00:36,050 INFO  ~ Backend used: [CUDA]; OS: [Linux]\r\n18:00:36,050 INFO  ~ Cores: [16]; Memory: [17.8GB];\r\n18:00:36,050 INFO  ~ Blas vendor: [CUBLAS]\r\n18:00:36,051 INFO  ~ Device Name: [GeForce GTX 1080]; CC: [6.1]; Total/free memory: [8511881216]\r\n18:00:36,051 INFO  ~ Device Name: [GeForce GTX 1080]; CC: [6.1]; Total/free memory: [8513978368]\r\nTRACE:TrainNews:learn():RRestoration du word2vec:/usr/bases/bypath/learn/models/word2vec_20180404_fr.model\r\nTRACE:TrainNews:learn():Reseau neuronal 1 couche\r\n18:01:13,990 INFO  ~ Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]\r\nTRACE:TrainNews:learn():Reseau construit\r\n18:01:15,699 DEBUG ~ Starting application default Akka system: application\r\n18:01:16,701 INFO  ~ Application started (Prod)\r\n18:01:16,802 INFO  ~ Listening for HTTP on /0:0:0:0:0:0:0:0:2020\r\n18:01:16,803 INFO  ~ DL4J UI Server started at http://localhost:2020\r\nTRACE:TrainNews:learn():Statistiques dans /usr/software/download/dl4jnewsstats.viz\r\n18:01:17,574 INFO  ~ StatsStorage instance attached to UI: FileStatsStorage(/usr/software/download/dl4jnewsstats.viz)\r\nTRACE:TrainNews:learn():Multi GPU:2\r\n18:01:17,635 INFO  ~ Creating new AveragingTraining instance\r\nTRACE:TrainNews:learn():Monitoring pret\r\nStarting training\r\nEpoch 0 begins....\r\n18:01:17,635 INFO  ~ Using workspaceMode ENABLED for training\r\n18:01:17,636 INFO  ~ Creating asynchronous prefetcher...\r\n18:01:17,661 INFO  ~ Starting ParallelWrapper training round...\r\n18:01:19,924 INFO  ~ Score at iteration 0 is 0.0\r\n18:01:19,924 INFO  ~ Device: [1]; ETL: 192 ms; iteration 0; iteration time: 0 ms; samples/sec: Infinity; batches/sec: Infinity; \r\n18:01:20,008 INFO  ~ Score at iteration 0 is 2.198418435731776\r\n18:01:20,008 INFO  ~ Device: [0]; ETL: 127 ms; iteration 0; iteration time: 0 ms; samples/sec: Infinity; batches/sec: Infinity; \r\n18:01:23,579 INFO  ~ Averaged score: 1.1000835462843037\r\n18:01:29,107 INFO  ~ Averaging time: 8342 ms\r\n18:01:32,402 INFO  ~ Averaged score: NaN\r\n18:01:37,978 INFO  ~ Averaging time: 8389 ms\r\n18:01:41,127 INFO  ~ Averaged score: NaN\r\n18:01:46,654 INFO  ~ Averaging time: 8340 ms\r\n18:01:50,609 INFO  ~ Averaged score: NaN\r\n18:01:56,186 INFO  ~ Averaging time: 8390 ms\r\n18:01:59,749 INFO  ~ Averaged score: NaN\r\n18:02:05,325 INFO  ~ Averaging time: 8389 ms\r\n18:02:08,656 INFO  ~ Averaged score: NaN\r\n\r\n\r\n\r\n\r\n\r\nwith **ParallelWrapper.workspaceMode(WorkspaceMode.NONE)** \r\n\r\nIt hangs shortly after start:\r\n\r\nEpoch 0 begins....\r\n17:27:05,851 INFO  ~ Using workspaceMode NONE for training\r\n17:27:05,852 INFO  ~ Creating asynchronous prefetcher...\r\n17:27:05,855 INFO  ~ Starting ParallelWrapper training round...\r\n\r\n....then nothing.\r\n\r\n\r\n\r\n\r\nIn both cases , the GPU are flat:\r\n```\r\n\r\nEvery 1,0s: nvidia-smi                                                                                                                                  qwam-ai: Fri Jun 29 17:44:09 2018\r\n\r\nFri Jun 29 17:44:09 2018\r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 396.24.02              Driver Version: 396.24.02                 |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 1080    Off  | 00000000:0A:00.0 Off |                  N/A |\r\n| 29%   63C    P2    44W / 180W |   1167MiB /  8117MiB |      2%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  GeForce GTX 1080    Off  | 00000000:0B:00.0 Off |                  N/A |\r\n|  0%   50C    P8     6W / 180W |    907MiB /  8119MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n\r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n|    0      2000      C   /usr/software/jdk/bin/java                  1157MiB |\r\n|    1      2000      C   /usr/software/jdk/bin/java                   897MiB |\r\n+-----------------------------------------------------------------------------+\r\n\r\n\r\n```\r\n\r\nHere is our code with the highlighted modification we made to switch to ParallelWrapper.\r\n\r\n\r\n```\r\n\r\n    public void learn(String datapath, String word2vecpath, String statspath, String modelpath, String layer,\r\n                      int neval, int ngpu, int navg,\r\n                      boolean ishalf) throws Exception {\r\n\r\n        int batchSize = 50;     //Number of examples in each minibatch\r\n        int nEpochs = 50;        //Number of epochs (full passes of training data) to train on\r\n        int truncateReviewsToLength = 300;  //Truncate reviews with length (# words) greater than this\r\n        int lstmLayerSize = 200;\r\n\r\n        // Half datatype 200% performance pour 50% de RAM GPU en moins\r\n        if (ishalf) {\r\n            System.out.println(\"TRACE:TrainNews:learn():Precision HALF\");\r\n            DataTypeUtil.setDTypeForContext(DataBuffer.Type.HALF);\r\n        }\r\n\r\n\r\n        // 18/06/2018 IMR Amelioration du parallelisme des GPU\r\n        CudaEnvironment.getInstance().getConfiguration()\r\n            .allowMultiGPU(true)\r\n            .setMaximumDeviceCacheableLength(1024 * 1024 * 1024L)\r\n            .setMaximumDeviceCache(6L * 1024 * 1024 * 1024L)\r\n            .setMaximumHostCacheableLength(1024 * 1024 * 1024L)\r\n            .setMaximumHostCache(6L * 1024 * 1024 * 1024L);\r\n\r\n\r\n        // this will limit frequency of gc calls\r\n        // tweak GC frequency to 10 s\r\n        Nd4j.getMemoryManager().setAutoGcWindow(10000);\r\n\r\n        System.out.println(\"TRACE:TrainNews:learn():RRestoration du word2vec:\" + word2vecpath);\r\n        wordVectors = WordVectorSerializer.readWord2VecModel(new File(word2vecpath));\r\n\r\n        TokenizerFactory tokenizerFactory = new DefaultTokenizerFactory();\r\n        tokenizerFactory.setTokenPreProcessor(new CommonPreprocessor());\r\n\r\n        NewsIterator iTrain = new NewsIterator.Builder()\r\n            .dataDirectory(datapath)\r\n            .wordVectors(wordVectors)\r\n            .batchSize(batchSize)\r\n            .truncateLength(truncateReviewsToLength)\r\n            .tokenizerFactory(tokenizerFactory)\r\n            .train(true)\r\n            .build();\r\n\r\n        NewsIterator iTest = new NewsIterator.Builder()\r\n            .dataDirectory(datapath)\r\n            .wordVectors(wordVectors)\r\n            .batchSize(batchSize)\r\n            .tokenizerFactory(tokenizerFactory)\r\n            .truncateLength(truncateReviewsToLength)\r\n            .train(false)\r\n            .build();\r\n\r\n        int inputNeurons = wordVectors.getWordVector(wordVectors.vocab().wordAtIndex(0)).length;\r\n        int outputs = iTrain.getLabels().size();\r\n\r\n        tokenizerFactory = new DefaultTokenizerFactory();\r\n        tokenizerFactory.setTokenPreProcessor(new CommonPreprocessor());\r\n        //Set up network configuration\r\n        MultiLayerConfiguration conf = null;\r\n\r\n            System.out.println(\"TRACE:TrainNews:learn():Reseau neuronal 1 couche\");\r\n            conf = new NeuralNetConfiguration.Builder()\r\n                .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)\r\n                // Removed in dl4j-1.0-beta\r\n                //.iterations(1)\r\n                // deprecated in dl4j-1.0-beta\r\n                //.updater(Updater.RMSPROP)\r\n                .updater(new RmsProp())\r\n                // Removed in dl4j-1.0-beta\r\n                //.regularization(true)\r\n                .l2(1e-5)\r\n                .weightInit(WeightInit.XAVIER)\r\n                .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue).gradientNormalizationThreshold(1.0)\r\n                // Removed in dl4j-1.0-beta\r\n                //.learningRate(0.0018)\r\n                .updater(new Adam(0.0018))\r\n                .list()\r\n                // LSTM uses CUDA, GravesLSTM don't\r\n                //.layer(0, new GravesLSTM.Builder().nIn(inputNeurons).nOut(200)\r\n                .layer(0, new LSTM.Builder().nIn(inputNeurons).nOut(200)\r\n                    // SOFTSIGN not supported by CUDA, TANH better\r\n                    .activation(Activation.TANH).build())\r\n                .layer(1, new RnnOutputLayer.Builder().activation(Activation.SOFTMAX)\r\n                    .lossFunction(LossFunctions.LossFunction.MCXENT).nIn(200).nOut(outputs).build())\r\n                .pretrain(false).backprop(true).build();\r\n\r\n\r\n        // ParallelWrapper will take care of load balancing between GPUs.\r\n        MultiLayerNetwork net0 = new MultiLayerNetwork(conf);\r\n        net0.init();\r\n\r\n\r\n        net0.setListeners(new ScoreIterationListener(1));\r\n\r\n        System.out.println(\"TRACE:TrainNews:learn():Reseau construit\");\r\n\r\n\r\n            // monitor UI\r\n            net0.setListeners(new ActivationsListener(net0, 20));\r\n            net0.setListeners(new GradientsAndParamsListener(net0, 100));\r\n\r\n            // Listener de visualisation UI\r\n            //Initialize the user interface backend\r\n            UIServer uiServer = UIServer.getInstance();\r\n            //Configure where the network information (gradients, score vs. time etc) is to be stored. Here: store in memory.\r\n\r\n            StatsStorage statsStorage;\r\n            if (statspath.equals(\"memory\")) {\r\n                System.out.println(\"TRACE:TrainNews:learn():Statistiques en memoire\");\r\n                statsStorage = new InMemoryStatsStorage();\r\n            } else {\r\n                System.out.println(\"TRACE:TrainNews:learn():Statistiques dans \" + statspath);\r\n                statsStorage = new FileStatsStorage(new File(statspath));\r\n            }\r\n            //Attach the StatsStorage instance to the UI: this allows the contents of the StatsStorage to be visualized\r\n            uiServer.attach(statsStorage);\r\n            //Then add the StatsListener to collect this information from the network, as it trains\r\n            // add a performanceListener and reduce iterations of listeners\r\n            int NBITER = 1000;\r\n            net0.setListeners(new StatsListener(statsStorage), new ScoreIterationListener(NBITER), new PerformanceListener(NBITER));\r\n\r\n        System.out.println(\"TRACE:TrainNews:learn():Multi GPU:\" + ngpu);\r\n        ParallelWrapper **net** = new ParallelWrapper.Builder(net0)\r\n            // DataSets prefetching options. Set this value with respect to number of actual devices\r\n            .prefetchBuffer(24)\r\n            // set number of workers equal to number of available devices. x1-x2 are good values to start with\r\n            .workers(2)\r\n            // rare averaging improves performance, but might reduce model accuracy\r\n            .averagingFrequency(3)\r\n            // if set to TRUE, on every averaging model score will be reported\r\n            .reportScoreAfterAveraging(true)\r\n            // 2 options here: NONE, ENABLED\r\n            .workspaceMode(WorkspaceMode.ENABLED)\r\n            // build wrapper\r\n            .build();\r\n\r\n        System.out.println(\"TRACE:TrainNews:learn():Monitoring pret\");\r\n\r\n        System.out.println(\"Starting training\");\r\n        int ieval = 0;\r\n        for (int i = 0; i < nEpochs; i++) {\r\n\r\n            System.out.println(\"Epoch \" + i + \" begins....\");\r\n            net.fit(iTrain);\r\n\r\n            System.out.println(\"Epoch \" + i + \" complete.\");\r\n\r\n            iTrain.reset();\r\n\r\n            if ((ieval == neval) || (i == nEpochs)) {\r\n                System.out.println(\"Starting evaluation test\");\r\n\r\n                Evaluation evaluation = net0.evaluate(iTest);\r\n\r\n                System.out.println(evaluation.stats());\r\n                ieval = 0;\r\n            } else {\r\n                System.out.println(\"No Evaluation for this epoch\" + ieval + \"/\" + neval);\r\n            }\r\n\r\n            System.out.println(\"Write model\");\r\n            // IMR 14/11/2017 modelpath parametrable\r\n            //ModelSerializer.writeModel(net0, datapath + \"NewsModel.net0\", true);\r\n            System.out.println(\"TRACE:TrainNews:learn():Sauvegarde du modele dans:\" + modelpath);\r\n            ModelSerializer.writeModel(net0, modelpath, true);\r\n            ieval++;\r\n        }\r\n\r\n        System.out.println(\"----- Example complete -----\");\r\n    }\r\n\r\n\r\n```\r\n\r\nWe are just like kids on Christmas day discovering that their toy is not working.\r\nDear Santa Claus, can you help us  ? ;-)\r\n \r\n"}}
{"actor":{"display_login":"ivan4github"}}