{"action":"created","issue":{"url":"https://api.github.com/repos/pmix/master/issues/204","repository_url":"https://api.github.com/repos/pmix/master","labels_url":"https://api.github.com/repos/pmix/master/issues/204/labels{/name}","comments_url":"https://api.github.com/repos/pmix/master/issues/204/comments","events_url":"https://api.github.com/repos/pmix/master/issues/204/events","html_url":"https://github.com/pmix/master/issues/204","id":185973917,"number":204,"title":"Compress large PMIx values","user":{"login":"hjelmn","id":1226817,"avatar_url":"https://avatars.githubusercontent.com/u/1226817?v=3","gravatar_id":"","url":"https://api.github.com/users/hjelmn","html_url":"https://github.com/hjelmn","followers_url":"https://api.github.com/users/hjelmn/followers","following_url":"https://api.github.com/users/hjelmn/following{/other_user}","gists_url":"https://api.github.com/users/hjelmn/gists{/gist_id}","starred_url":"https://api.github.com/users/hjelmn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hjelmn/subscriptions","organizations_url":"https://api.github.com/users/hjelmn/orgs","repos_url":"https://api.github.com/users/hjelmn/repos","events_url":"https://api.github.com/users/hjelmn/events{/privacy}","received_events_url":"https://api.github.com/users/hjelmn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-28T17:09:01Z","updated_at":"2016-10-28T17:11:15Z","closed_at":null,"body":"It might be beneficial to compress large PMIx strings before transferring or storing them. This could lead to a large decrease in the memory footprint."},"comment":{"url":"https://api.github.com/repos/pmix/master/issues/comments/256975909","html_url":"https://github.com/pmix/master/issues/204#issuecomment-256975909","issue_url":"https://api.github.com/repos/pmix/master/issues/204","id":256975909,"user":{"login":"hjelmn","id":1226817,"avatar_url":"https://avatars.githubusercontent.com/u/1226817?v=3","gravatar_id":"","url":"https://api.github.com/users/hjelmn","html_url":"https://github.com/hjelmn","followers_url":"https://api.github.com/users/hjelmn/followers","following_url":"https://api.github.com/users/hjelmn/following{/other_user}","gists_url":"https://api.github.com/users/hjelmn/gists{/gist_id}","starred_url":"https://api.github.com/users/hjelmn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hjelmn/subscriptions","organizations_url":"https://api.github.com/users/hjelmn/orgs","repos_url":"https://api.github.com/users/hjelmn/repos","events_url":"https://api.github.com/users/hjelmn/events{/privacy}","received_events_url":"https://api.github.com/users/hjelmn/received_events","type":"User","site_admin":false},"created_at":"2016-10-28T17:11:15Z","updated_at":"2016-10-28T17:11:15Z","body":"Here is a quick and dirty implementation that works:\r\n\r\n```diff --git a/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/pack.c b/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/pack.c\r\nindex 2543eec..0c009e9 100644\r\n--- a/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/pack.c\r\n+++ b/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/pack.c\r\n@@ -35,6 +35,7 @@\r\n #include \"src/util/error.h\"\r\n #include \"src/util/output.h\"\r\n #include \"src/buffer_ops/internal.h\"\r\n+#include <zlib.h>\r\n \r\n pmix_status_t pmix_bfrop_pack(pmix_buffer_t *buffer,\r\n                               const void *src, int32_t num_vals,\r\n@@ -300,26 +301,53 @@ pmix_status_t pmix_bfrop_pack_string(pmix_buffer_t *buffer, const void *src,\r\n     pmix_status_t ret = PMIX_SUCCESS;\r\n     int32_t i, len;\r\n     char **ssrc = (char**) src;\r\n-\r\n     for (i = 0; i < num_vals; ++i) {\r\n         if (NULL == ssrc[i]) {  /* got zero-length string/NULL pointer - store NULL */\r\n-        len = 0;\r\n-        if (PMIX_SUCCESS != (ret = pmix_bfrop_pack_int32(buffer, &len, 1, PMIX_INT32))) {\r\n+         len = 0;\r\n+         if (PMIX_SUCCESS != (ret = pmix_bfrop_pack_int32(buffer, &len, 1, PMIX_INT32))) {\r\n             return ret;\r\n-        }\r\n-    } else {\r\n-        len = (int32_t)strlen(ssrc[i]) + 1;\r\n-        if (PMIX_SUCCESS != (ret = pmix_bfrop_pack_int32(buffer, &len, 1, PMIX_INT32))) {\r\n+         }\r\n+       } else {\r\n+         z_stream strm;\r\n+         char *tmp = NULL;\r\n+         int32_t deflated_len = 0;\r\n+\r\n+         len = (int32_t)strlen(ssrc[i]) + 1;\r\n+         if (PMIX_SUCCESS != (ret = pmix_bfrop_pack_int32(buffer, &len, 1, PMIX_INT32))) {\r\n             return ret;\r\n-        }\r\n-        if (PMIX_SUCCESS != (ret =\r\n-            pmix_bfrop_pack_byte(buffer, ssrc[i], len, PMIX_BYTE))) {\r\n+         }\r\n+\r\n+         if (len > 512) {\r\n+           tmp = malloc (len);\r\n+\r\n+           memset (&strm, 0, sizeof (strm));\r\n+\r\n+           strm.next_in = ssrc[i];\r\n+           strm.avail_in = len;\r\n+\r\n+           strm.avail_out = len;\r\n+           strm.next_out = tmp;\r\n+\r\n+           deflateInit (&strm, 9);\r\n+           deflate (&strm, Z_FINISH);\r\n+           deflateEnd (&strm);\r\n+\r\n+           deflated_len = len - strm.avail_out;\r\n+           if (PMIX_SUCCESS != (ret = pmix_bfrop_pack_int32(buffer, &deflated_len, 1, PMIX_INT32))) {\r\n+             return ret;\r\n+           }\r\n+         }\r\n+\r\n+         if (PMIX_SUCCESS != (ret =\r\n+                              pmix_bfrop_pack_byte(buffer, tmp ? tmp : ssrc[i], tmp ? deflated_len : len, PMIX_BYTE))) {\r\n             return ret;\r\n+         }\r\n+\r\n+         free (tmp);\r\n+       }\r\n     }\r\n-}\r\n-}\r\n \r\n-return PMIX_SUCCESS;\r\n+    return PMIX_SUCCESS;\r\n }\r\n \r\n /* FLOAT */\r\ndiff --git a/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/unpack.c b/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/unpack.c\r\nindex fb6fcac..fdad563 100644\r\n--- a/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/unpack.c\r\n+++ b/opal/mca/pmix/pmix3x/pmix/src/buffer_ops/unpack.c\r\n@@ -32,6 +32,7 @@\r\n #include \"src/util/output.h\"\r\n #include \"src/buffer_ops/types.h\"\r\n #include \"src/buffer_ops/internal.h\"\r\n+#include <zlib.h>\r\n \r\n pmix_status_t pmix_bfrop_unpack(pmix_buffer_t *buffer,\r\n                                 void *dst, int32_t *num_vals,\r\n@@ -370,15 +371,42 @@ pmix_status_t pmix_bfrop_unpack_string(pmix_buffer_t *buffer, void *dest,\r\n         }\r\n         if (0 ==  len) {   /* zero-length string - unpack the NULL */\r\n             sdest[i] = NULL;\r\n-        } else {\r\n-            sdest[i] = (char*)malloc(len);\r\n-            if (NULL == sdest[i]) {\r\n-                return PMIX_ERR_OUT_OF_RESOURCE;\r\n-            }\r\n-            if (PMIX_SUCCESS != (ret = pmix_bfrop_unpack_byte(buffer, sdest[i], &len, PMIX_BYTE))) {\r\n-                return ret;\r\n-            }\r\n-        }\r\n+           continue;\r\n+       }\r\n+\r\n```"}}
{"actor":{"display_login":"hjelmn"}}