{"action":"created","issue":{"url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/3","repository_url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS","labels_url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/3/comments","events_url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/3/events","html_url":"https://github.com/kimoonkim/kubernetes-HDFS/pull/3","id":326654646,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkwNjg3NzMw","number":3,"title":"Introduces a uber chart","user":{"login":"kimoonkim","id":5914904,"node_id":"MDQ6VXNlcjU5MTQ5MDQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5914904?v=4","gravatar_id":"","url":"https://api.github.com/users/kimoonkim","html_url":"https://github.com/kimoonkim","followers_url":"https://api.github.com/users/kimoonkim/followers","following_url":"https://api.github.com/users/kimoonkim/following{/other_user}","gists_url":"https://api.github.com/users/kimoonkim/gists{/gist_id}","starred_url":"https://api.github.com/users/kimoonkim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kimoonkim/subscriptions","organizations_url":"https://api.github.com/users/kimoonkim/orgs","repos_url":"https://api.github.com/users/kimoonkim/repos","events_url":"https://api.github.com/users/kimoonkim/events{/privacy}","received_events_url":"https://api.github.com/users/kimoonkim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-05-25T20:30:29Z","updated_at":"2018-06-06T20:48:30Z","closed_at":null,"author_association":"OWNER","pull_request":{"url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/pulls/3","html_url":"https://github.com/kimoonkim/kubernetes-HDFS/pull/3","diff_url":"https://github.com/kimoonkim/kubernetes-HDFS/pull/3.diff","patch_url":"https://github.com/kimoonkim/kubernetes-HDFS/pull/3.patch"},"body":"Introduces an uber-chart, `charts/hdfs-k8s`, that specifies other subcharts as dependencies. Zookeeper chart from the incubator repo is also specified as a remote dependency. See `requirements.yaml` file under the uber-chart dir for more details.\r\n\r\nThis allows a one-liner command like `$ helm install -n my-hdfs charts/hdbs-k8s` to launch all charts.\r\n\r\nIt turns out these subcharts run together at the same time, which exposed a few existing bugs. Datanode pods hang when namenode pods start a bit later, as JVM DNS caches remember, for too long, the namenode DNS entries do not exist. We fixed the bug by adding a liveliness probe that fails if datanode jmx says it failed to register with namenodes. Datanodes will fail fast, restart and eventually succeed.\r\n\r\nSimilarly, a namenode bug was fixed. Namenodes formatting would fail if zookeeper or journalnodes are not up and running. Previously, namenode pods fall into a weird state where the medata is created but zookeeper dir is not. We fixed the bug by creating marker files inside PVs and resume from where it failed before. Again, namenode pods would fail fast and eventually succeed.\r\n\r\nThe uber-chart allow us to consolidate a lot of default values and some template code into the `values.yaml` and `templates/_helpers.tpl` under `charts/hdfs-k8s`. Some of the clean-up is done in this change. Others are left for future work.\r\n\r\nSee `README.md` under the uber-chart dir for more details."},"comment":{"url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/comments/395208654","html_url":"https://github.com/kimoonkim/kubernetes-HDFS/pull/3#issuecomment-395208654","issue_url":"https://api.github.com/repos/kimoonkim/kubernetes-HDFS/issues/3","id":395208654,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTIwODY1NA==","user":{"login":"scotthew1","id":5482907,"node_id":"MDQ6VXNlcjU0ODI5MDc=","avatar_url":"https://avatars1.githubusercontent.com/u/5482907?v=4","gravatar_id":"","url":"https://api.github.com/users/scotthew1","html_url":"https://github.com/scotthew1","followers_url":"https://api.github.com/users/scotthew1/followers","following_url":"https://api.github.com/users/scotthew1/following{/other_user}","gists_url":"https://api.github.com/users/scotthew1/gists{/gist_id}","starred_url":"https://api.github.com/users/scotthew1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scotthew1/subscriptions","organizations_url":"https://api.github.com/users/scotthew1/orgs","repos_url":"https://api.github.com/users/scotthew1/repos","events_url":"https://api.github.com/users/scotthew1/events{/privacy}","received_events_url":"https://api.github.com/users/scotthew1/received_events","type":"User","site_admin":false},"created_at":"2018-06-06T20:48:29Z","updated_at":"2018-06-06T20:48:29Z","author_association":"NONE","body":"sorry it looks like one of the journalnodes still wasn't up at the time. once all of those came up, it's back to the original problem\r\n\r\n```\r\nSTARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41; compiled by 'jenkins' on 2016-01-26T00:08Z\r\nSTARTUP_MSG:   java = 1.8.0_131\r\n************************************************************/\r\n18/06/06 20:43:46 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\r\n18/06/06 20:43:46 INFO namenode.NameNode: createNameNode [-format, -nonInteractive, hdfs-k8s]\r\nFormatting using clusterid: CID-aff01b13-8093-4506-a56e-de318d937132\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: No KeyProvider found.\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: fsLock is fair:true\r\n18/06/06 20:43:46 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000\r\n18/06/06 20:43:46 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=false\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: The block deletion will start around 2018 Jun 06 20:43:46\r\n18/06/06 20:43:46 INFO util.GSet: Computing capacity for map BlocksMap\r\n18/06/06 20:43:46 INFO util.GSet: VM type       = 64-bit\r\n18/06/06 20:43:46 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB\r\n18/06/06 20:43:46 INFO util.GSet: capacity      = 2^21 = 2097152 entries\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: defaultReplication         = 3\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: maxReplication             = 512\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: minReplication             = 1\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: encryptDataTransfer        = false\r\n18/06/06 20:43:46 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: supergroup          = supergroup\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: isPermissionEnabled = true\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: Determined nameservice ID: hdfs-k8s\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: HA Enabled: true\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: Append Enabled: true\r\n18/06/06 20:43:46 INFO util.GSet: Computing capacity for map INodeMap\r\n18/06/06 20:43:46 INFO util.GSet: VM type       = 64-bit\r\n18/06/06 20:43:46 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB\r\n18/06/06 20:43:46 INFO util.GSet: capacity      = 2^20 = 1048576 entries\r\n18/06/06 20:43:46 INFO namenode.FSDirectory: ACLs enabled? false\r\n18/06/06 20:43:46 INFO namenode.FSDirectory: XAttrs enabled? true\r\n18/06/06 20:43:46 INFO namenode.FSDirectory: Maximum size of an xattr: 16384\r\n18/06/06 20:43:46 INFO namenode.NameNode: Caching file names occuring more than 10 times\r\n18/06/06 20:43:46 INFO util.GSet: Computing capacity for map cachedBlocks\r\n18/06/06 20:43:46 INFO util.GSet: VM type       = 64-bit\r\n18/06/06 20:43:46 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB\r\n18/06/06 20:43:46 INFO util.GSet: capacity      = 2^18 = 262144 entries\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000\r\n18/06/06 20:43:46 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10\r\n18/06/06 20:43:46 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10\r\n18/06/06 20:43:46 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: Retry cache on namenode is enabled\r\n18/06/06 20:43:46 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis\r\n18/06/06 20:43:46 INFO util.GSet: Computing capacity for map NameNodeRetryCache\r\n18/06/06 20:43:46 INFO util.GSet: VM type       = 64-bit\r\n18/06/06 20:43:46 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB\r\n18/06/06 20:43:46 INFO util.GSet: capacity      = 2^15 = 32768 entries\r\nRunning in non-interactive mode, and data appears to exist in QJM to [10.47.128.8:8485, 10.36.128.17:8485, 10.41.128.59:8485]. Not formatting.\r\n18/06/06 20:43:47 INFO util.ExitUtil: Exiting with status 1\r\n18/06/06 20:43:47 INFO namenode.NameNode: SHUTDOWN_MSG:\r\n/************************************************************\r\nSHUTDOWN_MSG: Shutting down NameNode at beads-24/10.128.8.137\r\n************************************************************/\r\n+ rm -rf /hadoop/dfs/name/current\r\n+ exit 1\r\n```"}}
{"actor":{"display_login":"scotthew1"}}