{"action":"created","issue":{"url":"https://api.github.com/repos/clustervision/trinityX/issues/153","repository_url":"https://api.github.com/repos/clustervision/trinityX","labels_url":"https://api.github.com/repos/clustervision/trinityX/issues/153/labels{/name}","comments_url":"https://api.github.com/repos/clustervision/trinityX/issues/153/comments","events_url":"https://api.github.com/repos/clustervision/trinityX/issues/153/events","html_url":"https://github.com/clustervision/trinityX/pull/153","id":194624908,"number":153,"title":"HA","user":{"login":"jflf-CV","id":17725623,"avatar_url":"https://avatars.githubusercontent.com/u/17725623?v=3","gravatar_id":"","url":"https://api.github.com/users/jflf-CV","html_url":"https://github.com/jflf-CV","followers_url":"https://api.github.com/users/jflf-CV/followers","following_url":"https://api.github.com/users/jflf-CV/following{/other_user}","gists_url":"https://api.github.com/users/jflf-CV/gists{/gist_id}","starred_url":"https://api.github.com/users/jflf-CV/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jflf-CV/subscriptions","organizations_url":"https://api.github.com/users/jflf-CV/orgs","repos_url":"https://api.github.com/users/jflf-CV/repos","events_url":"https://api.github.com/users/jflf-CV/events{/privacy}","received_events_url":"https://api.github.com/users/jflf-CV/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-09T15:34:27Z","updated_at":"2016-12-13T17:55:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/clustervision/trinityX/pulls/153","html_url":"https://github.com/clustervision/trinityX/pull/153","diff_url":"https://github.com/clustervision/trinityX/pull/153.diff","patch_url":"https://github.com/clustervision/trinityX/pull/153.patch"},"body":"This pull request contains the core HA code and configuration for TrinityX.\r\n\r\n\r\n**Mandatory read**:\r\n\r\nThe `ps_controller_storage.rst` document, which describes all the storage possibilities. There is also a fair amount of documentation in the new reference configuration file: `controller-ha.cfg`.\r\n\r\nThe following setups have been tested:\r\n\r\n- non-HA, `none`\r\n- non-HA, `export`\r\n- non-HA, `dev`\r\n- HA, `export`\r\n- HA, `dev`\r\n- HA, `drbd`\r\n\r\nIn all HA tests, both the primary and secondary were tested.\r\n\r\n\r\n**Notes**:\r\n\r\n- The configuration file for DRBD (`/etc/drbd.d/global_common.conf`) is now documented. It should be tuned by the engineer on site for maximum performance on the specific hardware. The values included are calculated for machines connected by a GigE link, and where that link is the slowest point in the DRBD data path.\r\n\r\n- The default IPs are now the same a for a Bright installation, as required by Daniele.\r\n\r\n- When installing a DRBD setup, Pacemaker will try to start the slave resource on the secondary controller as soon as it's up, even before DRBD is actually installed and configured there. When running `pcs status` at the end of the `pacemaker` PS on the secondary controller, the output will look like this:\r\n```\r\n Resource Group: Trinity\r\n     trinity-ip\t(ocf::heartbeat:IPaddr2):\tStarted hactrl1\r\n     wait-for-device\t(ocf::heartbeat:Delay):\tStarted hactrl1\r\n     trinity-fs\t(ocf::heartbeat:Filesystem):\tStarted hactrl1\r\n     trinity-nfs-server\t(ocf::heartbeat:nfsserver):\tStarted hactrl1\r\n Resource Group: Trinity-secondary\r\n     trinity-secondary\t(ocf::heartbeat:Dummy):\tStarted hactrl2\r\n Master/Slave Set: Trinity-drbd [DRBD]\r\n     DRBD\t(ocf::linbit:drbd):\tFAILED hactrl2 (unmanaged)\r\n     Masters: [ hactrl1 ]\r\n\r\nFailed Actions:\r\n* DRBD_stop_0 on hactrl2 'not installed' (5): call=32, status=complete, exitreason='none',\r\n    last-rc-change='Fri Dec  9 16:00:58 2016', queued=0ms, exec=47ms\r\n```\r\nThis is completely normal, as indeed DRBD isn't installed. The `shared-storage` PS will take care of that, and restart the slave resource.\r\n\r\n- Some of the post scripts in the new configuration files are commented out. This is because they haven't been updated yet for HA.\r\n\r\n\r\n**Known issues**:\r\n\r\n- The DRBD kernel module package (`kmod-drbd84`) follows the RHEL kernel, which, at the time of writing, is more recent than the CentOS kernel. Yum fails when trying to install the latest DRBD release on CentOS 7.2, and an older release must be installed by hand. This is best done when the configuration switches to stepping mode, as all the additional repos have already been installed.\r\n\r\nWorkaround (on both controllers):\r\n```\r\nyum -y install kmod-drbd84-8.4.8-1_1.el7.elrepo && \\\r\necho kmod-drbd84* >> /etc/yum/pluginconf.d/versionlock.list\r\n```\r\n\r\n**NOTE: as of 12/12/2016 this is no longer an issue, as kernel 3.10.0-514.2.2.el7 is now in the CentOS 7 repos and works with the latest DRBD release.**\r\n\r\n- In some instances, the updated `nfsserver` RA will still start with 8 threads instead of 256. I believe that it is related to a race condition between the installation of the new configuration file, and the NFS server starting. Restarting the NFS server brings back the 256 threads. I believe that issue to apply only at installation time, i.e. I haven't seen it any time later. I'm keeping it in the know issues list until it is widely confirmed that it doesn't happen later in real life.\r\n\r\nWorkaround (on the primary controller):\r\n```\r\npcs resource disable trinity-nfs-server\r\nsleep 5s\r\npcs resource enable trinity-nfs-server\r\n```\r\n\r\n- Pacemaker's timing in starting resources is at best capricious, shall we say. There is no straightforward way of waiting for a resource to come up and be sure that it's really up (looking at you DRBD). I have included delays here and there at the most problematic points, including a Delay resource to make sure that the kernel is up to speed with the new device before attempting to mount the FS. It works in my VMs, but may not work on real hardware. This is the reason why the whole critical section is in stepping mode: **the engineer must make sure that all resources for each step came up before moving to the next step.** Each step assumes that the previous one has completed correctly, and that its resources are available. If it's not the case the faulty resources need to be fixed, or deleted entirely and the postscript reran.\r\n\r\nWorkaround:\r\n`pcs status` ad nauseam.\r\n"},"comment":{"url":"https://api.github.com/repos/clustervision/trinityX/issues/comments/266811785","html_url":"https://github.com/clustervision/trinityX/pull/153#issuecomment-266811785","issue_url":"https://api.github.com/repos/clustervision/trinityX/issues/153","id":266811785,"user":{"login":"jflf-CV","id":17725623,"avatar_url":"https://avatars.githubusercontent.com/u/17725623?v=3","gravatar_id":"","url":"https://api.github.com/users/jflf-CV","html_url":"https://github.com/jflf-CV","followers_url":"https://api.github.com/users/jflf-CV/followers","following_url":"https://api.github.com/users/jflf-CV/following{/other_user}","gists_url":"https://api.github.com/users/jflf-CV/gists{/gist_id}","starred_url":"https://api.github.com/users/jflf-CV/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jflf-CV/subscriptions","organizations_url":"https://api.github.com/users/jflf-CV/orgs","repos_url":"https://api.github.com/users/jflf-CV/repos","events_url":"https://api.github.com/users/jflf-CV/events{/privacy}","received_events_url":"https://api.github.com/users/jflf-CV/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T17:55:55Z","updated_at":"2016-12-13T17:55:55Z","body":"**DURING THE PRIMARY INSTALL ONLY**: if something goes badly wrong during the critical section and you can't fix it and want to start fresh, rather than restoring a previous VM snapshot and going through the whole process again, do this:\r\n\r\n```\r\npcs cluster destroy\r\nrm -fr /trinity/ /etc/trinity.* /root/secondary /etc/drbd.d/trinity_disk.res /var/run/drbd/drbd-* /etc/exports.d/trinity* /etc/sysconfig/nfs* /run/sysconfig/nfs-utils\r\n```\r\n\r\nThis will clean up your system and you can start again. You can also comment out all the PS before the critical section, they have already run (re-running them won't cause any harm though, it will only waste some time).\r\n\r\nIf you try that for the secondary install you will waste both primary and secondary setups..."}}
{"actor":{"display_login":"jflf-CV"}}