{"action":"created","issue":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/1113","repository_url":"https://api.github.com/repos/rcbops/rpc-openstack","labels_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/1113/labels{/name}","comments_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/1113/comments","events_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/1113/events","html_url":"https://github.com/rcbops/rpc-openstack/issues/1113","id":158285824,"number":1113,"title":"Plugin rabbitmq_status.py not properly excluding notifications queues on returned metrics.","user":{"login":"hybridpollo","id":1617447,"avatar_url":"https://avatars.githubusercontent.com/u/1617447?v=3","gravatar_id":"","url":"https://api.github.com/users/hybridpollo","html_url":"https://github.com/hybridpollo","followers_url":"https://api.github.com/users/hybridpollo/followers","following_url":"https://api.github.com/users/hybridpollo/following{/other_user}","gists_url":"https://api.github.com/users/hybridpollo/gists{/gist_id}","starred_url":"https://api.github.com/users/hybridpollo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hybridpollo/subscriptions","organizations_url":"https://api.github.com/users/hybridpollo/orgs","repos_url":"https://api.github.com/users/hybridpollo/repos","events_url":"https://api.github.com/users/hybridpollo/events{/privacy}","received_events_url":"https://api.github.com/users/hybridpollo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/impacts-maas","name":"impacts-maas","color":"d4c5f9"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/impacts-support","name":"impacts-support","color":"d4c5f9"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/prio-1","name":"prio-1","color":"eb6420"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/status-needs-review","name":"status-needs-review","color":"cccccc"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/type-bug","name":"type-bug","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"hughsaunders","id":253506,"avatar_url":"https://avatars.githubusercontent.com/u/253506?v=3","gravatar_id":"","url":"https://api.github.com/users/hughsaunders","html_url":"https://github.com/hughsaunders","followers_url":"https://api.github.com/users/hughsaunders/followers","following_url":"https://api.github.com/users/hughsaunders/following{/other_user}","gists_url":"https://api.github.com/users/hughsaunders/gists{/gist_id}","starred_url":"https://api.github.com/users/hughsaunders/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hughsaunders/subscriptions","organizations_url":"https://api.github.com/users/hughsaunders/orgs","repos_url":"https://api.github.com/users/hughsaunders/repos","events_url":"https://api.github.com/users/hughsaunders/events{/privacy}","received_events_url":"https://api.github.com/users/hughsaunders/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2016-06-03T03:48:49Z","updated_at":"2016-06-14T15:56:26Z","closed_at":null,"body":"Greetings,\r\n\r\nI don't submit these very often,  it seems that this maybe the right place for this, otherwise then any guidance is always welcomed.   The maas plugin rabbitmq_status.py looks to not be properly excluding notification queues in the metrics reported.    Its possible this has to do with rabbitmq now using vhosts per each openstack component starting in 12.x.x.x implementations, but I will include as much  information as I can below.\r\n\r\n##### What version (commit) do you encounter this issue on?\r\n\r\n```\r\nroot@xxxxxx-infra01:/opt/rpc-openstack/rpcd# git describe --tags\r\nr12.1.1\r\n```\r\n\r\n##### What type of issue is this (bug, feature, documentation, regression)?\r\nBug\r\n\r\n##### What is the expected behavior?\r\nrabbitmq_status.py properly excluding notification queues in the metrics.\r\n\r\n##### What is actually happening?\r\nMaaS alerts for rabbit are triggering with the following alarm, when all queues are at 0 except the notification.* queues:\r\n\r\nREASON: RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above 100\r\n\r\n##### How can we reproduce this issue?\r\nHere is an example rabbitmq node with a growing notifications queue on the /ceilometer vhost.\r\n\r\n```\r\nroot@xxxxxx-Infra01_rabbit_mq_container-db618bcd:/var/log/rabbitmq# for x in $(rabbitmqctl list_vhosts | awk '$1 !~ '/Listing'/ { print }') ; do echo \"Backed up queues for vhost $x\" ; rabbitmqctl list_queues -p $x | awk '$2 > 1 { print }' ; echo \" \" ; done\r\nBacked up queues for vhost /\r\nListing queues ...\r\n \r\nBacked up queues for vhost /ceilometer\r\nListing queues ...\r\nnotifications.sample\t212\r\n \r\nBacked up queues for vhost /cinder\r\nListing queues ...\r\n \r\nBacked up queues for vhost /glance\r\nListing queues ..\r\n \r\nBacked up queues for vhost /heat\r\nListing queues ..\r\n \r\nBacked up queues for vhost /keystone\r\nListing queues ..\r\n \r\nBacked up queues for vhost /neutron\r\nListing queues ..\r\n \r\nBacked up queues for vhost /nova\r\nListing queues ..\r\n```\r\n\r\nMetrics returned by the rabbitmq_status.py plugin:\r\n\r\n```\r\nroot@xxxxxx-infra01:/usr/lib/rackspace-monitoring-agent/plugins# ./rabbitmq_status.py -H 172.29.238.65 -n xxxxxx-Infra01_rabbit_mq_container-db618bcd -U maas_user -p xxxxxxxxxxxxxx\r\nstatus okay\r\nmetric rabbitmq_notification_messages int64 0 messages\r\nmetric rabbitmq_uptime int64 816711536 ms\r\nmetric rabbitmq_messages int64 212 messages\r\nmetric rabbitmq_ack int64 5962130 messages\r\nmetric rabbitmq_deliver_get int64 5962131 messages\r\nmetric rabbitmq_max_channels_per_conn int64 1 channels\r\nmetric rabbitmq_deliver int64 5962131 messages\r\nmetric rabbitmq_sockets_total int64 58889 fd\r\nmetric rabbitmq_publish int64 5854773 messages\r\nmetric rabbitmq_fd_used int64 342 fd\r\nmetric rabbitmq_mem_used int64 217246960 bytes\r\nmetric rabbitmq_fd_total int64 65535 fd\r\nmetric rabbitmq_disk_free_alarm_status uint32 1\r\nmetric rabbitmq_proc_used int64 4422 processes\r\nmetric rabbitmq_mem_limit int64 53976499814 bytes\r\nmetric rabbitmq_mem_alarm_status uint32 1\r\nmetric rabbitmq_sockets_used int64 321 fd\r\nmetric rabbitmq_msgs_excl_notifications int64 212 messages\r\nmetric rabbitmq_messages_unacknowledged int64 0 messages\r\nmetric rabbitmq_messages_ready int64 212 messages\r\nmetric rabbitmq_proc_total int64 1048576 processes\r\n```\r\n\r\n##### How can we verify your issue is addressed?\r\nEnsuring the rabbitmq_status.py plugin is properly reporting metrics including the exclusion of the notfication queues.\r\n\r\n##### If this is a large issue, what can be done immediately to alleviate the issue while working on a longer-term solution?\r\nN/A not a large issue at the moment but it might be as we upgrade customers to Liberty.\r\n\r\n##### What is your evaluation on the importance of this issue, and why?\r\n\r\n\r\n##### What branches / milestones do you believe this issue should target?\r\nThis is the first time seeing this on 12.x.x environments.  \r\n\r\n###### Remove branches (lines) you do not wish to target.\r\n- [ ] Master\r\n- [ ] liberty-12.2\r\n- [ ] liberty-12.1"},"comment":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/comments/225928068","html_url":"https://github.com/rcbops/rpc-openstack/issues/1113#issuecomment-225928068","issue_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/1113","id":225928068,"user":{"login":"mancdaz","id":562262,"avatar_url":"https://avatars.githubusercontent.com/u/562262?v=3","gravatar_id":"","url":"https://api.github.com/users/mancdaz","html_url":"https://github.com/mancdaz","followers_url":"https://api.github.com/users/mancdaz/followers","following_url":"https://api.github.com/users/mancdaz/following{/other_user}","gists_url":"https://api.github.com/users/mancdaz/gists{/gist_id}","starred_url":"https://api.github.com/users/mancdaz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mancdaz/subscriptions","organizations_url":"https://api.github.com/users/mancdaz/orgs","repos_url":"https://api.github.com/users/mancdaz/repos","events_url":"https://api.github.com/users/mancdaz/events{/privacy}","received_events_url":"https://api.github.com/users/mancdaz/received_events","type":"User","site_admin":false},"created_at":"2016-06-14T15:56:26Z","updated_at":"2016-06-14T15:56:26Z","body":"```\r\n15:12:14 ********************** Run Holland ***********************\r\n15:12:14 + run_holland\r\n15:12:14 ++ sudo lxc-ls\r\n15:12:14 ++ grep galera\r\n15:12:14 ++ head -n1\r\n15:12:14 + sudo lxc-attach -n jrpcaioiad-d1a_galera_container-26712479 -- /bin/bash -c 'holland bk'\r\n15:12:14 No handlers could be found for logger \"holland.core.util.bootstrap\"\r\n15:12:14 + HOLLAND_RC=1\r\n```"}}
{"actor":{"display_login":"mancdaz"}}