{"action":"created","issue":{"url":"https://api.github.com/repos/cloyne/network/issues/114","repository_url":"https://api.github.com/repos/cloyne/network","labels_url":"https://api.github.com/repos/cloyne/network/issues/114/labels{/name}","comments_url":"https://api.github.com/repos/cloyne/network/issues/114/comments","events_url":"https://api.github.com/repos/cloyne/network/issues/114/events","html_url":"https://github.com/cloyne/network/issues/114","id":327410772,"node_id":"MDU6SXNzdWUzMjc0MTA3NzI=","number":114,"title":"Device: /dev/twa0 [3ware_disk_13], 2 Currently unreadable (pending) sectors","user":{"login":"mitar","id":585279,"node_id":"MDQ6VXNlcjU4NTI3OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/585279?v=4","gravatar_id":"","url":"https://api.github.com/users/mitar","html_url":"https://github.com/mitar","followers_url":"https://api.github.com/users/mitar/followers","following_url":"https://api.github.com/users/mitar/following{/other_user}","gists_url":"https://api.github.com/users/mitar/gists{/gist_id}","starred_url":"https://api.github.com/users/mitar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mitar/subscriptions","organizations_url":"https://api.github.com/users/mitar/orgs","repos_url":"https://api.github.com/users/mitar/repos","events_url":"https://api.github.com/users/mitar/events{/privacy}","received_events_url":"https://api.github.com/users/mitar/received_events","type":"User","site_admin":false},"labels":[{"id":53496003,"node_id":"MDU6TGFiZWw1MzQ5NjAwMw==","url":"https://api.github.com/repos/cloyne/network/labels/broken","name":"broken","color":"fc2929","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-05-29T17:08:16Z","updated_at":"2018-10-21T22:17:01Z","closed_at":null,"author_association":"MEMBER","body":"For some time now server3 has 2 unreadable (pending) sectors. This is good, because it means the problem is not growing and we can keep those disks. But this should still be fixed.\r\n\r\n```\r\nThis message was generated by the smartd daemon running on:\r\n\r\n   host name:  server3\r\n   DNS domain: cloyne.org\r\n\r\nThe following warning/error was logged by the smartd daemon:\r\n\r\nDevice: /dev/twa0 [3ware_disk_13], 2 Currently unreadable (pending) sectors\r\n\r\nDevice info:\r\nST3750640NS, S/N:3QD0B5XJ, FW:3.AEE, 750 GB\r\n```\r\n\r\nSo the issue here is that disk detected two sectors to be bad. This is why they are pending. You should resolve those pending sectors and get disk to remove them from use and replace them with space sectors. The process is a bit involved, but it is fun:\r\n* you first remove the offending disk from the RAID (you leave the other mirror disk in RAID, so there is no service disruption)\r\n* once out of RAID, you override the whole disk (`dd` tool) with write blocks of size equal to disk blocks with zeros\r\n* once you override an offending sector with zeros, disk knows what is the right content of that sector and swaps the spare sector in, lowering this count (and probably increasing \"uncorrectable\" count)\r\n* then you have to reformat disk back for mdn RAID (`fdisk` tool)\r\n* and reattach it back to RAID so that mirror is restored and original data is copied back\r\n\r\nI suggest you go through and figure our all commands to do this and then log them here."},"comment":{"url":"https://api.github.com/repos/cloyne/network/issues/comments/431709064","html_url":"https://github.com/cloyne/network/issues/114#issuecomment-431709064","issue_url":"https://api.github.com/repos/cloyne/network/issues/114","id":431709064,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTcwOTA2NA==","user":{"login":"clonm","id":6488743,"node_id":"MDQ6VXNlcjY0ODg3NDM=","avatar_url":"https://avatars3.githubusercontent.com/u/6488743?v=4","gravatar_id":"","url":"https://api.github.com/users/clonm","html_url":"https://github.com/clonm","followers_url":"https://api.github.com/users/clonm/followers","following_url":"https://api.github.com/users/clonm/following{/other_user}","gists_url":"https://api.github.com/users/clonm/gists{/gist_id}","starred_url":"https://api.github.com/users/clonm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clonm/subscriptions","organizations_url":"https://api.github.com/users/clonm/orgs","repos_url":"https://api.github.com/users/clonm/repos","events_url":"https://api.github.com/users/clonm/events{/privacy}","received_events_url":"https://api.github.com/users/clonm/received_events","type":"User","site_admin":false},"created_at":"2018-10-21T22:17:01Z","updated_at":"2018-10-21T22:17:01Z","author_association":"CONTRIBUTOR","body":"Before:\r\n```\r\nroot@server3:/home/cloyne# sudo smartctl -a -d 3ware,13 /dev/twa0\r\nsmartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-130-generic] (local build)\r\nCopyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org\r\n\r\n=== START OF INFORMATION SECTION ===\r\nModel Family:     Seagate Barracuda ES\r\nDevice Model:     ST3750640NS\r\nSerial Number:    3QD0B5XJ\r\nFirmware Version: 3.AEE\r\nUser Capacity:    750,156,374,016 bytes [750 GB]\r\nSector Size:      512 bytes logical/physical\r\nDevice is:        In smartctl database [for details use: -P show]\r\nATA Version is:   ATA/ATAPI-7 (minor revision not indicated)\r\nLocal Time is:    Sat Oct 20 11:09:59 2018 PDT\r\nSMART support is: Available - device has SMART capability.\r\nSMART support is: Enabled\r\n\r\n=== START OF READ SMART DATA SECTION ===\r\nSMART overall-health self-assessment test result: PASSED\r\nSee vendor-specific Attribute list for marginal Attributes.\r\n\r\nGeneral SMART Values:\r\nOffline data collection status:  (0x82) Offline data collection activity\r\n                                        was completed without error.\r\n                                        Auto Offline Data Collection: Enabled.\r\nSelf-test execution status:      (   0) The previous self-test routine completed\r\n                                        without error or no self-test has ever \r\n                                        been run.\r\nTotal time to complete Offline \r\ndata collection:                (  430) seconds.\r\nOffline data collection\r\ncapabilities:                    (0x5b) SMART execute Offline immediate.\r\n                                        Auto Offline data collection on/off support.\r\n                                        Suspend Offline collection upon new\r\n                                        command.\r\n                                        Offline surface scan supported.\r\n                                        Self-test supported.\r\n                                        No Conveyance Self-test supported.\r\n                                        Selective Self-test supported.\r\nSMART capabilities:            (0x0003) Saves SMART data before entering\r\n                                        power-saving mode.\r\n                                        Supports SMART auto save timer.\r\nError logging capability:        (0x01) Error logging supported.\r\n                                        General Purpose Logging supported.\r\nShort self-test routine \r\nrecommended polling time:        (   1) minutes.\r\nExtended self-test routine\r\nrecommended polling time:        ( 202) minutes.\r\nSCT capabilities:              (0x003d) SCT Status supported.\r\n                                        SCT Error Recovery Control supported.\r\n                                        SCT Feature Control supported.\r\n                                        SCT Data Table supported.\r\n\r\nSMART Attributes Data Structure revision number: 10\r\nVendor Specific SMART Attributes with Thresholds:\r\nID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE\r\n  1 Raw_Read_Error_Rate     0x000f   105   088   006    Pre-fail  Always       -       65372609\r\n  3 Spin_Up_Time            0x0003   087   085   000    Pre-fail  Always       -       0\r\n  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       135\r\n  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       2\r\n  7 Seek_Error_Rate         0x000f   087   060   030    Pre-fail  Always       -       499263873\r\n  9 Power_On_Hours          0x0032   055   055   000    Old_age   Always       -       39562\r\n 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0\r\n 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       137\r\n187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0\r\n189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0\r\n190 Airflow_Temperature_Cel 0x0022   061   044   045    Old_age   Always   In_the_past 39 (Min/Max 38/43)\r\n194 Temperature_Celsius     0x0022   039   056   000    Old_age   Always       -       39 (0 13 0 0 0)\r\n195 Hardware_ECC_Recovered  0x001a   062   047   000    Old_age   Always       -       228011716\r\n197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       2\r\n198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       2\r\n199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0\r\n200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0\r\n202 Data_Address_Mark_Errs  0x0032   100   253   000    Old_age   Always       -       0\r\n\r\nSMART Error Log Version: 1\r\nNo Errors Logged\r\n\r\nSMART Self-test log structure revision number 1\r\nNum  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error\r\n# 1  Extended offline    Completed without error       00%     16244         -\r\n\r\nSMART Selective self-test log data structure revision number 1\r\n SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS\r\n    1        0        0  Not_testing\r\n    2        0        0  Not_testing\r\n    3        0        0  Not_testing\r\n    4        0        0  Not_testing\r\n    5        0        0  Not_testing\r\nSelective self-test flags (0x0):\r\n  After scanning selected spans, do NOT read-scan remainder of disk.\r\nIf Selective self-test is pending on power-up, resume after 0 minute delay.\r\n\r\n```"}}
{"actor":{"display_login":"clonm"}}