title: In recovering context, new erasure PGs stay inactive+incomplete
body: <p>The problem (if you want to see 'status' first, please see output at the end of post)</p>
  
  <p>In my production cluster with slowly recovering erasure-coded pools, </p>
  
  <p>new erasure pools are not usable because all their pgs remain "creating+incomplete" seemingly forever.</p>
  
  <p>My existing pools recovery will take a very very long time (several months), but I expect to be able to create new 'clean' pools immediately to receive new data in a resilient manner.</p>
  
  <h1>What I tried :</h1>
  
  <p>ceph osd pool create newpool 128 128 erasure myprofile</p>
  
  <p>rados --pool newpool put anobject afile
  ==> This blocks</p>
  
  <p>ceph pg ls-by-pool newpool incomplete ==> all my pgs are listed</p>
  
  <p>ceph pg 15.1 query
  ==> state ; "creating+incomplete"
     "up" and "acting" contain only the osd '1' as first element, and 'null'(2147483647) at all other positions.</p>
  
  <p>Please note that osd '1' on my platform is the most loaded one (it has almost two times the number of PGs than other OSDs)</p>
  
  <h2>Full context</h2>
  
  <ul>
  <li>Luminous 12.2.0 (Ubuntu 16.04) migrated from jewel</li>
  <li>existing erasure pools with a legacy of very degraded situation (all OSDs are now up, but 83% objects are misplaced, 13% is degraded)</li>
  <li>recovery time is estimated to a year</li>
  <li>my erasure profile is 12 + 3</li>
  </ul>
  
  <p>==> I would like to start writing in a new "clean" pool, while the existing pools will slowly recover.</p>
  
  <p>=====================================
  ceph status</p>
  
  <p>cluster:
      id:     b5ee2a02-b92c-4829-8d43-0eb17314c0f6
      health: HEALTH_WARN
              1314118857/1577308005 objects misplaced (83.314%)
              Reduced data availability: 10 pgs inactive, 10 pgs incomplete
              Degraded data redundancy: 203997294/1577308005 objects degraded (12.933%), 492 pgs unclean, 342 pgs degraded, 279 pgs undersized</p>
  
  <p>services:
      mon: 28 daemons, quorum 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28
      mgr: 37(active), standbys: 38, 12, 45, 33, 29, 10, 11, 22, 31, 47, 15, 36, 40, 32, 41, 24, 44, 34, 27, 28, 43, 35, 39, 16, 25, 26, 2, 9, 3, 4, 8, 23, 19, 17, 5, 42, 6, 21, 7, 20, 30, 13, 18, 14, 46, 1
      osd: 47 osds: 47 up, 47 in; 512 remapped pgs</p>
  
  <p>data:
      pools:   3 pools, 522 pgs
      objects: 100M objects, 165 TB
      usage:   191 TB used, 466 TB / 657 TB avail
      pgs:     1.916% pgs not active
               203997294/1577308005 objects degraded (12.933%)
               1314118857/1577308005 objects misplaced (83.314%)
               155 active+undersized+degraded+remapped+backfill_wait
               140 active+remapped+backfill_wait
               114 active+undersized+degraded+remapped
               63  active+recovery_wait+degraded+remapped
               30  active+clean+remapped
               10  creating+incomplete
               9   active+recovery_wait+undersized+degraded+remapped
               1   active+undersized+degraded+remapped+backfilling</p>
  
  <p>io:
      client:   291 kB/s rd, 0 B/s wr, 14 op/s rd, 16 op/s wr
      recovery: 6309 kB/s, 2 objects/s</p>
  
  <hr>
  
  <hr>
  
  <p>ceph health detail</p>
  
  <p>HEALTH_WARN 1314114780/1577303100 objects misplaced (83.314%); Reduced data availability: 10 pgs inactive, 10 pgs incomplete; Degraded data redundancy: 203992956/1577303100 objects degraded (12.933%), 492 pgs unclean, 342 pgs degraded, 279 pgs undersized
  OBJECT_MISPLACED 1314114780/1577303100 objects misplaced (83.314%)
  PG_AVAILABILITY Reduced data availability: 10 pgs inactive, 10 pgs incomplete
      pg 15.0 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.1 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.2 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.3 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.4 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.5 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.6 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.7 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.8 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
      pg 15.9 is creating+incomplete, acting [1,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647] (reducing pool tester min_size from 12 may help; search ceph.com/docs for 'incomplete')
  PG_DEGRADED Degraded data redundancy: 203992956/1577303100 objects degraded (12.933%), 492 pgs unclean, 342 pgs degraded, 279 pgs undersized
      pg 1.e2 is stuck unclean for 9096318.249662, current state active+undersized+degraded+remapped, last acting [1,2147483647,2147483647,5,40,8,28,47,13,12,29,10,23,2147483647,35]
      pg 1.e3 is stuck undersized for 11585.111340, current state active+undersized+degraded+remapped, last acting [1,3,2147483647,47,11,21,32,46,28,23,2147483647,13,2147483647,19,26]
      pg 1.e4 is stuck undersized for 11588.194871, current state active+undersized+degraded+remapped+backfill_wait, last acting [26,6,23,46,18,30,2147483647,25,38,29,13,45,9,35,20]
      pg 1.e5 is stuck undersized for 11588.374341, current state active+undersized+degraded+remapped+backfill_wait, last acting [14,40,2147483647,22,18,17,29,31,28,43,34,19,33,15,32]
      pg 1.e6 is stuck undersized for 11584.602668, current state active+undersized+degraded+remapped, last acting [1,38,40,2147483647,46,14,2147483647,23,7,44,15,39,8,21,28]
      pg 1.e7 is stuck undersized for 11578.574380, current state active+undersized+degraded+remapped, last acting [1,13,2147483647,37,29,33,18,2147483647,9,38,23,16,42,2147483647,3]
      pg 1.e8 is stuck undersized for 11571.385848, current state active+undersized+degraded+remapped, last acting [1,23,2147483647,7,36,26,6,39,38,2147483647,29,11,15,2147483647,19]
      pg 1.e9 is stuck undersized for 11588.254477, current state active+undersized+degraded+remapped+backfill_wait, last acting [13,44,16,11,9,2147483647,32,37,45,17,20,21,40,46,2147483647]
      pg 1.ea is stuck undersized for 11588.242417, current state active+undersized+degraded+remapped+backfill_wait, last acting [25,19,30,33,2147483647,44,20,39,17,45,43,24,2147483647,10,21]
      pg 1.eb is stuck undersized for 11588.329063, current state active+undersized+degraded+remapped+backfill_wait, last acting [29,39,2147483647,2147483647,40,18,4,33,24,38,32,36,15,47,12]
      pg 1.ec is stuck undersized for 11587.781353, current state active+undersized+degraded+remapped+backfill_wait, last acting [35,37,42,11,2147483647,2147483647,30,15,39,44,43,46,17,4,7]</p>
  
  <p>[...]</p>
  
  <pre><code>pg 3.fa is stuck unclean for 11649.009911, current state active+remapped+backfill_wait, last acting [36,15,42,4,21,14,34,16,17,8,39,3,2,7,19]
  pg 3.fb is stuck undersized for 11580.419670, current state active+undersized+degraded+remapped, last acting [1,7,16,9,19,39,2147483647,33,26,23,20,8,35,40,29]
  pg 3.fd is stuck unclean for 11649.040651, current state active+remapped+backfill_wait, last acting [17,21,8,26,15,42,46,27,7,39,14,35,4,29,25]
  pg 3.fe is active+recovery_wait+degraded+remapped, acting [22,8,45,18,10,46,33,36,16,7,17,34,43,1,23]
  pg 3.ff is stuck unclean for 11649.056722, current state active+remapped+backfill_wait, last acting [33,46,47,17,37,4,40,34,28,43,3,44,13,2,11]
  </code></pre>