{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/6658","repository_url":"https://api.github.com/repos/influxdata/influxdb","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/6658/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/6658/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/6658/events","html_url":"https://github.com/influxdata/influxdb/issues/6658","id":155458851,"number":6658,"title":"atomic.AddUint32 in UUID generation causing global lock (significantly slowing down startup in HEAD)","user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdata/influxdb/labels/category/performance","name":"category/performance","color":"006b75"},{"url":"https://api.github.com/repos/influxdata/influxdb/labels/performance-testing","name":"performance-testing","color":"006b75"}],"state":"open","locked":false,"assignee":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-05-18T09:44:53Z","updated_at":"2016-05-18T14:32:31Z","closed_at":null,"body":"Running a startup on HEAD with a large dataset shows clearly (in perf) that the overwhelming majority of time is spent in atomic.AddUint32. This only appears because of performance improvements in HEAD; this does not appear in 0.13 stable (which spends most of its time in runtime.mapiternext and runtime.indexbytebody).  This was introduced in \"Replace code.google.com/p/go-uuid with TimeUUID from gocql\" (62434fb87) at the end of March.\r\n\r\nSince I have issues loosing data in HEAD, i've not been able to see if this also is a significant user of CPU once influxd is started, but it seems plausible that it will, particularly under high loads. This code is, I think, in ./uuid/uuid.go and effectively introduces a global lock around anything that wants a UUID:\r\n\r\n```\r\n376bc8ce (Mint          2015-09-18 22:47:15 -0500 54) // FromTime generates a new time based UUID (version 1) as described in\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 55) // RFC 4122. This UUID contains the MAC address of the node that generated\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 56) // the UUID, the given timestamp and a sequence number.\r\n376bc8ce (Mint          2015-09-18 22:47:15 -0500 57) func FromTime(aTime time.Time) UUID {\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 58)   var u UUID\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 59)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 60)   utcTime := aTime.In(time.UTC)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 61)   t := uint64(utcTime.Unix()-timeBase)*10000000 + uint64(utcTime.Nanosecond()/100)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 62)   u[0], u[1], u[2], u[3] = byte(t>>24), byte(t>>16), byte(t>>8), byte(t)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 63)   u[4], u[5] = byte(t>>40), byte(t>>32)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 64)   u[6], u[7] = byte(t>>56)&0x0F, byte(t>>48)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 65)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 66)   clock := atomic.AddUint32(&clockSeq, 1)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 67)   u[8] = byte(clock >> 8)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 68)   u[9] = byte(clock)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 69)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 70)   copy(u[10:], hardwareAddr)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 71)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 72)   u[6] |= 0x10 // set version to 1 (time based uuid)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 73)   u[8] &= 0x3F // clear variant\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 74)   u[8] |= 0x80 // set to IETF variant\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 75)\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 76)   return u\r\n62434fb8 (Matt Robenolt 2015-03-30 08:56:05 -0700 77) }\r\n```\r\n\r\nThis is clearly putting a global lock for a good reason (to ensure we dont get duplicate UUIDs within the same nanosecond), but this should be fairly simple to improve while either guaranteeing, or making extremely unlikely, that we have duplicate IDs.\r\n\r\nThere are various ways to improve this that occur to me; since (AFAIK) nobody cares about the time to the nanosecond for these, one way to do this is to have a bunch of goroutines producing these atomic values, sharing the nanoseconds in each millisecond between them. This would still give timestamps accurate to the ms (or a fraction of ms, whatever we wanted).\r\n\r\nThe second option would be to remove the node MAC (which isnt adding a lot), and replace it with a random number (per nanosecond), which is (from a quick read of https://www.ietf.org/rfc/rfc4122.txt allowed). This of course would make it technically possible for a duplicate value to appear, but fantastically unlikely (technically, restarting the process today with a different time could also cause this, i'm not actually sure what is more likely!). \r\n\r\nThoughts? This is low hanging fruit for performance optimization I think... \r\n\r\nTagging @jwilder FYI; I found this while testing his PR and @mattrobenolt who I think (?) wrote this code. \r\n\r\nRaw perf data (this remains pretty much static during the whole startup). Our dataset is a few TB all on SSD, with loads of CPU cores. \r\n\r\n```\r\nSamples: 4M of event 'cycles', Event count (approx.): 430757131359\r\n  28.53%  influxd                       [.] sync/atomic.AddUint32\r\n  11.64%  influxd                       [.] strings.Index\r\n   5.80%  influxd                       [.] scanblock\r\n   4.72%  [kernel]                      [k] _raw_spin_lock\r\n   4.60%  influxd                       [.] runtime.MSpan_Sweep\r\n   2.33%  influxd                       [.] github.com/influxdata/influxdb/tsdb.(*DatabaseIndex).Series\r\n   1.82%  influxd                       [.] runtime.memmove\r\n   1.76%  influxd                       [.] runtime.aeshashbody\r\n   1.54%  influxd                       [.] runtime.memeqbody\r\n   1.52%  [kernel]                      [k] down_read_trylock\r\n   1.39%  influxd                       [.] runtime.mallocgc\r\n   1.34%  influxd                       [.] runtime.mapaccess1_faststr\r\n   1.22%  influxd                       [.] runtime.deferreturn\r\n   1.18%  influxd                       [.] runtime.newdefer\r\n   1.14%  influxd                       [.] runtime.writebarrierptr\r\n   1.10%  influxd                       [.] runtime.cas64\r\n   1.02%  influxd                       [.] runtime.xchg\r\n   0.95%  influxd                       [.] runtime.readvarint\r\n   0.89%  influxd                       [.] github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).WalkKeys\r\n   0.82%  [kernel]                      [k] page_fault\r\n   0.80%  influxd                       [.] github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).KeyAt\r\n   0.78%  [kernel]                      [k] up_read\r\n   0.76%  influxd                       [.] runtime.findfunc\r\n   0.74%  influxd                       [.] runtime.gentraceback\r\n   0.73%  influxd                       [.] runtime.releasem\r\n   0.69%  influxd                       [.] getfull\r\n   0.66%  influxd                       [.] runtime.freedefer\r\n   0.55%  influxd                       [.] runtime.acquirem\r\n   0.53%  influxd                       [.] runtime.atomicload64\r\n   0.53%  influxd                       [.] runtime.memclr\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/220044937","html_url":"https://github.com/influxdata/influxdb/issues/6658#issuecomment-220044937","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/6658","id":220044937,"user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"created_at":"2016-05-18T14:32:31Z","updated_at":"2016-05-18T14:32:31Z","body":"@jwilder yes. No panic, but still missing data. "}}