[09:36] <meena> that seems like an interesting omission…
[13:42] <Odd_Bloke> meena: What does?
[13:43] <Odd_Bloke> LongLiveCHIEF: Can you confirm what version of netplan you had in that image (`dpkg -l netplan.io`)?
[13:44] <LongLiveCHIEF> sure, one moment
[13:46] <LongLiveCHIEF> 0.99-0ubuntu3 for arm64
[13:46] <LongLiveCHIEF> q
[13:46] <LongLiveCHIEF> q!
[13:46] <LongLiveCHIEF> whoops
[13:48] <Odd_Bloke> :)
[13:48] <Odd_Bloke> OK, bummer, that's the latest version.
[13:48] <Odd_Bloke> (So my hopes that this problem would be solved by someone else have been dashed. ;)
[13:48] <LongLiveCHIEF> yeah... :sad:
[13:49] <LongLiveCHIEF> there's still many hopes in that regard. That particular bucket of hope is bottomless
[13:50] <LongLiveCHIEF> something interesting came up in #raspberrypi over the weekend about rfkill disabling wifi early on when wireless region wasn't set
[13:50] <LongLiveCHIEF> in wpa
[13:51] <LongLiveCHIEF> ...and it did get me thinking
[13:52] <LongLiveCHIEF> is there a way to halt cloud-init and drop into a shell before a specific stage?  or to log output of a specific command after/before a stage?
[14:03] <Odd_Bloke> LongLiveCHIEF: OK, I've confirmed that waveform, the Ubuntu Pi person, is still looking into the wifi issue.
[14:03] <Odd_Bloke> I don't think we have a good way of doing something like that, no.
[14:04] <LongLiveCHIEF> no worries, i have my ways
[14:07] <LongLiveCHIEF> forward my info to the ubuntu pi person if he wants someone in the field to test and report.  my gh is longlivechief
[14:50] <Odd_Bloke> blackboxsw: I think we can improve your force-push method to not need a tracking file: we already have all that information in git history.  I'm putting something together now.
[15:29] <mutantturkey> hi guys. i am building a custom image with packer. this gets build on digitalocean, and then reused on digitalocean. what this means is the first timeit runs it gets information from cloud-init regarding networking information, but when i instantiate  the image on a new virutal server that original network information is no longer correct
[15:29] <mutantturkey> the problem is that, when i create a new server with this image, the FIRST time it boots, it has the incorrect information for networking
[15:30] <mutantturkey> however, i think this is because the networking information is inputted into /etc/network/interfaces.d/* AFTER network services are already running. So the first time, it'll not have a net conncetion, but correct values, and then subsequently it will work just fine
[15:30] <mutantturkey> so i am looking for an idea on how to solve this. My one thought was to move /etc/init.d/network to a later sysvinit stage
[15:39] <blackboxsw> +1 Odd_Bloke
[15:40] <Odd_Bloke> mutantturkey: What distribution are these images for?  And how are you installing cloud-init?
[15:40] <blackboxsw> mutantturkey: I'm not super familiar with packer builds. but do you have the opportunity to run 'cloud-init clean --logs' before shutting down that image and redeploying it?
[15:43] <Odd_Bloke> blackboxsw: `clean` wouldn't remove network configuration, right?
[15:44] <blackboxsw> Odd_Bloke: no but it would tell cloud-init to re-run network config on next boot (the first redeploy of that created image)
[15:44] <blackboxsw> I was making an assumption that mutantturkey's guess on /etc/network/interfaces.d/* being written AFTER networking services are running is probably unlikely
[15:44] <Odd_Bloke> cloud-init should detect that its a new instance without a `clean`, though, so doing that could mask a more general issue on DO.
[15:45] <Odd_Bloke> If cloud-init isn't plumbed into the boot sequence correctly, it's plausible (and perhaps likely) that it would run after networking services.
[15:45] <blackboxsw> good point. it *could* be a general issue with DO, but I was presuming it was an issue with how the image is being created with packer
[15:46] <Odd_Bloke> (And if we're talking about SysV init, then this presumably isn't an Ubuntu image, where I would have higher confidence that cloud-init will get plumbed in correctly simply via installation.)
[15:46] <blackboxsw> Odd_Bloke: that's a good though too. yeah I guess if the before=networking-target.online (or whatever it's called) wasn't present then right, we could be in that camp
[15:47] <blackboxsw> or sysV init too right
[15:47] <Odd_Bloke> Right, except we get no Before= in a SysV world. :p
[15:54] <Odd_Bloke> blackboxsw: https://paste.ubuntu.com/p/kbC4XkHq3H/ is my iteration of the force-push proposal.
[16:37] <blackboxsw> Odd_Bloke: yes excellent
[16:38] <blackboxsw> Odd_Bloke: though we need to limit that globbing to just debian/patches/*cpick*
[16:38] <Odd_Bloke> blackboxsw: When would we have any cherry-picks in the daily branch that we want to retain?
[16:38] <Odd_Bloke> Oh, I suppose for non-devel releases?
[16:43] <blackboxsw> Odd_Bloke: any release can have release-specific packaging, config disabling changs
[16:43] <blackboxsw> and those debian/patches would not contain a *cpick* or fox-cpick prefix
[16:43] <mutantturkey> blackboxsw: i do have that opportunity
[16:44] <mutantturkey> would that fofce on next boot
[16:44] <mutantturkey> thats perfect
[16:44] <mutantturkey> this is sysvinit, this is debian buster with a bunch of custom stuff on top
[16:44] <blackboxsw> like https://github.com/canonical/cloud-init/tree/ubuntu/xenial/debian/patches
[16:44] <mutantturkey> i can do a rebuild will take an hour or two, dwill that clean thing work?
[16:44] <blackboxsw> Odd_Bloke: ^
[16:45] <blackboxsw> mutantturkey: the clean thing removes all semaphores that marked that networking was created in the first place.... but as Odd_Bloke mentoined, it might be masking some other pre-instance change that cloud-init tries to rely on when you are re-launching your created image
[16:46] <mutantturkey> okay, then let me try that
[16:46] <mutantturkey> again, sysvinit is more opaque so i dont really want to muck with it if possible :-)
[16:47] <blackboxsw> mutantturkey: and ok on sysvinit, and Debian, so as Odd_Bloke mentioned the sysvinit dependency ordering may not have been created in your debian image properly to ensure cloud-init writes network before network is brough up by the system, without a strict startup dependency added, cloud-init *could* be writing the network config too late as you guessed
[16:47] <mutantturkey> what does --logs do
[16:47] <mutantturkey> wouldn't i want t o keep my logs
[16:48] <blackboxsw> mutantturkey: --logs rms /var/log/cloud-init.log /var/log/cloud-init-output.log
[16:48] <mutantturkey> alright im giving it a swing
[16:49] <blackboxsw> which should allow you to distill what cloud-init does on 'first boot' during your first relaunch
[16:49] <Odd_Bloke> blackboxsw: The presence of the semaphores isn't relevant if cloud-init isn't running at the right point in boot; we would expect the semaphore to be ignored for a new instance boot anyway.
[16:49] <Odd_Bloke> mutantturkey: Installing cloud-init from the Debian archive?
[16:49] <blackboxsw> Odd_Bloke: agreed, but it is useful to compare the last boot stage timestamp against the dmesg or journalctl logs
[16:50] <blackboxsw> I was only thinking to avoid any previous boot logs from during the image creation process
[16:55] <mutantturkey> Odd_Bloke: yes
[16:55] <mutantturkey> okay, blackboxsw i will look at the logs. Clearing out the old config seems like it would be useful to force cloud init to redo its business.
[17:07] <blackboxsw> Odd_Bloke: also I don't think we can individually just git log --online through all commits as we have had times where we needed to fixup interrelated cherrypicks and lack of a ordered set of commits across all cpick* patches would mean we can't back them out without conflicts partway through
[17:11] <blackboxsw> actually oops. right git log debian/patche/*cpick* will order that for us properly anyway
[17:19] <mutantturkey> that did not appear t owork, rebooting now
[17:26] <mutantturkey> well now it's online but refusing ssh, i think something stuck let me try to reboot again
[17:37] <Odd_Bloke> blackboxsw: Yep, that should take care of it.
[17:37] <blackboxsw> Odd_Bloke: good, tested and looks good. I'm resolving your review comments now
[17:37] <Odd_Bloke> Cool, thanks!
[17:51] <blackboxsw> Odd_Bloke: I think I'm done https://github.com/canonical/uss-tableflip/pull/45
[17:56] <Odd_Bloke> blackboxsw: Did you push?  I'm seeing a lot of unresolved comments.
[18:03] <blackboxsw> hrm, maybe not. checking again and going through comments
[18:43] <blackboxsw> I had missed about 5 comments Odd_Bloke pushed the changes
[20:27] <blackboxsw> Goneri: +1 on your branch https://github.com/canonical/cloud-init/pull/305/files with a minor nit. Just needs that and a rebase against master and we are good to go
[20:29] <Goneri> blackboxsw, done
[20:30]  * blackboxsw just twiddles thumbs waiting on Travis thanks Goneri will land today
[20:41] <Odd_Bloke> falcojr: Thanks for the CLA PR, as you can see, I've landed it. :)
[20:41] <falcojr> cool, thanks
[20:49] <Odd_Bloke> lucasmoura: Thanks for the CLA PR!
[21:07] <mutantturkey> alright got sidetracked today but
[21:07] <mutantturkey> first boot with cloud-init clean doesn't fix the problem
[21:07] <mutantturkey> first boot i can verify has the correct *updated* configs
[21:07] <mutantturkey> but it's not applied (i can see that via ifconfig)
[21:08] <blackboxsw> Goneri: merged thanks !
[21:09] <mutantturkey> restarting networking does fix theproblem
[21:09] <blackboxsw> mutantturkey: so Odd_Bloke's suggestion that maybe the deb didn't wire the proper SysV init script dependencies in your image might be the issue there
[21:09] <mutantturkey> yep
[21:09] <mutantturkey> theoretically i think i understand how sysviinit works but
[21:09] <mutantturkey> the goal is, networking should run after cloud init
[21:10] <blackboxsw> we expect to have a 4 stages wired in the right order during startup something akin to this https://github.com/canonical/cloud-init/tree/master/sysvinit/debian
[21:12] <mutantturkey> so those are all different init scripts in /etc/init.d right?
[21:15] <mutantturkey> okay I have those files cool
[21:15] <blackboxsw> mutantturkey: yep.
[21:15] <mutantturkey> so during boot this should all go to syslog?
[21:15] <blackboxsw> I was just checking an older release of cloud-init on trusty to confirm separate config files in /etc/init
[21:16] <mutantturkey> yes i do have those
[21:19] <Goneri> thanks blackboxsw :-)
[21:24] <mutantturkey> okay blackboxsw we have a lead!
[21:24] <mutantturkey> it appears none of these are configred
[21:24] <blackboxsw> mutantturkey: guaranteed to go to /var/log/cloud-init.log not sure where debian typically logs cloud-init emitted logfiles besides the default
[21:24] <mutantturkey> if i read rc*.d right, anything that starts with S means 'start"
[21:24] <blackboxsw> s/besides/in addition to/
[21:24] <mutantturkey> k means, kill at tha trun level
[21:24] <mutantturkey> networking is in the rc.S runlevel
[21:24] <mutantturkey> so looks like cloud-init's 4 things aren't gettng run at startp
[21:24] <blackboxsw> that sounds like reasonable failure path ;)
[21:25] <mutantturkey> though clearly something is running because cloud-init does LOG
[21:26] <mutantturkey> ooff
[21:26] <blackboxsw> mutantturkey: cat /run/cloud-init/status.json that'll tell you what stages did run (on newer cloud-init you can run "cloud-init analyze show")
[21:28] <Odd_Bloke> blackboxsw: Note that trusty is upstart, not SysV.
[21:29] <blackboxsw> oopsie, right-o though our collection of init scripts mentioned here https://github.com/canonical/cloud-init/tree/master/sysvinit/debian should be all-inclusive of what we expect to see on debian right
[21:29] <mutantturkey> S12networking
[21:29] <mutantturkey> that's so whack
[21:30] <Odd_Bloke> blackboxsw: I wouldn't assume so (particularly as those haven't been touched in 7 years).
[21:37] <mutantturkey> well that makes non sense
[21:37] <mutantturkey> let me jump into #debian and seewhatup
[21:41] <mutantturkey> blackboxsw: alright,here we go. it's at runelevel S, which is like single user mode, happens before evreything else
[21:42] <mutantturkey> that's before _all_ the cloud service bits run
[21:42] <mutantturkey> which well makes since to me because other services might try to bind to a specifc IP etc and fail if networking is not up
[21:45] <mutantturkey> so, givne that, is there a move to make here
[21:48] <mutantturkey> so i think "cloud-init writes config files, but doesnt run ifup or the like to configure networking after then config is changed'
[21:52] <mutantturkey> bso my simple thought is another init file called /restart-networking-hack that runs on runelevel 6 that reruns ifup