[00:03] <smoser> stokachu, still there
[00:03] <smoser> ?
[00:03] <smoser> http://paste.ubuntu.com/6211669/
[00:05] <smoser> so that "works for me". and I tested by following doc/devel/README.txt
[00:05] <smoser> and my 'launch' command
[00:05] <smoser> ./tools/launch $BOOTIMG --publish $ROOTTGZ --add config -- curtin -v install --config config/my-network.conf "PUBURL/${ROOTTGZ##*/}"
[02:09] <roaksoax> bigjools: if you can review the moonshot stuff would be greatlt appreciated
[02:09] <bigjools> roaksoax: I am overloaded with stuff at the moment, I will ask jtv to look
[02:10] <jtv> What needs reviewing?
[02:10] <roaksoax> thanks
[02:10] <roaksoax> lp:~andreserl/maas/moonshot_enablement
[02:10] <jtv> OK
[02:10] <roaksoax> jtv btw is maas-import-ephemerals done?
[02:14] <jtv> Just about.  We need to back up the old pserv.yaml before overwriting it, the "arches" setting needs to be in a place where it can be shared between the import scripts, and some settings can do with clearer names.  But that's all easy stuff.
[02:14] <jtv> With those changes in place, we can land the switchover branch that is already pending.
[02:15] <roaksoax> awesome
[02:15] <roaksoax> jtv: thanks for the hardwork
[02:16] <jtv> All in a good cause!
[02:16] <jtv> Oh, I see Raphaël has the back-up-pserv.yaml branch up for review already.
[02:16] <jtv> Excuse me while I do reviews...
[02:18] <jtv> Argh.  Nope.  That looks buggy.  OK, so back to plan A.  :)
[02:19] <jtv> Nope, nope, it works.  Go Raphers!
[02:22] <jtv> roaksoax: that mechanism for including scripts in the userdata is really paying off, isn't it?
[02:22] <roaksoax> jtv: yeah i love it...
[02:22] <roaksoax> wanna put maas_enlist there when i rewrote it to pythin
[02:23] <bigjools> can we bring maas-signal back there too?
[02:25] <roaksoax> bigjools: xan you subscribe maas-maintainers to djorm-ext-pgarray .. kinda critical
[02:25] <roaksoax> bigjools: and maas signal has always been in maas the reason why we ship itnwith maas-enlist is for end user usage on remote nodes
[02:26] <bigjools> yes I was getting mixed up
[02:27] <bigjools> roaksoax: what do you need the subscription for?
[02:27] <roaksoax> bigjools: it is already.. to be able to promote the package to main
[02:27] <roaksoax> it needs bug subscriber
[02:28] <bigjools> roaksoax: I don't understand what you want
[02:29] <roaksoax> bigjools: nevermind it is done already
[02:30] <roaksoax> bigjools: but add maas-maintainers as a subscriber for launchpad.net/ubuntu/+source/curtin
[02:30] <bigjools> roaksoax: bug subscriber you mean?
[02:31] <bigjools> I will not add teams as subscribers to things as that's pretty obnoxious, people should subscribe themselves
[02:32]  * jtv comes up from email, gasps for breath, goes under again
[02:37] <roaksoax> bigjools: well maas-maintaoners maintain those packages
[02:37] <roaksoax> not just me
[02:37] <roaksoax> thats why it is subscribes
[02:37] <bigjools> that's fine but team subscriptions suck
[02:37] <roaksoax> thats a maas dependenxy that *we* maintain
[02:37] <bigjools> I subscribed myself, feel free to ask the red guys
[02:37] <roaksoax> as it is in ubuntu only for maas
[02:40] <bigjools> now, which sprint track is maas in at SF ...
[02:42] <jtv> roaksoax: bigjools also wrote some script to detect whether the host is physical or virtual...  He checked for qemu initially, but I think he found a more general way.
[02:42] <bigjools> jtv: I didn't :)
[02:42] <jtv> Oh blast.  Maybe it was allenap then.
[02:42] <bigjools> there's a package called imvirt that we should use
[02:43] <bigjools> jtv: well I wrote a commissioning script
[02:43] <bigjools> but we need to generalise
[02:43] <jtv> Found it.  roaksoax, have a look at src/metadataserver/models/commissioningscript.py and search for "notvirtual"
[02:44] <jtv> I see now that it does mention qemu...
[02:44] <jtv> Still, may be useful to compare, to see if we get consistent answers.
[03:31] <stokachu> smoser: nice testing that now
[03:38] <smoser> stokachu, oh. in-target wont work in your version
[03:38] <smoser> :-(
[03:38] <smoser> but this should
[03:40] <smoser> http://paste.ubuntu.com/6212161/
[03:40] <smoser> and i'm off to bed now.
[03:47] <stokachu> smoser: thanks will test some more
[04:59] <stokachu> interesting the web-ui doesn't update the nodes start/stop status if commissioning/starting nodes through maas-cli
[05:00] <stokachu> and of course just after i type that it shows up
[05:00]  * stokachu is tired apparently
[06:52] <bigjools> jtv: we need to fix the Makefile so "make doc" does not try to rebuild the database.  Very annoying :/
[06:54] <jtv> Urgent?
[06:54] <rvba> jtv: did you merge lp:~rvb/maas/backup-config ?
[06:54] <jtv> No...  I assumed you had.
[06:54] <jtv> Because it's landed.
[06:55] <rvba> Weird, I didn't
[06:55] <rvba> jtv: is everything okay for me to land the branch that makes the switch to the new script?
[06:55] <jtv> Not _quite_.  I need two simple reviews!
[06:56] <rvba> Okay, I'll review them now.
[06:56] <rvba> I really want us to do the switch so that I can do another round of testing.
[06:57] <jtv> Thanks.  I'll be back by call time.
[06:57] <jtv> Yes, quite.
[07:27] <rvba> jtv: all reviewed, let's land!
[07:29] <bigjools> rvba, jtv: apparently juju-core now supports maas-tags as constraints
[07:50] <rvba> jtv: should I land the switch now?
[07:50] <jtv> rvba: let me just check if my last branch landed — previous attempt failed because of the prerequisite.
[07:51] <rvba> k
[07:51] <jtv> Ah yes, that's landed.
[07:51] <jtv> So go ahead!
[07:51]  * rvba lands
[08:14] <rvba> bigjools, jtv, allenap: All of the migration-related branches have landed… and the branch that switches to the new script has landed as well… a package is being built in the daily ppa… test away!
[08:15] <allenap> jtv: Thanks for looking at that PostgreSQL thing. I suspect we just take the hit first time around.
[08:18] <jtv> allenap: I don't know much about the situation, but it seems reasonable...  Or maybe there's some heuristic that will get most of the rows you want to delete.
[08:28] <rvba> bigjools, allenap, jtv: the package is ready.  fresh outta the oven.
[08:28] <jtv> Does it work?
[08:28] <allenap> rvba: What jtv said.
[08:28] <rvba> Well, we all need to test it!
[08:30] <jtv> And is this in the daily PPA?
[08:30] <rvba> jtv: yes
[08:59] <bigjools> allenap: have you got a quick example of how to use the LLDP constraints?
[09:00] <rvba> bigjools: do you have time to have a look at: https://code.launchpad.net/~rvb/maas/fpi-button/+merge/190049
[09:00] <bigjools> sure
[09:00] <rvba> ?
[09:00] <rvba> Ta
[09:01] <bigjools> rvba: s/curtin/fast installer/ on the buttons
[09:02] <rvba> bigjools: well, I thought you were okay with both… I choose curtin because it is shorter.
[09:02] <bigjools> rvba: it's too jargon-y for the UI
[09:02] <allenap> bigjools: From Juju, no; Dan said he was going to talk to mramm about getting someone to sort that out on their side.
[09:02] <bigjools> allenap: they said it was working I hear
[09:02] <rvba> bigjools: then we need to re-work the wordings entirely, otherwise it won't fit
[09:02] <allenap> bigjools: With regards to a tag, yes, I'll find something...
[09:02] <bigjools> allenap: cheers
[09:03] <bigjools> rvba: how many characters do we have?
[09:03] <rvba> bigjools: let's say that what we have now fits.
[09:03] <bigjools> ok but new people are going to say "wtf is curtin?"
[09:04] <rvba> And we don't have much more space.
[09:04] <bigjools> dude!
[09:04] <bigjools> display = "Use the default installer"
[09:04] <bigjools> is longer :)
[09:04] <rvba> bigjools: the critical piece is display_bulk.
[09:05] <bigjools> display_bulk = "Mark the selected nodes as using the default installer"
[09:05] <bigjools> s/default/fast/ in the other button
[09:05] <rvba> I'm testing that right now.
[09:05] <bigjools> ok
[09:05] <rvba> bigjools: s/default/fast/ in the other button?
[09:06] <rvba> d-i is not fast :)
[09:06] <bigjools> the UseCurtin one
[09:06] <bigjools> you already have something *longer* for the d-i button
[09:06] <rvba> "Use the fast path installer" does not fit on the node's page.
[09:07] <bigjools> s/path// then
[09:07] <rvba> "Use the default installer" fits
[09:07] <bigjools> so two options:
[09:07] <bigjools> "Use the fast installer"
[09:07] <bigjools> and
[09:07] <bigjools> "Use the default instaler"
[09:07] <bigjools> ok?
[09:07] <rvba> Yeah, that's the best option.
[09:07] <bigjools> great
[09:07] <rvba> Mark the selected nodes as using the fast installer/Mark the selected nodes as using the default installer
[09:08] <rvba> That fits.  It's a bit tight but okay.
[09:08] <bigjools> FWIW I got caught out the other week when I assumed should_use_fastpath_installer was a property :(
[09:09] <allenap> bigjools: //lldp:chassis/lldp:id[@type="mac"]/text() = "20:4e:7f:94:2e:10"
[09:10] <allenap> would select a machine that's connected to the router with that MAC.
[09:10]  * bigjools blinks
[09:10] <bigjools> remind me which API field...
[09:10] <allenap> bigjools: Tag definition.
[09:10] <bigjools> ok
[09:11] <allenap> bigjools: routers are a special case for acquire(); it now accepts connected_to and not_connected_to arguments, which are MACs.
[09:11] <allenap> Or multiple MACs.
[09:12] <bigjools> allenap: nice
[09:12] <allenap> bigjools: rvba did that bit.
[09:12] <bigjools> allenap: this sounds like it needs its own doc
[09:12] <bigjools> too much for release notes
[09:12] <allenap> bigjools: Yep.
[09:13] <bigjools> Ok when I land the release notes, someone can add a :ref: to the new doc
[09:14] <rvba> bigjools: I pushed the change to my fpi-button branch.
[09:15] <bigjools> rvba: should I be concerned that the ordering changed in test_views_nodes.py?
[09:16] <bigjools> oh you changed the code being tested.   sorry
[09:16] <rvba> bigjools: the ordering?  AFAIK I only changed the error messages.
[09:16] <bigjools> yeah my bad
[09:16] <bigjools> approved¬
[09:17] <bigjools> approved!
[09:17] <rvba> Ta
[09:17] <rvba> ¡Ta!
[09:21] <bigjools> ¡Olé!
[09:23] <rvba> allenap: I'll rebuild the package soon, if you want to land your lldp to be part of the build you can do so now (unless you want to tweak the UI some more before landing).
[09:25] <bigjools> https://code.launchpad.net/~julian-edwards/maas/release-notes-13.10/+merge/190061
[09:25] <bigjools> hello gmb
[09:25] <gmb> Hi bigjools.
[09:26] <gmb> bigjools: My account on Kanban appears to have been deactivated… I'm assuming that wash;'t you and am contacting their help desk, but just to check… ;)
[09:26] <allenap> rvba: I probably won't be ready in time.
[09:26] <bigjools> gmb: oh didn't anyone tell you yet?
[09:26]  * bigjools joking
[09:26] <gmb> Laugh? Nearly shat.
[09:27] <bigjools> allenap: can you check my release notes branch please?
[09:27] <allenap> bigjools: Sure.
[09:27] <bigjools> gracias
[09:27] <bigjools> or ¡gracias!
[09:28] <rvba> allenap: okay, no worries.  We can build a package quickly with https://code.launchpad.net/~julian-edwards/+recipe/maas-daily-saucy so just build a package when you land you branch and then we'll test it.
[09:28] <rvba> your* branch
[09:28] <bigjools> gmb: you're still marked as a board user for the maas board so that's all very odd
[09:29] <gmb> bigjools: Hmm… sounds like some kind of password / authentication snafu. ISTR this happening earlier this year to awd and domas on the green squad board, actually.
[09:30] <bigjools> ¿yay?
[09:33] <gmb> bigjools: If there were a phrase to accurately confer "We are about the only corporate users of that particular half-maintained web app," then I would use that. ¿yay? will have to do.
[09:34] <bigjools> I thought there were many users
[09:34] <bigjools> anyway time to put kids to bed, back in a bit
[10:16] <jtv> rvba: the new ephemerals script doesn't care.
[10:16] <jtv> It'll just take releases that are in the stream, and optionally filter those.
[10:17] <jtv> While maas-import-pxe-files won't care anymore because it  skips failing downloads.
[10:18] <rvba> jtv: okay, makes sense.
[10:25] <allenap> rvba: Do you think you'll have time to review https://code.launchpad.net/~allenap/maas/lldp-commissioning-results-in-ui/+merge/189663 again today?
[10:26] <rvba> allenap: sure… is it much different from the branch Jeroen reviewed?
[10:27] <allenap> rvba: Yes, it's almost all new.
[10:28] <rvba> allenap: okay, I'll have a look… could you provide a screenshot of the page?
[10:30] <rvba> jtv: allenap: could any of you take care of fixing ./etc/maas/import_pxe_files please?
[10:31] <jtv> rvba: by removing the RELEASES setting?
[10:31] <rvba> jtv: that or adding Saucy.
[10:31] <jtv> OK
[10:32] <jtv> Testing is going to be hard though.  :(
[10:32] <jtv> I have a large instance, but it's not large enough.
[10:32] <rvba> You need to symblink to /mnt/
[10:32] <rvba> symlink*
[10:32] <rvba> Plenty of space there right?
[10:32] <jtv> Oh, that worked for a "large"!?
[10:32] <rvba> Yeah.
[10:33] <jtv> Great.
[10:34] <jtv> Oh dear
[10:34] <jtv> I think our config conversion is broken...
[10:35] <rvba> :/
[10:35] <jtv> Need a different way to get the variables.
[10:36] <rvba> allenap: I'm a bit confused by merge_details_cleanly/merge_details.  Does the usage of merge_details_cleanly() mean the change in backward incompatible?
[10:37] <rvba> is*
[10:39] <allenap> rvba: http://people.canonical.com/~gavin/ui/lldp-commissioning-results-in-ui/ for screenshots.
[10:39] <rvba> Thanks.
[10:40] <allenap> rvba: merge_details_cleanly() generates the XML without using the lshw details as the document root (so that they're in there once without a namespace, and once in the 'lshw' namespace).
[10:41] <allenap> rvba: When tagging you get both, so that old expressions work.
[10:41] <allenap> rvba: But we want to encourage people to use only the namespaced XML from here on.
[10:41] <rvba> allenap: okay, thanks; makes sense.
[10:58] <allenap> rvba: Do you think I should add highlighting support to the XML? It's very easy with highlight.js, and looks good.
[10:58] <rvba> allenap: look like a nice addition, but probably not critical for today :)
[10:58] <rvba> allenap: I can use a little help with the testing ;)
[10:58] <allenap> rvba: Okeydoke :)
[11:02] <rvba> allenap: jtv seems to be busy with the migration stuff, if you could fix (and, most importantly test) ./etc/maas/import_pxe_files, that would be one thing off the list.
[11:02] <jtv> allenap, rvba — either of you willing to review a simple branch?  https://code.launchpad.net/~jtv/maas/use-unexported-config-vars/+merge/190089
[11:03] <jtv> Yeah, this is distracting me from the /etc/maas/import_pxe_files change I'm afraid.
[11:03] <jtv> I've got another thing to fix here.
[11:03] <rvba> jtv: I'll take it.
[11:03] <jtv> Merci beaucoup.
[11:07] <allenap> jtv: I have one small suggestion for the env branch, hang on...
[11:07]  * jtv hangs on
[11:10] <allenap> jtv: Done, on the mp itself.
[11:10] <jtv> OK
[11:10] <jtv> Thanks
[11:16] <allenap> rvba: Can I hit the build button again?
[11:16] <rvba> allenap: go! :)
[11:17] <jtv> allenap: I added the quoting, and will sleep better because of it.  But not the "&&," because AFAICT the script never ran with "-e" in the first place.
[11:18] <allenap> jtv: Okay, cool.
[11:18]  * allenap wonders why bash scripts don't default to -eu ...
[11:19] <jtv> Because dried grapes are hilarious.
[11:20] <allenap> jtv: Que?
[11:21] <jtv> Wait... what was that other word for dried grapes?
[11:21] <allenap> jtv: raisins... reasons...?
[11:21] <jtv> And I don't mean hilarious...  I mean that other word that's sometimes used instead.
[11:22] <allenap> jtv: depressing?
[11:22] <jtv> Normally means something like insane, loudly irrational...
[11:22] <jtv> Ah yes.  Hysterical.
[11:22] <jtv> Raisins are hysterical.
[11:23]  * allenap goes to have lunch because that's normal.
[11:23] <jtv> In Unix shell tradition, bash is an interactive shell.  You don't want it to exit on the first mistake.
[11:23] <jtv> Had it been developed purely as a batch shell, things might have been different...
[11:23] <jtv> dash is more like that.
[11:24] <jtv> In that it's not very accommodating to humans.
[11:25] <allenap> bash accommodates all human error.
[11:25] <jtv> Exactly.
[11:25] <jtv> Because it's meant to support interactive use.
[11:25] <jtv> Try running your terminals with "bash -e"...
[11:26] <jtv> Actually, that might be a good way to train novices.
[11:26] <jtv> Direct, merciless, effective.
[11:26] <jtv> Drop and give me 50!
[11:29] <rvba> roaksoax: we landed the branch that switches from the old script to the new script.  Testing is in progress but it's looking okay so far… could you test upgrading the package from older versions?  Something weird happened, I upgraded (using our daily ppa) from a package built one hour ago to a package built 10 minutes ago (the difference was only 2 revisions) and after the upgrade the 'maas-cluster-celery'
[11:29] <rvba> service was down.
[11:44] <jtv> allenap, rvba: another small branch needs reviewing...  If you'd be so kind: https://code.launchpad.net/~jtv/maas/load-immutable-config/+merge/190095
[11:44] <jtv> This is the other fix for the config conversion.
[11:54] <jtv> roaksoax: is maas-cluster-controller really supposed to depend on maas-dhcp?
[11:54] <jtv> Isn't that a Recommends?
[11:55] <rvba> jtv: I'll review your branch
[11:55] <jtv> Thanks.
[11:55] <jtv> Having a bit of trouble with the latest packaging branch though: can't install maas-cluster-controller without also installing the DHCP server!
[11:56] <rvba> I think roaksoax told me they wanted to make maas-dns and maas-dhcp mandatory and not optional.
[11:56] <rvba> roaksoax will confirm this.
[11:56] <jtv> Isn't that going to wreak havoc on people just trying it out in small networks?
[11:57] <rvba> No really, if you don't configure an interface to be "managed" the dhcp server won't be started.
[11:57] <rvba> Not* really
[11:57] <jtv> Phew.
[12:04] <jtv> Thanks allenap & rvba for your reviews.  With this landed, the config conversion works.
[12:10] <allenap> rvba: The package looks like it's used revno 1676, which is from yesterday. What am I missing?
[12:10] <rvba> 1.4+bzr1676+dfsg-0+1689+209~ppa0~ubuntu13.10.1
[12:11] <rvba> "1.4+bzr1676+dfsg-0" comes from the packaging.
[12:11] <rvba> 1689 is the upstream revision used.
[12:11] <rvba> 209 is the packaging revision.
[12:11] <allenap> rvba: No one would believe me if I said packaging is arcane :)
[12:12] <rvba> allenap: look at the recipe: it uses a funky pattern to create the package name.
[12:12] <rvba> allenap: {debupstream}-0+{revno}+{revno:packaging}~ppa0
[12:12] <allenap> rvba: And {debupstream} is where the weird bit comes from.
[12:13] <rvba> allenap: yeah, that comes from the packaging's first line of debian/changelog.
[12:15] <allenap> smoser: You're logged into the garage maas. Are you using it?
[12:18] <jtv> allenap, rvba: I'm done as of r1689.  Unless there's anything urgent that either of you need help with.
[12:18] <stokachu> smoser: filed bug 1237215
[12:21] <smoser> allenap, i have a node in use, why?
[12:22] <allenap> smoser: I want to test the latest daily package.
[12:22] <smoser> allenap, garage maas really isn't the place to do that.
[12:22] <allenap> jtv: Nope, I'm good :)
[12:22] <allenap> smoser: Why not?
[12:23] <smoser> because people expect for it to work.
[12:23] <rvba> jtv: I'm good too :).
[12:23] <smoser> cts people are expecting to be using that this week.
[12:24] <allenap> smoser: I thought the deal was: you put your name in the calendar, then it's yours, and you leave it in the state you found it (or working, if it wasn't).
[12:24] <allenap> smoser: My name's in the calendar :)
[12:25] <smoser> allenap, i don't really think "go crazy and put it back" is a sustainable plan on a shared resource.
[12:25] <allenap> smoser: What's it there for then?
[12:27] <smoser> use of maas.
[12:27] <smoser> not development of maas.
[12:27] <smoser> how will you put it back to the state which you got it ?
[12:27] <smoser> do you know the state that its in?
[12:27] <smoser> do the packages correctly downgrade?
[12:28] <smoser> how can you reasonably expect to "put it back the way you got it"
[12:28] <smoser> honest question.
[12:29] <matsubara> allenap, you can use the QA lab to do this sort of testing. it's supposed to be used like that, you use and throw away, next person  (or automated test) will re-build the whole thing anyway)
[12:30] <allenap> smoser: I've not used the garage MAAS in anger before, so I would expect someone to have documented this by now. I know I've seen it broken more times than I've seen it working, but that's perhaps because I get asked to look with it is broken.
[12:30] <allenap> s/with/when/
[12:30] <smoser> you've seen it broken because of what i'm saying above.
[12:31] <smoser> so my solution is not to let people break it.
[12:31] <rvba> matsubara: I'm currently testing things in the lab, the idea was to parallelize the testing.
[12:31] <matsubara> rvba, I see. ok then. I'll let you guys fight for the resource then :-)
[12:31] <smoser> i'm perfectly fine if we can come up with a "put it back into known state" button.
[12:31] <smoser> but we dont have that now
[12:31] <smoser> and others will expect to be able to use that shared resource.
[12:32] <smoser> i'd love a "redeploy garage maas" button.
[12:32] <caribou_> Has the functionality in 1.2 to add archives for newly provisioned machines been dropped in Maas 1.4 ?
[12:33] <smoser> rvba, https://bugs.launchpad.net/curtin/+bug/1237215
[12:34] <stokachu> caribou_: boot images you mean?
[12:34] <smoser> hm..
[12:34] <caribou_> stokachu: nope, there is a button in the config section to add entries to the sources.list; it's no longer there  in 1.4
[12:35] <stokachu> ah
[12:53] <rvba> smoser: is the curtin installer supposed to work on ARM nodes?
[12:53] <smoser> no.
[12:53] <rvba> Okay. Thanks.
[12:54] <smoser> it has no way of setting up bootloading on arm nodes (unimplemented).
[12:54] <smoser> rbasak, could add that.
[13:10] <allenap> rvba: MAAS supports one Juju environment per *user*, is that right, not per API key?
[13:11] <rvba> allenap: that's how it's supposed to work indeed.  But there is that bug we talked about recently in juju-core that makes the maas provider take down *all* the instances it can get his hands on when bootstrapping an env.
[13:12] <allenap> rvba: I'm thinking about the user preferences page, where it says "You'll need a separate MAAS key for each Juju environment".
[13:13] <allenap> rvba: That's not just misleading, that's wrong :-/
[13:13] <rvba> allenap: it's both indeed.
[13:13] <allenap> rvba: I shall file a bug for it.
[13:17] <smoser> allenap, i think that the real blocker is if you're admin its a global space
[13:18] <smoser> regular users can be separated, but if an admin tries to use juju, he'll wipe out all nodes
[13:19] <danwest> hi all, know today is going to be tough but I was curious if anyone has tested juju-core, maas tag constraints yet?
[13:19] <allenap> smoser: I still thought that an admin would only destroy those nodes allocated to him/her. The problem is that Juju destroys nodes that have been allocated independently of Juju.
[13:20] <allenap> danwest: Not yet.
[13:20] <danwest> my understanding is that for hostname placement you need to create a tag with the hostname in it
[13:20] <allenap> danwest: Speaking for me.
[13:20] <danwest> which just seems wrong
[13:20] <allenap> danwest: Eugh, I hope not.
[13:20] <danwest> allenap: going to ping juju folks to check if they know
[13:21] <smoser> allenap, well, i thought the bug was that if you juju bootstraped then juju would release all nodes that it could release.
[13:21] <smoser> and as admin that is "all nodes"
[13:21] <allenap> smoser: I'll check.
[13:27] <danwest> allenap: confirmed with juju-core folks that new maas constraints would indeed require a hostname tag to use hostname as a constraint
[13:29] <danwest> allenap: which means to make it 'just work' we would need to auto-gen a hostname tag but concerned that is unmaintainable
[13:29] <allenap> smoser: You're right. That's fairly awful.
[13:30] <allenap> danwest: It should be fixed in juju-core.
[13:31] <danwest> allenap: agreed but I fear there are arguments against it, at a minimum we need to document this behavior/requirement in maas release notes
[13:31] <allenap> rvba: In juju-core, maasEnviron.AllInstances (and therefore maasEnviron.instances) should not return all instances when handed an empty slice, it should return only allocated instances. Does that sound correct?
[13:32] <allenap> danwest: Yeah, there's nothing we can do to fix that in time for release, other than documenting it.
[13:32] <danwest> agreed, thx
[13:32] <allenap> danwest: Do you know who's looking after the release notes for MAAS?
[13:32] <danwest> allenap: thought it was bigjools, no?
[13:32] <allenap> danwest: Of course, yes. I'll email him.
[13:33] <danwest> roaksoax: ^^?
[13:33] <smoser> the key release note there is "do not be admin"
[13:33] <rvba> allenap: yes, but instances *allocated to this environment*.
[13:33] <smoser> one thing that sucks about "do not be admin" is that there is no way that i'm aware of to create a user that is non-admin
[13:33] <smoser> other than the GUI (which i dont consider useful)
[13:34] <allenap> rvba: One step at a time :) Firstly we just want to only select instances allocated to the calling user, not just all instances.
[13:35] <rvba> allenap: That's correct.
[13:35] <allenap> rvba: Cool. I might see about fixing that then.
[13:35] <roaksoax> danwest: bigjools is
[13:35] <danwest> roaksoax: thx
[13:37] <roaksoax> rvba: so ... did you guys land anything that would affect commissioning? i tested it yesterday with saucy on amd64 with no issues whatsoever
[13:38] <rvba> roaksoax: we didn't change anything in this area.
[13:38] <roaksoax> rvba: then maybe something with the image you used?
[13:39] <rvba> roaksoax: I don't know, I tested using a fresh instance and AFAIK, we check the md5sum of the images we download.
[13:39] <rvba> roaksoax: but I'm happy to test again.
[13:41] <roaksoax> rvba: sure! ill also test but did yesterday with my moonshot enablement and worked just fine
[13:43] <rvba> roaksoax: I just tested again, same result: "Failed tests,"
[13:44] <roaksoax> k
[13:45] <roaksoax> rvba: is there console output of the commissioning?
[13:45] <rvba> roaksoax: no
[13:46] <roaksoax> rvba: well i think it would be a good idea to obtain it
[13:46] <rvba> roaksoax: no sure how to do that in the lab
[13:46] <rvba> not*
[13:49] <roaksoax> rvba: maybe the new import ephemerals has something to do with it
[13:49] <rvba> roaksoax: possibly… let's see if you see the same problem…
[13:50] <stokachu> smoser: http://paste.ubuntu.com/6213826/ everything works except for the network_commands, i was hoping to just append to the existing /etc/network/interfaces but im assuming it doesn't work like that?
[13:50] <smoser> stokachu, well, theres not a lot of value in appending i dont think
[13:51] <smoser> but if you want, just don't override builtin
[13:51] <smoser> and make your entry run after builtin
[13:51] <smoser> i think you will run before builtin as it is, and it would overwrite yours.
[13:51] <stokachu> so would putting it in late_commands solve this?
[13:52] <smoser> have to look. i wish i'd have made 'builtin' '00builtin'
[13:53] <smoser> but i think if you just chainge '102_add_vlans' to 'c102_add_vlans' it'll be fine.
[13:53] <smoser> oh, and remove your overwriting of builtin
[13:53] <stokachu> ok ill try both ways and see what happens
[13:55] <smoser> stokachu, you can definitely do it in late_commands.
[13:56] <smoser> that paste bin doens't look right hough
[13:56] <smoser> you dont have a clsing ] on 'maas:'
[13:57] <stokachu> smoser: yea it chopped it off when copying from vi
[13:57] <stokachu> i verified its there
[14:05] <rvba> roaksoax: btw, we need to re-add "saucy" in etc/maas/import_pxe_files.
[14:05] <roaksoax> yup
[14:08] <stokachu> smoser: here is what is currently working http://paste.ubuntu.com/6213897/
[14:08] <stokachu> it adds the vlan information into /etc/network/interfaces now
[14:10] <smoser> i wish i had done a archive format for config.
[14:10] <smoser> such that you could put your config changes separate from maas's stuff.
[14:10] <smoser> so you didn't have to modify maas's late_command
[14:10] <smoser> but could just add your own
[14:11] <smoser> you can do that with multiple configs, but you only have one  config here.
[14:11] <stokachu> yea so far just one
[14:18] <smoser> what i should have done (and will do in the future) is support an archive format for config.
[14:18] <smoser> ie, 'cloud-config-archive' format.
[14:18] <smoser> so that woudl be:
[14:18] <smoser> #curtin-config-archive
[14:18] <smoser> - |
[14:18] <smoser>   maas stuff here
[14:18] <smoser> - |
[14:18] <smoser>   stokachu's stuff here
[14:18] <stokachu> that would be sweet
[14:20] <smoser> jamespage, are you around ?
[14:38] <AskUbuntu> MaaS dhcp cannot be detected at node installation | http://askubuntu.com/q/355847
[14:46] <smoser> roaksoax, let me know if there is anything i can do to help you
[14:47] <roaksoax> smoser: will do once I give this a test :)
[14:47] <roaksoax> rvba: ok, so i'm building a package I;'ll use to test the moonshot stuff + the commissioning stuff
[14:47] <roaksoax> rvba: other than that, how many more branches are we expecting to land?
[14:49] <roaksoax> rvba: ok this is definitely something new and weird
[14:49] <roaksoax> in my test cluster from yesterday i had machines commissioned
[14:49] <roaksoax> all fine
[14:49] <roaksoax> now I look at show "failed tests"
[14:58] <roaksoax> smoser: "ci-info: no authorized ssh keys fingerprints found for user ubuntu.ec2:" ??
[14:58] <smoser> probably not a big deal.
[14:58] <smoser> thats ephenmeral boot?
[14:58] <smoser> or install
[14:59] <roaksoax> smoser: yeah, commissioning.. and commissioning fails
[14:59] <smoser> well thats unrelated to commissionoing failure
[14:59] <smoser> its just telling you that the user it configured did not have ssh keys
[14:59] <smoser> ie, you cant get in here :)
[14:59] <roaksoax> well then that's probably why
[14:59] <roaksoax> it is failing
[14:59] <roaksoax> so something must have changed...?
[15:05] <rvba> roaksoax: the only thing we need to fix is /etc/maas/import_pxe_files.
[15:05] <roaksoax> rvba: ok
[15:05] <rvba> roaksoax: the cleanest way to do it is probably to remove the RELEASE variable entirely.  This way, we will download all the supported releases (the script now copes with missing images).
[15:06] <roaksoax> rvba: if the scripts do that then it would be fine, but keep in mind it might not work on precise (since precise maybe doesn't know that saucy is a relese yet but rather development)
[15:06] <roaksoax> smoser: ok
[15:07] <roaksoax> smoser: so I see this issue again
[15:07] <roaksoax> ci-info: no authorized ssh keys fingerprints found for user ubuntu.ec2:
[15:07] <roaksoax> the machine failed to enlist
[15:07] <smoser> that is unrelated
[15:07] <rvba> roaksoax: you're talking about the backport right?
[15:07] <smoser> i dont know why you're not enlisting. but it has nothing to do with the fact that the user has no keys.
[15:08] <roaksoax> smoser: that's what I'll check in a bit
[15:09] <roaksoax> rvba: yup
[15:09] <rvba> roaksoax: good point, I'll just add "saucy" to the list then.
[15:17] <rvba> roaksoax: on second thoughts, aren't we support to SRU distro-info everytime a series is released?
[15:17] <rvba> roaksoax: I mean distro-info --supported on precise currently returns l/p/q/r/s
[15:17] <roaksoax> rvba: we do
[15:17] <roaksoax> but what if it is not therE?
[15:18] <roaksoax> wouldn't be the first time
[15:18] <roaksoax> :)
[15:18] <roaksoax> so I think we should go through the safe path
[15:18] <roaksoax> either way i don't mind
[15:18] <rvba> What do you mean "it is not there"?
[15:18] <roaksoax> just dont remove RELEASES
[15:18] <roaksoax> just comment it out
[15:18] <rvba> If the image is not there?
[15:18] <roaksoax> rvba: yeah if the image is not there it will fail
[15:19] <rvba> roaksoax: no, the script deals gracefully with that now
[15:19] <roaksoax> ok
[15:19] <rvba> I'll test it, just for safety, and propose a fix.
[15:20] <roaksoax> rvba: ok, so is enlisting also failing for you?
[15:20] <roaksoax> rvba: i haven't even upgraded to the latest maas
[15:20] <roaksoax> and enlist fails
[15:21] <rvba> roaksoax: I haven't tested enlisting using saucy
[15:22] <roaksoax> rvba: could you also try and see if it fails
[15:22] <rvba> roaksoax: ok
[15:30] <rvba> roaksoax: I just got 4 nodes enlisted using Saucy's (commissioning) image.
[15:31] <roaksoax> rvba: ok, try to commission them
[15:31] <rvba> roaksoax: that is the part that will fail…
[15:32] <roaksoax> still failing?
[15:33] <rvba> …still commissioning…
[15:33] <roaksoax> i can't even enlist a node... this is completely weird
[15:34] <rvba> roaksoax: 4 "failed tests"
[15:36] <rvba> roaksoax: https://code.launchpad.net/~rvb/maas/re-enable-saucy/+merge/190159
[15:40] <roaksoax> rvba: ok maas-import-ephemerals doesn't seem to be working either
[15:40] <rvba> roaksoax: !
[15:42] <roaksoax> rvba: i run maas-import-pxe-files
[15:42] <roaksoax> and only downloads the pxe images
[15:42] <roaksoax> and that's it
[15:43] <rvba> roaksoax: I'm running it in a canonistack instance right now and I'm seeing it installing the ephemeral images…
[15:43] <rvba> roaksoax: I'm using the daily package from our ppa
[15:43] <roaksoax> hdl@vm18-2:~/roaksoax/backdoor-image$ sudo maas-import-ephemerals
[15:43] <roaksoax> hdl@vm18-2:~/roaksoax/backdoor-image$
[15:44] <roaksoax> rvba: and I upgraded
[15:44] <roaksoax> yours might be a clean install?
[15:44] <rvba> roaksoax: yes, clean install
[15:45] <roaksoax> ok so there's something there that needs to be fixed for an upgrade maybe
[15:47] <roaksoax> rvba: ok, you will need to connect to the image using ssh
[15:47] <roaksoax> rvba: but you will need to enable the user/pass for it
[15:47] <roaksoax> kinda following https://lists.launchpad.net/maas-devel/msg00808.html
[15:48] <roaksoax> \but since I don't know where the images are stored now
[15:48] <roaksoax> you'll need to figure it out
[15:48] <roaksoax> rvba: and is this only in saucy? other releases works fine?
[15:48] <rvba> roaksoax: yes
[15:52] <roaksoax> rvba: ok, so just try to connect to the image
[15:52] <roaksoax> and block is power off
[15:52] <roaksoax> and we can look from there
[15:52] <roaksoax> because here i can't even
[15:52] <roaksoax> obtain the ephemerals
[15:57] <roaksoax> rvba: is there no output when i run maas-import-ephemerals?
[15:57] <roaksoax> rvba: if I ctrl+c the import
[15:57] <roaksoax> i think it wont try to run it again
[15:57] <rvba> roaksoax: there is usually plenty of output
[15:57] <roaksoax> rvba: not for me i guess
[15:57] <roaksoax> i don't see anything
[15:57] <roaksoax> rvba: and these are upgrades
[15:57] <roaksoax> so there's something messed up real bad
[15:58] <roaksoax> i thought we had agreed that ephemerals config file would be *separated* from pserv.yaml?
[15:58] <rvba> roaksoax: I think Julian decided otherwise
[15:59] <roaksoax> bigjools: ^^
[16:02] <roaksoax> rvba: show me a pserv.yaml with various releases enabled
[16:03] <roaksoax> i really don't like this
[16:03] <roaksoax> -_-'
[16:05] <rvba> roaksoax: I don't have one handy (because I always install from fresh) but it should contain something like: http://paste.ubuntu.com/6214348/
[17:09] <allenap> roaksoax: I'm working on a bson encoding problem that's preventing tags from being evaluated. I have to feed my kids now, but I'll be back to work on it soon. When has this got to be done by?
[17:11] <roaksoax> allenap: tonight :)
[17:11] <allenap> roaksoax: Cool :)
[17:35] <adam_g> roaksoax, http://people.canonical.com/~agandelman/failed_metadata.png any iea?
[17:35] <adam_g> this is during comissioning
[17:35] <roaksoax> adam_g: we are looking at commissioning issues right now...
[17:36] <roaksoax> adam_g: did you upgrade or just magically started happening?
[17:36] <roaksoax> smoser: ^^
[17:37] <adam_g> roaksoax, upgraded on monday and added two new declared nodes, those are not comissioning
[17:38] <smoser> what OS booted for commissioning?
[17:38] <roaksoax> adam_g: ok so i had nodes commissioning/enlistment working yesterday and today they magically stopped working, so might be related. I was using saucy for enlistment/commissioning though
[17:38] <adam_g> http://paste.ubuntu.com/6214708/ is from apache log
[17:39] <adam_g> (err thats maas log)
[17:40] <adam_g> smoser, trying to figure out what booted, by logs
[17:46] <adam_g> roaksoax, hmm. the node appeared to be commissioned okay even with those errors. appears ready in MAAS
[17:48] <roaksoax> adam_g: uhmm is ipmi discovered? are the cpu's/ram/etc being displayed?
[17:52] <adam_g> roaksoax, yup
[17:52] <roaksoax> adam_g: then i guess it works just fine then :/
[17:58] <AskUbuntu> how to stop entire openstack so that i can run maas server on my localhost | http://askubuntu.com/q/355914
[18:15] <smoser> adam_g, thats your clock is broken
[18:15] <smoser> and cloud-init was adjusting its timestamps to fix oauth failed
[18:16] <jamespage> smoser, I am now - sorry - no internet access all day
[18:17] <smoser> :)
[18:34] <AskUbuntu> how to run openstack and maas server on localhost together | http://askubuntu.com/q/355931
[18:50] <roaksoax> rvba: let me know when around
[18:51] <roaksoax> [    0.000000] Command line: nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:maas-precise-12.04-amd64-ephemeral-20121008 iscsi_target_ip=<IP_ADDRESS> iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=:::
[18:51] <roaksoax> :maas-enlist:BOOTIF ro root=/dev/disk/by-path/ip-<IP_ADDRESS>:3260-iscsi-iqn.2004-05.com.ubuntu:maas:maas-precise-12.04-amd64-ephemeral-20121008-lun-1 overlayroot=tmpfs cloud-config-url=http://<IP_ADDRESS>/MAAS/metadat
[18:51] <roaksoax> a/latest/enlist-preseed/?op=get_enlist_preseed log_host=<IP_ADDRESS> log_port=514 console=ttyS0,9600 initrd=amd64/generic/precise/commissioning/initrd.gz BOOT_IMAGE=amd64/generic/precise/commissioning/linux BOOTIF=0
[18:51] <roaksoax> 1-38-ea-a7-0f-6e-ba
[18:51] <roaksoax> err
[18:51] <roaksoax> sorry
[19:08] <roaksoax> rvba: ok so i think we are in big trouble...
[19:08] <roaksoax> allenap: ^^
[19:09] <roaksoax> rvba: so basically, I selected saucy to be the release for commissioning, first it PXE booted into precise.., and when I noticed that I rebooted the machine, and it then pxe booted into saucy.
[19:09] <allenap> roaksoax: So... what are we talking about?
[19:10] <roaksoax> rvba: once enlisted, i told it to commission... however, it PXE booted into precise again (when it should have been saucy, and obviously commissinnig failed) so I told it to PXE boot again and it now did on saucy
[19:10] <roaksoax> rvba: and commissioned in saucy, but with failed tests
[19:11] <roaksoax> allenap: simple enlistment/commissioning
[19:12] <allenap> roaksoax: Is the problem that the commissioning series keeps changing?
[19:13] <roaksoax> allenap: yes, but apart from that commissioning finishes "successfully" but shows Failed Tests on the maas webui
[19:13] <roaksoax> which is what rvba reported in the morning
[19:16] <roaksoax> allenap: so if I tell maas to enlist/commission with saucy, in the first attempt uses precise and obviously fails to do it
[19:16] <roaksoax> allenap: so I reboot the node, and the second attempt does use saucy
[19:16] <roaksoax> which works and enlists/commissions
[19:16] <allenap> roaksoax: How do you tell it to commission with saucy? Via the global option?
[19:17] <roaksoax> allenap: yeah, on the webui settings "Default distro series used for commissioning"
[19:17] <roaksoax> allenap: which is what i always have used
[19:17] <roaksoax> allenap: and never had problems with
[19:17] <roaksoax> allenap: so this is something recent... from yesterday to today
[19:18] <allenap> roaksoax: Very odd :-/
[19:18] <allenap> roaksoax: I'll look soon; right now I'm this >< close to figuring out a weird bson issue.
[19:18] <roaksoax> probably related to maas-import-ephemerals, which is also being problematic
[19:20] <allenap> roaksoax: Is it going to be a problem to prevent python-bson-ext from being installed at the same time as maas?
[19:21] <allenap> It's a C module that behaves weirdly in Apache+WSGI.
[19:28] <roaksoax> allenap: maybe, we cannot control ordering of installation
[19:40] <allenap> roaksoax: See https://bugs.launchpad.net/maas/+bug/1237615 for the problem. Right, I'll look at this commissioning issue now.
[20:06] <roaksoax> allenap: ok, I;ll have a look in a bit
[20:06] <allenap> roaksoax: Thanks.
[20:19] <stokachu> oo we should do a responsive layout so people can manage their servers from their phone/tablet!
[20:49] <allenap> roaksoax: I think you've set the wrong target branch on https://code.launchpad.net/~andreserl/maas/maas_saucy_fix_commissioning/+merge/190239... I'm not reviewing 107234 lines :)
[20:51] <roaksoax> allenap: lol let me see
[20:54] <roaksoax> allenap: https://code.launchpad.net/~andreserl/maas/maas_saucy_fix_commissioning/+merge/190244
[20:54] <roaksoax> done
[20:54] <roaksoax> that should fix the failed trsts stuff
[20:54] <roaksoax> but not the pxe booting intonprecise when it shpuld be saucy
[20:58] <allenap> roaksoax: Yeah, I don't understand that bit at all.
[20:59] <roaksoax> allenap: ill look at it
[20:59] <roaksoax> thanks though!
[21:00] <allenap> roaksoax: Credit to you. I'm off now, I'm going to use my last energy to get upstairs to bed.
[21:02] <roaksoax> lol goodnight
[21:46] <rvba> roaksoax: I tested your fix for the commissioning issue and it worked!  Congratulations for fixing this!
[22:28] <bigjools> morning
[23:10] <roaksoax> bigjools: howdy!! regression day fixing today :)
[23:23] <roaksoax> bigjools: do you know anything about the bson issue allenap ? i don't seem to understand fully
[23:23] <roaksoax> since it seems we don't even depend on bson
[23:24] <bigjools> roaksoax: I have no idea and I'm still catching up on email so don't know if I was told about it
[23:25] <roaksoax> bigjools: do you know when python-bson was introduced as a dependency to maas and why the packaging was never updated to depend on it?
[23:26] <bigjools> roaksoax: not entirely sure, I think it was in the last few weeks
[23:27] <roaksoax> uhmmm
[23:27] <bigjools> roaksoax: oh looks like allenap added it recently for the lldp work
[23:28] <bigjools> sorry I am going to be a bit slow today, not feeling well
[23:30] <roaksoax> bigjools: np.. he did leave this bug #1237615 before he left
[23:30] <roaksoax> but I don't fully understand it
[23:30] <bigjools> lookng
[23:32] <stokachu> should we break up http://maas.ubuntu.com/docs/development/preseeds.html into sub-headings for Debian Installer and Curtin installer?
[23:33] <stokachu> the documentation there now doesn't apply to curtin at all since preseed_xinstall doesnt exist
[23:35] <stokachu> or maybe even do a new page dedicated to curtin
[23:37] <stokachu> hmm, even have a heading in the TOC as like 'Node Enlistment, Commission, and Install'
[23:37] <bigjools> stokachu: yeah it needs updating
[23:37] <stokachu> bigjools: im just putting my thoughts out here as im going to take a stab at updating that part of the doc
[23:38] <bigjools> thank you, that would be great
[23:38] <stokachu> bigjools: if i make some doc changes within the TOC will that upset anyone or would it be open for discussion in an MP?
[23:39] <bigjools> roaksoax: it looks like wsgi is possibly compiled differently to python
[23:39] <bigjools> or some mismatch somewhere at least
[23:39] <bigjools> stokachu: please agree on changes with us before starting
[23:39] <bigjools> ie suggest something and we'll review before you commit to something
[23:40] <stokachu> bigjools: do you want that in a bug or just here on irc
[23:40] <bigjools> stokachu: bug would be ideal, tag it with "doc"
[23:40] <stokachu> bigjools: sounds good ill do that
[23:40] <bigjools> great, thanks very much
[23:40] <stokachu> np
[23:42] <bigjools> roaksoax: http://emptysqua.re/blog/python-c-extensions-and-mod-wsgi/
[23:42] <bigjools> that might fix it
[23:43] <roaksoax> bigjools: right, so we would need to test that
[23:43] <roaksoax> too late now I think to take care of that
[23:43] <bigjools> roaksoax: has the upload deadline passed?
[23:44] <roaksoax> bigjools: kinda... but I'm uploading now
[23:44] <bigjools> heh
[23:44] <bigjools> sadly allenap did not give us a direct way to re-create the problem in the bug so I don't know what triggers it
[23:45] <roaksoax> bigjools: exactly :)
[23:45] <roaksoax> bigjools: but I just noticed that python-bson wasn't even a depends for maas
[23:45] <roaksoax> package
[23:45] <roaksoax> so I just added it
[23:45] <bigjools> :(
[23:45] <roaksoax> so we will see if an install of that causes any issues
[23:48] <roaksoax> bigjools: my concern now is that if I add the depends on python-bson it might trigger the issue
[23:52] <bigjools> roaksoax: probably
[23:53] <bigjools> roaksoax: the bug talks about python-bson-ext though


[00:39] <infinity> nacc: Huh.  dh-php didn't survive long.  Is there a new hotness now?
[00:39] <infinity> nacc: Or is it just that we don't ship any extensions in main?
[00:40] <infinity> nacc: (demoted, though)
[00:41] <nacc> infinity: yeah :) ... not necessarily anything new, but it's only a build-time dependency for most packages
[00:42] <nacc> infinity: thanks! that will help clear up some stuff in proposed
[00:49] <nacc> infinity: i'll try and check back after dinner, i ran those steps locally and the final result was what I expected (and all three passed tests at the end)
[00:50] <nacc> infinity: will that automatically get picked up by update_excuses? dh-php is currently stuck as an unsatisfiable dep (xml2) due to the old component mismatch. Or does someone need to poke it?
[01:11] <infinity> nacc: It'll sort itself out.
[01:27] <nacc> infinity: ok, great, thanks!
[03:59] <slangasek> lputils.PackageMissing: Could not find binaries for 'libuhnspell-1.3-0/None' in yakkety
[03:59] <slangasek> oh the irony
[07:16] <LocutusOfBorg> please RM pepperflashplugin-nonfree on i386 thanks (following the upstream and debian removal)
[07:17] <LocutusOfBorg> also vmtk on powerpc
[20:36] <jbicha> could the status on https://launchpad.net/ubuntu/vivid be set from "supported" to "obsolete"
[21:04] <cjwatson> jbicha: uh it's complicated
[21:05] <cjwatson> jbicha: (short answer: sorry, no, not yet, basically due to phones and snappy 15.04, but hopefully in the not too distant future)
[21:05] <jbicha> ok, I spoke up in case it was just an oversight
[21:05] <cjwatson> not an oversight, it's definitely annoying that we can't right now