[16:00] <danilos> me
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <sinzui> me
[16:01] <matsubara> who's here today?
[16:01] <stub> yo
[16:01] <danilos> me
[16:01] <flacoste> me
[16:01] <matsubara> bigjools, intellectronica, rockstar, Ursinha ?
[16:01] <bigjools> meeeee
[16:01] <Ursinha> me
[16:02] <matsubara> hmm I think rockstar is flying to London, isn't he?
[16:02] <flacoste> yep
[16:02] <matsubara> ok, let's move on without Code and Bugs
[16:02] <matsubara> [TOPIC] Agenda
[16:02] <MootBot> New Topic:  Agenda
[16:03] <matsubara>  * Next meeting
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs
[16:03] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:03] <matsubara>  * DBA report (DBA contact)
[16:03] <matsubara>  * QA Stats of the week
[16:03] <matsubara> [topic] next meeting
[16:03] <MootBot> New Topic:  next meeting
[16:03] <matsubara> so, no meeting next week?
[16:03] <Ursinha> matsubara, will be live!
[16:03] <danilos> or live meeting?
[16:03] <mpt> We'll do it live!
[16:03] <bigjools> oh man
[16:04] <bigjools> our secret identities will be blown
[16:04] <matsubara> forgot the LOSA
[16:04] <stub> You mean I have to wear pants??
[16:04] <matsubara> herb: around?
[16:04] <herb> me
[16:04] <Ursinha> lol
[16:04] <danilos> stub: that's never been necessary for live meetings
[16:04] <matsubara> ok, will any of the LOSAs attending the Epic?
[16:04]  * sinzui will still be in his jim-jams
[16:04] <danilos> matsubara: I don't think the timezone suits them :)
[16:04] <herb> matsubara: I don't think so, no.
[16:04] <danilos> herb: really?
[16:05] <herb> danilos: I know I'm not.
[16:05] <matsubara> ok, we can have the meeting there then and ask one of the LOSAs to join us
[16:05] <herb> I'm fairly certain spm isn't.  That only leaves mthaddon and he hasn't mentioned anything to me about it.
[16:06] <matsubara> i guess he's not going by his reply to mwhudson_ climbing email
[16:06] <flacoste> mthaddon isn't coming
[16:06] <flacoste> matsubara: right
[16:06] <matsubara> we'll sort it out there and let you know herb
[16:07] <matsubara> [topic] * Actions from last meeting
[16:07] <MootBot> New Topic:  * Actions from last meeting
[16:07] <matsubara> * Ursinha to keep an eye on OOPSes regarding bug 277129, to confirm that the bug is really fixed.
[16:07] <flacoste> actually
[16:07] <flacoste> that fix was only deployed to staging
[16:07] <Ursinha> yes
[16:07] <flacoste> so that watch was meaningless
[16:07] <Ursinha> yes again
[16:07] <flacoste> i can say now that it is fix
[16:07] <flacoste> fixed
[16:07] <flacoste> but that we have a fall-out for it :-)
[16:08] <flacoste> the DoomedTransaction OOPS you saw this morning
[16:08] <matsubara> the other two items are for intellectronica
[16:08] <flacoste> and that gary is working on
[16:09] <matsubara> and another item for me, which i did (matsubara to find or file a new bug for the core dump issue and add info discussed today there, subscribe niemeyer, kiko and someone from ubuntu
[16:09] <matsubara> )
[16:09] <matsubara> thanks flacoste
[16:09] <flacoste> good progress has been made on that one
[16:09] <flacoste> it was in reviewed last time i checked
[16:10] <matsubara> great
[16:10] <matsubara> I guess I'm not receiving notifications for it
[16:11] <matsubara> maybe I forgot to subscribe myself
[16:11] <matsubara> anyway, moving on
[16:11] <flacoste> i,m not sure there was progress report on the bug
[16:11] <matsubara> flacoste: hmm, I'll update it after the meeting then.
[16:12] <matsubara> [action] matsubara update core dump bug with progress made last week
[16:12] <MootBot> ACTION received:  matsubara update core dump bug with progress made last week
[16:12] <matsubara> thanks
[16:12] <matsubara> [topic] * Oops report & Critical Bugs
[16:12] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:12] <herb> flacoste, matsubara what was the bug number for that so I can subscribe myself?
[16:12] <matsubara> Ursinha: floor is yours
[16:12] <matsubara> herb: let me look
[16:12] <Ursinha> okay
[16:13] <Ursinha> 7 bugs, not sure if it'll be pointless but will list all them anyway
[16:13] <Ursinha> 3 for foundations: bug 284458, 284418, 284403
[16:13] <Ursinha> 1 for shipit: 284448, salgado already told that it's only caused when users handcraft URLs, need to set priority on it
[16:13] <Ursinha> 1 for translations: 284429
[16:13] <Ursinha> 1 for soyuz: 264319
[16:13] <Ursinha> oops
[16:13] <Ursinha> bug 284448
[16:13] <Ursinha> bug 284429
[16:14] <Ursinha> bug 264319
[16:14] <Ursinha> 1 for bugs: bug 186262
[16:14] <bigjools> I'll take 264319
[16:14] <matsubara> flacoste: can you comment on 284458?
[16:14] <Ursinha> bigjools, thanks
[16:15] <danilos> Ursinha: we'll look into 284429
[16:15] <matsubara> the other two from foundations are being diagnosed/addressed already
[16:15] <Ursinha> danilos, thanks
[16:15] <flacoste> matsubara: that's a bug for SOyuz
[16:15] <danilos> Ursinha: jtv was looking at it just before he left
[16:15] <flacoste> it's not API code
[16:15] <flacoste> it's soyuz code invoked through the API
[16:15] <bigjools> flacoste: is it just a case of returning a different exception?
[16:15] <flacoste> yep
[16:15] <flacoste> bigjools: NotFound
[16:15] <bigjools> ok, easy
[16:16] <matsubara> flacoste: right, I thought the NotFoundError was a generic thing
[16:16] <bigjools> I got that one too Ursinha
[16:16] <flacoste> not in this case
[16:16] <matsubara> all right, thanks flacoste
[16:16] <Ursinha> bigjools, ok, I'll take not
[16:16] <Ursinha> bigjools, do you want me to assign it to you?
[16:16] <bigjools> Ursinha: I'll do it right now when I target it
[16:16] <flacoste> 284418 is something to fix when we deploy
[16:16] <flacoste> i assigned it to canonical-losas
[16:16] <Ursinha> bigjools, ok, thanks
[16:16] <flacoste> i think Tom is the one who would fix it
[16:17] <matsubara> herb: bug 95822
[16:17] <herb> matsubara: thanks.
[16:17] <flacoste> (we need to remove the .pyc after renaming the directory and before starting the app server)
[16:17] <flacoste> gary is handling 284403
[16:18] <flacoste> and that's the fall-out from the fix to the timeout-related exception
[16:18] <matsubara> I guess that's all Ursinha ?
[16:19] <Ursinha> salgado, what about bug 284448?
[16:19] <sinzui> I'm not certain about the priority
[16:19] <matsubara> Ursinha: salgado already said it's low priority and he's not supposed to attend this meeting :-)
[16:19] <Ursinha> matsubara, old habit :) I'll set the bug to low
[16:20] <Ursinha> sinzui, why?
[16:20] <sinzui> It interferes with bookmarks. We should be redirecting in the example given.
[16:21] <sinzui> The 404 page is needed, but the users who have used shipit.edubuntu should know they can access their information at shipit.ubuntu
[16:21] <Ursinha> sinzui, can you set the proper priority on that, please?
[16:22] <sinzui> I'm not certain how many users the problem really affect to know whether we should be redirecting
[16:22] <sinzui> Ursinha: I think seeing the frequency of oopses help me to know the priority
[16:22] <Ursinha> sinzui, so we should keep it low and see if it happens more often?
[16:22] <matsubara> Ursinha: bring it up again if the OOPS count for that one increases too much
[16:23] <Ursinha> ok, I'll feed the bug with more oopses if they happen
[16:23] <Ursinha> and bring it here
[16:23] <Ursinha> there's only the bugs' bug left
[16:23] <sinzui> Ursinha: I would definitely make it high if this happens almost every day...but I think in this case the problem is that the user need to get to shipit.ubuntu
[16:23] <matsubara> sinzui: so far we had like 3 OOPSes, so doesn't seem to happen often
[16:24] <matsubara> intellectronica: around?
[16:24] <Ursinha> intellectronica, bug 186262
[16:24] <intellectronica> matsubara: yes
[16:24] <Ursinha> intellectronica, hi :)
[16:24] <Ursinha> oh my
[16:24] <Ursinha> intellectronica, https://launchpad.net/bugs/186262
[16:25] <intellectronica> ouch
[16:25] <intellectronica> wow, this is quite old, is it a comeback?
[16:26] <matsubara> intellectronica: yeah, looks like. mthaddon tried to file a bug recently and got a timeout
[16:26] <sinzui> Is it really High if the bug is old?
[16:26] <Ursinha> intellectronica, yes, it seems
[16:26] <Ursinha> I've seen it lately
[16:26] <Ursinha> the oopses, that is
[16:26] <intellectronica> Ursinha: ok, assigning to myself and will investigate
[16:26] <mthaddon> matsubara, I know someone else got it recently too (a non-canonical employee)
[16:26] <Ursinha> intellectronica, ok, thanks
[16:27] <Ursinha> nothing else from me, then
[16:27] <intellectronica> Ursinha: is this something that happens consistently?
[16:27] <Ursinha> intellectronica, I was able to reproduce
[16:27] <matsubara> intellectronica: yes, depending on what you use as the bug summary
[16:27] <intellectronica> anyway, we can discuss later
[16:27] <matsubara> and it's workaroundable
[16:27] <matsubara> so, I guess that's why people are not complaining so loudly :-)
[16:27] <matsubara> thanks for taking it intellectronica
[16:28] <matsubara> we have only 1 critical bug which is fix committed
[16:28] <Ursinha> no critical bugs, also, besides one fix committed
[16:28] <matsubara> actually should be fix released by now
[16:29] <Ursinha> thanks guys, you can move on matsubara
[16:29] <matsubara> thanks everyone
[16:29] <matsubara> [topic] * Operations report (mthaddon/herb/spm)
[16:29] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:29] <herb> * Monday (2008-10-13): Did a cherry pick to lpnet* to update the shipit pages for intrepid.
[16:29] <herb> * Last night (2008-10-16): Rolled out 2.1.10
[16:29] <herb> * Over the last week we've had to restart codebrowse 7 times because it was unresponsive.
[16:30] <matsubara> very well executed rollout, btw. Congrats! 21 min downtime only
[16:30] <herb> I'll look into whether or not there are any bugs filed for the codebrowse issue.  if not I'll take care of that after the meething.
[16:30] <matsubara> [action] herb to find or file a bug about codebrowse being unresponsive and needing restarts
[16:30] <MootBot> ACTION received:  herb to find or file a bug about codebrowse being unresponsive and needing restarts
[16:30] <herb> matsubara: lifeless and mthaddon worked out a change in procedure.  really seemed to help
[16:31] <mthaddon> we've actually had a 21 minutes rollout before (1.2.4), but I definitely agree the new rollout method is much better and quicker)
[16:32] <matsubara> herb: indeed. the only downside is that oops reports are less informative now, but flacoste will email you guys about it
[16:32] <flacoste> that's done
[16:32] <matsubara> cool
[16:32] <matsubara> anything else for herb?
[16:32] <flacoste> are we having a re-roll?
[16:32] <herb> I haven't heard anything about that.
[16:32] <herb> doesn't mean we aren't
[16:32] <flacoste> matsubara, Ursinha?
[16:32] <flacoste> what's your feeling?
[16:33] <flacoste> personnally, candidate for it would only be for the DoomedTransaction problem
[16:33] <mthaddon> OOPS reports less informative?
[16:33] <matsubara> flacoste: it'd be nice to have the oops thing fixed and I guess we don't need to re-roll for that, right?
[16:33] <flacoste> matsubara: just restart
[16:33] <mthaddon> is this because we make build in another directory, or unrelated?
[16:33] <matsubara> the other issues so far don't look that critical
[16:33] <flacoste> mthaddon: it is, i sent you an email about it
[16:33] <matsubara> in any case we'll be checking the oops summaries the whole day, and if anything comes up, we'll let you know
[16:34] <flacoste> DoomedTransaction problem is the ghost of the old timeout problem
[16:34] <flacoste> so would happen as often as it
[16:34] <mthaddon> flacoste, ok, see the email here
[16:34] <flacoste> we could cherry-pick the fix
[16:34] <flacoste> or re-roll
[16:34] <Ursinha> flacoste, well, so far I guess that only a CP would be enough
[16:35] <flacoste> since we are still in r-c
[16:35] <flacoste> mode
[16:35] <flacoste> i think it's easier for the LOSA to do a re-roll than a CP
[16:35] <flacoste> mthaddon, herb: is that right?
[16:35] <sinzui> flacoste: the old timeout problem as in bug #244957?
[16:35] <herb> it's easier on pqm. :)
[16:35] <herb> if we re-roll we don't need to run the test suite on the release+patch.
[16:35] <flacoste> matsubara: no as in bug 277129
[16:36] <matsubara> sinzui: ^
[16:36] <flacoste> herb, mthaddon: what time would be good for a re-roll today?
[16:37] <mthaddon> flacoste, any time is fine - re-roll to where?
[16:37] <flacoste> app-servers
[16:37] <mthaddon> flacoste, yeah, anytime is fine
[16:37] <stub> After the fix exists might be best though
[16:37] <flacoste> :-)
[16:37] <mthaddon> flacoste, sooner the better, but really no difference
[16:37] <mthaddon> and what he said :)
[16:38] <flacoste> so unless anything else comes up, we can target a re-roll after gary's land his fix
[16:38] <matsubara> deal
[16:38] <flacoste> which should happen later this afternoon
[16:39] <matsubara> so, anything else for LOSAs?
[16:39] <matsubara> thanks herb, mthaddon
[16:39] <matsubara> moving on
[16:39] <matsubara> [topic] * DBA report (DBA contact)
[16:39] <MootBot> New Topic:  * DBA report (DBA contact)
[16:39] <stub> Load on the db seems to have decreased a little after the rollout, although it is too early to tell for sure. Hopefully this is a sign of good work and not broken systems :)
[16:39] <stub> The replicated staging db rebuild work has been done and awaiting review. I don't see any problems getting this landed and switched on Monday, time in London permitting.
[16:39] <stub> Also in London, we need to find time with Mark to discuss some of the db design that was bounced this cycle (bzr Job system, milestone/productrelease linking).
[16:39] <stub> fini.
[16:40] <matsubara> all right. any questions for stub?
[16:40] <danilos> what are you wearing?
[16:40] <danilos> uhm, wrong channel
[16:40] <danilos> ;)
[16:40] <matsubara> heh
[16:41] <matsubara> thanks stub
[16:41] <matsubara> [topic] * QA Stats of the week
[16:41] <MootBot> New Topic:  * QA Stats of the week
[16:41] <Ursinha> :)
[16:41] <matsubara> so this is a new section, and I have no stats for this week
[16:41] <matsubara> but the idea is to bring up some number about triage and needstesting items from the current week and last week
[16:42] <matsubara> kinda show how we're progressing during the cycle
[16:42] <matsubara> what you guys think? keep, bag, change?
[16:42] <danilos> sounds nice, not sure how you are going to present it live :)
[16:42] <danilos> maybe using a drawing board?
[16:42] <Ursinha> danilos, never heard of ascii-art? :)
[16:43] <matsubara> that's all from me.
[16:43] <danilos> Ursinha: ascii-art is going to be a long stretch during Epic, though I'd like to see him try :)
[16:43] <matsubara> danilos: I'll come up with a way :-)
[16:43] <matsubara> or probably postpone it to the next IRC meeting :-)
[16:43] <danilos> ok, thanks
[16:43] <matsubara> ok, anything else before I close?
[16:44] <Ursinha> not from me
[16:44] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:44] <matsubara> #endmeeting
[16:44] <MootBot> Meeting finished at 10:44.
[16:44] <danilos> thank you as well, bye bye :)
[16:44] <matsubara> just in time