Mozilla Messaging - NOC

Buildbot downtime : Aug 5th 07:00-08:00 PDT

2009-08-04T12:55:00.001-07:00

For a short duration, on August 5th, between 07:00-08:00 PDT, the Thunderbird buildbot master server will be restarted to close bug 501272 - [buildbot] Increase RAM of buildbot master to 1GB. Our buildbot master has been running very low on memory, and has started to experience more swaping than is acceptable. This downtime should be fairly short and once back, our buildbot master should resume business as usual, except happier.

2 buildbot maintenance windows this week

2009-05-10T17:49:00.000-07:00

In the coming week, there will be 2 buildbot maintenance windows that could close the tree for 2-4 hours each. The reason for these is to allow for distruptive (from tree green-ness point of view) build systems and buildbot changes.

After these, the new hardware and resources available to our build infrastructure should be ready for use. As usual, work on these will be tracked in bugzilla, and if you encounter any problems, or have any good reason why this should be postponed, let it be known.

9AM PST, Monday, May 11th, 2009 - bug 492298
9AM PST, Thursday, May 14th, 2009 - bug 492297

[Edited: Fixed incorrect times due to backwards EDT -> PST conversion]

Hardware move complete

2009-05-01T00:48:00.000-07:00

It took a little longer than initially planned for, but it's finally complete. All our hardware is now sitting in our brand new space, with room to grow.

The hardware upgrade also went really well, so our overall build capacity has been greatly increased today.

The builders are just starting to pick up builds again, so I expect it will take a little bit more time for the trees to go green again.

That's all folks!

Downtime: Moving Hardware - 2009/04/30

2009-04-28T22:13:00.000-07:00

On Thursday, April 30th, 2009, we will be moving all our gear to a different location, so there will be downtime of all Mozilla Messaging services spread through the day.

Internet facing services should be down for 1-2 hours, and these include:

www.mozillamessaging.com
planet.mozillamessaging.com
SpreadThunderbird
All other *.mozillamessaging.com sites

Build services will be down for a longer period of time, and it might require closing the tree for a few additionnal hours, while it gets itself back to it's usual green.

As a positive side-effect of this move, we'll have more room to expand our capacity in the future. Plus, this includes a planned series of hardware upgrades that will be happening at the same time, a perfect occasion, since we have to power everything down anyways.

When completed, our build infrastructure will have close to 4x more computing resources at its disposal, yummy!

More information will be posted to this blog, as the move progresses.

Also, you can track progress on this issue by watching bug 490578

As usual, we always try and minimize outwardly visible downtime, but this time around, it can't be completely avoided.

[Update: Re-scheduled to April 30th]

Warning : MPT Colo Network issues

2009-04-27T08:43:00.000-07:00

[Mon Apr 27 08:45:37 PDT 2009]

The primary Mozilla Colo in San Jose has experienced networking issues. While this was hapenning, there was some spurious build bustage, as various services *.mozilla.org would sometimes timeout.

[Update: Mon Apr 27 10:13:59 PDT 2009: All is back to normal]
[Reported: Mon Apr 27 08:45:37 PDT 2009]

Resolved - Intermittent Network Issues - 2009/04/18

2009-04-18T07:12:00.000-07:00

Starting at around 8:30 EST this morning, our main firewall has started experiencing some problems, and as a result, network connectivity is degraded. I am seeing highly variable packet drop rates, sometimes reaching up to > 80 %.

This means that currently, pretty much all *.mozillamessaging.com and *.spreadthunderbird.com will be slow at best, and might display hangs and time-outs.

Apologies all around, and I'll post an update as soon as this situation is resolved.

[UPDATE: 17:05 EDT - Issue resolved]
[UPDATE: 13:15 EDT - It's hapenning again]
[UPDATE: 11:20 EDT - Things are looking normal again]

Downtime: OS X Builders

2009-02-19T09:25:00.000-08:00

Around 2-3 PM PDT today, the OS X builders will be disabled temporarly so they can benefit from a RAM upgrade. This means the Tinderbox trees will not see OS X builds for that duration. Also, as the builders get terminated, they might introduce spurious burning to the trees, please ignore them.

Tracking bug is 474794

Rolling Build Infrastructure Outage

2009-01-21T07:39:00.000-08:00

In light of the various bustages/instabilities from last week, I will be performing a few sweeping system-wide changes to the build infrastructure at Mozilla Messaging.

What this means is that today, every single build host will be at least stoped+started, and possibly rebooted too.

This should not induce bustage to the Trees, but it does mean that for a certain time window, various builders will be offline, so they won't be picking up new code pushes. For this reason, it was judged safer to hold the Trees closed while this is hapenning. That's Thunderbird, Thunderbird3.0 and Sunbird.

The work will be tracked in bug 474600, where the progress of the outage will be tracked. If you notice problems with builders during that period and you think it's related to this work, add a comment there.

Also, we'll try and keep tree closure to a minimum.

So Long : See you on January 19th

2009-01-08T19:16:00.000-08:00

Just a quick note to point out I'll be heading out on vacation tomorrow morning. I am returning to regular active duty on Monday, January 19th.

If, in the meantime, something ~~breaks~~ needs my attention, here are some simple steps to follow:

File a bug
Determine if it's an emergency. If so, goto step 3, else STOP
Ask in #maildev about that bug, and point out it is an emergency

At that point, somebody in there will determine if it's worth paging me for. In the case it is, some magical incantations will be performed, I will be notified/paged, and your bug should get closed.

In the meantime, please, try and not break the trees please.

OS X dep/nightly builder upgraded to XCode 2.5

2008-12-23T07:42:00.000-08:00

This upgrade should have little impact. I've upgraded the OS X builder responsible for the daily dep builds as well as the nightlies to XCode 2.5. This is a minor upgrade, and was necessary to enable full dwarf-2 debugging support in the upcoming builds.

See bug 468622 for the details.

Tinderbox: s/Thunderbird/Thunderbird3.0/g

2008-12-16T12:12:00.000-08:00

As you might know already, Thunderbird development since Beta 1 has switch over to following the mozilla-1.9.1 branch of mozilla-central. Now that we are getting ready to bring up new builders building against mozilla-central again, it was decided to split them up into 2 distinct Tinderbox trees.

Thunderbird3.0 will be tracking releases/mozilla-1.9.1 and Thunderbird will keep on tracking mozilla-central. Shortly, all existing builders will be moving to the Thunderbird3.0 Tinderbox tree, and as the new builders come online, they will start showing up in the Thunderbird tree.

www.mozillamessaging.com svn moved

2008-11-04T10:33:00.000-08:00

This post is to inform anybody that happens to have a checkout of the www.mozillamessaging.com: site that it has moved to our new top-level SVN area on svn.mozilla.org

You need to switch over your checkout to the new location if you want to keep your local modifications intact and be able to check them back in later. It's simple


$trunk/ > svn info | grep URL
URL: https://svn.mozilla.org/projects/mozillamessaging.com/trunk
$trunk/ > svn switch https://svn.mozilla.org/mozillamessaging.com/sites/mozillamessaging.com/trunk
At revision 19494.

Done, it's that simple. Please do so, otherwise, any future attempts at updating will end up emptying your checkout (except for modified/added files), most likely not what you'd want.

Houston, we have a problem

2008-09-23T04:08:00.000-07:00

There is current some fairly serious and wide-spread outage at the San Jose colo, where our trusty rack is hosted. I am not entirely sure what the extent of the damage is, but one thing is for sure, expect service outages.

On the upside, our website is pretty much the only piece that doesn't seem to be affected, yay! [Update: e-mail @mozillamessaging.com also survived]

On the downside, pretty much everything else was, booo.

I'll post more details as things get themselves sorted out.

Status update #4 (unittests)

2008-09-23T11:32:00.000-07:00

Another update, and the last one. The unittest boxes are back and building right now, that's:

Linux comm-central check

Win2k3 comm-central check

MacOSX 10.4 comm-central check

At this point, Nagios is back to all-green, so everything should be back to normal after this morning exciting failures

Apologies to all those who were affected by this interruption of service

If you still notice something broken, please, hurry up and file a bug about it

Status update #3

2008-09-23T10:43:00.000-07:00

Build machines are back in business, not the unittest ones yet. That should include the Lightning build boxes.
Specifically, the following builders are now back and building code:

cb-sb-win32-tbox:
tb-linux-tbox
tbnewref-win32-tbox
bm-xserve07
cb-xserve03
cb-sb-linux-tbox

Status update #2

2008-09-23T10:36:00.001-07:00

l10n dashboard is back in business
l10n buildbot is running and I just forced a re-compare of all 43 locales
buildbot masters are back in business

mail-test services are back in business

2008-09-23T10:23:00.000-07:00

Status update for users of the mail-test service at mozillamessaging.com.

It's back in business, please don't hesitate to file a bug if you notice anything out of the ordinary.

Hello World

2008-06-01T21:43:00.000-07:00

Hello World indeed!

The fine folks @ Mozilla Messaging are working hard to bring Thunderbird to the next level.

As part of this goal, they needed help to build out a system infrastructure separate from the existing one at Mozilla Corporation. The plan is to self-host more and more in the future.

It makes me proud to say that I'll be the first one to help achieve that goal. Hopefully, I won't be alone for too long.

This blog will be used to track status updates about the Mozilla Messaging Infrastructure, downtime warnings and other general information.