The hardware behind


Bugzilla Physical Layout Diagram
Click the image for a full-size (readable) version





16 Replies to “The hardware behind”

  1. What sort of traffic does Bugzilla get to warrant this sort of setup, in terms of hits/sec or queries/sec or whichever metric is appropriate?

    I don’t really have a feel for how big BMO is.

    1. @tom jones: We cache some of the content (css, javascript, and images that are part of the site layout rather than bug content) to help it perform better, since connectivity isn’t always the greatest there. We have a datacenter in Beijing, so it made sense to use it. There wasn’t enough hardware there to fully-replicate the site though.

  2. @Alex: the average amount of traffic that gets could easily be handled on one webserver and one slave database server, however, it tends to be pretty “bursty”.

    For example, every so often a major bug will get fixed (or some controversy will arise from one) and a bug will get linked from major news sources.

    Another example is triage meetings for our development teams. Because of our distributed nature (last I heard around 40% of our employees are remote, and that doesn’t count a lot of the volunteers that are involved as well), and when you have a telephone conference call to discuss a bug list and everyone on the call suddenly loads that same buglist in their browser at once that’s a lot of query traffic.

    These situations don’t happen often, but it was important to us to keep the site responsive when it does happen, so there’s sufficient hardware in place to handle these bursts.

    Gerv Markham just posted some stats I dug up the other day for the kind of usage we have, which is over at

  3. The stats Gerv posted only give you a feel for people who create content in Bugzilla, it doesn’t count people who only browse. I’m seeing if I can come up with some numbers for that, too.

  4. OK, so our metrics guys tell me we aren’t actually recording any pageview stats for Bugzilla, but a quick grep of the weblogs tells me we’re getting roughly a million hits per day from about 70,000 unique IP addresses.

  5. Awesome information, always wondered what the BMO infrastructure looked like exactly.
    With this setup, do PHX transactions need to wait for acknowledgment from the SJC1 master to commit? If so, did you notice delays from it?

  6. @Michael Kurze: The masters have blind two-way replication between them. MySQL won’t duplicate transactions because it filters out binlog entries that were generated by its own server ID. You can write to either master and it will immediately take, and this does mean you can potentially have insert ID conflicts. We deal with this by having the ACLs set up so that the load balancers are the only sources allowed to connect to MySQL, and both load balancers are set up to only allow writes to go to one of the masters (and always the same one). We can swap masters at any time by switching which one is active in the load balancers (but this does require taking the site offline long enough to make sure all of the connections to the one you’re moving off of complete before the one you’re moving onto is brought online, to avoid conflicts).

  7. Because of the amount of performance improvement we don’t really care… it’s a necessary cost for the performance we need. That said, SSD devices aren’t nearly as write-limited a they once were. Modern SSD drives have a high enough re-write capacity to last almost as long as a platter-style hard drive.

  8. Hi Dave,

    I apologize for reviving this 2-year old topic 🙂 but I’m currently going through an exercise of making my company’s Bugzilla scale and be more fault-tolerant. While I do have experience with both Bugzilla and Web/MySQL services to that matter, I still learn a lot from that and would be really grateful if you could clarify several things in your setup for me, like:

    1) Your web servers are load-balanced, with Bugzilla, does that mean you have several Bugzillas (web part of it) connecting to the same DB? I know BZ can run like that for the most part (thanks to mid-air collision detection), but what about multiple bug editing – there’s no mid-air collision detection for that? Or it is implemented differently, somehow?

    2) With attachments stored on NAS and mounted via NFS to those multiple web servers, have you seen any problems with locking/data integrity (e.g. when several people work on the same bug and modify attachments)?

  9. @Serge:
    1) Yes. The mid-air collision detection works exactly the same no matter how many web heads you have – whoever gets to the database first to store the change wins, and everyone after that gets a collision. The attempt to change it is made when the user submits the change, not when they load the bug to view it. This is no different than two people trying to edit at the same time on the same webserver from different browsers.
    2) No, since the attachment metadata is still stored in the database, and the same mid-air protections apply (and in our case, we’re actually storing the attachments themselves in the database, the NAS is only used for configuration, chart data, etc.

Leave a Reply

