Yesterday our platform experienced capacity issues for a 4 hour period. During this time, most functionality was either slow or unavailable. We are very sorry for the frustration and problems this caused our users. We know how critical TicketLeap is to you and no amount of unplanned downtime is acceptable. I want you to know how seriously we take this situation.
The problems on TicketLeap yesterday occurred in association with an on-sale we conducted for the wildly popular event “Comic-Con International.” While we knew the event was going to put significant demand on our system, we did not expect the traffic we received.
In December, we conducted a test sale of 1,000 tickets in coordination with Comic-Con International to gauge the traffic to the platform and to work out any problems in a controlled fashion. After the test run, we made various optimizations to meet this demand level and scaled up our architecture with Amazon Web Services significantly.
The traffic we received yesterday was several orders of magnitude higher than our high end estimate. Due to the heavy strain on the system, users for all events across our system received “Over Capacity” errors. This prevented ticket buyers from buying tickets and it prevented event organizers from managing their events.
Throughout the day, we worked to adjust the scale of our infrastructure with Amazon in an effort to meet this demand. Unfortunately, we discovered a bottleneck that didn’t allow the platform to scale as needed. We are working with the highest levels of leadership at Amazon on this issue and wrote a detailed technical write-up about Saturday for those who are interested. Also,