Recently, we implemented a hotfix in ChartBuilder to limit how many setlists are loaded at once. This change has been working well over the past few weeks while we architect a more permanent solution that will eliminate the underlying server issues ChartBuilder has been causing.
Unfortunately, this past Sunday the hotfix proved insufficient and resulted in server slowdowns. In response, we reverted that change and reinstated a limit of 20 upcoming setlists. This was a temporary measure that restored normal server performance.
The good news is that we have a permanent fix ready to deploy to ChartBuilder tomorrow. While it did not resolve the issue today, it will address the root cause going forward, and for today’s impact, we sincerely apologize. --- What Happened:
In preparation for the Christmas season, we temporarily increased the amount of ChartBuilder data returned by the API to support planning workflows. This additional buffer was intended to be short‑lived, but because the corresponding ChartBuilder client update was delayed, the expanded payload remained in place longer than expected.
During peak traffic this morning, the increased data volume caused ChartBuilder requests to place excessive load on the API. This created a cascading slowdown that affected overall system responsiveness. --- How We Resolved It:
Once the root cause was confirmed, we immediately reverted ChartBuilder’s API response back to its standard behavior of returning the next 20 items. This change reduced load significantly and restored normal performance. --- Long‑Term Fix:
ChartBuilder has a client update scheduled for release tomorrow morning. This update removes the need for the temporary buffer entirely and prevents this class of issue from recurring. Once that release is live, the API will no longer need to serve the expanded dataset. --- Moving Forward:
We’re reviewing our internal processes to ensure temporary configuration changes don’t remain in place longer than intended, especially during high‑traffic seasons. We’re also improving our monitoring around payload size and request patterns so we can detect and mitigate similar issues even faster.