The Good, The Bad, and the League: 6/27 - 7/10
_Your semi-weekly dose of server problem-os, NA League news, and other!_
9.14 will be here shortly, with a bunch of changes to some of your favorite champions. Arcade pass is still live for a few weeks, TFT is still fun, and there was a bunch of server burps over the last two weeks.
* **Mac voice still not working properly** Bug assigned to a specialist. (Core bug is still being worked on).
* **Stuck in Spectator (7/2, ~ongoing)** NOC is notified by PS that players are still getting stuck in spectator after a change that was supposed to fix the problem. Problem ain’t fixed, possibly resolved with the upcoming 9.14 patch.
**Server Stuff: **
* **Delay with Missions (6/28, ~54 hours)** Automated monitoring flags an alert that the missions system is being overloaded. NOC pings and starts triaging with the proper product team. Draining the backlog will take time, but no stats are lost. Over the next two days, the large backlog is properly stored and the alert stops.
* **Payment system erroring out (6/29, ~10 minutes)** Automated alerting notices that errors are too high for one payment system. Before the NOC could dig into what’s going on, the errors drop back to normal levels.
* **Login Queue increased in LAX area (6/29, ~68 minutes)** NOC is notified that there’s a login queue for players early in the morning. Investigation leads to discovering affected players are within the LAX area and only there. Further looking finds a single router that ran out of memory, causing the delay. Rebooting the router fixed the memory issue, and the login queue disappeared.
* **League web content links in Japanese (7/1, ~31 minutes)** Templates used with the client are updated, and shortly afterwards the NOC is notified that na.leagueoflegends.com is in Japanese as the default language. NOC also confirms the issue exists on all other regions. NOC reverts the template change, and all languages reset appropriately.
* **Arcade Tokens aren’t being rewarded (7/1, ~16 hours)** NOC is notified (after the first set of missions were active) that there’s problems with the Arcade Missions. Triaging the issue identifies an incorrect start date, the root cause of the problem. The missions are corrected server side and redeployed.
* **Delay with Missions (7/5, ~90 minutes)** Automated monitoring flags an alert that the missions system is being overloaded. NOC pings the associated teams and starts an investigation to ensure the dependent services aren’t overloaded with data again in the future, causing client systems to lag or fail.
* **High Error Rate (7/9, ~33 minutes)** Automated monitoring flags an alert that errors for a dedicated system is exceeding critical levels. Error rate then drops back to 0% after 30 minutes. NOC investigates, finding a single server was the source of the errors, but are unable to discover why without associated product teams digging into the problem. NOC hands off the issue to that team.
**Game Stuff: **
* **Emergency Micropatch breaks TFT (6/27, ~27 minutes)** A micropatch to TFT to adjust item drops on PVE rounds is rolled out. Right afterwards, reconnections spike across all TFT regions. Coincidence? We think so, so the micropatch is rolled back and reconnections drop back to normal levels. Micropatch was analyzed to find why that occured.
* **Game start latency increased (6/27, ~4 hours)** NOC notices that game latency starts to increase to non-ideal levels. NOC responds by adjusting the TFT queue throttle to 5 minutes, which lowers the game latency back to acceptable levels. Over the next few hours, the throttle is reduced back down and removed.
* **3rd Party System Issues (7/2, ~45 minutes)** NOC is notified that a 3rd party system is having issues world-wide. NOC can only watch as the associated system impacts other systems that interface with it. Compensation mode is enabled and tickers are updated. Games start to disconnect, login issues appear, dogs start living with cats, and then the service starts to recover. Everything gets back to normal within a dozen minutes or so.
* **Qiyana disabled due to bug (7/5, ~19 hours)** NOC is notified that Qiyana has a large bug and may need to be disabled (thanks Q + Liandry’s). Live Producers approve the disable, and Qiyana is turned off while a hotfix is created. Hotfix is made, tested, and deployed, and then Qiyana is re-enabled on all servers.
**Player Support Issues:**