Master Server Maintenance Thread

Started by Led, June 26, 2016, 10:38:09 AM

Previous topic - Next topic
June 26, 2016, 10:38:09 AM Last Edit: July 16, 2016, 02:16:47 PM by Anyder
12:30 PM central time 6/26/16

edit:  maintenance complete, let me know if there are any issues
Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Were there any issues before? I found that the servers loaded a lot faster myself

Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Quote from: Led on June 26, 2016, 12:56:21 PM
Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

Is all I noticed was the TPS went up from 30 to 40. Although it did seem like better connection as I managed to rack up a lot more kills than usual ;)
Feel free to contact me on Steam, I answer instantly (Unless I'm offline):<br /><br />YouTube: https://www.youtube.com/user/roblox751123

tps is set at 40 for the ICW6, not related to MS restart
Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Quote from: Led on June 26, 2016, 12:56:21 PM
Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

Doesn't feel any faster than it was before, so idk

July 06, 2016, 11:18:30 AM #6 Last Edit: July 06, 2016, 11:38:30 AM by Anyder
Performing some firewall tasks to see if we can somehow block DDoS attacks. A friend of mine and I are testing this right now, if anyone has any problem in the meantime, post here --

EDIT: Finished it!
Anyder | Talent, Ops & Culture | SWBF & Player Engagement
Email: communityambassador@swbfgamers.com
SWBFSpy Discord: http://discord.swbfspy.com
SWBFSpy Info: http://info.swbfspy.com

Switch reboot
Jul 15 2016 04:02:32 PM PT

At approximately 5:49pm CDT, our primary aggregation switch in Chicago rebooted itself, causing a disconnection event for all customers at that location.

This is the second time this has happened. In both cases, we have been unable to determine a cause of the switch failure.

In response to this, to prevent further service interruptions, we will be working to replace this switch in the near future. We will post a maintenance event when we have final information on when the maintenance will be performed.
Anyder | Talent, Ops & Culture | SWBF & Player Engagement
Email: communityambassador@swbfgamers.com
SWBFSpy Discord: http://discord.swbfspy.com
SWBFSpy Info: http://info.swbfspy.com

Is there any Maintenance going on with SWBFSpy lately? There's next to no servers across both games.
Feel free to contact me on Steam, I answer instantly (Unless I'm offline):<br /><br />YouTube: https://www.youtube.com/user/roblox751123

Quote from: TheGangstarTY on July 28, 2016, 04:04:47 AM
Is there any Maintenance going on with SWBFSpy lately? There's next to no servers across both games.
Unless we post something, there are no maintenance events going on.

[spoiler]
Subject and date   Description
(No maintenance events recorded)


Subject and date   Description
(No events recorded)


Subject and date   Description
(No bandwidth events recorded)

[/spoiler]
Anyder | Talent, Ops & Culture | SWBF & Player Engagement
Email: communityambassador@swbfgamers.com
SWBFSpy Discord: http://discord.swbfspy.com
SWBFSpy Info: http://info.swbfspy.com

Upcoming router maintenance early 8/5
Aug 04 2016 12:17:19 PM PT    Between 3am and 5am CDT on Friday, August 5, we plan to update the OS on one of our routers in Chicago, which will require a reboot. We expect for this to cause approximately 10 minutes of connectivity loss for your service at this location.

We are performing this update because the new software release should offer higher performance than the old one, which will make the router able to better-handle particularly high-packet-rate DDoS attacks. We have seen several extremely large DDoS attacks against a customer in Chicago today that have saturated all of our upstream links, and would have needed to be null-routed no matter what, but they caused some additional impact because this router could not process all of the traffic internally.

If there is a problem with the new software release, we will roll back to the older version.
Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Location down
Aug 08 2016 02:33:53 PM PT    Our primary aggregation switch in Chicago appears to have crashed, causing the location to go offline. We are currently investigating.

We already have a new switch on order to replace this one, since it has crashed twice in the past.

Update @4:43pm CDT: We are waiting for an on-site technician to arrive at the cabinet and physically survey it to determine what has happened. The main possibilities are that the switch has failed or that the PDU that it is plugged into has failed. If it is the latter, it should be a quick fix once the technician arrives; if the former, it may take a little longer for us to rearrange cables.

The replacement switch that we ordered is higher-end, with additional redundancy, including redundant PSUs.
Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Intermittent packet loss over Telia


We have been noticing intermittent packet loss over one of our upstreams in Chicago (Telia) tonight, on the inbound from some endpoints (from New York/Ashburn, at minimum). We have contacted Telia to ask them about this. Clients whose ISP reaches us via Telia may notice skipping or other performance problems with services in Chicago.
Anyder | Talent, Ops & Culture | SWBF & Player Engagement
Email: communityambassador@swbfgamers.com
SWBFSpy Discord: http://discord.swbfspy.com
SWBFSpy Info: http://info.swbfspy.com

August 25, 2016, 02:16:04 PM #13 Last Edit: August 25, 2016, 02:27:12 PM by Led
Master Server Maintenance Has commenced.  and is now over, seems to work  :cheers:
Quote from: Abraham Lincoln. on November 04, 1971, 12:34:40 PM
Don't believe everything you read on the internet

Charter packet loss

Aug 25 2016 01:29:49 PM PT   We have received reports from Charter customers that they are seeing intermittent packet loss to our Chicago location. The data we have seen so far suggest that the problem is occurring between Charter and Voxel (Internap's peering network). We have emailed Internap to ask them to investigate, and we have also emailed Charter.
Anyder | Talent, Ops & Culture | SWBF & Player Engagement
Email: communityambassador@swbfgamers.com
SWBFSpy Discord: http://discord.swbfspy.com
SWBFSpy Info: http://info.swbfspy.com