[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
Field Day Forums • Server Outage this Morning
Page 1 of 1

Server Outage this Morning

Posted: Tue Jan 03, 2017 10:01 am
by djgagnon
This morning, from approximately 7:30am Central to 8:45, the main ARIS server was not functional.

Rackspace had to reboot one of the host servers, which ungracefuly powered off our database servers. Once I booted them up in the correct order, they resumed normal functionality.

Despite the outage, I believe the process when pretty well. I received text and email warnings as soon as the machines had an issue and was able to identify the issue within 3o minutes.

Outages will always happen, so when they do I like to see it go this way.

David

Update: Server Outage this Morning

Posted: Tue Jan 03, 2017 10:49 am
by djgagnon
Upon closer inspection, this issue was not fully resolved. We are working to troubleshoot and restore service now. I'll update this thread as I learn more.

David

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 11:03 am
by djgagnon
Update:

I believe that the cluster of 3 database servers came out of sync as of 11-11-2016 when one of the three machines went offline. We were unaware that a single machine had failed because alerts are only sent if then entire cluster fails.

This morning, when a second database server failed, the cluster did finally come down and generate alerts.

My theory is that the server I (randomly) picked to start the cluster back up was actually offline for over a month. When it became the master, the other two machines actually synced against old data.

I am currently restoring a backup from last night at 1AM Central time. The server will be offline for approximately 1 hour while this completes.

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 11:41 am
by djgagnon
The restore from backup completed successfully and data between 11-11-2016 and 1AM 12-2-2017 Central Time was restored.

Unfortunatly, this means that 7 hours of data could have been lost. Luckily, from other logs, it doesn't appear anything happened during that time of any consequence.

I'm currently re-syncing the database cluster. The overall server should be restored in the next 15 minutes.

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 11:58 am
by djgagnon
The database was restored and the server is now functioning properly.

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 1:27 pm
by chasleff
Hello,

We are having trouble getting in to the editor or using the app. Failing at Login even with QR code game start. arisgames.org also looks to be down. tested at 1:25pm on 1/3/2017 using chrome

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 1:47 pm
by djgagnon
Confirmed.

When server 3 synced up it took 1 and 2 down. I'm booting it back up again now. It takes about 10 minutes.

David

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 2:08 pm
by djgagnon
The database cluster is once again up and running. It is only using 2 nodes which gives us basic redundancy but 1/3 less capacity than usual.

I'm going to wait until 9pm this evening to attempt to bring the third machine back online.

Re: Server Outage this Morning

Posted: Tue Jan 03, 2017 2:27 pm
by chasleff
We are now able to login and our stuff is there.