[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 494: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 5312: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3925)
Server Outage this Morning - Field Day Forums
Server Outage this Morning
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 10:01 am

    This morning, from approximately 7:30am Central to 8:45, the main ARIS server was not functional.

    Rackspace had to reboot one of the host servers, which ungracefuly powered off our database servers. Once I booted them up in the correct order, they resumed normal functionality.

    Despite the outage, I believe the process when pretty well. I received text and email warnings as soon as the machines had an issue and was able to identify the issue within 3o minutes.

    Outages will always happen, so when they do I like to see it go this way.

    David
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Update: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 10:49 am

    Upon closer inspection, this issue was not fully resolved. We are working to troubleshoot and restore service now. I'll update this thread as I learn more.

    David
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Re: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 11:03 am

    Update:

    I believe that the cluster of 3 database servers came out of sync as of 11-11-2016 when one of the three machines went offline. We were unaware that a single machine had failed because alerts are only sent if then entire cluster fails.

    This morning, when a second database server failed, the cluster did finally come down and generate alerts.

    My theory is that the server I (randomly) picked to start the cluster back up was actually offline for over a month. When it became the master, the other two machines actually synced against old data.

    I am currently restoring a backup from last night at 1AM Central time. The server will be offline for approximately 1 hour while this completes.
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Re: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 11:41 am

    The restore from backup completed successfully and data between 11-11-2016 and 1AM 12-2-2017 Central Time was restored.

    Unfortunatly, this means that 7 hours of data could have been lost. Luckily, from other logs, it doesn't appear anything happened during that time of any consequence.

    I'm currently re-syncing the database cluster. The overall server should be restored in the next 15 minutes.
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Re: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 11:58 am

    The database was restored and the server is now functioning properly.
  • chasleff
    Posts: 44
    Joined: Tue Oct 18, 2016 2:46 pm

    Re: Server Outage this Morning

    by chasleff » Tue Jan 03, 2017 1:27 pm

    Hello,

    We are having trouble getting in to the editor or using the app. Failing at Login even with QR code game start. arisgames.org also looks to be down. tested at 1:25pm on 1/3/2017 using chrome
    Charles Leffingwell
    ARISE Serious Game Programmer & Designer

  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Re: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 1:47 pm

    Confirmed.

    When server 3 synced up it took 1 and 2 down. I'm booting it back up again now. It takes about 10 minutes.

    David
  • User avatar
    djgagnon
    Site Admin
    Posts: 132
    Joined: Tue Sep 20, 2016 10:52 pm
    Location: Madison, WI
    Contact:

    Re: Server Outage this Morning

    by djgagnon » Tue Jan 03, 2017 2:08 pm

    The database cluster is once again up and running. It is only using 2 nodes which gives us basic redundancy but 1/3 less capacity than usual.

    I'm going to wait until 9pm this evening to attempt to bring the third machine back online.
  • chasleff
    Posts: 44
    Joined: Tue Oct 18, 2016 2:46 pm

    Re: Server Outage this Morning

    by chasleff » Tue Jan 03, 2017 2:27 pm

    We are now able to login and our stuff is there.
    Charles Leffingwell
    ARISE Serious Game Programmer & Designer