Top

Website / Server Updates & Maintenance

Comments

24 comments

  • Avatar
    Jim Tarber

    The larger update is scheduled for Monday night, "after hours" (in quotes because we're global!), but there will be other brief periods where the website may be unavailable for a minute or two.

    We just had two of those very brief outages in the last hour as we did some updates to address HTTPS/SSL protocol restrictions and a problem with the SSL certificate chain.

    This update (already in place) may fix some security warnings on some devices and browsers (mostly mobile browsers I believe).

    There is still at least one more brief outage coming tonight, then the larger one on Monday night after hours.  We may need further updates after that one, and will continue that on Tuesday night if necessary.

    All of this is to strengthen the security of inworldz.com, to stay ahead of the problems.

  • Avatar
    Moonrise Azalee

    That sounds awesome :)

  • Avatar
    Yichard Muni

    Pawsomderfull!

    I think it will be time to close posting messages in the old forum. I still found russian scams 2 days ago, so that you can count this as a security issue too :-)

  • Avatar
    Jim Tarber

    Tonight's web server updates are complete.

    We still have a lot of upgrades and updates to do, but at least we're not getting the C rating we were getting 12 hours ago. Looking good now:

    And more updates on this machine and others over the next couple of days.

    One thing that was not clear before: the updates on Monday night will require a restart of the main login/web server machine, which for now means we need to restart all regions after that, which means we'll have a full grid restart Monday night, into Tuesday morning.  The regions will not be getting a region server update, they just need to be restarted, so they should all be restarted in something like 4 hours or so, meaning they should complete by about 3am IW time.

  • Avatar
    Jim Tarber

    Note also that these security updates mean that you can no longer connect to inworldz.com over HTTPS with some older browsers and devices. For example, Android 4.4.2 should work, but earlier versions not.  If you're still running Internet Explorer 10 or earlier, you'll need another browser other than Internet Explorer or something newer than Chrome 49 (although if you are running IE on XP or Vista for Internet browsing, this is probably long overdue).  Same thing goes with Safari 5 on OS X 10.6, or Safari 6 on 10.8, although Safari 6 on iOS 6.0.1 should be fine.

    All versions of Android 4.4.2 or later, Safari 7 or later, Chrome 50 or later, IE 11 or later should be fine, and as I mentioned on iOS 6 with Safari 6 (or later for each).  All modern versions of Firefox (since FF 31) should be fine.

    The main factor for the compatibility updates there is a requirement for HTTPS over TLS 1.2.  That is currently now enforced on inworldz.com for https connections; I think we need that now for our commerce activity (Payment Card Industry Data Security Standard), although Elenia and I need to discuss where business requirements meet technical ones.

     

  • Avatar
    Benski Trenkins

    Sent a notice with link to this topic in the Welcome Group.

  • Avatar
    Jim Tarber

    Thanks Ben.

    Also if anyone does experience any trouble with certain devices or browsers, it would be interesting to hear, just to make sure our understanding of what is affected by the security updates matches our expectations.  I'm hoping that anything affected is old enough that nobody here is using it (or that at least there's an upgrade they can do to continue).

  • Avatar
    Yichard Muni

    Seems to work well with Opera.

     

    There should be no more old browsers, since they all have automated updates.

  • Avatar
    Jim Tarber (Edited )

    I'd love to think that but I don't think that is the case for some mobile devices and operating systems. People running Internet Explorer on an older Windows or Windows Mobile device are most likely the ones most affected by these updates to our security, however they are for the most part very necessary.

  • Avatar
    Benski Trenkins

    Works just fine on my Android phone. The warning is gone (Chrome, Android v7.1)

  • Avatar
    Ferrator Montoya

    Will this do anything to address FPS's lower than 5 in Firestorm?

    Lower than 15 FPS kin Singularity?

  • Avatar
    Jim Tarber

    First clarification, we have scheduled region restarts; there is no code change or upgrade involved.

    But even when the next grid server update comes, absolutely not.  "FPS" is "viewer FPS" (assuming you aren't referring to "SimFPS" or "PhysicsFPS").  Viewer FPS is dependent on your viewer, the viewer-side hardware (especially your graphics card), your graphics driver, your Internet connection speed (the viewer is coded to include that in your "frame"), and how much content is within view.  Nothing server-side affects that other than to not show your viewer all of the content, and that is controlled by the draw distance setting.

    If you're seeing less performance, overall (not just once in a while) with FS vs Singularity, and it's not different content, different region, different position or facing/rotation, then it is probably a difference in the graphics settings.  Take a close look at the settings in both.  It might be as simple as different draw distances, but it's probably not that easy.  15 fps is pretty low; 5 fps is almost unusable.  There's a lot more going on in your case than anything on the server side. You'll need to do some specific test to determine what is slowing down the viewer frame, and/or enable some viewer-side performance debugging.  If you're getting 5 fps with FS a lot, you might want to report that and work with the FS team to try to track down if it's a limit of your hardware or a problematic setting or FS problem. But if you're getting 15 fps with Singularity (poor performance there too), the problem is probably not a FS problem.  But the FS folks can help you identify where the problem is. For example, there is a "Developer" menu option under Consoles called "Fast Timers" (Ctrl-Shift-9) that gives you a very detailed breakdown of where exactly the viewer is spending all its time in each frame.  That's nothing a grid update, especially a restart is going to help with.

  • Avatar
    Ferrator Montoya (Edited )

    Aha!!

    OK, that is a good place to start checking. What are the specific items I would be looking for as trouble spots?

    What I know is that, about the upgrade before last the viewer performance for ME [might only be me, but I have heard similar complaints] took a real dive into the unusable range. 

    Looks like FS is performing just as poorly on other grids also. So yeah, what do I look for in CTRL-SHIFT-9?

    That and I TRY ot get in touch with the FS people. This is close to a deal breaker for me.

  • Avatar
    Jim Tarber

    I think you're mixing unrelated or coincidental observations, because as I explained above, it's difficult for the grid software to affect viewer FPS even if we tried. The only way I can think for a grid to affect FPS is to flood the network with packets, which might be the case only while downloading region contents on entering a new region, or if still loading your inventory.  By unrelated or coincidental, I mean something else changed at about the same time and it just appears to have similar timing with a grid change. For example, a Windows Update, or a new graphics driver, etc.  Or something related to your antivirus, or something else on the machine.  I know there have been a couple of new nvidia drivers recently.  If you have graphics driver software installed that can check for an update... do so.  If there's a noticeable change without a hardware change, it might be a driver.  Otherwise, probably something else on that machine like I mentioned above.

    There are no specific items to check that I know of (but I'm not a viewer guy) other than the Fast Timers list, but that is a "tree" of items. You open the top level (think "root folder") and find the "child items" with the biggest numbers causing the frame time to be slow, then open that, find the biggest child item of that, then open that etc. With an FPS of 5-15, that means something is occupying the frame for 60-200ms, which is a long time for a graphics frame.  So look for an item that has a time longer than say 20ms, there shouldn't be too many, probably one at the first level under the root.  The actual items in the Fast Timers tree probably won't mean anything to you (or even me) and would require someone on the viewer team (Firestorm in this case) to investigate why that item would have a long frame time on that viewer.

    At any rate, none of this is related to Monday's security updates, and I really don't want to disrupt in this thread any further with a viewer support question. You should really file a support question with the viewer team.  They are pretty good when they jump on a problem, and FPS is a Big Deal to viewer folks.

  • Avatar
    Ferrator Montoya

    Alright, agreed.

    Looks like FS is performing just as poorly on other grids also. 

    That and I TRY to get in touch with the FS people. This is close to a deal breaker for me.

  • Avatar
    Jace Alexander

    I just switched to the IW Viewer.   I've never had better frame rates or been able to use such high graphics settings as I can with the in-house viewer.

  • Avatar
    Yichard Muni

    Today PCs are complex machines with many layers and functions. Checking things such as a new driver which installed itself (in the background, without notice) is well beyond the abilities of the average lay user. What would be interesting is a kind of monitor function in the viewer, checking for changes in the PC or dysfunctions like a new driver breaking the graphic rendering. This could be added to the viewer, or a standalone program.

    Don't ask me to do this, though. It is out of my abilities.  But a PC diagnosis tool would be very useful in many domains. Especially if it is independent of the OS.I remember there were some, at a time, and people rushed over them. But they were not updated.

  • Avatar
    Arkady Arkright

    What hardware are you on, Ferr ? I recall a case some time back of someone getting low frame rates, it turned out that their PC was defaulting to use the basic graphics chip on the motherboard rather than the expensive graphics card sitting in an expansion slot doing nothing.

  • Avatar
    Jim Tarber

    [Aside: Before someone replies in this thread (too), Ferr has thankfully moved the discussion about graphics cards and frame rates to another thread here. Please do not reply in this thread regarding that topic.]

    _____

    Reminder: This message is just a remind everyone that there be a planned outage tonight disrupting much of the grid starting sometime after 10 PM InWorldz time.  The outage may be brief, but may last as long as a couple of hours, and will be followed by rolling restarts of all regions.  I'll be trying to minimize the disruption but it is likely to be about an hour or so.

    See the announcement post here for more details.

  • Avatar
    Jim Tarber (Edited )

    The scheduled maintenance on the inworldz.com main website and the central grid services will begin shortly.

    At various points over the next couple of hours, users will experience outages of the website and some grid operations (including logins) will be disrupted.

    I'd advise you to avoid any significant operations over the next couple of hours until the central grid services updates are complete. (I'll post another message when that happens.)

    After that point, rolling restarts (no upgrades) will commence for all regions.

    All updates and restarts should be complete by morning (probably by 6 AM InWorldz time).

  • Avatar
    Jim Tarber

    After considerable struggles with incompatible Apache web server configuration changes, the website upgrades are complete now, as well as system upgrades.  However the next step is to restart the machine for the system updates.

    So web server updates should be complete as soon as the web server system updates are finished, but during this time the central grid services will be unavailable. Grid operations may be disrupted.  Take care in doing significant operations such as purchases and inventory offers, especially where no-copy items are involved.  Logins will be disrupted during this time, so you'll know when it has completed if you can log back in again.

    If all goes well, the website and logins should be back in a few minutes, but there may be more system updates to be applied so it could be anywhere from a 15 minute disruption, to over an hour.

    I'll post another update when this disruptions to the core services are complete.  Once we reach that point, rolling restarts of all regions will begin.

  • Avatar
    Jim Tarber

    System, security and software updates are complete, at least all of the updates planned for tonight.

    InWorldz central grid services are back up again now as well, and logins have resumed.

    Rolling restarts for all regions will begin shortly and should take about 5-6 hours to complete.

    So all regions should be restarted and back up by about 6 AM InWorldz time.

  • Avatar
    Jim Tarber

    So the restarts took a lot longer than I estimated (actually started a lot later than I had planned), but finished about 9 AM InWorldz time (about 2.5 hours ago).  All regions should be up now and everything should be running normally again.

  • Avatar
    Yichard Muni

    nice!

    well, this will probably not bring any visible change, but safety will be improved

     

    by the way the old forum is still open, and this is still a safety breach (due to spam and phishing posts)

Please sign in to leave a comment.