Almost all of our display status have gone "offline" in the last hour

  • 1
  • Problem
  • Updated 3 years ago
  • Not a Problem
Managing Chromeboxes in kiosk mode. All chromeboxes are latest RV app and are managed in the Chrome Console. Noticed many of them suddenly went "offline" at 14:25 EST. When I went to investigate in the console at 14:30, I got Error 1000s with every click in the chrome console.  Made me think it might be Google profile services.

Bit then as of about 15:20EST, all but 3 production displays are showing offline in risevision. These are spread across 5 venues in 3 cities and almost all of them are exhibiting a sudden offline status.

Chrome Console showed them to be checking in until 10 minutes ago. Now they are all offline there as well.  Technician is being dispatched to put eyes on a few venues.
Photo of Dave

Dave

  • 886 Points 500 badge 2x thumb

Posted 3 years ago

  • 1
Photo of Blake Freeman

Blake Freeman, Official Rep

  • 36,116 Points 20k badge 2x thumb
Dave,

Can you please email support@risevision.com with a few of the display IDs that were affected?

We do have several managed Chrome OS devices up in our office and I'm not seeing any strange activity. I would like to see what our logging on your display IDs has to say.
Photo of Blake Freeman

Blake Freeman, Official Rep

  • 36,116 Points 20k badge 2x thumb
I received your email, thanks Dave.

We'll be looking into this.
Photo of PCPC PCA

PCPC PCA

  • 70 Points
@Dave - Do you happen to have any locations in the Dallas-Fort Worth area? If so, there was a major internet outage... some of Level3 pipes were down. They are back-up now, and we are working as normal now. 
Photo of Blake Freeman

Blake Freeman, Official Rep

  • 36,116 Points 20k badge 2x thumb
Dave,

I would be interested to know this as well. I have been unable to replicate this issue at all, and we have not received any reports of displays offline en masse. Let me know!
Photo of C Robert Cone

C Robert Cone

  • 382 Points 250 badge 2x thumb
This happens at my school fairly frequently,  the displays show the google sign in screen and restarting the player or display in the rise vision app does nothing, I have to manually reset the chromebox with the power button, break out of the rise vision app from starting, sign in using a different account, then restart then chromebox.  We have seen this when there has been no network.
Photo of Dave

Dave

  • 886 Points 500 badge 2x thumb
So we had a strange convergence of trouble:

Locally, the network that provides the central pipe from the venues over a MAN to the internet , began dropping connections/packets a few days ago. Over the last 24hrs, this escalated and presented in risevision like the majority of the display's were down and a handful were up. This did happen to all venues at once nor did it drop whole venues at once.

This also caused the Chrome Management Console to report contradictory info to what we saw in Risevision.

At the exact same time there was a failure with the Google Chrome Device Management Console App and the GCM began to exhibit errors. Because of this coincidental timing, we're attributed the mass display failure to this GCm problem.

Network or GCM may not have been the root cause. As of 0730 EST, all displays came back up normally and began to display content.

GCM error experineced yesterday has also apparently been resolved by Google.
Photo of Blake Freeman

Blake Freeman, Official Rep

  • 36,116 Points 20k badge 2x thumb
Dave,

Thanks for the update! I'm glad to hear it wasn't something on our end!