This is the story of when Dray, Nate, and I spent the better part of two days trying to figure out a bug. Hopefully, you won’t run into this same bug with your applications, but if you do perhaps this can be of some help.
Shortly after the release of iOS 10, we heard users of one of our web applications were no longer able to play videos. We were quickly able to reproduce the issue, and observed that videos were not working with iOS 10, but were working as expected with iOS 9. This is when the journey to find the root of the problem began, what we discovered was more bizarre than expected.
To back up briefly, I should explain how these videos are served, as they are not just requested using a publicly accessible URL. The videos are stored on Amazon S3 and delivered by CloudFront. To restrict video access, the source URL given to the video player is a URL to our application, where authorized users are redirected to a short-lived, signed CloudFront URL.
The first step we took to find the problem was to check if any errors were reported to the browser’s JavaScript console. In order to do so using the tools Apple provides, it means connecting an iOS device to a computer and then use the developer tools of Safari on the computer. Upon doing so, we noticed messages like the following in the console (which didn’t help us very much):
VIDEOJS:
"ERROR:"
"(CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED)"
"The media could not be loaded, either because the server or network failed or because the format is not supported."
Next step we took was to look at the network requests being sent from the phone’s browser. Unfortunately, that didn’t help us much either, because Safari doesn’t show request headers, response headers, nor response status codes for video requests.
At this point, not knowing where to look for the problem, a couple of my coworkers started making ever increasingly complex test pages, trying to find the specific situation that caused the problem. We spent a lot of time thinking the problem was in the redirection, perhaps when combined with HTTPS requests, but were never able to actually pin down the problem.
After not being able to create a standalone test page that reproduced the issue in iOS 10 devices, we decided to look elsewhere for the problem. We still really wanted to see the details of the requests for the video files, but with Safari’s developer tools not providing those details, something else was needed. We wanted to use an HTTP proxy, but there were a couple problems with doing that: the phone’s network traffic would need to flow through the computer running the HTTP proxy, and we were dealing with HTTPS requests (which means the requests were encrypted).
To solve the first of those problems, my phone needed to use my computer as its wireless access point; which brings up another problem: a Mac can use its WiFi radio to connect to a wireless network or to create a wireless network, not both at the same time, but my computer would still need access to the Internet. Luckily, I own a Thunderbolt Ethernet Adapter and after finally finding an Ethernet port in our office, the first problem was solved: my computer would connect to the Internet via Ethernet and then create a wireless network for my phone to use.
As for our second HTTP proxy problem (reading encrypted HTTPS traffic), we were nicely surprised to learn that the HTTP Proxy we were wanting to use already had a solution to that. We were using Charles Web Debugging Proxy, and it includes instructions on how to capture and display HTTPS traffic, which mostly revolves around installing a root SSL certificate on the computer and on the phone. This is certainly not something you’d want to do under normal circumstances, as we were essentially performing a Man-in-the-middle attack on ourselves.
After getting all of that set up, it didn’t take us long to narrow in on the
problem. The first thing we noticed was that the request to our application for
the video, which we expected to respond with a redirect to CloudFront, was
responding with 401 Unauthorized status. The next thing we noticed was that
the Cookie HTTP header in the request didn’t look right, it contained much
less data than all the other requests to our application. After that, one of my
coworkers discovered a bug had been reported with
WebKit, the
underlying engine in Safari and many other browsers.
So, the problem ended up being that the browser was not sending the cookies our application was relying on when making requests for media elements.
If your application has a similar problem, the solution will be application specific: wait for Safari to be fixed, remove authentication/authorization from those requests, or use a different authorization scheme. As of this writing, Safari on iOS 10 has still not been fixed.
Hindsight is 20/20
Looking back, there were some things that could have helped us narrow in on the
problem sooner. In production, we direct all of our log messages to
Logentries, and had we looked more closely we would
have seen the 401 Unauthorized responses for the video requests. That would
not have told us why our application was responding that way, but at least it
would have narrowed our focus.
What would have been of most help is if Safari showed request and response headers for video requests. It shows these details for other request types, and Google Chrome shows these details for video requests. If that data was presented in the developer tools, we would not have needed to use an HTTP proxy.