Just a wild guess: Could the enterprise in-house code signing certificate have expired? That would lock out the app. Imagine that happening mid-flight!
(The iOS enterprise in-house code signing certificates, which only Apple can issue, always seem to come with 1-2 year hard expiration dates)
Not a problem, really. FAA regulations require all sorts of backups and emergency procedures. Even my dad's little four-seat general aviation plane has backups for every critical instrument.
This issue prevents you from taking off (out of caution), but if it happened in-flight everyone'd know what to do.
Your last sentence is key, and seems to be something that everybody forgets about. You have a great deal of redundancy so that a problem in the air doesn't get you killed. But the whole point of that redundancy is defeated if you're relying on it from the beginning of the flight. Therefore it's common, and reasonable, to have issues that wouldn't cause any trouble in the air, but which prevents you from flying to begin with.
Imagine if the speedometer in your car failed. You shouldn't drive it until you get it fixed, but it won't kill you if it happens while you're on the highway.
Normally you reach a decision on the kind of redundancy and technological diversity by looking at certain risks and then decide if you only need one, two of the same, or two or more of differently implemented/built devices using different methods of operation.
For (made up example, I don't know anything in particular about aviation, but know the general approach from working with controls for nuclear powerplants) navigation one might conclude that GPS and old-fashioned UHF radio navigation is technologically different enough that an event taking down one means of navigation (say, jamming of GPS at 1.5 GHz, spread spectrum, digital, ...) will not affect UHF radio (at analog modulation on a few hundred MHz). So one decides that you have GPS and UHF, displayed on different monitors/indicators, powered by separate power-busses in the airplane.
For paper documentation, they'll probably claim: Bring two binders with the procedures to the plane, store at different sides of the cockpit.
For iPads one might have decided that they must not be connected to any network while on the plane, so that data cannot be deleted from the pilot's and first officer's devices. But have they considered the "common cause failure" of two iPads, not being connected to any network, not communicating with each other, with full batteries, probably with passed selfchecks (checksums for stored data on the flash)... to really fail at the same time because of an expired certificate?
More over, isn't this just a manual? I mean, for most normal situations, the pilots know what to do, so even if the backup also died, chances are pretty good that they would land without problems, I imagine.
There are so many possible scenarios that are encountered in flight that there is no way to have them all memorized. There are certain "memory items" that a pilot must know, but many procedures for dealing with both routine and emergency problems require the non-flying pilot to pull out a quick-reference handbook or checklist to be sure everything is done in the proper order and nothing is overlooked. Even routine departures and landings follow a checklist.
A lot of newer aircraft have built in checklists now[0]. For a bunch of reasons (e.g. safety, automatic double-checking, and expedience).
Yes, it would be non-optimal if an aircraft had an issue without a handbook, but assuming that communications haven't been lost, someone on the ground can easily read out items.
Maps too, from what I hear. The problem is scalability. If no one carried maps, everyone would have to talk verbally and take notes with ATC for five minutes for every landing to make sure nothing had changed.
If only a couple planes pester ATC then the system will keep up. It would be safe enough. What are the odds of an inflight emergency requiring a diversion at any instant? Pretty low. So if it happened to one plane they'd figure it out verbally. However requiring all ATC and all pilots to talk over all maps at all landings isn't scalable.
Something similar happens with manuals. One plane loses a system and its ipad, you call home on the radio or satphone or contact a fellow pilot on 123.45 MHz or call ATC for help. This doesn't scale to every plane in the air at the same time for every mostly minor maintenance writeup. Maybe you can ignore error messages this flight, after all most measurable things are not important, just like in business. Or maybe engine code 3141Z.kjdf.subsection2 means the engine is about to explode on the wing, but you've got no manuals...
Something often overlooked is megacorp analysis paralysis. You'll be formally written up if you don't file an engine operation temperature profile thingy form or some meaningless catering receipts or whatever according to boring corporate policy, and it will be brought up at your annual performance review, affect your promotion chances, etc. This scales pretty well when one ipad breaks and you use the copilots ipad or call your boss and argue and he issues you an indulgence or the boss takes dictation of the form for you or you refuse to fly (its only one flight after all). This doesn't scale well when every ipad in a company stops working.
Personally, if I were in a plane over water and I lost all my papers out the window, no fuel estimates, no weight and balance, no maps, no flight plan, no list of obstacles, no nav plan, no notes, nothing at all, I'd probably panic and head for the closest coastal airport (and where is that, without any docs?) If one plane does this, it scales VERY well. ATC calms you down, vectors you in, gives you some geographical advice, and you land. If every plane in the air does this simultaneously, you'll overload ATC and probably get someone killed.
This applies everywhere to all industries not just planes.
> If every plane in the air does this simultaneously, you'll overload ATC and probably get someone killed.
We've already had a test run for this, and it worked fine. 9/11 involved exactly that scenario - every flight in or heading to the US had to make for the nearest airport and land immediately.
Aeronautical charts extrapolate from this because being in command of an aircraft is really just commanding a different kind of vessel. Port, starboard, manifest, fix, captain, first officer, and hull are all valid aviation terms, for example, with roots in the sea. A chart is not a map.
Thank you! I'd have trouble explaining to someone what the precise difference is but now I can wear my consequentialist hat and explain that they are different because they are used differently.
The flight plan is programmed into the airplane's avionics, as well as on a printed hard copy in the cockpit. The app is supplemental. It can not be used as a primary navigation reference, per FAA regulations.
Nothing in AA IT would surprise me. I did some contracting there. Their technology groups are so horribly f'ed up. The only client I've had worse was the government.
I hope there isn't a guy sitting somewhere who accidentally rejected the certificate :) I think that also locks everyone out. If the iPad's all crashed though it seems unlikely. Although it's definitely something that could be easily missed.
Ah, but when you read stories like this, I wouldn't trust the reporter (or even the interviewee) to be 100% accurate. "All crashed" could mean anything from a dead battery to being PIN locked to failure to launch because of an expired code signing cert :P
Good point and definitely possible. In fact I've done it myself - switched computer rejected the old certificate and suddenly nobody could access the app. Fortunately at the time it was only a few people and a beta so not too big of a problem.
> The pilot came on and said that his first mate’s iPad powered down unexpectedly, and his had too, and that the entire 737 fleet on American had experienced the same behavior.
user mode programs causing power down is not an app bug
Nothing safety critical would have been entrusted to the iPad. The pilots would be able to deal with the problem with either ATC support (anything the pilots can read to each other, ATC can read to them) or backup papers.
Yes, if you're on the ground, you won't be taking off without the stuff, but that's obvious when it can be fixed by the pilot getting out for a few minutes.
People don't screw around with plane safety. The failure mode of 'our iPads turned off' will have been considered as a prelude to their being introduced.
> Nothing safety critical would have been entrusted to the iPad.
I'm afraid I must disagree here. The iPad is a replacement for a bunch of paper manuals and maps and charts which are the backups to the ones contained in the flight computers. So if your iPad doesn't work any more, you don't have a backup anymore and that is safety critical.
It is not safety critical. In the unlikely event of both flight computers failing (many aircraft carry a backup FMC/FMS, but even a single computer is far more reliable than anything made on HN), and the consumer iPad which is barely FAA approved as a backup not working, any pilot can land the aircraft with their eyes closed. You think reference materials for the aircraft are the only thing keeping you alive and everything is hopeless if the pilot loses his weight tables? They're specifically trained when they get type rated and have all of the important figures memorized. Not ideal, sure, but "not safe" is another ballgame.
EFBs have a long history in the cockpit. Charts are required to be carried by FAR, but honestly, a scheduled commercial flight genuinely needing to consult one in anger is very rare. That was the impetus to get the FAA to approve iPads for EFB use in the first place, I think, was pilots making the case that they're needed so infrequently that an iPad is fine.
Threads like this make me wish there was more general awareness of what goes on in the cockpit, because I think it would go a long way toward assuaging unnecessary fear. Aviation is one of those rare regulation success stories where pretty much everything has been thought of.
Kind of weird, there historically have been similar 'do it the old school way because its more reliable' fallbacks for other things in air transport. What happened to the planes that were mid flight when this bug occurred?
Aircraft approaches are issued every 90 days. To stay up to date you need to have a book of these 'plates' for every region of the U.S, or a subscription to a service that sends you paper updates. If this happened in flight they would either get ATC to read them the plate, or check with ATC that whatever outdated plate they had was still accurate.
Obviously though this is a pretty big technical error on the part of Jeppeson (the app provider). It's pretty clear that it shouldn't be possible for the apps to stop working due to something done remotely. There is another provider, called ForeFlight, which I think does a better job, and I was glad to see they weren't involved.
Thanks for elaborating on the issue. I didn't know that the approaches are issued in that short time-interval. That makes the use of an app like this at least comprehensible.
AFAIK this is more of a regulatory thing than a direct safety issue. This has to work to be allowed to take off. When in flight though, they have no choice but to keep flying.
The device not working could have become an issue if something unexpected happened though.
All the threats mentioned there - jamming, spoofing, impersonation, etc. - can already happen with the unencrypted radio calls it's supplementing. With a hundred dollars or so worth of radio, you can hop on your local airport's ATC frequency and hold the mic down, or pose as the controller.
I suppose the argument is, when there's a person in the loop it's easier to notice funny business. If someone hops on the ATC frequency, the odds are someone will notice more quickly.
When it's mediated by a computer, you tend to trust it as a magic box of truth- and injecting subtle errors is going to be easier with ADS-B than on a voice channel, isn't it?
It is outrageous that flights could be stopped because of a reliance of shiny iThings and the software running on them.
Paper versions do not run out of juice.
There is a place for shiny tech, but the flight deck is not it. Paper worked, and we should go back to it.
> An Electronic Flight Bag, which replaces more than 35 pounds of paper-based reference material and manuals that pilots often carried in their carry-on kitbag, offers numerous benefits for American and its pilots.
> ... removing the kitbag from all of our planes saves a minimum of 400,000 gallons and $1.2 million of fuel annually based on current fuel prices. Additionally, each of the more than 8,000 iPads we have deployed to date replaces more than 3,000 pages of paper previously carried by every active pilot and instructor. Altogether, 24 million pages of paper documents have been eliminated
> ... All American pilots now enjoy the benefits associated with replacing their heavy kitbags – one of the airline's biggest sources of pilot injuries – with a 1.35-pound iPad. The digital format also requires less time to update each of the six or more paper manuals found in each pilot's kitbag, as manual paper revisions take hours to complete every month, compared to the minutes it takes for electronic updates.
Paper comprises an entire suitcase. Charts, airport docs, aircraft manuals, maint logs, regulations, etc etc. Updates are frequent and also paper. The fuel savings alone from not carrying the paper is substantial. As long as there's a backup tablet, it should be okay. If they get lost they can always call in for vectors.
Well, airlines have to very aggressively optimise for efficiency with everything they do. They have huge revenues and very little in the way of profit (about 5%).
It'd be foolish for them to not save money when they can.
If you're curious, check out https://www.foreflight.com/, it's available from the app store and is marketed towards GA pilots. I've used it with some success on a domestic US flight with the inflight WIFI.
EDIT: I just checked ForeFlight had a version update on 4/27. I doubt American is using the same software on their tablets, but then again you never know.
Certification for GA and large jet commercial passenger service are two different things. GA pilots can and often do use various portable navigational and other devices that are in some ways more sophisticated than what commercial airliners have, but they are not certified for commercial use.
Eh, yes and no. The FAA requires operators to get approval for iPad EFBs individually, and does not require GA pilots to do any such thing. "Certified," however, has a special meaning in aviation. iPads are not certified hardware, period. Pilots must still use their aircraft avionics (which are certified) as their primary means of navigation. iPads can be supplemental only, for both general aviation and commercial operations.
Thanks for the better title on HN. If you look at most reports on the issue, the title mentions "An iPad glitch" making it sound the iPads are at fault rather than a software issue.
Yes, the app crashed. That was no doubt an app bug. The iPads also crashed. That's eyebrow-raising, and definitely an iPad glitch.
The airline will fix the bug and it won't happen again. But relying on a closed, unauditable platform to keep any sort of safety-critical data in these conditions seems absolutely unconvincing to me.
These enterprise apps aren't vetted by Apple, so they could very well be using unsafe APIs Apple doesn't recommend and would block during an App Store review process. And there are tons of ways to crash machines from userspace. Also, it's likely the iPads are running an old version of iOS. And for all we know, the iPads are jailbroken and all bets are off. Finally, it sounds like AA's software provider was able to deliver a patch, so I don't think it's just a matter of iOS being "unauditable". What OS would they be using that met your auditability standards? Even Android is not totally open.
It's not safety-critical data; that absolutely wouldn't be allowed. If something were to go wrong in flight, the air traffic control would have no problem getting the plane down with no problems.
Furthermore, you have 2 iPads in the cockpit, so it could only be a problem if both fail at the same time. This seems like it'd be a more rare occurrence in the safety case.
In terms of whether open source could have helped, I doubt it very much. Stopping a bug like this (beyond extensive in-house testing) is to largely declare the platform and software bug-free (individual audits typically focus on properties like security, which are easy to target), and a full operating system (including UI) is far too large to effectively audit in that way.
Certainly iOS gets far more user milage than any free alternative, which is a big advantage there.
I'd have far more faith in Apple's display system than I would the eminently auditable Gnome (this is all assuming that the FAA did not have access to the iOS source, which is unknown at this time).
Can anybody here estimate the money lost due to late flights, crews, etc? I don't know where to start from but I'd love to compare it with the costs of keep using paper.
Seems a tad generous considering a back-of-the-envelope calculation puts the average amount contributed to GDP per US citizen per hour at around $6.
From an airline point of view, if their schedule is sufficiently flexible to accommodate the delays without flight cancellations and the delays are insufficiently long to require passengers to be compensated, the biggest costs will be additional fuel if the delays involve the aircraft being put in a holding pattern at the other end and additional time or penalty charges levied by the airports. Staffing isn't that big a deal even if they're on overtime.
Though if you’re travelling for business - you’re probably not doing "normal" work. Travelling to sign big deals, and so on. Even a few large deals would skew that number upwards.
The DOT numbers are based on median hourly income of airline travelers, rather than their contribution to the economy. You can read the report I linked for their detailed methods.
As a former iOS Developer for American I can say that while there is a rather large effect on the entire network, it pales in comparison to the savings gleaned from having the iPads.
I couldn't give specific numbers on the lost revenue here. It could be anywhere from a few thousand to several hundred thousand dollars. But there are people at the airline who analyze the cost of delays and they can be pretty significant - more so if this occurred at fleet launch in the early morning (though that doesn't seem to be the case).
Is it me or is this headline misleading. As in AA for one reason or the other has a faulty app that refuses to launch or crashes the iPad. How is this Apple's fault if you want to be fair. This is like saying 7 bakerys had to close down for the week because their HP desktop crashed , when really they did not upgrade their software. What happened to investigative reporting rather than clickbait?
Perhaps they chose too complex a platform, given the use case. Paper, while it has its own failure modes, is pretty failsafe, especially for information that doesn't change during the course of the flight.
I wonder if there is an in-between solution that would provide the benefits of an electronic display without introducing as many failure modes as a device like an iPad which is designed primarily for consumer use.
Don't forget about the savings for American Airlines. No more paper maintenance manuals, saving lots of weight, on every flight. The ipad powering down will probably result in a claim to the supplier of the ipads. Of course it is embarrassing, but think of the environmental benefits of saving all that weight.
35 pounds, or around 16kg. That is rather heavy...
I'm thinking that they should not all carry iPads, but have one iPad and one other device from a different manufacturer running different software but with the same information, for redundancy purposes. The weight of carrying an iPad and backup device is still tiny compared to what it replaced.
> and one other device from a different manufacturer running different software but with the same information, for redundancy purposes
This sounds like a great idea in theory, but as far as I know in practice isn't ideal. Problems include less familiarity with the second system/more training time required, issues with the second system may not be discovered since it's used less, and less incentive to get the first system right in the first place.
What would perhaps be better is a backup running a previous software version, maybe.
> they didn't have any contingency plan as backup.
Do you have any Source of your claim that there was no contingency plan?
FAA regulations seems to indicate that there are plans:
"(5) Procedural means.
g. Procedural Mitigations. If one or more onboard EFBs fail, resulting in loss of function
or the presentation of false or hazardously misleading information, a contingency plan or process
will need to be in place to provide the required information."
The same articles says:
"The pilot told us when they were getting ready to take off, the iPad screens went blank, both for the captain and copilot, so they didn’t have the flight plan,” " (emphasis mine)
What I meant is taht they didn't have a second electronic contingency plan. Probably they had to switch to paper on those flights (otherwise the impact would have been minimal)
My wife works as cabin crew for a major international airline. A few years ago she was grateful that they abolished the requirement that all crew (an A380 has around 30 crew) had to bring their manuals for every flight. The book was about 3kg and rather bulky so took up most of the room in her flight bag where she was also supposed to keep a jacket, jumper, pair of shoes (they switch from heels when boarding to flats during flight), hat, make up and various bits of documentation.
Which sounds like BS / urban legend. $40.000 for (generously) 1 gram x 300 passengers = less than a pound per flight?
[UPDATE] not sure for fuel, but might be true in olive prices, given these numbers:
Their website says 6,700 flights per day. Let's round it to 6000.
6,000 flights a day -> It's 2,190,000 flights a year x 100 meals per flight x 1 olive saved -> 219,000,000 olives saved.
This list http://sizes.com/food/olives.htm gives about 80-150 olives per pound (depending on the kind of olives), let's say 120 on average.
This makes 219,000,000 olives -> 1,825,000 pounds.
If they get them in bulk for like $1 or even $0.1 per pound, that's still an impressive $182,500 saved per year!
(Not sure if all flights include meals though. I used 100 meals per flight as a guesstimation could be higher/lower depending on the duration of most flights and the percs included).
I don't know how accurate that is, but it passes a basic smell test for me.
An olive is about 6 grams, so the per-olive fuel consumption is about 0.0001 gallons. Jet fuel currently costs about $1.70/gallon, so that's about 0.017 cents per olive in fuel.
To save $40,000 in fuel per year by reducing olives, you'd be flying about 200 million fewer olives.
American flies about 500 million passengers per year, so $40,000 in fuel savings could be achieved with about 0.4 olives saved per passenger.
I have no idea if this actually happened, but the numbers look about right!
I'll use yours: 1 gallon for 100 pounds of weight.
If an olive is 6 grams (0.013 pounds), then 100 pounds need: 7,692 olives.
For $1.7/gallon, this gives: .00022 dollars fuel cost per olive (or 77 times less than your estimation).
For 500 million passengers with one olive less, we got: 500 million less olives x 0.00022 = $110,000 in savings.
Taken from another angle, 300 passengers * 0.00022 savings per olive/passenger is: 0.066 savings per flight.
From the 6000 flights/day they have on their site, this gives: 6000 * 365 * 0.066, or a saving of $144,540, so same ballpark.
They seem to save at least $80-$100,000 per year in both fuel and olive costs by removing just one olive.
Of course while it sounds amazing to us, it involves 6000*365 flights / 500 million passengers a year.
Which means that this seemingly "large" number (when compared to say, our salary) is actually peanuts compared to their operating costs. Like, by removing one olive per passenger, they reduce their operating costs per flight by less than .000025%.
Your estimate and mine are almost the same. 0.017 cents is $0.00017 is close to $0.00022. The difference is because I did some heavy intermediate rounding, what with olive weight being highly approximate to begin with.
You're right that in relative terms, the savings is tiny. But absolute numbers count as well. If it saves $40,000/year then that means one full-time employee can spend something like 25-50% of his time working on this problem and they still come out ahead.
While I am sure there are plenty of advantages to a full on app. I kind of wonder what the downside of a lot of that content being just PDFs is. Seems like there are less things to go wrong with that approach.
One of the first things I was taught in a freshman CIT class was "Don't forget about paper! It has its place!" Not saying pilots should have to carry around heavy paper charts...but yeah.
I think I remember reading here that an airport was able to function even after there computers went down since they kept backup printouts of entire flight schedules. Seems like a pretty big contrast.
Because it is not mission-critical in the first place. If you think the flight plans on the iPad is the sole critical factor in the overall flight operation, you're mistaken.
There are fallbacks, there are fallbacks even for the fallbacks and finally, the pilots themselves are trained in the event of total failures on all ends.
In addition, it was a software glitch. We don't know what kind of glitch it was but if it was, it doesn't matter if they were using proper-graded billion dollar worth of products, software glitches exist everywhere.
I think that is for passengers during the critical phases (takeoff/landing) so that they are not distracted.
A few airlines have switched over to digital maps for airports, since they offer some substantial advantages (less weight, more up to date, etc).
If you want to have a look at one of these apps, its on the public app store (https://itunes.apple.com/de/app/lido-iroutemanual-aeronautic...) - though I could not find a source that AA is using the LH Systems app (and if it was the source of the problem, I guess there would be several more stories about it.
edit: Source: did some work for LH Systems, but not affiliated with them.
"Was" being the correct word; airlines have become more and more lenient with that, and I'm sure airplane manufacturers do better shielding and error correction.
My anecdotal evidence supports OP's and againts yours. My Ipad crashes more frequently with each iOS upgrade. My wife's Ipod touch is even more prone to crash that the Ipad. I just gave up with both iOS and Android (I gave the Ipad to my son). It turns out I only needed a phone and a kindle paperwhite after all.
Actually this seems extraordinarily unlikely. Your proposition is that a no-iOS zone was maliciously created in multiple highly secure and surveilled areas across the country (ie airports) but despite the fact that the hack should affect all iOS devices in range, there were no reports whatsoever of any passenger devices (nor the pilots iPhones for that matter) being disabled.
while I fully agree with your last part, the first part of your argument is dubious at best. it would be trivial to get away with a malicious access point in an airport. they helpfully provide charging stations for your electronics, and all it takes is an identically sized and shaped power brick to drop in a malicious access point.
It would be trivial to get away with it in one airport. Doing it in multiple airports spread across the destinations where AA flies simultaneously would be significantly less trivial.
I've observed in the wild a few issues where updates to MDM software or bugs in applications crash iPads. MDM/MAM control layers usually have problems right after iOS releases.
AppStore apps can cause issues too. Chrome had an issue last year that crashed some of our iPad2s.
(The iOS enterprise in-house code signing certificates, which only Apple can issue, always seem to come with 1-2 year hard expiration dates)