Early in the development of the 737 MAX, engineers gathered at Boeing's transonic wind tunnel in Seattle to test the jet's aerodynamics using a scale model with a wingspan comparable to that of an eagle.
The testing in 2012, with air flow approaching the speed of sound, allowed engineers to analyze how the airplane's aerodynamics would handle a range of extreme maneuvers. When the data came back, according to an engineer involved in the testing, it was clear there was an issue to address.
Engineers observed a tendency for the plane's nose to pitch upward during a specific extreme maneuver. After other efforts to fix the problem failed, the solution they arrived at was a piece of software ̵
This is the story, including previously unreported details, or how Boeing developed MCAS, which played a critical role in two airliners nose-diving out of the sky, killing 346 people in Ethiopia and off the coast of Indonesia.
Extensive interviews with people involved with the program, and a review of proprietary documents, show how Boeing originally designed MCAS as a simple solution with a narrow scope, then altered it late in the plane's development to expand its power and purpose. Still, a safety-analysis led by Boeing concluded that there would be little risk in the event of an MCAS failure – part of an FAA-approved assumption that pilots would respond to an unexpected activation in more than three seconds.
The revised design allowed MCAS to trigger on the inputs of a single sensor, instead of two factors considered in the original plan. Boeing engineers considered that lack of redundancy acceptable, according to proprietary information reviewed by The Seattle Times, because they calculated the probability of a hazardous MCAS malfunction to be virtually inconceivable.
As Boeing and the FAA advanced the 737 MAX toward production , they limited the scrutiny and testing of the MCAS design. Then they agreed not to inform pilots about MCAS in manuals even though Boeing's safety analysis expected pilots to be the primary backstop in the event the system went haywire.
The grounding of the MAX has entered it in the 15th week. Safety officials around the world are scrutinizing the changes to MCAS that Boeing has proposed to ensure such accidents won't happen again. And they are assessing what training pilots may need on the new system.
"Safety is our top priority," Boeing said in a statement. “Through the work we are doing now in partnership with our customers and regulators to certify and implement the software update, the 737 MAX will be one of the safest airplanes ever to fly.”
This investigation examines what is known about the origins and operation of MCAS ahead of the final official accident investigation reports, expected late this year for Lion Air Flight 610 and next year for Ethiopian Airlines Flight 302.
Wind tunnel and simulator tests
Though Boeing was locked into a plan to revamp its popular 737 model, the Seattle wind-tunnel tests in 2012 revealed a problem
During flight tests to certify an airplane, pilots must safely fly and extreme maneuver, a banked spiral called a wind-up turn that brings the plane through a stall . It was probably never experienced the normal commercial flight, it could occur in pilots for some reason needed to execute a steep banking turn.
Engineers determined that on the MAX, the force the pilots feel in the control column as they execute this maneuver would not smoothly and continuously increase. Pilots who pull back forcefully on the column – sometimes called the stick – might suddenly feel a slacking of resistance. An FAA rule requires that the plane handle with smoothly changing stick forces
The lacquer or smooth feel was caused by the jet's tendency to pitch up influenced by shock waves that form over the wing at high speeds and the extra lift provided by the pods around the MAX engines, which are larger and faster forward on the wing than on previous 737s.
This was verified in early simulator modeling, with plans tested in scenarios at about 20,000 feet of altitude , According to one of the workers involved.
While the problem was narrow, it proved difficult to cope with. The engineers first tried tweaking the plane's aerodynamic shape, according to two workers familiar with the testing. They placed vortex generators – small metal vanes on the wings – to help modify the flow of air, trying them in different locations, in different quantities and different angles. They also explored altering the shape of the wing.
Two people familiar with the discussions said 737 MAX chief test pilot Ray Craig preferred such a physical solution to solve the plane's aerodynamics. Philosophically, Boeing had long established efforts to create automated actions such as stick-pusher – a device used on some aircraft without pilot action pushing the control column forward to lower the jet's nose – that would seize control of a situation from the pilot, according to one of the people.
But the aerodynamic solutions didn't produce enough effect, the two people said, and so the engineers turned to MCAS.
It was simple in concept but powerful
In the middle of a wind-up tower, the software would automatically swivel up the leading edge of the plane's entire horizontal tail, known as the horizontal stabilizer, so that the air flow would push The tail up and correspondingly push the nose down.
As the pilot pulled on the control column, this uncommanded movement in the background would counter the jet's tendency to pitch and smooth out the feel of the column through. out of the maneuver.
An engineer recalled Craig testing MCAS for the first time in the simulator.
“Yeah! This is great, ”Craig gushed after seeing how MCAS responded, according to the engineer. (Craig left Boeing before the operation of MCAS was revised.) This original version of MCAS, according to two people familiar with the details, was activated only in distinct sensors such as: A high angle of attack and a high G force.
Angle of attack is the angle between the wing and the oncoming air flow. G-force is the plane's acceleration in the vertical direction.
How much MCAS moved the tail when it was activated by the angle of attack and the jet's speed, said one of the people familiar with the MCAS design who, like many of the sources in this story, asked for anonymity because of the sensitivity of ongoing investigations.
The fix didn't stir much controversy.
Another Boeing plane, the KC-46 Air Force tanker , a software-driven system similarly moves the stabilizer into a wind-up tower and even has the same MCAS name, though the design is very different.
Boeing's failure analysis
When Boeing was ready to certify the 737 MAX, the laid out plan for MCAS in documents for the FAA.
Under the proposal, MCAS would trigger in narrow circumstances. It was designed to address potentially unacceptable nose-up pitching moment at high angles of attack at high airspeed, "Boeing told the FAA in a proprietary system safety assessment reviewed by The Times.
In a separate presentation made for foreign safety regulators that was reviewed by The Times, Boeing described MCAS as providing a nose down command to oppose the pitch up. Command is limited to 0.6 degrees from trimmed position. ”
Two people involved in the initial design plans for MCAS said the goal was to limit the system's effect, giving it as little authority as possible. That 0.6-degree limit was embedded in the company's system safety review for the FAA.
The Boeing submission also included an analysis that calculated the effect of possible MCAS failures, with each scenario characterized as either a minor, a major or a hazardous. Failure to determine how much redundancy must be built in to prevent the event
Virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the "major failure" requirement, which is that The probability of a failure must be less than 1 in 100,000
A major failure is not expected to produce any serious injuries and is defined more as something that would increase the cockpit crew's workload. Such systems are usually allowed to rely on a single input sensor.
Boeing analyzed what would happen if, in normal flight mode, MCAS triggered inadvertently up to its maximum authority and moved the horizontal stabilizer to the maximum 0.6 degrees.
It also calculated what would happen on a normal flight if somehow the system kept running for three seconds at its standard rate of 0.27 degrees per second, producing 0.81 degrees of movement, thus exceeding the supposed maximum authority.
Why three seconds? This is the period of time that FAA guidance says it should take a pilot to recognize what's happening and start to counter it.
Boeing assessed both of these failure modes as "major." Finally, the analysis looked at the inadvertent operation of MCAS during a wind-up turn, which was assessed as "hazardous," defined in a cold actuarial analysis as an event causing serious or fatal injuries to a small number of people, but short of release the plane (that's called "catastrophic").
Hazardous events typically demand more than one sensor – except when they are outside normal flight conditions and unlikely to be encountered, such as a wind-up turn.
According to a document reviewed by The Seattle Times, Boeing's safety analysis calculated this hazardous MCAS failure to be almost inconceivable: Given the improbability of an airliner experiencing a wind-up turn, compounded by the unlikelihood of MCAS failing while it happened, Boeing came up with a Probability for this failure of about every 223 hours of flight. In its first year of service, the MAX fleet logged 118,000 flight hours. acceptable in all circumstances
In flight test, MCAS changes
About a third of the way through flight testing in 2016, as first reported by The Seattle Times in March, Boeing made substantial changes to MCAS.
The flight test pilots had found another problem: The same lacquer or smooth stick forces was also occurring in certain low-speed flight conditions. To cover that issue too, engineers decided to expand the scope and power of MCAS
Because at low speed a control surface must be deflected more to the same effect, engineers increased the power of the system at low speed from 0.6 degrees or stabilizer nose-down deflection to 2.5 degrees each time it was activated. On the stabilizer, maximum nose down is about 4.7 degrees away from level flight. So with the new increased authority to move the stabilizer, just couple of iterations of the system could push it to that maximum.
Because there are no excessive G-forces at low speed, the engineers removed the G-force factor as a trigger. But that meant MCAS was now activated by a single angle-of-attack sensor.
One of the people familiar with MCAS's evolution said the system designers didn't see any need for additional sensor or redundancy because the hazard assessment had determined that an MCAS failure in normal flight would only qualify in the "major" category for which the single sensor is the norm.
"It wasn't like it was to cover some safety or certification requirement," the person said . “The trigger isn’t a safeguard. It tells (the system) when to operate. ”
While the changes were dramatic, Boeing did not submit the revised system safety assessment to the FAA.
An FAA spokesman said the safety agency did not require a new "The change to MCAS did not trigger an additional safety assessment because it did not affect the most critical phase of flight, considered to be higher cruise speeds," he said.
The person familiar with the details of MCAS 'evolution said Boeing did the extra analysis of the new low-speed, higher-authority changes. He said the effect of the potential failures at low speed was less, and so did not add any risk to the prior analysis. So the documents sent to the FAA with the failure analysis were not revised.
"You turn in the answer," he said. "You do not have to document all your work."
MCAS as it was actually differentiated in another way from what was described in the safety analysis turned in to the FAA.
The failure analysis didn't appear to consider the possibility that MCAS could trigger repeatedly, as it did on both accident flights. Moving multiple times in 0.6 or 2.5 increments depending on the speed, it effectively had unlimited authority if pilots did not intervene.
Two former Boeing test pilots described a culture of pressure within the company to limit flight testing, which can delay projects at a time when orders are stacking up, costing the company money. Matt Menza, a different pilot who did test flights on the MAX , remembered times when test pilots at Boeing would have the chance to thoroughly examine systems in what he called a "system-safety murder board" to explore all the potential failures. But the general corps of test pilots did not have a lot of technical details about the MCAS design, such as the single-sensor input.
Boeing never flight-tested a scenario in which a broken angle-of- attack sensor triggered MCAS on its own, instead relying on simulator analysis, according to a person familiar with the process. One of the forms test pilots expressed bewilderment that the angle-of-attack was never explored in the air.
A variety of employees have described internal pressures to advance the MAX to completion, as Boeing tried to catch up with the hot -selling A320 from rival Airbus
Mark Rabin, an engineer who did flight-testing work unrelated to the flight controls, said there was always talk about how delays or even one day can cost substantial amounts. Rabin said, "It was all about loyalty," said Rabin. "I had a manager tell me," Don't rock the boat. You don’t want to be upsetting executives. ”
Do pilots need more training?
Boeing's system safety analysis of MCAS, working out the failure probabilities, assumes that the pilots will take steps in response to anything that arises, and will do so quickly.
The pilots' struggles to control their planes before both MAX crashes suggest that the FAA's three-second guidance for expected pilot response time, upon which part of Boeing's system safety analysis was based, needs to be carefully reassessed.  "If the three seconds are not the right amount of time to be able to catch a runaway stabilizer, and it actually takes seven seconds, then … we need to understand that," said the person familiar with the details of MCAS When MCAS is activated in the cockpit and moves the horizontal stabilizer, a large wheel beside each pilot is mechanically connected to the stabilizer starts to spin. This is the manual trim wheel. As a last resort to stop a stabilizer moving uncommanded, a pilot can grab and hold the wheel.
The person familiar with MCAS said the wheel will spin noisily and fast, 30 or 40 times, for each activation. Meanwhile the stabilizer movement will increase the force needed to hold the control column, by about 40 to 50 pounds for a 2.5 degree movement. Such uncommanded movement that won't stop is referred to as a "runaway stabilizer."
Boeing has said that to deal with this, pilots need first to have basic hand-flying skills – pull the nose up to where you want it , then use the thumb switches on the yoke that connect electrically to the stabilizer to neutralize the forces – and then shut off MCAS with a pilot checklist procedure on how to handle a "runaway stabilizer."
Off-set sensor failure set off multiple alerts causing distraction and confusion from the moment of takeoff, just before MCAS kicked in.
On the Ethiopian Airlines flight, for example, a "stick shaker" noisily vibrated the pilot's control column throughout the flight, warning the plane was in danger of a stall, which it wasn't; a computerized voice repeating a loud "Dank!" warned that the jet was too close to the ground; a "clacker" making a very loud clicking sound signaled by the jet was going too fast; and multiple warning lights told the crew that the speed, altitude and other readings on their instruments were unreliable.
Exactly what pilot training for MCAS is appropriate has become a big issue that threatens the grounding of the MAX.
While the FAA and US airlines seem ready to clear the plane to fly with just iPad training for American pilots on the MCAS fixes, some foreign regulators for more intensive simulator training for all pilots on how to handle a runaway stabilizer.
Early in the process of selling the MAX, according to two people familiar with the discussions, Boeing promised to give Southwest Airlines a substantial rebate for every plane if the MAX required simulator training.
One former MAX Worker, Rick Ludtke, said the rebate reported to him by managers was $ 1 million per plane, a figure representing another employee is roughly accurate.
A Southwest spokesperson said, "We do not discuss publicly the specific details of our contractual agreements. , ”But added that“ the purchase of an aircraft is a significant investment, and guarantees for various items… are incorporated into every 737 contract. ”
Ludtke and two other forms workers described internal pr. essences during the MAX certification to avoid any changes to the design of the plane that might cause the FAA to lean on a simulator mandate.
It became a significant point of attention for Michael Teal, the 737 MAX program manager, and Keith Leverkuhn , vice president and general manager of the 737 MAX program, according to a person involved in the discussions. They felt confident based on past experience that the MAX would be approved without simulator training, but they were wary, according to the worker.
Meanwhile, Boeing's chief technical pilot on the MAX, Mark Forkner, was also facing pressure, according to another person involved in the project. The person recalled Forkner as frequently anxious about the deadlines and pressures faced in the program, going to some of his peers in the piloting world for help. As first reported by The New York Times, Forkner suggested to the FAA that MCAS not
"Mark never dreamed of anything like this could happen," said Forkner's attorney, David Gerger. "He puts safety first – at this job and in the Air Force."
U.S. pilot unions have expressed concern at the omission of MCAS from the manual. One reason is that when MCAS activates, it changes somewhat the response of the airplane.
For example, there is a cutout switch in the control column so that when a pilot pulls or pushes in the opposite direction to a runaway stabilizer, it electric power to the stabilizer. When MCAS is active, this cutout switch does not work, which could surprise a pilot who didn't know about the system.
Boeing ultimately won the FAA's approval to give pilots just an hour of training through and iPad about the differences between the MAX and the previous 737 generation. MCAS was not mentioned
The FAA, after internal deliberations, also agreed to keep MCAS out of the manual, reasoning that MCAS was a software code that operates in the background as part of the flight-control system, according to an official familiar with the discussions.
A single sensor
Boeing has avoided accepting direct blame in public, saying MCAS was only one link in a chain of events. Its leaders have also said that MCAS was designed according to the standard procedures it has used for years.
The 737 MAX was certified in accordance with the identical FAA requirements and processes that have governed certification of previous new airplanes and derivatives. The FAA considered the final configuration and operating parameters of MCAS during MAX certification, and concluded that it included all certification and regulatory requirements, ”Boeing said in a statement.
The most controversial detail of the MCAS design has been the reliance on a single angle-of-attack sensor. On both of the deadly flights, everything started with a faulty sensor. In the second crash in Ethiopia, the data trace strongly suggests that the sensor was destroyed in an instant, likely at a bird strike.
There are two such sensors, one on either side of the fuselage. Why didn't Boeing, especially after discarding the G-force as a trigger, use both angle-of-attack sensors?
The thinking was that requiring input from two angle-of-sensors would mean that if one failed the system would not function.
That has implications not only for safety but for airline costs. If the system is down, a pilot might fly into a situation where it needed and find it unavailable. Or the airline might have to take the plane out of service and lose money.
Both factors point to a principle of not adding complexity: Keep a system as simple as possible. customer's operations, ”said the person familiar with the MCAS details. You don't want to "increase the risk that the system fails when you need it."
In this case, if simple as possible meant as minimal as the safety regulations allow. Since Boeing's system safety analysis found that one sensor was acceptable, that's what it went with.
But that's not the logic followed for a system on the KC-46 Air Force tanker, also called MCAS.
Boeing says the MCAS systems on the MAX and on the tanker share only a name and a similar function, and have completely different avionics.
But they both move the horizontal stabilizer to smooth the pilot stick forces in a wind-up turn. Air Force spokeswoman Ann Stefanek says "MCAS on the KC-46 has two sensors and the system compares the two readings."
Boeing's proposed update to MCAS for the MAX will have the same.
Last Sunday at the Paris Air Show, Boeing CEO Dennis Muilenburg reiterated the company's position that while the original MCAS was properly designed, "we know we can improve it."
The fixes include relying on two sensors rather than one, limiting MCAS to one rather than multiple activations, and revising the software.
“We are confident that they will result in a safe airplane, one of the safest airplanes ever to fly, and that MCAS will not contribute to a future accident, "he said.