The exhaust valve is by far the most likely component of a piston aircraft engine to fail catastrophically in flight. When an exhaust valve fails, combustion ceases in the associated cylinder. The engine loses power and starts running rough. In a single-engine airplane, this usually results in a precautionary landing—on-airport if you’re lucky, off-airport if you’re not.
Once in a while, an exhaust valve failure has even more serious consequences. If the engine is turbocharged, the liberated valve fragment can take out the turbocharger on the way out. Rarely, it can get wedged into the piston crown, shatter the piston, and cause a total power loss. The results are sometimes fatal.
That’s why it’s so important to detect exhaust valve failures early while they are still in the incipient stage, well before they can cause serious problems. The traditional way of doing this is via the annual compression test, but that hasn’t proven to be a very reliable method because loss-of-compression typically doesn’t occur until the sick valve is very close to failing.
A borescope inspection is vastly better and can detect failing valves a lot earlier. AOPA publishes a great poster to help mechanics understand what to look for. Unfortunately, most shops don’t do regular borescope inspections, and the ones that do typically do them only at the annual inspection. That may not be often enough to prevent exhaust valve failures reliably, especially for engines that fly a lot.
It seemed to us that there had to be a better way. For engines equipped with a recording digital engine monitor—and nowadays that’s well over half the piston fleet—we think the answer lies in the cutting-edge field of predictive analytics.
Predictive Analytics
Predictive analytics is the use of data, statistical algorithms and machine learning techniques to identify the likelihood of future outcomes based on historical data. Instead of just using data to analyze what happened, the goal of predictive analytics is to assess what is likely to happen in the future.
Over the past decade, remarkable advances in this technology have been made in the airline industry. Boeing has been in the forefront with its groundbreaking Airplane Health Management program for the 777 and 787, but Airbus, GE, Pratt & Whitney and Rolls-Royce have been investing heavily in this area. The resulting capability to use aircraft sensor data to predict incipient component failures has been quite remarkable.
At Savvy, we felt that piston GA ought to be able to benefit from this technology, too. So in 2014, we launched a program whose goal was to predict incipient exhaust valve failures based on digital engine monitor data, and we dubbed it FEVA™—Failing Exhaust Valve Analytics.
Humble beginnings (FEVA 1.0)
Over our many years of looking at graphical depictions of digital engine monitor data from piston aircraft engines, we occasionally observed an unusual pattern—a very slow, rhythmic EGT oscillation with a frequency of roughly one cycle per minute—that seemed to correlate strongly with burned exhaust valves. All of our human analysts were trained to keep an eye out for this pattern, and to alert our clients whenever they saw it so that the corresponding cylinder could be borescoped to check the condition of the exhaust valve. More often than not, cylinders exhibiting this peculiar EGT oscillation turned out to have exhaust valves that were in trouble.
Of course, of the more than 3,000,000 flights whose data has been uploaded to the SavvyAnalysis platform, only a small fraction were examined by our human analysts. So in 2014, we decided to create a computer algorithm—now referred to as FEVA 1.0—to scan every flight uploaded to the platform for this EGT oscillation pattern. Whenever the algorithm spotted the pattern in the data (as it did in EGT #5 in the chart above), it would alert one of our analysts to look at the flight. If the analyst agreed that the EGT pattern looked suspicious, we’d contact the client and suggest a borescope inspection of the cylinder.
FEVA 1.0 was essentially an “expert system”—an algorithm that attempts to emulate the decision-making ability of a human expert. We back-tested it on hundreds of actual flights, comparing its output to the findings of our human analysts. We tweaked and re-tweaked the algorithm to get it to alert on the same flights as our human analysts did. We then fine-tuned it to achieve an optimum balance between maximum detection sensitivity and minimum false-positive rate.
Not good enough
FEVA 1.0 was definitely a step in the right direction, and gave us the ability to alert our clients to incipient exhaust valve problems that might well have otherwise been missed. But it wasn’t good enough. Clients would sometimes send us borescope images of valves that were obviously very sick and close to failure, and ask us “why didn’t FEVA alert me about this?” That’s embarrassing.
The problem was that FEVA 1.0 was just too narrow-minded. While it’s true that cylinders exhibiting slow rhythmic EGT oscillation usually have a failing exhaust valve, the converse is not true. Cylinders with failing exhaust valves don’t always exhibit these telltale EGT oscillations.
For example, if the valve’s rotator cap stops working properly or if the valve starts sticking in its guide due to deposit buildup, the valve might start heating unevenly and burning badly, but the EGT would not oscillate. FEVA 1.0 wouldn’t detect such failures. For that matter, Savvy’s human analysts might not be able to spot them, either.
It became apparent that any algorithm capable of reliability predicting incipient exhaust valve failures would need to look for more than just EGT oscillations. It would probably need to look at more than just EGT. It might need to look for complex patterns and interrelationships among data obtained from many different sensors.
But, what patterns and interrelationships should we be looking for? Other than EGT, what sensor data should we be looking at? We had no clue. Nor, for that matter, did Continental, Lycoming, Rotax, or anyone else in the industry.
“I bet we could get the computer to figure this out!” exclaimed Savvy’s vice president Chris Wrather over brunch one day. Chris brings a unique blend of skills to Savvy—entrepreneur, airplane owner, instrument pilot, A&P mechanic, computer programmer and Ph.D.-level mathematician. As luck would have it, Chris had just completed a distance-learning course on machine learning from Stanford University.
A better mousetrap (FEVA 2.0)
Suppose you wanted to create a computer algorithm that could look at a photo and determine whether it was of a man or a woman. One way of doing that would be to assemble a group of experts and ask them to come up with a set of rules that could be programmed into the computer: e.g., men have broader shoulders and more facial hair, women have more curves and fancier hairdos. Another way would be to feed the computer thousands of photos, each one identified as male or female, and let the computer figure out for itself how to tell which is which. The former is called an “expert system” while the latter is called “machine learning.”
Machine learning has a couple of distinct advantages over the expert system approach. One is that computers are a lot better at figuring out complex patterns and interrelationships among a large number of input variables than humans are. Another is that machine learning models keep getting smarter and smarter over time as they get fed more and more training data.
Chris became convinced that a FEVA 2.0 based on machine learning was a better way to go. The plan was to program the computer with a generalized machine-learning algorithm known as the “random forests” model whose inputs would include pretty much everything we could get from the aircraft’s digital engine monitor—more than 30 different input variables (for monitors that capture that many).
The model would then be “trained” by feeding it data from thousands of actual GA flights, some involving engines known to have failing exhaust valves, others from engines with valves known to be healthy. These flights would constitute “the training set.” For each flight in the training set, we would tell the model whether the flight was from a healthy-valve engine or a sick-valve engine and, for a sick-valve engine, which cylinder or cylinders had failing valves. It would then be up to the computer model to figure out how to tell the difference.
After a long period of training and back-testing, we deployed the new FEVA 2.0 machine-learning model about a year ago. It proved superior to FEVA 1.0 in every respect. It caught more failing exhaust valves. It generated fewer false alarms. Also, in contrast to FEVA 1.0’s simple go/no-go output, the FEVA 2.0 model yielded a numeric “risk score” for each cylinder that could range anywhere between very low risk to very high risk. We advised owners of engines with one or more cylinders having above-average FEVA 2.0 risk scores to have those cylinders borescoped at the earliest convenient opportunity.
Over the past year, we’ve been carefully accumulating more and more flight data that we are able to correlate with actual valve health findings obtained from borescope inspections and cylinder removals. In that fashion, we are growing our training set so we can continue to train the FEVA model to improve its predictive performance.
Better is now smarter (FEVA 2.1)
This week, we’re delighted to announce the release of FEVA 2.1, a significantly improved version of the FEVA machine learning model that provides a 20% improvement in predictive performance compared with FEVA 2.0 (as measured using our test dataset). This latest version of FEVA differs from its predecessor in two important ways:
- FEVA 2.1 expends the list of input variables to 35+ including several new ones that FEVA 2.0 did not consider; and
- FEVA 2.1 draws its input data from an aircraft’s entire recent flight history, rather than from only the most recent flight as was the case of FEVA 2.0.
As a result of these improvements plus our expanded training set, FEVA 2.1 is both 20% better at predicting failing valves (“sensitivity”) and 20% less likely to make a false prediction (“positive predictive value”) than FEVA 2.0. In addition, successive FEVA 2.1 reports will be more consistent over time than was the case with FEVA 2.0.
As was the case with FEVA 2.0, FEVA 2.1 predictive reports will be available for all piston aircraft enrolled in SavvyMx, SavvyQA, and SavvyAnalysis Pro.
The proof is in the pudding
To illustrate how far FEVA has come since its introduction seven years ago, a case study may be instructive. In 2015, Wayne purchased a beautiful 2008 DA-40 Diamond Star. Savvy managed the prebuy for Wayne, and has been managing the maintenance ever since. Wayne has been religious about uploading the data captured by his Garmin G1000 to the SavvyAnalysis platform.
In July 2018, Wayne had an annual inspection performed at a shop in Truckee, California. When the shop performed a compression test on the plane’s Lycoming O-360 engine, the compression on cylinder #2 was 0/80! A borescope inspection revealed a seriously burned exhaust valve in that cylinder that probably would have failed soon had it not been caught at the annual. Here’s what it looked like with the borescope:
Each of Wayne’s flights had been scanned by FEVA, and none raised a red flag that the #2 exhaust valve might be failing. The question is: why not?
Keep in mind that this took place in 2018, two years before Savvy introduced the FEVA 2.0 machine learning model. At that point in time, FEVA was still an expert system programmed to alert whenever it found a slow, rhythmic EGT oscillation that we knew was reliably indicative of a failing exhaust valve.
Let’s look at the EGT data from Wayne’s last flight in the DA-40 before he put it in the shop for its 2018 annual inspection:
The orange trace in the chart is EGT #2. It sure looks like something odd is going on there, doesn’t it? Let’s look closer at the cruise segment of the flight:
Oh yeah! Something is definitely wonky with EGT #2. But it doesn’t exactly look like the sort of slow, rhythmic oscillation that FEVA 1.0 was programmed to look for. It looks more like a FAST, rhythmic oscillation. Let’s zoom in a bit more to see just how fast:
At this magnification, it’s easy to see that EGT #2 is oscillating at a frequency of about one cycle per 10-11 seconds. This is six times faster than the one cycle per minute oscillation that FEVA 1.0 was designed to look for. In fact, FEVA 1.0 was specifically programmed to ignore EGT oscillations faster than one cycle per 30 seconds based on the “expert assumption” that oscillations faster than that almost had to be noise. No wonder it didn’t generate an alert for Wayne’s failing exhaust valve.
Would the machine learning model have done better? We wondered the same thing, so we ran Wayne’s flights prior to the 2018 annual inspection through FEVA 2.1’s machine learning model. Here’s what came back:
Wow! The FEVA 2.1 model returned an off-the-charts risk score for cylinder #2, one of the highest risk scores we’ve ever seen the model produce. How exactly did it come to the conclusion that the #2 cylinder was at extraordinarily high risk for exhaust valve failure? That’s hard to say.
The FEVA 2.1 model uses more than 35 input variables to make its predictions. It looks for hidden patterns in the data, some of which might not be apparent to the human eye. It’s very difficult to identify which specific variables are causing the model to predict a higher failure probability for a particular cylinder because the model is extremely complex. The machine learning approach used by FEVA 2.1 is enormously powerful. The drawback is that it is difficult to tease out the effects of individual variables in order to arrive at an explanation of why the model made the prediction that it did.
What we can say is that FEVA 2.1 did not have access to information from the compression test or the borescope inspection. It did look at EGTs and a whole bunch of other things derived from engine monitor data, and it looked at them over a series of flights in order to come up with its prediction that the #2 exhaust valve was at high risk of failing.
An exhaust valve failure can be an expensive and dangerous event. Over the past seven years, the FEVA feature of SavvyAnalysis has scanned uploaded data from more than three million GA flights for the telltale signs of a burned exhaust valve whenever engine data is uploaded to the platform. On more than 60 occasions, FEVA has predicted exhaust valve failure before it happened.
In early 2020, Savvy achieved a quantum leap forward with the deployment of the FEVA 2.0 machine learning model. This was the first practical application of machine learning technology to GA maintenance. It greatly improved FEVA’s predictive accuracy and sensitivity compared to the original expert system approach.
The just-released FEVA 2.1 upgrade incorporates more predictive variables from an aircraft’s entire recent flight history, resulting in predictions that are 20% more accurate and sensitive. This cutting-edge technology is available to subscribers to SavvyMx, SavvyQA, and SavvyAnalysis. Savvy is working hard to keep making our predictive analytics smarter and more accurate, and to extend it to other safety-critical areas such as sticking valves.
If you’re flying a piston single—especially one with a four-cylinder engine—isn’t it bordering on irresponsibility NOT to take advantage of this failure-prediction technology that might just save your bacon someday?
You bought a plane to fly it, not stress over maintenance.
At Savvy Aviation, we believe you shouldn’t have to navigate the complexities of aircraft maintenance alone. And you definitely shouldn’t be surprised when your shop’s invoice arrives.
Savvy Aviation isn’t a maintenance shop – we empower you with the knowledge and expert consultation you need to be in control of your own maintenance events – so your shop takes directives (not gives them). Whatever your maintenance needs, Savvy has a perfect plan for you: