16:00
S1: Industrial point of view of CBM at Amphi Fournel (AF)
Chair: Marta Ribeiro and Mathieu Ducousso
16:00
20 mins
|
Fleet-level Maintenance Cost and Availability Analysis based on Discrete Event Simulations
Sin-seok Seo
Abstract: Maintenance has consistently been a significant expense driver across various industries, including production [1] and aeronautics. Empirical investigations reveal that maintenance activities account for between 15% and 70% of total production costs within manufacturing industries [2]. Further research demonstrates that approximately 33% of every dollar allocated to maintenance in the United States is wasted due to unnecessary activities related to maintenance [3]. Despite these findings, the majority of the industry continues to depend on conventional maintenance policies. Moreover, manufacturers of aeronautic equipment have begun transitioning towards a subscription-based maintenance charging model, under which clients provide a consistent periodic fee, and the manufacturer absorbs the actual costs of maintenance. Considering this shift in charging models, there is a heightened necessity to minimize maintenance expenses while simultaneously ensuring maximum availability.
Historically, maintenance policies have undergone considerable evolution, and they persist as challenging subjects of research within the industry [4]. In the initial stages, the industry heavily relied on reactive maintenance, also referred to as corrective maintenance or run-to-failure. Subsequently, focus shifted towards preventive maintenance, an approach that takes repair actions prior to any occurrence of failure. This methodology progressed into Condition-Based Maintenance (CBM), where maintenance decisions relied on the condition indicators of machinery obtained commonly through measurement systems. Modern approaches such as predictive maintenance and Prognostics and Health Management (PHM) leverage condition monitoring data to predict the Remaining Useful Life (RUL) of a machine, and to make decisions based on this forecast.
These new maintenance policies allow for the reduction of maintenance costs while increasing availability [1]. However, it is challenging to ascertain how much we can improve with a new maintenance policy or which maintenance parameters — such as the periodicity of repairs — would offer the best trade-off between cost and availability.
A maintenance policy simulation tool, referred to as mp-sim (Maintenance Policy SIMulator), has been developed with the capacity to address these questions [5]. Mp-sim can simulate a spectrum of maintenance policies including periodic, condition-based, and predictive models in conjunction with advanced degradation models. It is a Python-based tool designed on the principles of Discrete Event Simulation (DES), effectively simulating the performance of systemic processes as a discrete series of events in time. This unique operating feature enables simulation time to jump directly to the timed occurrence of succeeding events, thus significantly enhancing computational efficiency [6] [7] [8]. Mp-sim’s functionality extends to a comparison of different maintenance policies, analysis of trade-offs between expense and availability, comprehensive visualization assessments, and maintenance policy optimization, among other applications.
Mp-sim's principal use case involves the study of the trade-off relationship between the maintenance costs and the fleet-level availability for aircraft or helicopters, while exploiting various maintenance strategies. In some instances, there may be a desire to minimize maintenance costs at the expense of availability, or vice versa. Alternatively, an optimal equilibrium might need to be sought between the two parameters. We plan to perform a live demonstration of the aforementioned use case using the mp-sim tool at ICCBMA 2024.
|
16:20
20 mins
|
Applying Survival Analysis Algorithms for CBM in Aerospace: A Study on how Censoring Rates Influence Model Performances
Ghanem Bahrini, Sebastien Rzarakivony, Morgane Barbet-Massin
Abstract: As the aerospace industry seeks innovative approaches to ensure equipment reliability and extend operational lifespans, survival analysis emerges as a key statistical tool in realizing these objectives. This research contributes to the aerospace sector by utilizing survival analysis techniques to predict the time to failure of critical components, such as engines, This aligns with the principles of Condition-Based Maintenance by offering predictions on when maintenance is needed. A significant challenge herein is the notion of censoring, which arises when the event of interest does not occur within the observation period. This scenario implies that the equipment has continued to function without failure up to a specific analysis date, however, the exact moment when maintenance might be required remains unknown. Survival analysis models were specifically designed to address this characteristic, allowing for the incorporation of censored data. However they were often developed for medical data, where censoring rate is lower than for aerospace data. Indeed, as aerospace is very safe, censoring rate can reach above 90%. Our study delves into how the censoring rate may affects the models' performance
Our methodology adopts an innovative approach to constructing semi-synthetic datasets for simulating synthetic censoring scenarios, inspired by the process outlined by (S Qi, 2023)[1]. Starting with real-world survival data, we first analyze censoring distributions to derive the probability density function (PDF) for censoring. This PDF is then adjusted to randomly generate a variety of synthetic censoring rates, from 20% to 95%. By selecting only non-censored instances and combining them with calculated synthetic ones, we create several datasets, each with various censoring rates. Note that these datasets are presented in a tabular format, encapsulating cross-sectional data with covariates that differentiate individuals. Besides its covariates, each individual is described by either the time to an event's occurrence or the time to censoring, denoted by a binary indicator of censoring.
We explore the influence of censoring rate for several survival analysis algorithms, we use the Cox proportional hazards model [2] (Cox, 1972), as it remains the most commonly used algorithm. We also examine the Weibull Accelerated Failure Time (AFT) model, providing a parametric complement to the Cox model, and the Random Survival Forest, an ensenmble method for survival analysis using decision trees [3] (Ishwaran, 2008). Additionally, we incorporate the DeepHit model [4], (Changhee Lee, 2018), which employs a deep neural network to directly learn the distribution of survival times, accommodating complex interactions within the data and marking a significant departure from traditional survival analysis techniques.
To assess the performance of these algorithms under various censoring rates, we utilize several evaluation metrics. We calculate the concordance index[5] (Harrell, Califf, Pryor, & Lee, 1982). This index measures the extent to which the sequence of actual survival times aligns with the predicted risk orderings. Furthermore, we employ the integrated Brier score [6] (Ulla B. Mogensen, 2012), and D-calibration [7] (Humza Haider, 2020), to estimate the calibration accuracy of the different algorithms under varying censoring rates. Additionally, our analysis incorporates the MAE-PO [1], which adeptly estimates the Mean Absolute Error (MAE) for survival datasets that include censored subjects. This metric is particularly valuable as it not only ranks models based on their performance but also provides an approximation to the true MAE, which is otherwise challenging to calculate directly due to the presence of censored individuals.
Our results will demonstrate how censoring rates affects algorithm performance and reveal whether certain algorithms exhibit varying effectiveness depending on the encountered censorship rates.
|
16:40
20 mins
|
Text-mining applied on maintenance reports for aviation equipment maintenance support
Céline Berthou, Manon Bedere, Yannick Evra, Jean-Frédéric Diebold, Dong Quan Vu, Héléna Vorobieva
Abstract: In the context of aviation equipment maintenance, it has been observed that a high number of equipment are mistakenly returned to the supplier due to failure. Indeed, due to maintenance time constraints, operators tend to send an excessive number of equipment for repair or replacement in order to quickly fix the failure within the allotted time. This generates additional costs for the manufacturer. Following this observation, this work aims to assist the maintenance operator diagnose failures more quickly and identify the actual equipment in need of repair or replacement, thereby reducing maintenance costs and improving the operational availability of the equipment in service. In this context, a diagnostic tool for failure and maintenance support is being implemented. It is enriched with field expertise by capturing and capitalizing on domain knowledge and feedback from maintenance operators through natural language processing (NLP) technologies applied to maintenance reports.
To achieve this, a NLP model is used to automatically classify maintenance reports according to various variables such as the reported failure type, the actual faulty equipment to be repaired, and the maintenance actions to be performed to correct the failure. Several pretrained models are tested, mainly transformer-based models like BERT [1] and its variants. We labelled a database of maintenance reports to specialize the model for the specific task, evaluate and compare the performance of different tested models. We tested several fine-tuning strategies on the pretrained vanilla BERT model and are currently experimenting the SafeAeroBERT model [2] already specialized in the aerospace domain using the ASRS [3] public database. Regarding vanilla BERT experiments, we show that the best strategy is the fine-tuning of its last four layers.
We address three main difficulties encountered during the training. First, vanilla BERT or even specialized BERT models like SafeAeroBERT, were trained on good quality data, when real industrial datasets include a lot of very specific words, acronyms, abbreviations from aerospace domain (sometimes even specific to only one company), with poor grammatical and orthographical sentences. To address this problem, we applied large data pre-processing, including the creation of a dictionary of acronyms and abbreviations to replace them with their full meaning before training the model, so that the model can better understand the context. The second difficulty encountered is the small size of the available training database, which is less than three thousand labelled maintenance reports. Therefore, we applied several data augmentation techniques. We have observed that conventional techniques (translation, synonyms, etc.) remain applicable within the scope of aeronautical language. Finally, the training database is highly imbalanced, with frequent cases well represented, when rare cases have few samples. Even with data augmentation, without specific training strategy, the model may struggle to predict classes with few train samples, while they may be important from a business perspective (the most severe failures are often the rarest) and need to be predicted accurately. To better predict these infrequent cases, we modify the vanilla loss function of the model during the fine-tuning. Among the tested losses, the focal loss [4] greatly improves the prediction of small classes.
In the end, the achieved rate of accurate classifications reaches approximately 90%.
In the context of using this maintenance support tool by maintenance operators, it is necessary to have a confidence indicator associated with the model's predictions to indicate to the operator the level of confidence that can be placed on the results. Such score can for example reflect poor representation of some classes on the training set, poor writing of the maintenance report or its misunderstanding by the model. Thanks to a confidence score, the operator will know that verifications should be carried out to validate the model's prediction.
This work is part of the theme of conformal prediction applied to classification. Several conformal prediction approaches are being tested, including the False Discovery Rate (FDR) approach [5]. This statistical approach involves estimating a threshold that represents a good compromise between the estimated FDR (false positive rate predicted by the model) and the estimated FRR (false recall rate corresponding to false negative rate predicted by the model). The goal is to simultaneously minimize these two rates, but improving one leads to the degradation of the other, as we can see in Fig.1. If the selected threshold is high, the FDR will be small and the FRR will be high. The model will predict fewer labels, but they will be more reliable. On the other hand, if the selected threshold is low, the FDR will increase and the FRR will decrease. The model will predict more labels, but with a high risk of false positives. In our application case, the aim is to propose to the maintenance operator reliable labels to be prioritized (e.g. labels corresponding on the orange operating point in Fig.1), as well as other less prioritized labels that can be considered if the failure is not corrected following the initial maintenance actions based on the prioritized labels (e.g. additional labels when using red operating point in Fig.1). This will guide the operator in maintenance by prioritizing the equipment to be checked.
These various works aim to improve the automatic classification model of maintenance reports integrated within the maintenance support tool, which is currently being industrialized in aviation equipment maintenance workshops. This will allow a better fleet availability management by facilitating a faster understanding and maintenance of the failures detected during health monitoring.
|
17:00
20 mins
|
Airframe Digital-twins in Airbus Defence and Space
Daniel Iñesta, Jose Luis Melchor, Pablo Caffyn, Jaime Garcia
Abstract: The main Airbus Defence and Space platforms are fitted with Operational Loads Monitoring (OLM) Systems, that collect the necessary data to obtain stress time histories at the selected analysis locations. These stress time histories are then used to calculate the severity in terms of crack initiation and crack growth per location and per each individual aircraft. The result is a fatigue and crack growth severity factor map along the Airframe for each MSN. Additionally, Airbus Defence and Space has the information related to the aircraft condition, in terms of Concessions, Directives, Service Bulletins and Major Repairs per MSN.
Combining the information provided by the OLM systems and knowing the aircraft condition it is possible to assess the current structural integrity status for each individual aircraft, and automatically update the maintenance, e.g limitations in terms of ALS or repairs with limitations, according to its real usage.
The information is presented over a map of the major structural components, therefore it is easy to have an overview per MSN. This can help assess the new in-service issues, detect repetitive damages and facilitates the corresponding root cause analysis.
Other advantage of using the airframe digital-twins is to be able to assess the changes in terms of usage of the aircraft (performing prognosis) and assess the potential life extension of the structure.
|
17:20
20 mins
|
Closed-loop Digital Twin Framework for Maintenance Optimization of Industrial Systems
Sin-Seok Seo, Guillaume Doquet
Abstract: A major challenge for performing Condition Based Maintenance (CBM), (be it predictive or
systematic) on an industrial system is to monitor its state of health during the entire lifecycle.
In particular, aircraft engines (for both airplanes and helicopters) are notoriously hard to
monitor (Bastard, et al., 2016), given the relative sparsity of sensors on board, the impact of
external conditions (e.g. air temperature, humidity and pollution), as well as the complex
interactions between modules (such as turbine and compressor), the behaviour of which
depends on wear. As a result, typical simplified degradation models struggle to perform well
enough for practical use in predictive maintenance strategies. Improving the accuracy of engine
health monitoring techniques and the efficiency of fleet maintenance policies is therefore a high
priority for the aeronautics industry.
A Digital Twin (DT) could be a powerful tool in service of this industrial need, shadowing in
real-time its Physical Twin (PT): the real-life engine. Starting with sensor measurements
collected on the PT, informative health indicators can be extracted or calculated from the data
and stored in the DT. Using these health indicators, estimation of a Remaining Useful Life
(RUL) of the DT can be followed. Thereafter, the estimated RUL can be fed to a predictive
maintenance policy, tasked with advising repair operations. Finally, these recommendations
can be sent back to the PT, thereby closing the loop between physical and digital worlds
(Thelen, et al., 2022).
SAFRAN is involved in an ambitious DT research program gathering both academic and
industrial partners (TwinPlus, 2023). In this program, our work is concerned with building a
generic DT platform (from the algorithmic design to the software implementation) following
the aforementioned pipeline. We have built a prototype of our proprietary end-to-end DT
solution, equipped with a user-friendly interface to easily create and simulate health monitoring
and maintenance scenarios. Thanks to its genericity, the architecture can be used to instantiate
DTs of any industrial system, provided an adequate PT. Most notably, given that a key factor
for de-carbonation of the aerospace industry is the electrification of aviation, we are particularly
interested in monitoring fleets of batteries (Almuhtady, et al., 2014). We plan to perform a live
demonstration of our DT platform at ICCBMA2024, in which the potential of the approach will
be illustrated by performing multi-criteria maintenance optimization based on simulated battery
data obtained using the PyBAMM tool (Sulzer, et al., 2021)
|
17:40
20 mins
|
Embraer perspective on the adoption of CBM solutions for fixed-wing aircraft systems
Lucinete A. L. Di Santis, Ricardo Rulli, Pedro Tostes, Carlos A. G. Carneiro
Abstract: Modern fixed-wing aircraft usually have much more data acquisition, storage, and transmission capabilities than previous ones. By addressing the right requirements, these can allow for a better implementation and use of Aircraft Health Monitoring (AHM) concepts and solutions, seeking to increase the efficiency of maintenance operations and fleet availability, and to reduce maintenance burden and costs. As aviation is a complex system, enhanced operational performance is a challenge, especially in large and mixed fleets. Due to the increasing volume of operational data, the maturation of digital technologies, the increasing number of studies and regulations, the feasibility of AHM adoption has been benefited.
In the context of aircraft health monitoring there is Condition-Based Maintenance (CBM) philosophy which aims to use information related to the health condition of aircraft systems and structures to identify optimal maintenance interventions over time [1], either for unscheduled or scheduled maintenance. In other words, CBM makes use of the data from the AHM solution to allow the execution of maintenance interventions only when they are required, in the best location and with adequate resources, aiming to minimize the impacts of aircraft downtime due to maintenance, providing a window of opportunity for the operator to decide the most convenient time for intervention.
Recent issuance of an advisory circular, AC 43-218 by the FAA (U.S. Federal Aviation Administration) [2], has given more light to the application effort of AHM methods and Integrated Aircraft Health Management (IAHM) process and systems. In addition, the evolution of PHM (Prognostics and Health Monitoring) for systems, and SHM (Structural Health Monitoring) for structures, two technologies that are pillars of AHM and CBM, brought much more knowledge about the technical aspects related to CBM deployment into the current and future aircraft maintenance environment.
The work presented herein aims to discuss about Embraer perspective on the adoption of CBM solutions for aircraft systems, its connection with documents A4A (Airlines for America) issue paper IP-180 [3] and FAA AC 43-218, and the impacts of CBM for unscheduled maintenance in terms of aircraft availability, and for scheduled maintenance in terms of the creation of alternative procedures for classic maintenance procedures.
Along the years Embraer has studied and developed PHM algorithms focusing on systems’ fault diagnosis and data analytics to predict failure trends, to support operation management. With the evolution of monitoring capacity, evolution of digital technologies, breadth of discussion in various forums and publications related to predictive maintenance, these enabled Embraer to investigate PHM implementation process to determine the remaining useful life (RUL) of aircraft systems’ equipment with the objective to predict failures and indicate an adequate timeframe for maintenance interventions [4]. Different approaches have been used by Embraer to determine equipment’s remaining useful life, such as, identifying patterns with historical maintenance field data, stochastic analyses, and test bench data. This kind of solution is intended to keep or improve the system’s reliability level and maintain the continued airworthiness of the aircraft during its operational life cycle.
Regarding the challenges for CBM adoption, in the Embraer perspective, the following stand out: the OEM's ability to provide inherent AHM capability to the product and supportability, the airline's ability to apply AHM in its processes and the equipment supplier's ability to support AHM operation.
Studies indicate that unscheduled maintenance can be benefited from the adoption of CBM. By having information about the prediction of failure events before they happen with an adequate anticipation will allow the aircraft operator to decide when and where to perform a maintenance intervention in advance, to minimize operational and cost impact, avoiding the possibility of delay or cancelation flight due to maintenance. The focus is to plan for an optimal maintenance intervention before failure.
Additionally, studies on the use of AHM in scheduled maintenance have also been conducted with the objective to make the maintenance tasks execution time more flexible. The A4A IP-180 document brings recommendations to identify AHM alternative/hybrid procedures that covers the classic maintenance procedures [3]. In the Embraer perspective, due to the current AHM readiness/maturity level its application for scheduled maintenance is only recommended for non-safety Failure Effect Categorization (FEC). Depending on the criticality, demonstration of adequate level of reliability for each application is paramount for effective and efficient CBM solutions which can be proposed as alternative means of compliance for replacing classic maintenance procedures.
The introduction of Artificial Intelligence (AI) algorithms for determining RUL have also been studied [5], and the results indicate that additional development is still required specially with the objective to interpret and explain AI outputs at an acceptable level. Also, there is a need for sufficient failure data for adequate AI responses, which cannot be achieved without enough systems’ time of operation.
Depending on the maintenance type (unscheduled or scheduled), the application’s level of criticality, the coverage of failure modes and the algorithm’s capability and reliability, initial CBM solutions can start to deliver a more efficient and less costly maintenance capability. By gaining confidence and proving their feasibility, these applications can evolve to include more complex use cases, in the future.
|
|