Anomaly Detection in Metal-Textile Industries

1. Introduction

The use of image processing techniques for industrial quality inspection systems has a long tradition and is applied in the geometric measurements of produced parts as well as defect detection, contamination detection, and feature adherence either in 2D and 3D [1, 2, 3]. With the availability of computing power and ever increasing storage capacity, the application of machine learning approaches in this field has become feasible for use cases where classical industrial image processing failed. One of these areas is metal textiles that pose multiple challenges on the image processing pipelines when it comes to the quality inspection of the textiles and industrial parts than integrate these textiles, respectively.

Machine learning approaches would formulate this quality inspection task as a classification problem. Typically, the machine learning model is trained on a dataset that contains images of the respective classes in a more or less even distribution. However, many times this conflicts with the production process itself: Firstly, the datasets that can be generated will almost always be highly skewed to the class of parts that are error free (called PASS parts, and FAIL for the faulty parts). This is obvious, as otherwise the production would not be economically attractive. Secondly, the FAIL parts often must be decomposed into multiple classes themselves, e.g., weave errors, dents, welding errors, etc. This increases the class distribution imbalance with respect to the FAIL class even more. For all FAIL classes, further decisions and actions have to be taken, e.g., scrapping vs. repairing on a per class level. These decisions and actions often involve the assessment of the degree of damage in the FAIL parts and the existing processes for further treatment of parts belonging to these classes. This knowledge is often available only implicitly in the experience of the quality control inspectors. Hence, instead of trying to address this multi-class problem with a fully automated solution a one-class problem, using anomaly detection in a human-in-the-loop process is more promising.

As a main contribution of this paper, we describe a use case of an optical inspection process in metal-textile industries which need to provide a nearly 100% PASS rate. We propose an approach where the automated inspection system reliably detects parts that are undoubtedly PASS parts and only sends cases in doubts to the human quality inspector. This way, the inspection payload of the human inspector, i.e., the number of parts to be inspected by the human inspector, is reduced. This in turn leads to a better quality of delivered parts and better working conditions for the inspection worker. This work is an extended version of our previous work published in [4].

The rest of the paper is organized as follows. In the next section, we review some related work with respect to optical inspection and anomaly detection. Section 3 presents our solution proposing a STFPM architecture for a one-class anomaly detector and shows how it has been integrated into the inspection process of the GKD company. In Sections 4 and 5, we discuss our work and conclude with an outlook on future research and future application areas.

2. Related work

In the following, we review work that is related to our work from several different perspectives. Our approach aims at reducing the worker’s load in the process of quality inspections while we remain at a quality level of near to 100%. Therefore, we review related work showing other applications of the Six Sigma approach taken in our metal-textile application use-case first. Then, we review works from the area of making decisions in optical inspections systems, and in general, an review works from the area of anomaly detection systems, in particular.

Six Sigma is a set of techniques and tools for process improvement introduced for manufacturing training programmes in the late 1980. While its original focus was on process electronics industries, it spread out to many sectors and is still widely used today. It comes with a set of statistical tools for improving existing processes. Statistical tools help identifying non-conforming products in the range of parts per million (PPM). It means, on the other hand, that establishing a process following the Six Sigma approach, the output of defective parts is at the level of 3.4 PPM and below [5]. Six Sigma approaches in production settings involve a quality control process that is integrated in the production process [6]. Quality control can be performed automatically, manually or as a mixture of both approaches. If a manual inspection is part of the quality control process, the process itself loses its stationary nature due to human factors induced, such as stress, fatigue, or individual variances. These have their cause in the repetitiveness of the process and the required long time spans of concentration required by the worker. It turns out that manual quality control can only detect between 60 and 80% of defective parts. At the same time, the quality inspection process takes up to 10% of the whole labour costs [7]. The above-mentioned factors can even result in a performance reduction below this value [8]. Furthermore, the structure of this work can have negative impact on the workers’ health, especially when visual inspection is involved [9]. With such detection rates, it is hard to establish a Six Sigma process based on human inspection alone. In a fully automated process, on the other hand, this factor depends on the performance of the underlying algorithm, which can also not achieve the required quality of a Six Sigma quality control process. Therefore, a combined approach as proposed in this work seems beneficial to come closer to the goals of achieving a Six Sigma quality control process. Usually, the final stage in a typical Six Sigma process includes the inspection of the final product to check either its quality and/or the process [7]. In this paper, we focus on the control process which, in turn, focuses on a single product entity checked by optical inspection.

Statistically, an optical inspection system (OIS) for quality control can be seen as a classifier with a specific operating characteristic. The performance goal of the quality control system is to distinguish passed from failed parts while reducing the total number of wrong classifications. The amount of admissible errors depends on the application domain. Six Sigma allows for 3.4 PPM of false negative decisions (part is FAIL, but decision is PASS) (cf. the red area under the gray curve in Figure 1). The number of false positives (part is PASS but decision is FAIL) should be as small as possible to reduce the negative economical impact. An OIS relies purely on visual inspection of the parts produced to make this decision. This can either be a simple image taken by a camera but can, such as in our case, also be a combination of different types of digital images and a visual inspection by a human quality control expert.

Automated OIS can be distinguished into parameter-based systems that check specific parameters such as dimensions based on a static definition of parameters. These are typically hard coded into the inspection algorithm and require a calibrated camera system. Model-based systems check a part against an expected visual appearance for both possible decisions. The decision takes the most probable of the two classes or checks against the PASS quality level by thresholding the deviation from the optimal case. This approach is known as anomaly detection (AD).

The approaches of anomaly detection in OIS can be separated in three different areas: (1) one-class classification, (2) reconstruction-based methods, and (3) representation-based methods. In a typical production process, a high amount of normal products are produced. This is a challenging task for modeling anomaly detection, because it generates a highly imbalanced dataset with much more images of PASS parts than images of FAIL parts. Most state-of-the-art solutions are working with convolutional neural networks (CNN) that require large training sets (see, for instance, [10]). To make a machine learning classifier unbiased in its decision, the datasets should be balanced with respect the number of examples in the different classes. Alternatively, the classification problem can be reformulated as a one-class-classification problem that is trained only on the normal (PASS) images that count for the majority of the data in the dataset. The most common methods for one-class classification are, for instance, shown in [10, 11, 12]. For training the network, high quality and in the best case balanced datasets are crucial and not easy to get. In [13], a dataset of about 1000 images of aluminum part with 102 defect-free and 23 defective images is presented. Rippel and Merhof [14] mentions (besides giving an overview of the field of anomaly detection methods) four more publicly available datasets for anomaly detection with a dataset size ranging from 1300 to 5300 images. In the area of detecting anomalies for photovoltaic cells, [15] proposed a large dataset which consists of more than 36.000 images of good and defective photovoltaic cells with eight different defective categories.

The main two classes of CNN-based anomaly detection methods are reconstruction-based methods and representation-based methods, both with capabilities for image-level and pixel-level predictions. In the former case, we speak of anomaly detection while in the pixel-level case one speaks of anomaly segmentation. This is based on the assumption that a model which was trained with images of O.K. parts is not able to reconstruct defective parts as well. In representation-based methods, anomalies are being detected on distributions from the extracted features of a neural network. Reconstruction-based methods, on the other hand, are founded on generative models. The three main models are auto-encoder (AE), generative adversarial network (GAN), and normalizing flow networks (NF), where NFs can be seen as a mixed form. In traditional AE, the input and the output of the network are compared with L2 norm or structural similarity loss. Other authors who deployed GANs used generated images for training. The high generalization capabilities of GANs are restricted introducing pseudoanomalies in a self-supervised fashion [16, 17]. Especially, self-supervised methods shift the problem of the high generalization to a bias of pseudoanomalies, which results in poor performance in benchmarks.

A special case of reconstruction-based models are normalizing flow models, which show very good results and have the capabilities to estimate likelihoods by learning transformations between densities and given distributions. The learning is done by an invertible mapping function which transforms basic probability functions in multiple steps into the target distribution. By being invertible, data sampled from this learnt representation can be projected back into the original space. Representation-based methods differ from reconstruction-based methods in their comparison between normal and abnormal images which is done in the feature space instead of the image space. They typically have two parts: feature extraction and feature comparison between data points and expected distributions. Because of the separation between feature extraction and comparison, there is a big freedom in choosing the neural network backbone. Rippel et al. [18] showed in their work that neural networks which were pre-trained on ImageNet can indeed generate meaningful features. Pre-trained features of single layers in a pyramidal way were proposed by Cohen and Hoshen [19]. The training is done by storing all layer-wise aggregated features. For inference, a simple kNN search is done which yields a patchwise, multiscale feature comparison with the maximum distance as anomaly score. Aggregating the ideas of Cohen et al. and Rippel et al. [18] by simply calculating a Gaussian on every patch and taking the Mahalanobis distance between inferred and stored features, Defard et al. [20] improved the performance on the MvtecData set in PaDIM by reducing kNN search time in combination with a randomized dimension reduction. Due to the Gaussian-based outlier sensitivity, PatchCore [21] introduced core set sampling which reduces the kNN search space up to 99%. This family of algorithms has been further improved: SOMAD [22] puts a self-organizing feature map in the PaDIM setup, SA-PatchCore [23] is using transformer-based self-attention mechanisms to get a more global receptive field of view for detecting co-occurence anomalies, and Kim et al. [24] sped up the calculation of covariance matrix in PaDIM. Another group of methods is the group of knowledge distillation-based approaches. In these approaches, an untrained student network is trained to learn the layer-wise feature representations of a pre-trained teacher network, which has often the same architecture as the student network and is shown only normal images. The assumption is similar to the one in reconstruction-based networks: By minimizing the difference in the representation on only normal images, the difference between student and teacher should be larger when an abnormal image is shown. An important method which we also deploy in our work presented here is the student-teacher feature pyramide matching (STFPM) proposed by Wang et al. [25].

3. Technical solution

In this section, we describe our technical solution for the human-AI coworking optical quality inspection process for metal-textile filters by GKD-Gebr. Kufferath AG. In the next section, we first recap prerequisites that need to be met for a data-driven project following a CRISP-DM approach. In Section 3.2, we outline the current process and how the data for training our STFPM network were acquired. In Section 3.3, we define our model and show how it is being trained. Then, the integration of our approach into the QC process of GKD is explained in Section 3.4.

3.1 Project requirements following CRISP-DM

According to the standard process for this type of data driven projects, CRISP-DM [26], the requirements for a technical solution must be defined during the business analysis process step. These requirements must be quantifiable, if possible, and include also the requirements for the potential deployment, i.e., the operation of the designed solution. The core requirements and constraints are listed in very condensed form in Table 1.

Name	Rationale	Quantification
Reduction of workload	QA workers shall be relieved from analyzing filters that are safely PASS parts, where safely means, that the Six Sigma conditions are not violated	Reduction: Min: >0% Plan: >50% Wish: >66%
No degradation in ‘fail’ rate	Even if Six Sigma conditions are kept, the rate of ‘fail’ parts should not increase	Fail rate: Min: <5% Plan: <4% Wish: <3%
Reduction of false positives	There is an assumption that at the end of shifts, the fatigue of QA workers leads to unwanted ‘fail’ decisions (false positives). With reduced workload, there should be an improvement in the human decision process.	FP reduction: Min: <0% Plan: <1% Wish: <2%
Application in production process	The integration of the anomaly detector should not slow down the overall production process, even if labelling the human-controlled parts is included	Rate reduction: Min: <5% Plan: <1% Wish: <0%
Minimize setup modifications	Modifications to the existing production environment should be kept as small as possible	Modifications: Min: Mechanics, Touchscreens, network image storage Plan: N/A Wish: N/A

Table 1.

Requirements for the use case. Setup modifications were fixed during analysis, and hence, there is no variation in potential execution.

As has been discussed in [27], the use of any particular method from artificial intelligence imposes a set of specific requirements. In our case, rather generally speaking, all (supervised) machine learning methods need data to be trained. This is especially true for image-based classification tasks like in our case here. What is more, the quality of the training data drastically influences the quality of the resulting model and in turn the quality of the solution and the success of the overall application (Figure 2).

3.2 Current process and dataset generation

The technical solution for the use cases starts with the generation of a dataset that can be used for training an image-based anomaly detector. Based on a previous failed attempt to inspect the parts automatically with classical industrial image processing approaches, a setup that could capture images of the parts produced already existed in the company (cf. Figure 3).

The filters are produced in a closed manufacturing cell where two line cameras are capturing the filter while it is under production on a turntable. The first camera captures the gray values, and the second camera captures the distance of the filter elements to the camera’s chip plane. Thus, two images showing the unwind filter are available per produced part with 10,500×1408 pixels (Figure 4a and b) and 10,500×1409 pixels for the depth image, respectively (Figure 4c and d). Images are horizontally and vertically aligned, so that there is a direct correspondence between image regions.

While the image capturing happens in-line with the production, the speed of production is determined by the speed of inspection by the QC-workers that have to manually inspect 100% of the filters produced.

Production errors can occur in

• the mesh,

• the transition between mesh and weld,

• the weld,

• the transition between weld and shell, and

• the shell.

Hence, errors are possible in image parts that are highly textured and low structured and vice versa.

3.3 Model building for the anomaly detector

While there are well-established approaches for anomaly detection, e.g., local outlier factor [28], one-class support vector machines [29], and isolation forest [30], to name a few, an anomaly detector for image-based human-in-the-loop (HITL) approaches should not only take a decision if an anomaly exists. It should, if possible, also show the position of the anomaly detected to guide the quality control worker in the further inspection process.

The latter requirement can be fulfilled by reconstruction-based methods and knowledge distillation-based method, as mentioned in the related work section. The well-known texture affinity (over structure) of convolutional neural networks [31] is beneficial for the problem at hand. While the woven metal mesh can be seen as a large texture, the welded areas of the filter have virtually no texture at all, which hold true for the depth image as well.

The machine learning model used here is a student–teacher feature pyramid model [25] (STFPM) that gets trained on images of PASS class filters only. Alternatives have also been researched and are subject for continuous improvement [4]. The local reconstruction error can be visualized using heat map colormaps. This reconstruction error image is shown to the quality control workers to guide them to the potential source of the anomaly. As shown in Figure 5, the anomaly is exactly at the predicted position.

3.3.1 Training

The STFPM training uses two networks with an identical architecture. Each network receives the input image, or a batch of input images, respectively. The training set consists of PASS images only. A preprocessing stage performs a patching of the images to reduce the input size and the total size of the networks parameters. The teacher network (marked blue in Figure 5 is pre-trained with frozen weights, while the student (marked orange in Figure 5) has its weights randomly initialized and gets trained. The target of the student is to reproduce the output in the latent spaces by minimization of error on the different scales l1l2l3.

3.3.2 Inference

During inference, both networks receive the input image. Now only the component-wise differences are taken. At the respective latent layers l1l2l3, the differences can be seen in Figure 5. Each feature map is scaled up to the size of the lowest feature map l1↑, and the results are component-wise multiplied Π, producing the output image as map of the reconstruction error.

Figure 6 gives a more detailed view on the anomaly heatmap. While the welding error is easily to spot for humans, the small mesh error only becomes apparent in the heat map image.

3.4 Process integration

Process integration may not only consider the production process itself, but also the influence of the process change to the human personnel involved in the total process. To achieve that, a development process was designed that integrates the technical part, together with the organizational and process part plus the analysis of human factors [32]. This is necessary as a human-in-the-loop approach must consider the mutual influences of the human and technical parts of a solution. Using this method, the final process was designed and changes to the existing process steps applied. The final process is shown in Figure 7.

The anomaly detection process now involves a predefined decision boundary for the anomaly score. This boundary determines whether a part is identified as PASS or for further inspection ‘INSP’ (Figure 7a and b). As can bee seen from Figure 7, the newly introduced class ‘INSP’ includes all parts, where the anomaly score is exceeding the threshold. This still includes a subset of correctly produced parts, which would lead to an Type 1 classification error in a fully automated system. For these parts, the QC worker has the final call and can also take further action depending on the class and severity of the error. This reduces scrap and hence improves the process’ efficiency.

The PASS parts that come out of the anomaly detection stage contain a portion of type 2 classification errors. These must not exceed the Six Sigma threshold including a safety margin for errors that might be introduced by the QC worker.

The whole HITL process can be seen in Figure 8. As depicted, the part of the reduction in workload for the QC worked can be used to label the data of the manual inspection. Thus, over time, the dataset is augmented with properly labeled data. This labeled data, although small in size, can be used for automatic threshold determination for the anomaly detector. So instead of designing the anomaly detector with a safe margin threshold, this threshold can be constantly rebuilt, taking the labeled dataset into account. This reduces the workload even more while keeping the quality requirements fulfilled.

4. Discussion

As it turn out, quality control processes following a Six Sigma approach, where the optical quality inspection has to sort out all (or near to all) defective parts, are hard to come by. Human inspection can only reach up to 60–80% inspection rate. Today, even with a technical AI-based solution, the margins of Six Sigma are very hard to meet. We therefore propose a combined approach where the optical inspection is done by the human only for parts that surely are non-O.K. parts. Instead of inspecting 100% of all parts, only a fraction of parts need to be inspected by the human worker following the STFPM approach proposed in this work. This way, the cognitive load of the worker is reduced during her shift alleviating the process to find all of the defective parts that have been produced in her shift. While definitely the stress level for the inspection worker is reduced, it is still to be shown to which level the worker is relieved. This will be further investigated in future works. As for now, we can only state that the QC worker involved in this work was positive about the reduction of the number parts to be inspected. While we get positive feedback from the shop floor, there are some challenges that need to be further addressed in the future. One of them is the problem of interfering with a running inspection system. The worker are used to their processes and need to instantly switch to a different process. Another complication is that the visual representation of the anomaly from the technical inspection process is not familiar to the worker in the first place. Here, we need to find good and intuitive representations for the worker in the QC process. Another important issue is to also look into other possible models such as ROCKET [33] which make use of random convolutional kernels for time series classification. First experiments of deploying such an architecture for 2D data look very promising.

5. Conclusion

In this paper, we presented an approach to deploying a student–teacher feature pyramid model for anomaly detection metal-textile filters used in automotive exhaust gas filtering. The quality requirements are following a Six Sigma approach, i.e., no defect parts may be shipped to the customer. The quality inspection was done 100% visually by a human inspection worker as technial inspection systems installed at the production plant could not meet the quality requirements by themselves. We therefore redesigned the QC process in such a way: an STFPM-based anomaly detector trained on good and defective parts is separating the surely PASS parts from other parts that might have problems, i.e., where an anomaly has been detected. Only those parts are further inspected by the human inspector. First tests show that an reduction of workload of 50% is achievable, with a margin left to further improvement without violating the Six Sigma requirements. These values have been measured in multiple testing scenarios with a smaller dataset of 352 test images. The threshold for the anomaly decision was then moved so that no false negatives were produced. The true positives (pass parts) were then upscaled to the real production values, from which a workload reduction of 69.4% was calculated. Even when an additional safety margin is added, reduction was above 50%. Final results will be taken on a larger set of images from a longer period of production that will also account for parameter shifts, e.g., in the cameras and lighting.

In conclusion, we see a very positive way forward of deploying AI systems on the shop floor by following a human-in-the-loop approach. The AI-based inspection system deals with the large mass of undoubtedly PASS parts and detects any form of anomaly. With the human in the loop, there is no need to further specify different fault classes, and a one-class classification process is sufficient. This has also the positive effect that the training process of the AI-based anomaly detector is simplified as fewer data for distinguishing different anomaly classes is required. This also reduces the development costs of such systems for industry.

In this paper, we presented an extended version of the metal-textile anomaly detection use case which was presented in [4]. While we are very convinced that the approach is a way to help automating the QC process on the shop-floor, we need to further investigate on our work. For instance, we need to survey how much the automated inspection process relieves the worker and how this influences the outcomes of the quality process. Further as already mentioned, we also will look into other promising network architectures in our future work.

Acknowledgments

We acknowledge the support by the Federal Ministry of Education and Research (BMBF) under grant no 02L19C602 and GKD-Gebr. Kufferath AG for their cooperation and permission to use their images. We would like express our gratitude to our past and current WRIKsam staff for their contributions in this work, in particular, we would like to thank T. Arndt, M. Conzen, O. Galla, H. Köse, and M. Tschesche.