Evaluation of Interoperability of CNN Models between MATLAB and Python Environments Using ONNX Runtime Model

Literature review

Mishra et al. developed an automated robot car using artificial intelligence (AI), trained its neural network using AlexNet model, and used YOLO in the object detection phase, where ONNX format was used for practical deduction and judging components [2]. Namala et al. proposed a system using U-Net CNN architecture to solve the problem of hair segmentation in real-world scenarios. In human–robot interactions, where real-time interaction is essential, hair segmentation model using a standard U-Net model faces difficulty in real-time processing. Their proposed method involved Keras and PyTorch for segmentation, which was possible using ONNX, and the difficulty in real-time processing was resolved [3]. Vicente et al. [4] provided a review of machine learning tools that may be suited for deployment in embedded systems, from which they selected two representative tools. One was the well-established Python-based Scikit-Learn, the other was the interoperability-oriented ONNX Runtime. Comparison of their response times showed the considerable superiority of ONNX Runtime. As for actual applications using ONNX, Lin et al. [5] developed a fluorescent sensor array combined with deep learning techniques for real-time monitoring of meat freshness, in which SqueezeNet was applied for automatic identification of the freshness level of meat based on fluorescent sensor array images with high accuracy (98.17%) and further deployed it in various production environments such as personal computers, mobile devices, and websites using ONNX format. To reiterate, deep learning model converters such as ONNX can move designed models between frameworks and to runtime environments. However, Jajal et al. pointed out that conversion errors tend to compromise model quality and disrupt deployment, and that the failure characteristics of deep learning model converters are unknown, adding to the risk when deep learning interoperability technologies are used [6].

Duque et al. addressed the challenge of optimizing a VGG16 CNN for efficient weapon detection on computationally and memory-constrained devices. They utilized a strategic blend of techniques, including transfer learning, pruning, and quantization. The VGG16 architecture was chosen for its relevance in rapid and accurate weapon detection in crowded settings [7]. Wan et al. proposed improved VGG19-based transfer learning models for strip steel surface defect recognition, in which it is reported that the improved model performed significantly better in detecting surface seams defects, despite few samples and imbalanced datasets [8]. Al-khuzaie et al. reported that identifying acute lymphocytic leukemia is time-consuming and labor-intensive, and that the results are not always accurate. The authors investigated methods to improve efficiency and accuracy and to automate the diagnosis by analyzing images of leukemia cells and showed that using a VGG19-based CNN, they were able to obtain an accuracy as high as 99.49%, surpassing other models in terms of simplicity and performance [9]. In addition, Gill et al. investigated a model that could discriminate between normal and cataractous eyes using a VGG19-based CNN, and were able to identify cataracts with more than 90% accuracy [10].

It is evident from these studies that VGG19-based CNN models possess a high classification ability. This tendency was also observed in earlier studies [11]. However, the interoperability among different frameworks expected from ONNX has not been elucidated.

As for recent developments of robots for peg-in-hole tasks, Huang et al. proposed a method in which a compliant dual-arm robot completes a peg-in-hole assembly task, using a position control model during the phase when the arms are not in contact with the part, and a torque control model during the phase when they are in contact with the part. As a result, the system successfully completed the assembly task for different assembled parts with a maximum gap between the peg and the hole of 0.5 mm [12]. Jiang et al. proposed a measurement method for robot peg-in-hole prealignment using a combined two-level vision system. The assembly system and a global coordinate system calibration method based on a dynamic coordinate system were developed to execute an accurate transformation between the coordinate systems. The hole pose was predicted with the proposed image processing method and the hole edge matching method. Experimental data and analysis results demonstrated that the hybrid measurement system showed high precision in the local hole pose and global robot pose measurement accuracy [13]. However, defect detection methods for the workpieces were not considered or equipped.

Proposed method and evaluation

We developed a MATLAB application to efficiently design, train and test prediction models for various kinds of defects and feature detection tasks of industrial products, industrial materials, cultivated cell, steel spark test, and others [11, 14–16] utilizing CNN, 3D CNN, SVM, CAE, FCN, VAE, YOLO, and FCDD models. ONNX model import and export function was implemented in the application.

In this study, a VGG19-based transfer learning CNN model built on MATLAB was exported to an ONNX model in order to be used for the defect detection by a picking robot running on Python. Two user interfaces were developed for MATLAB and Python to ensure pixel-level compatibility and achieve satisfactory interoperability on both frameworks. Generally, when designing and testing a transfer learning-based CNN, the resolution of input images has to be downsized to match the input layer of the transferred powerful CNN model such as VGG19. Furthermore, when images are downsized, an interpolation parameter such as nearest neighbor, bilinear or bicubic should be suitably selected. Billa et al. proposed a novel CNN-based architecture for image resizing forensics, both from resizing detection and factor determination points of view in the presence of double-JPEG compression, in which image scaling utilizing bicubic interpolation was used [17].

The evaluation results of three types of interpolation methods were shown through classification experiments of an industrial product using the two user interfaces created. Moreover, the practicality of a robot incorporated with an ONNX model for defect detection is demonstrated through an application to a peg-in-hole task of small workpieces.

Design and training of VGG19-based transfer learning CNN models

Since the VGG19-based transfer learning CNN models have demonstrated high classification accuracy in previous experiments, the same design method is applied in this study to the tasks of classifying photographs of industrial products, industrial materials, cultivated cell and steel spark test. Original 156 non-defective and 196 defective images of an industrial product provided by a collaborative manufacturer were used for training a VGG19-based CNN as shown in Figure 1. Test images were generated by flipping the original images horizontally, which were then used to test the CNN after training, i.e., to evaluate the generalization performance.

Figure 1 shows the overall network structure and activations of the CNN model for binary classification, i.e., OK or NG, which is transferred based on VGG19. Each convolution block has a ReLU (Rectified Linear Unit) layer, and each fully-connected block has a dropout layer or softmax layer. Network parameters, training conditions, training time needed, and processing time after training are shown in Table 1. Following are the specifications of the computing environment: Intel(R) Core(TM) i9-10850K CPU 3.60 GHz, GPU: NVIDIA GeForce RTX 3090, main memory 64 GB. The forward calculation time needed for processing an image in this environment was 70 ms. The forward calculation time includes the interpolation time by nearest neighbor and the prediction time of the VGG19-based CNN or its ONNX model. The forward calculation time of the VGG19-based CNN on MATLAB and its ONNX model on Python were similar.

The original images used in this study had a resolution of 2590 × 1942 × 3, however, to additionally train the VGG19-based CNN using the target product’s images, the resolution of the images has to be downsized to 224 × 224 × 3 to fit the input layer of VGG19. Nearest neighbor, bilinear and bicubic methods were available for downsizing of original sized images on MATLAB environment. Since nearest neighbor downsizing only selects one of the closest points for downsizing, it is the simplest and requires minimal processing time. Bilinear downsizing considers the closest 2 by 2 neighborhood of known pixels and then the average value of these 4 pixels is outputted. This results in visually smoother images than nearest neighbor. Bicubic downsizing considers the closest 4 by 4 neighborhood of known pixels, i.e., a total of 16 pixels, resulting in generation of noticeably sharper images than the previous two methods. When bilinear or bicubic method is used for downsizing, pixel values different from original ones are forced to be included in downsized images.

Figure 2 shows the differences of pixel values between nearest neighbor and bilinear, nearest neighbor and bicubic, and bilinear and bicubic methods.

In this study, images downsized by nearest neighbor were used for training the CNN and testing the generalization ability, due to the fact that the image downsized by nearest neighbor does not have the required pixel values. Moreover, it was confirmed that pixel values computed by bilinear or bicubic were marginally different on MATLAB and Python. The changes in pixel values tend to bring out scores predicted by machine learning models. Thus, nearest neighbor was selected to ensure the reliability and reproducibility of predicted scores for defect detection between MATLAB and Python environments.

After training the CNN on MATLAB, the generalization ability was evaluated using the test images. Table 2 shows the confusion matrix, which confirmed a classification accuracy of 99%.

ONNX (Open Neural Network Exchange)

ONNX was developed by Microsoft and Facebook [1]. Prior to the development of ONNX, neural network models were trained on various frameworks and were customized for each framework, making it difficult for each model to be mutually used on other frameworks and the concept of interoperability of a CNN model among different frameworks had not been sufficiently established. However, with the development of ONNX, it become possible for engineers to convert trained CNN models to ONNX format models and interoperate each model on different frameworks, regardless of which framework they were trained on.

Export to ONNX model

The VGG19-based transfer learning CNN model built on MATLAB can be converted to an ONNX model by calling the exportONNXNetwork( ) function provided by ONNX runtime via the ONNX export button, as shown in Figure 3. In MATLAB, ONNX operator number from 6 to 14 is supported. In this study, the default value was set to 8, and generated an ONNX model file (file extension: onnx) which could be imported in Python.