Heart of machines Report
Editor: Demon King Author: Tsinghua University big data research center
A library of efficient and concise transfer learning algorithms has been opened by the Machine Learning Research Department of the Big Data Research Center of Tsinghua University Transfer-Learn, and the first sub-library has been released
The machine learning research department of big data research center of Tsinghua University has long been committed to transfer learning research. Recently, the project department has open source an efficient and concise transfer learning algorithm library based on pytorch: transfer learn. With this library, you can easily develop new algorithms or use existing algorithms.
Currently, the project releases the first sublibrary
Domain Adversarial Neural Network (DANN)
Deep Adaptation Network (DAN)
Joint Adaptation Network (JAN)
Conditional Domain Adversarial Network (CDAN)
Maximum Classifier Discrepancy (MCD)
Margin Disparity Discrepancy (MDD)
Background on adaptation in the field
At present, deep learning models surpass human performance in some computer vision and natural language processing tasks, but their success usually depends on large-scale labeled data. In practical application scenarios, tag data is often scarce.
One way to solve the problem of scarcity of labeled data is to generate training data by computer simulation, such as synthesizing training data using computer graphics technology (as shown below). In addition, relevant areas could be identified
However, in this scenario, the training data and test data are no longer subject to independent and identical distribution, which greatly reduces the accuracy of the deep network. In order to solve the generalization problem caused by data set offset, the concept of domain adaptation is proposed.
The goal of domain adaptation is to transfer the knowledge learned by machine learning models in source domain (Source) to target domain (Target). For example, in the case of computer simulation to generate training data, synthetic data is the source field, and real scene data is the target field. Domain adaptation effectively alleviates the dependence of deep learning on artificial labeled data and is widely concerned by academia and industry. At present, it has been widely used in many tasks, such as image classification, image segmentation, target detection, emotion analysis, machine translation and so on.
Wu Enda once said, "after supervised learning, transfer learning will lead to the next wave of commercialization of machine learning technology. According to Turing prize winner bengio, transfer ability is one of the basic abilities for further development of deep learning. With the application of product level machine learning into the field of data scarcity, the performance of the cutting-edge model obtained by supervised learning is greatly reduced, and domain adaptation becomes more and more important.
Depth domain adaptive methods mainly include the following three categories:
Confrontation training. Dann is the earliest work, which introduces domain discriminator to encourage feature extractors to learn domain independent features. Based on Dann, a series of methods are derived, such as conditional domain confrontation network cDAN, maximum classifier difference MCD.
Theoretical inspiration. Through strict theoretical derivation, the algorithm that can explicitly control the generalization error of transfer learning, such as interval divergence MDD, is obtained.
Dann network architecture diagram.
MDD network architecture diagram.
The above methods show good performance on experimental data. However, there are some problems in the open source implementation of adaptive methods in academia
Poor stability With the training, the accuracy of some countermeasures will be greatly reduced.
In view of these shortcomings, the original intention of dalib is to:Through a few lines of code, the user can apply the domain adaptive algorithm to the actual project without considering the implementation details of the domain adaptive module。
Dalib separates the domain adaptive loss function from the existing domain adaptive training code and encapsulates it in the form of pytorch cross entropy loss function to facilitate users' use.
The domain adaptive loss function is also decoupled from the model architecture and does not depend on the specific classification task, so the algorithm library can be easily extended to machine learning tasks other than image classification.
Using two lines of code, you can define a task-independent domain against loss function as follows:
There are some common modules in various domain adaptive loss functions, such as classifier module used in all algorithms, gradient inversion module and domain discriminator module used in confrontation training, kernel function module used in statistical distance.
These common modules are separated from the domain adaptive loss function provided. Therefore, in theIn dalib, users can customize their domain adaptive loss function just like building blocks。 For example, in the kernel method, users can customize Gaussian kernel functions or other kernel functions with different parameters, and then pass them into the calculation of multi-core maximum mean difference (mk-mmd).
At present, all modules and loss functions have provided detailed API documentation:
The research of domain adaptive algorithm often pays attention to the innovation or theoretical value of the method, but neglects the stability and reproducibility in implementation. In the process of reproducing the existing algorithms, the accuracy of some algorithms is unstable. DALIB solve these problems by improving the numerical calculation. The implementation is no longer carried out here. )
The test accuracy of dalib on common domain adaptive benchmark sets is higher than that reported in the original paper, and the accuracy rate on some data sets is even 14% higher. The following figure shows the test results on office-31 and visda-2017
The accuracy of different algorithms on office-31.
The accuracy of different algorithms on visda-2017.
Dalib algorithm library provides the test results of supported algorithms on office-31, office home and visda-2017, as well as complete test scripts. The research group of Professor Long Mingsheng of Tsinghua University believes that the open source algorithm library will help to promote the future research work of transfer learning.
the next version of domain adaptive algorithm library DALIB will support various complex settings of domain adaptive algorithms, including partial set domain adaptive tasks (Partial Domain Adaptation), open set domain adaptive tasks (Open-Set Domain Adaptation), generic domain adaptive tasks (Universal Domain Adaptation), etc. At the same time, multi-functional domain adaptive algorithms will be supported (Versatile Domain Adaptation).
Transfer learn is still in the initial stage of development. The research team said that with the continuous development of transfer learning direction, in the future, transfer learn algorithm library will continue to follow up the better algorithms in the new work, continue to expand and optimize, and provide a stable and reliable evaluation benchmark for transfer learning.
The current version is maintained by Jiang Junguang and Fu Bo, two students of teacher long Mingsheng's research group。 The school of software of Tsinghua University and the National Engineering Laboratory of big data system software provide strong platform support for the development of the algorithm library。