Research on the Quality of AI Medical Devices and Standardization Running Out of “Acceleration”

Research on the Quality of AI Medical Devices and Standardization Running Out of “Acceleration”
Currently, new generation information technologies represented by artificial intelligence, cloud computing, big data, 5G networks, etc. are rapidly developing and accelerating their integration with the medical device industry. Since the beginning of this year, the field of intelligent medical devices in China has achieved fruitful results, with multiple products approved for market launch. With the acceleration of clinical application of intelligent medical devices, the importance of quality research and standardization work is increasingly evident. Multi level and multi angle research work has been carried out both domestically and internationally, providing reliable basis for the full lifecycle management of intelligent medical devices.
Since this year, China’s artificial intelligence (AI) medical device industry has blossomed: in the fields of coronary artery blood reserve score calculation, electrocardiographic analysis, intracranial tumor MRI assisted diagnosis, diabetes retinopathy assisted diagnosis and so on, products have successively obtained Class III medical device registration certificates, and some products have entered the priority approval channel for innovative medical devices, Some products have been approved for overseas marketing
With the acceleration of clinical application of artificial intelligence medical devices, full lifecycle supervision has put forward more requirements for their quality research and standardization. Developed countries and regions are vying to initiate standard formulation and revision work, seizing the high ground of artificial intelligence medical device development. In the first half of this year, under the leadership of the National Medical Products Administration, the centralized unit of standardization technology for artificial intelligence medical devices in China actively carried out relevant research work around regulatory requirements and industrial demands, and achieved phased results.
Active standardization construction abroad
At present, the artificial intelligence industry is in a rapid development stage as a whole, and the standard formulation and revision in various countries and regions have entered an active period. Countries and regions such as the United States, Russia, and the European Union have proposed standard planning, as well as organizations such as the International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), the American Association for the Advancement of Medical Devices (AAMI), Global Diagnostic Imaging, and others Organizations such as the Medical Information Technology and Radiotherapy Industry Association (DITTA) are also actively expanding their presence. In the field of artificial intelligence medical device standards, drug regulatory agencies generally exercise centralized management responsibilities in various countries and regions.
The US Food and Drug Administration (FDA) is exploring the establishment of an agile regulatory framework for artificial intelligence medical device software products, with a particular focus on their change evaluation. At present, the FDA is participating in the development of international standards through platforms such as ISO and IEC, and collaborating with institutions such as AAMI and the British Standards Institute (BSI) to develop medical artificial intelligence terminology and classification schemes, as well as medical artificial intelligence confirmation process schemes. In addition, the FDA also participated in the IEEE Artificial Intelligence Medical Device Working Group led by the China Institute for Food and Drug Control (hereinafter referred to as the Institute), jointly developing IEEEP2801 and P2802 standards.
The Russian Health Supervision Agency has proposed 10 medical artificial intelligence standards in its National AI Standardization Program 2020, including terminology classification, scope of use, general testing, clinical trials, and data related requirements.
Other countries and regions are also actively promoting the research and development of supporting technology resources for artificial intelligence medical devices. The AI4EU project, which brings together 21 countries and regions in the European Union, aims to achieve the sharing and integration of technical resources such as data, algorithms, and testing, providing support and certification for establishing unified standards and development strategies.
The orderly promotion of standardization work in China
In order to fully leverage the professional and talent advantages of the expert group of the centralized unit, in April this year, the centralized unit established five research groups to carry out research work around five themes: artificial intelligence medical device terminology classification coding, data and annotation quality, software characteristics and network security, product quality evaluation, and quality management. Each research group is led by experts in the fields of education, clinical practice, regulation, scientific research, testing, and standardization, guiding standard pre research and refining and improving the standard system.
According to the Management Measures for Medical Device Standards, China’s artificial intelligence medical device standard system is planned in four directions: basic standards, management standards, method standards, and product standards.
The basic standards mainly focus on common issues in the industry, regulating the terminology, classification, coding, data quality, data annotation, and dataset of artificial intelligence medical devices. Basic standards are currently the main direction for developing standards related to artificial intelligence medical devices. According to the 2020 Medical Device Industry Standard Revision Plan of the National Medical Products Administration, the “Quality Requirements and Evaluation of Artificial Intelligence Medical Devices Part 1: Terminology” and “Quality Requirements and Evaluation of Artificial Intelligence Medical Devices Part 2: General Requirements for Datasets” have officially entered the drafting stage, and both standards belong to the basic standards. On the basis of preliminary work, in June this year, the Chinese People’s Procuratorate, together with 14 member units from clinical, scientific research, and industry, jointly prepared the draft for soliciting opinions on the above two standards and publicly solicited opinions. At the same time, technical verification work was initiated, and it is expected to complete the approval and application of the standards by the end of the year.
The management standards mainly address the special issues faced by the quality management of artificial intelligence medical device production, including risk management, algorithm development management, infrastructure management, product iteration change management, personnel management, and other aspects. The quality management research group of the centralized unit has conducted research and on-site visits on the production and research and development status of frontline enterprises, organized multiple discussions, consolidated industry consensus, and formed a draft standard around the special issues of deep learning development management. Currently, the standard is being applied for and approved.
The method standards mainly focus on the quality evaluation requirements of products and components, including product/system performance evaluation methods, product change evaluation methods, safety testing methods, etc., and also involve supporting content such as testing tool evaluation and annotation tool evaluation. The product quality evaluation research group of the centralized unit formulates a standard drafting plan based on the evaluation needs and common issues of hot products, and lays out the development of common evaluation methods and product specific evaluation method standards. On the basis of product testing practice, the Secretariat of the responsible unit has organized the preparation of multiple draft standards, some of which are currently being applied for standard approval.
Product standards can be divided into branches such as ophthalmology, respiratory science, orthopedics, etc. based on the clinical application direction of artificial intelligence medical devices, as well as into branches such as computer-aided diagnosis, computer-aided detection, computer-aided triage, computer-aided treatment, etc. based on the similarity of product functions, evaluation indicators, and data modalities. At present, the pre research work on product standards has been initiated, and the preliminary draft of relevant standards for coronary CT blood flow analysis, neurological imaging assisted diagnosis, and other directions has been formed. The responsible unit will dynamically layout product standards and initiate projects based on the number of products launched and regulatory requirements.
In April of this year, the responsible unit launched the 2021 collection of standards for artificial intelligence medical devices, which attracted widespread attention in the industry. A total of 38 proposals were received, covering basic standards, management standards, method standards, and product standards comprehensively. At present, the Secretariat of the centralized unit is handling the relevant matters of standard project application in accordance with the requirements of the Medical Device Standard Management Center of the National Medical Products Administration.
Preliminary results have been achieved in quality research
The scientific development of standard systems requires solid quality research as a backing. Understanding the quality of artificial intelligence medical devices is an important proposition in regulatory science. In order to expand the research perspective, the Chinese People’s Procuratorate closely collaborated with teams such as the Beijing International Mathematical Center to systematically review the market situation, product validation methods, clinical trial design, and evolution of evaluation ideas of artificial intelligence medical devices abroad. The differences and shortcomings in product quality evaluation in the past were analyzed, and standardization requirements were summarized. The results of this work have been officially published in international academic journals.
Throughout the development process of artificial intelligence medical devices and early computer-aided diagnosis/testing products, it can be found that interpretability and data integrity are important starting points for their quality evaluation, and they are also the main challenges faced by the current quality evaluation and inspection of such products. On the one hand, many artificial intelligence medical device products widely use deep learning algorithms, but their working principles lack intuitive explanations. To improve the evidence-based scientificity, the interpretability and transparency of the product quality evaluation process and results are very important. The requirements for structured, standardized, and traceable evaluation methods, evaluation processes, test sets, and test reports are becoming increasingly high. On the other hand, product quality evaluation should not only consider conventional algorithm performance indicators, but also pay increasing attention to generalization ability and robustness. It is necessary to flexibly configure detection schemes based on specific scenarios. These trends indicate that the quality evaluation of artificial intelligence medical device products has entered a new stage of standardization and personalization.
In order to enhance the quality evaluation capability of artificial intelligence medical device products and support the revision of standards, the Chinese People’s Procuratorate has successfully applied for the national key research and development plan project “Research and Application Demonstration of Medical Artificial Intelligence Product Full Life Cycle Testing Platform”, and held a kickoff meeting in July this year. The project aims to build a national level medical artificial intelligence product testing platform, providing technical services for various aspects of product lifecycle supervision. The testing platform is expected to be completed in 2022, forming at least 18 typical sample datasets under the new model of open database construction and public-private data integration. It provides application demonstrations for various industries such as enterprises, universities, hospitals, and regulatory agencies, helping various sectors of society develop high-quality, multi-modal, and multi scenario standard datasets, reducing enterprise research and testing costs, and providing technical verification for the formulation of relevant standards, Provide technical support for the regulation of artificial intelligence medical devices. At present, the project has made initial progress in data interfaces, software interfaces, architecture, testing methods, and other aspects. The China Inspection and Quarantine Institute will solicit application requirements from educational and research institutions, hospitals, enterprises, and regulatory agencies based on project progress.
Under the leadership of the National Medical Products Administration, the quality evaluation and standardization of artificial intelligence medical devices in China are being orderly promoted. With the joint efforts of the expert group of the responsible unit, the “three carriages” of standards, scientific research, and testing are being promoted in a coordinated manner, striving to enhance the support capacity for the evaluation and approval of artificial intelligence medical devices, as well as full lifecycle supervision, and jointly helping to promote the healthy development of artificial intelligence medical devices in China.