Russia’s first intellectual battle between doctors and artificial intelligence took place in the Skolkovo technology park. It became apart of the scientific-practical conference of the Interregional Association “X-Ray Cancer Diagnostics” on December 14, 2019.
All x-ray diagnostics services developers, who applied for participation, received 100 studies in their chosen field of diagnosis (radiography, computed tomography or mammography) a day before the battle. The task was to analyze all the images using their services and send the results to the organisators the next day. Four programs took part in the final stage of the battle: Third Opinion (Russia), IRYM (Russia), CELSUS (Russia) and HY Medical (China). AI programs, as well as doctors, had to analyze five digital images. They needed to not only check each for pathological changes, but also indicate their localization.
The battle began with a chest radiography analysis. Only one service competed with doctors. The results obtained by artificial intelligence coincided with the doctors opinion in four out of five cases. The program gave a false positive result analyzing the last image. It found a pathology, although it was not there. Rib injury complicated the image recognition, but the injury did not fall into the zone of interest marked by the service.
During the second round, participants analyzed mammograms. Two services competed with expert doctors at once. The results of the first two images analysis were different. The first servise showed the results that only partially coincided with the doctors′ opinion, when the second service′s results were identical to doctors′.
The following three mammograms were not easy, both for artificial intelligence and for doctors. Both services gave a false positive result. They found a malignancy where the medical experts saw a normal condition or a benign change. The doctors opinion from the public was also ambiguous about these images: half of them suspected pathology in the images.
The last round was dedicated to CT scan. The opinions of artificial intelligence and experts did not coincide in only one image. However, he raised doubts not only in AI, but also in public.
The Third Opinion company received the People’s Choice Award, according to the Battle results. The Chinese service HY Medical won the nomination “Doctor’s Assistant”. IRYM has been recognized as the most accurate program.
And the Grand Prix of the contest went to the CELSUS by Medical Screening Systems Ltd. According to the jury, the CELSUS system showed the highest overall result for all evaluation criteria.
The conclusions from the battle are obvious: artificial intelligence has not yet surpassed the doctors and cannot replace them. But now it can become an excellent physician assistant and exclude the human factor influence reading digital images.