A woman who’s just had one of her breasts entirely removed, which forever leaves her with a huge scar across half of her chest, might be very satisfied with the result. If this was her only option, what obviously most matters to her is to be rid of the disease. But for another, who has submitted to a more advanced breast surgery – that is equally successful in clinical terms, but also more conservative and reconstructive – might on the other hand find her new look highly unsatisfactory in spite of the aesthetic results of the surgery being flawless. Every time she looks at herself in the mirror, she hates what she sees, which adds more distress to the one already associated with the disease itself. This may seem paradoxical, but there actually are cases like these.

Now, a project dubbed Cinderella is posed to receive five million euros from the European Commission during the next four years, to ensure that the aesthetic evaluation of the results of breast cancer surgeries will never be as subjective or unrealistic as the two referred above. The conditions are ripe for this to become a reality, believes Maria João Cardoso, Cinderella project coordinator, radically improving the satisfaction – and the relationship with their body – of women who have been submitted to breast cancer surgery.

Some decades ago, surgeons preferred to err on the side of precaution by removing the whole affected breast (so-called mastectomy), since then the advances in breast oncology have radically changed the surgical treatment of breast cancer. “Twenty years ago, says Maria João Cardoso, who is principal surgeon at the Champalimaud Foundation’s Breast Unit, in Lisbon, “it was established that removing the whole breast or just ablating the tumor locally and doing radiotherapy afterwards – so-called conservative surgery – lead to the same results in terms of survival”.

Since then, in a large majority of cases, it stopped being necessary to remove the whole breast – and more recently, it became possible to introduce elements of plastic surgery to create a “oncoplastic surgery”. Namely, the patient’s own tissues started to be used to fill in the defects caused by the removal of the tumor when the breasts were small and the void would be too visible, and reduction of the healthy breast was proposed to compensate for the volumetric asymmetry caused by the surgery in larger breasts.

For Maria João Cardoso, the aesthetic aspects of breast cancer surgery could no longer be ignored. As a question of psychological health and quality of life for the patients, it was important to take them into account. The time had come to talk about beauty with the patients.

Actually, there is no reason for women who survive breast cancer not to be able to make an informed choice of the surgery that will best preserve – and even improve – their body image.

But the truth is that the resulting aesthetic quality of the different surgical approaches may vary from patient to patient. So, at the same time, the possibility to choose among a panoply of surgeries makes the choice more difficult.

Why? “Because there aren’t yet any objective criteria to predict the aesthetic results of each type of surgery for each patient”, says Maria João Cardoso. “The choice ends up being the surgeon’s and is imposed on the patient, who, confronted with a number of possible surgical options, hesitates and often feels confused about which type of surgery will provide the best aesthetic results in her own case.”

First algorithmic contributions

Maria João Cardoso has been trying to objectify those subjective aesthetic opinions for years (not only the patients’, but very often, also the doctors’), which can vary as much as in the examples mentioned above. Her idea is to completely automatize the process that will predict the aesthetic results of breast cancer surgeries and to show the probable results to the patients before the intervention – in virtual form – so as to facilitate their choice. Maria João Cardoso firmly believes that that is possible – and that it will not only effectively lead to the best possible results, but also to more realistic predictions, thus avoiding too low or too high expectations on the part of the patients.

The mere mention of the word “virtual” tells us that solving the problem entails (as was expectable) a digital, computerized approach, capable of automatically measuring the physical features of each patient and returning an equally automatical answer. Maria João Cardoso – together with Jaime Cardoso, from INESC TEC, in Porto –, have already developed a software, BCCT.core, which does part of the job, and have been improving it for a number of years.

“The BCCT.core algorithm, has been learning, starting from around 200 photographs of patients’ front-view torsos following breast cancer surgery, to automatically classify the aesthetic quality of the results”, says Maria João Cardoso. “In the beginning, we only had pictures of white women and so it was also necessary to increase the diversity of the photographs.” In its latest version, which is web-based, BCCT.core can distinguish, in 99% of the cases, really bad surgical results from really good results, with the same precision as that of a group of human experts. However, for intermediate cases, the automatic evaluation of aesthetic quality becomes more imprecise.

Given that postoperative photos, by themselves, do not allow for making predictions, Maria João Cardoso and her team started photographing breast cancer patients before and after surgery, and introducing the pictures, after anonymizing them, into a database. “Starting in 2013, we thus constructed an image platform”, she says. The team also collected other data from the patients, such as age, height and bra cup size, which were needed for the software to reach its “verdict”.

Even so, BCCT.core, which is now used at the Breast Unit of the Champalimaud Foundation and in more than 300 other centers around the world, requires a time-consuming procedure. The pictures have to be taken manually, the additional data manually introduced, and after that, the patients needs to meet with a nurse in order to get a realistic idea of what will happen. More to the point, for each type of surgery, the nurse starts by searching the database for an “excellent” result in a case whose physical features most resemble those of the patient and shows her the image.

But the patient also has to understand that things do not always go as planned. This is why, “in order to constrain the expectation generated, it is also necessary to persuade the patient to also look at an example of a less successful aesthetic outcome”, says Maria João Cardoso. This can be difficult for the patient, but it allows her to make an informed decision about the type of surgery she wishes to be submitted to. It’s a substantial progress compared to previous procedures, such as retouching photographs to give the patient an idea of the result, or even “having the doctor draw some sketches on a piece of paper”, in Maria João Cardoso’s own words.

After the surgery – immediately after cicatrization, and six months and a year after finishing their treatment – patients will receive questionnaires to allow the medical team to evaluate each woman’s degree of satisfaction with respect to the obtained aesthetic results.

Advanced AI is making its way

To be able to aesthetically categorize surgical results (into good, bad, etc.), BCCT.core draws on a certain measure of “primitive” artificial intelligence (AI). But due to the most recent advances in AI – in particular in the field of so-called deep learning algorithms, with their enhanced capacity for image recognition, it will finally be possible to put the most ambitious of Maria João Cardoso’s ideas to the test. The international project Cinderella, which is the most recent phase of this long effort, is now ready to go full-forward thanks to the European grant that has just been approved for it.

The Cinderella project essentially consists in transporting to a health web-based platform called CANKADO (also available as an Android and Apple app), all the information about every possible type of surgery and all the quality questionnaires. CANKADO will link with the image repository and the IA algorithm. After the creation of this online structure, a large clinical trial will be conducted to compare the level of patient aesthetic satisfaction when the choice of surgery type involved advanced AI with that of patients who used the “conventional” procedure to choose their surgery type. For that, patients who are to be submitted to breast surgery at the five centers participating in the study – the Champalimaud Foundation, the San Raffaele Hospital in Milão,  Heidelberg’s University Hospital, Gdansk’s University Hospital and the Tel Hashomer Medical Centre Sheba, near Tel Aviv – will previously be randomly divided into two groups, each using one of the two procedures.

There are several differences between these two ways of choosing the surgery type. “For one thing”, says Jaime Cardoso, “the new advanced AI algorithm that is being developed by INESC TEC will enable the prediction of the surgery results based solely on photographs (before and after surgery), thus avoiding having to manually collect additional data.”

Also, in contrast to patients in the “control” group, who will continue to receive explanations about the surgeries from a nurse, those in the “experimental” group will get the explanations through CANKADO.

For the latter kind of “training” to be autonomous, a module containing a series of short videos about the surgeries will be added to the app, which will be produced during the project’s first year. In the beginning, a nurse will explain to the patients how the app works and, in principle, “if what is in the app corresponds to the questions and doubts the patients have, important resources will be spared”, says Maria João Cardoso”.

In the control group, the patients will be able to see photographs if they wish to, but these will be randomly chosen and will only concern one example of one of the similar surgeries. On the other hand, for those who have access to the app, the photographs will be chosen by BCCT.core and the advanced AI algorithm, and presented to the patients on the screen of their mobile device, making the process automatic. “The AI algorithm will also adapt the selected images to the physical traits of the patient for her to identify even more with the predictions she is shown”, explains Jaime Cardoso.

This kind of approach profoundly changes the way to confront the choice of surgery, Maria João Cardoso points out. This is because, instead of being based on a meeting with the surgeon and a nurse, patients with access to the app will be able to show the more realistic predictions for each type of surgery to their loved ones and discuss the best option with them. This allows a greater empowerment not only of the patients but, if they so wish, of other persons selected by them as well.

Pink, the photographer robot

But something was still missing: the automatization of picture-taking, another time- and human resource-consuming manual process. Maria João Cardoso solved the problem by finding a company that would develop a medical robot adapted to the situation. “The robot’s name is Pink, and a prototype is being built by the Czech company PhotoRobot using photographic equipment from Canon”, says Maria João Cardoso. The patient places herself in front of an illuminated background and the robot takes the pictures in the necessary positions. It will even be able to obtain three-dimensional images, something which could be useful for the augmented reality applications that are also being developed at the Breast Unit, for example Breast 4.0 (https://fchampalimaud.org/pt-pt/news/tecnologia-de-realidade-aumentada-e-modelos-3d-personalizados-da-mama-utilizados-pela-primeira), points out the doctor and scientist. The date when the photos were taken will automatically be registered by the robot together with the pictures.

“The first robot, which will be housed at the Champalimaud Foundation, should be installed in our digital lab a few months from now”, she adds.

Currently, for want of time and resources, no one takes photos or makes predictions in such a personalized, automated way, of the aesthetic results of breast cancer surgery. But if the Cinderella project shows that this automatic procedure works, resulting in a higher level of satisfaction for patients, that will be a first big step toward enabling women who are to be submitted to breast cancer surgery to truly make an informed and personalized choice of their surgery, increasing their chances to like what they see in the mirror.