This doesn't at all surprise me. I used to work at Anderson in the Leukemia department, brought on to analyze their "big data." They wanted to know which cells were indicative of relapse but were only able to provide data on two patients who each received different treatments. Despite my best efforts, I could not convince the MDs who run the show to obtain more (and better) data to even begin to apply machine learning. They had this intrenched opinion that math was something that got sprinkled over your data and turned rubbish into Science papers. I wouldn't be surprised if IBM encountered the same problem.
Moreover, this whole story is in line with my sense of the institutional culture. There is a lot of me-too science that goes on there, repeating better work at more prestigious universities. I bet when my bosses heard the pitch from IBM, they thought "this is my chance to compete with Stanford." However, no thought was given to the data that would feed into this system as evidenced by the changes of scope and infrastructure.
As an MD, and therefore looking at that from the other side of the fence, I totally agree. Medical research as a field has become profoundly dishonest since the "publish or perish" principle has become the rule also applied to clinical hierarchical advancement.
People with no scientific education (and clinical medicine is definitely not science) should be forbidden to lead scientific projects!
>It seems “very strange” that the IT department was bypassed, and “very unusual” that payments were not based on measurable deliverables, says John Halamka, the chief information officer at Beth Israel Deaconness Medical Center in Boston. He also notes that payments seem to have been made from donations that had not yet been received.
...
But inside the University of Texas, the project was apparently seen as one that missed deadlines and didn’t deliver. The audit notes that the focus of the project was changed several times, first focusing on one type of leukemia, then another, then lung cancer. The initial plan was to test out the product out in pilots at two other hospitals. That never happened.
MD Anderson changed the software it uses for managing electronic medical records, switching to a system made by Epic Systems of Madison, Wis. It has blamed this new system for a $405 million drop in its net income. According to the audit report, the Watson product doesn’t work with the new Epic System, and must be revamped in order to be re-tested. The information in the MD-Anderson/Watson product is also now out of date.
---
1. Requirements changes kill projects.
2. Key stakeholders not being on board kill projects.
3. Big system changes kill projects.
---
IBM has issues for sure, but I wouldn't expect a favorable outcome for anyone given these headwinds.
Caveat, we don't know how many of these issues were exacerbated/caused by IBM.
People in the AI community have spoken out against Watson because it's more marketing than tech. Some feared that it was setting machine learning / AI up on a pedestal that would give false hopes. It looks like those fears are now realized. I hope this doesn't further entrench naysayers against machine learning in medicine. Watson was a rules-based system from the get go and (to my knowledge) only recently started adding deep learning.
IBM bought Merge Healthcare for $1B, presumably to gain access to vast amounts of their imaging data. Would love to see some developments on the deep learning front from them, but sadly I doubt it'll ever break through the fog of marketing/PR BS that always seems to surround Watson.
TL;DR: the oncology part of the Watson project appeared to be working fine. The integration with MD Anderson BSS/OSS systems was the killer, especially since MD Anderson changed horses mid stream.
Integration with Business Support Systems and Operational Support Systems is the part where the customers make their raw data available to Watson, and Watson provides analyses/diagnoses in return. This data will usually arrive in proprietary formats and need to be delivered in proprietary formats, meaning integration is a bespoke operation for each customer, and will need to be done from scratch if the customer changes BSS in the meantime.
"The disclosure comes at an uncomfortable moment for IBM. Tomorrow, the company’s chief executive, Ginni Rometty, will make a presentation to a giant health information technology conference detailing the progress Watson has made in health care, and announcing the launch of new products for managing medical images and making sure hospitals deliver value for the money, as well as new partnerships with healthcare systems. The end of the MD Anderson collaboration looks bad."
Perhaps this has not been a total research failure but a political move. Watson Healthcare may increase the efficiency of imagine analysis among competitors of MDA. Imaging is one of the biggest revenue generators for a hospital.
Moreover, this whole story is in line with my sense of the institutional culture. There is a lot of me-too science that goes on there, repeating better work at more prestigious universities. I bet when my bosses heard the pitch from IBM, they thought "this is my chance to compete with Stanford." However, no thought was given to the data that would feed into this system as evidenced by the changes of scope and infrastructure.