Sonam Dubey, Beghou Consulting 

What if we could flip the patient data iceberg to expose what is typically hidden from sight—the full narrative of a patient’s healthcare journey? In the 10% of the iceberg we see above the surface are the traditional data sources such as claims and diagnostic and procedure codes, which give only us a snapshot of the patient. It is the vast base underwater that provides the rich detail we need. When paired with advanced analytics, this data can help us reduce time to diagnosis, understand the “why” of patient behaviors and decisions, and identify patient cohorts for enhanced patient engagement and interventions.

The industry’s current approaches to patient data analytics focus on structured data (e.g., height, weight, standardized test results, diagnosis codes, medications, stages of disease diagnosis). Although this data helps capture the patient’s journey, it does not represent 100% of the patient’s interactions with the healthcare system. In fact, only 20% of patient data is captured in structured form, with the remaining 80% comprising unstructured data (e.g., physician notes, progress notes, infusion summaries, lab reports). Furthermore, according to RBC Capital Markets, the compound annual growth rate of data for healthcare will reach 36%, and there will be nearly 5,000 digital device interactions daily per person by 2025.

 All this will significantly increase the amount of unstructured data.

The combination of increasing variance, volume, and velocity of patient data overall and complex unstructured data, which is not easily codified, has limited the industry’s ability to understand the full patient journey. For life science companies developing niche and targeted therapies, the resulting lack of insight creates challenges for identifying the right patients and intervening at the right moments to support treatment journeys.

Moving the patient health paradigm from 'what' to 'why'

Structured data primarily answers the “what” of patient health—clinical diagnoses, current disease status, treatment initiation, and discontinuation. Unstructured data provides the context around their health—the “why” —a comprehensive longitudinal view of the patient, a road map to their physicians’ thought processes, and the ability to understand the clinical and socioeconomic aspects of patients’ reactions to therapies and diagnoses.

For example, clinical notes capture nuanced observations about patient behavior, drug responses, and lifestyle factors; laboratory narratives and genomic data reveal deeper biological patterns and potential therapeutic targets; and detailed pathology reports provide context around disease progression. Advanced natural language processing (NLP) algorithms can extract meaningful insights from these unstructured sources, converting them into analyzable data points. Combining these with structured data sources in a way that is compliant with privacy requirements can generate a more comprehensive patient data set.

Accelerating diagnoses using unstructured data

A delayed rare disease diagnosis can be not only mentally and emotionally draining for patients, but also financially draining. According to a study by The EveryLife Foundation for Rare Diseases, the “diagnostic odyssey” for a patient with a rare disease can result in an average of more than $220,000 in medical costs and lost income.

 Each year saved in diagnosis directly impacts patients. We have seen in our work with life sciences companies that it can take up to eight years and 15 (or more) clinical encounters for patients to secure a diagnosis.

Therefore, finding new ways to speed diagnoses could significantly improve patient outcomes and finances. Utilizing artificial intelligence (AI) on unstructured data can fill gaps in a patient’s diagnostic journey.

For example, relatively simple documentation of the progression of fatigue in primary care notes, when combined with other longitudinal elements of the patient journey, can be an early indicator of rare pediatric diseases with a genetic predisposition that might be asymptomatic at birth. NLP-based analysis of physician notes could flag the need for appropriate specialist referrals for further diagnostic tests, ultimately reducing the time to diagnosis.

For other diseases, the use of AI to analyze patient data could help reduce misdiagnosis or underdiagnosis. The biggest challenge for one of Beghou’s clients with a rare disease drug was finding patients and initiating treatment. Misdiagnosis often occurred because of the multiple levels of symptoms, most of which were mild in the initial disease phase, and considerable delays in the confirmatory genetic test for most patients. Therefore, instead of using only claims data, which identified very few patients, we identified patterns of symptoms from patient notes. These symptoms ranged from simple skin spotting, vertigo-like symptoms, decreased ability to sweat, gastrointestinal issues, kidney-related issues, and irregular heartbeat. Using AI models, a substantial number of patients, each with a bundle of these symptoms, were identified within regional hotspots, which enabled focused outreach and facilitated diagnosis.

Enhancing engagement and personalizing interventions

Detailed patient profiles can also be developed using advanced analytic techniques such as machine learning and predictive modeling on clinical data (e.g., electronic health records, lab results, treatment responses), behavioral patterns (e.g., medication adherence, appointment attendance, portal usage), and socioeconomic factors (e.g., education, income, access to care) from both structured and unstructured data.

These profiles can be used to identify distinct patient microsegments with unique needs and preferences based on subtle patterns in disease progression; objective and subjective treatment effectiveness; predictions of potential complications; engagement preferences and compliance tendencies; and barriers to care and access to resources.

Patient microsegments facilitate the development of more effective, personalized intervention strategies and patient-centric, outcome-driven engagement plans, such as simplified medication schedules for patients with complex situations, digital health tools for tech-savvy individuals, and high-touch support programs for those requiring additional assistance.

For example, for patient cohorts predicted to be less adherent to their treatment, providers can understand underlying factors such as adverse effects, social determinants of health, or misunderstandings about the importance of treatment. Appropriate intervention strategies could include adverse effect management, reminder notifications, digital app-based prompts, and education about the treatment itself.

Improving insight quality and reducing cycle times with GenAI: A real-world example

We used data science and AI to improve the quality of insights by 25% and reduce intervention implementation cycle times by 20% for a specialty client. The GenAI program enabled and scaled AI infrastructure across the enterprise and functions to enable timely interactions with the healthcare providers and patients. Raw data from various unstructured data sources were enriched with the client’s specific ontologies and medical vocabulary to ensure the model could address ill-framed or abstract questions and integrate insights using retrieval-augmented generation. The latter allows the model to handle large amounts of data with relative ease and provide contextually relevant responses for the brand from all data sources, with a focus on usability via conversational AI, executive summaries, insights dashboards, and semantic search.

In some ways, claims data sets represent only the tip of the iceberg. With rapid technological advancements, especially with GenAI, we are on the brink of a large-scale transformation where combining structured and unstructured data sets will truly revolutionize how we understand patients and their journeys, engage and provide meaningful support, and impact patient outcomes.

To turn this potential into reality, we must radically change how healthcare stakeholders use unstructured data. This relies not only on tech stack, data storage, AI, and machine learning advancements, but also on data security and structural and change management. Cross-functional collaboration with clinicians and data scientists is fundamental to:

Understanding the nuances of medical ontologies.

Normalizing advanced AI algorithm use across healthcare systems.

Optimizing storage across disparate systems using common data models.

Refining definitions in metadata to evolve with advancements in medicine.


Instituting change management across functions for data democratization.

With a holistic, collaborative approach to designing richer data ecosystems and implementing advanced technologies, we can gain an in-depth understanding of health-related behaviors and decisions. Interventions centered around the patient then become possible, providing benefits for patients, physicians, and life science companies alike. Are we ready?

Sonam Dubey and Vishal Singal are partners at Beghou Consulting.

1. RBC Capital Markets. The Healthcare Data Explosion. 2020. 

https://www.rbccm.com/en/gib/healthcare/episode/the_healthcare_data_explosion

2. EveryLife Foundation for Rare Diseases. The Cost of Delayed Diagnosis in Rare Disease: A Health Economic Study. 

https://everylifefoundation.org/wp-content/uploads/2023/09/CombinedInfo_PDF.pdf

Articles

Filling the gaps in the patient journey within an increasingly complex data environment. 