Data in Pharma: How Companies Turn Unstructured Data into Decisions (+ Real Cases)

While in 2012 the average clinical trial protocol contained fewer than one million data points, today, according to Tufts CSDD, there are over 6 million. And that is for just one trial. On top of this, there are usually safety reports, laboratory equipment data, CRM records, and regulatory documents.

As the volume of data grows rapidly, so does the complexity of finding answers. In a 2024 survey of 300 life sciences executives, half of the respondents stated that insights are challenging or very challenging to obtain, although the majority of them already use AI-based data analytics tools.

So how do you stay in control of a data flow this size and get the information when you need it? In this article, Relevant Software looks at the core types of pharma data, the challenges that turn it into a cost, the use cases where it pays off, and the steps to build a data foundation AI can rely on. We’ll also share real-life cases of pharma data management from our own projects.

Unstructured data in pharma: quick summary

• More than 80% of healthcare data is estimated to be unstructured. In pharma, it sits in protocols, case reports, MSL notes, publications, and submissions.
• McKinsey Global Institute puts the value of generative AI for pharma and medical products at $60 billion to $110 billion a year.
• Most of that value comes from document-heavy work. McKinsey estimates gen AI could cut clinical study report drafting time almost in half and speed up major regulatory submissions by about 40%.
• Medical-legal review is another bottleneck. The same McKinsey analysis estimates a 50% to 70% reduction in review time.
• In our projects, most of the effort goes into data prep: cleaning it, tracing it to the source, and governing it well enough for QA and compliance to sign off.

Core types of pharma data

Reports are only the visible part of pharma data. Underneath sit lab instruments streaming readings, trial databases, safety systems, regulatory archives, CRM notes from field teams, batch records, and supplier contracts. Each one holds a piece of the story about a product, from the first assay to the last prescription.

The diagram shows where each type appears across a product’s life. Look at the arrow at the bottom: data from the market flows back into development. That’s why none of these sources can be managed in isolation.

R&D and discovery data

R&D and discovery is the data a company uses to decide which compounds go into clinical trials. The problem is where it lives. Instrument output lands in LIMS or on local file shares, experiment notes in the ELN, study reports in document management, and literature in external databases. Before a selection meeting, someone has to pull all of it together, usually by hand.

A large pharma company can run 8,000 to 9,000 lab instruments and employ thousands of R&D scientists, all producing assay results, omics data, and instrument output every day. Next to that sit electronic lab notebooks, internal study reports and the scientific literature. PubMed alone holds more than 40 million citations and has been adding close to a million a year.

Anything missed in that step costs a lot later. About 90% of drug candidates fail in clinical trials, most often because the drug doesn’t work well enough or turns out too toxic. If an early warning sign sat in a report nobody pulled into the dataset, the company learns about it in Phase II or III, after spending a large share of the median $1.1 billion it takes to bring one new drug to market.

Clinical trial data

Trials produce the evidence an approval rests on, and the load keeps growing. Tufts CSDD found that a Phase III protocol collected about 3.56 million data points in 2020, up from roughly 929,000 in 2012. Most of it lands in EDC systems, lab databases, ePRO apps, and imaging archives. The rest lives in protocols, amendments, investigator notes, CRF comments, and monitoring reports.

Structured fields record what happened to each patient. Free text often carries the context: why a visit was missed, why a site enrolls slowly, why an endpoint was hard to measure. Sponsors need both for go/no-go calls, protocol amendments, and site selection.

Regulatory data

Every claim on a label has to trace back to evidence, and that evidence travels through submissions. In 2024, FDA’s drug center alone received 310,433 electronic submissions. Behind each one is a long paper trail: applications, answers to health authority queries, labeling changes, CMC documentation, and the SOPs that govern them.

Here, speed and traceability are the business case. A slow answer to an agency query can push a launch back, and every answer has to point to its exact source.

Safety and pharmacovigilance data

Safety data never stops arriving. FDA’s adverse event database takes in over 2 million reports a year. Some come as structured case forms. Many arrive as emails, call center transcripts, literature articles, and social media posts that someone has to read, code and assess.

Serious, unexpected reactions usually have to be reported within 15 calendar days, so intake speed matters as much as accuracy. And safety data reaches into every other area. A confirmed signal can change the label, trigger a new study, or send quality teams back to specific batches.

Real-world data

Claims, electronic health records, registries and wearables show how a drug performs outside a trial, with real patients and real prescribing habits. A lot of the useful detail sits in physician notes and discharge summaries instead of coded fields.

It’s also one of the hardest sources to assemble. Life sciences teams work with four data vendors on average just to get the data they need. Done well, real-world data supports market access, label expansion, and better trial design.

Medical affairs and commercial data

CRM activity logs and sales numbers show reach and uptake. To understand what doctors think, you need the text around them: MSL field notes, HCP questions, advisory board summaries and medical information requests.

This is often where evidence gaps first show up. A question that comes up in fifty conversations can turn into a new publication, a label update, or a post-approval study. That’s how commercial data ends up shaping development.

Manufacturing and quality data

Manufacturing runs on structured data: process parameters, sensor readings, MES records and QC results in LIMS. The text comes from people, in deviation reports, investigation notes, CAPA records, and batch record comments.

Together, they decide whether a batch gets released and whether supply stays on track. When the same root cause hides behind differently worded deviations at several sites, only the text shows it.

Supply chain and procurement data

Prices, stock levels, shipments and product identifiers such as PZN or NDC change by the hour. In our procurement analytics project, the platform handles millions of transactional records a day. Contracts, tenders, and supplier emails add terms the numbers miss: delivery windows, penalties, exclusivity clauses.

This is the data that decides whether a product is available where patients need it, and at what price.

How the pieces add up

Each type matters on its own, but the important questions usually cut across several. Say a few MSLs hear about dizziness after a dose increase. To understand it, you’d check the safety database for similar cases, the trial data for the same patient subgroup, the batch records for the lots those patients received, and the CRM for any shift in prescribing.

That’s four data types, owned by four teams, in four systems.

Primary challenges of pharma data and what they cost your business

That split between data types, teams, and systems is where the biggest challenges start. Safety, clinical, manufacturing, and commercial data each have their own owners, formats and access rules. Every time a question crosses those lines, someone has to stitch the pieces together by hand, and the answer waits.

In pharma, that wait has a price tag. Tufts CSDD estimates a Phase III trial costs about $55,716 a day to run, and each day a product reaches the market later is worth roughly $500,000 in lost sales.

Now imagine that your data is scattered throughout the chain: more than 10 million data silos across biopharma. In a 2024 survey of 300 life sciences leaders, teams said it takes seven months on average to get data into a usable state.

Put these numbers next to the cost of delay, and you will see that the conclusion you get is hard to ignore. Getting data ready takes months, while every day of waiting costs money.

Unreliable AI output on fragmented data

Many teams expect AI to close the gap. In practice, a model answers from whatever data it can reach, and it answers with confidence.

Ask an assistant how many serious adverse events were reported for a product this quarter. If it only sees the safety database, it misses cases still sitting in the intake inbox. If literature cases are duplicated, it counts them twice. If it searches a shared drive, it may quote an SOP that was replaced last year. The answer looks precise, cites a source, and is wrong.

That’s where unprepared data really costs you. People stop trusting the tool after the first bad answer, and the project stalls. Gartner predicts that through 2026, organizations will abandon 60% of AI projects that aren’t supported by AI-ready data. In the same research, 63% of organizations said they don’t have, or aren’t sure they have, the right data management practices for AI.

Unified, governed data is what turns AI into something a safety lead or a medical director can act on.

Data silos

Answering a cross-functional question can take weeks, because every answer needs data from three or four teams. If you have an AI tool and it’s connected to only one of those systems, it’ll give you an answer that covers half the picture. That’s why we start by connecting the sources behind your priority decisions and recording where every file comes from.

Slow path to usable data

When data takes months to prepare, decisions run on last quarter’s numbers. Analysts spend their week copying data between spreadsheets.

We take that manual work off your team with data pipeline development:

• Connectors pull new records from CRM, EDC, safety, and lab systems on a schedule or the moment they change.
• Validation rules then check each record for missing fields, wrong formats and mismatched units before it moves on.
• The data is mapped to one shared model, such as CDISC for trial data or OMOP for real-world data, with ML models suggesting field matches, and flagging likely duplicates for a person to confirm.
• Monitoring alerts the team when a source changes or a load fails, so problems surface the same day.

Duplicates and conflicting versions

When the same HCP shows up three times in CRM and two versions of an SOP sit in one folder, AI counts the duplicates, and confidently quotes the outdated document. To keep that from happening on your pharma project, Relevant Software deduplicates records, keeps one master version of each entity and sets quality rules every source has to pass.

Documents without structure

Scientists spend roughly half their time collecting and retyping data. Insights locked in PDFs never reach a dashboard, and AI can’t cite what it can’t read. We extract fields with OCR and LLMs and code them to medical vocabularies like MedDRA.

LLM accuracy in a medical context

One misread dosage or adverse event in a safety report or HCP letter means a compliance incident, a correction, and a tool nobody wants to use again. We ground models in your own documents, require a source for every answer, score confidence, and send uncertain outputs to a reviewer.

Validation and regulatory compliance

In pharma, an AI tool won’t reach production until inspectors can see what it does and how it was tested. Quality assurance on these projects covers test plans for AI outputs, versioned prompts and models, and a log of every output, which keeps validation and inspections on track.

Patient and HCP data privacy

One leak of patient data can mean GDPR or HIPAA penalties and damage that outlasts any fine. We run models inside your cloud or on your own servers and stop sensitive text from reaching prompts.

Legacy systems

Many pharma companies still run older LIMS, home-grown trial databases, or on-premise warehouses. They work, but they were designed long before today’s analytics and AI tools, and they don’t share data with them easily. So every new dashboard or AI assistant needs its own custom integration; those integrations break when something changes, and a big part of each project’s budget goes into plumbing instead of the use case itself.

We deal with this in two steps. First, we build a connector layer that reads data from the old system without touching how it works, so teams can use that data in the shared platform right away. Once the value is clear, we modernize the legacy system module by module.

Pilots that never reach production

The budget goes into demos that never pay back, and the team gives up on AI before it gets a fair test. We plan for production from the first sprint, with real data volumes, audit trails, and uptime targets.

All of this adds up to a lag between what the data says and when anyone acts on it. In R&D, it shows up as researchers waiting on data from other teams, and our whitepaper on how AI streamlines research data flows looks at how to remove those bottlenecks.

Big data, analytics, predictive models and RWE: how each one works with pharma data

Once data is connected and cleaned, the next question is what to do with it. Pharma teams usually reach for four approaches: big data, data analytics, predictive analytics, and real-world evidence. They often get mixed up, but each one does a different job with the data and answers a different question.

Big data

Big data in the pharmaceutical industry deals with datasets that are too large or too diverse to be processed by a single database. These usually originate from pharmaceutical research, patient medical records, genomics, medical imaging and wearable devices, and come in a wide variety of formats: sequencing files, images, sensor data streams, tables and free-text data.

For businesses, the benefit lies in preserving data that would otherwise be lost. Raw sequencing results, images, and sensor data streams remain searchable for many years. When your team has a new question, they may not launch a new study or collect data from scratch. They will first check what the company already has.

Technically, handling this takes three things:

• storage spread across many servers instead of a single database;
• computing power that grows with the volume of data;
• pipelines that process different data types at once: DNA sequences, images and sensor signals.

Most of the team’s work on such a project comes down to two decisions: which tools will process the data and how to lay it out in storage. The goal is to keep queries fast and stop the cloud bill from growing unpredictably as the data grows.

Data analytics

The next step is examining raw data to find patterns, relationships, and insights a team can act on. That’s the core job of data analytics.

The teams that get the most value here are the ones that manage to combine different sources. You can get useful insights from enrollment numbers, and just as much from investigator notes and monitoring reports. The problem is that notes are free text, while the numbers sit in a table. Once you pull the key details out of the text and put them into fields (reason, date, site), the numbers and the explanation can appear in the same report.

Most businesses already have numbers their teams argue about, and the best move is to start with those. From there, we agree on one definition for each metric and build a semantic layer, a layer on top of the data where those definitions live. Only then do we build the dashboards.

Predictive analytics

Predictive analytics uses historical data to estimate what’s likely to happen next. Typical pharma questions: which patient subgroups are most likely to respond to a drug, which sites will miss enrollment targets, which patients may drop out, how demand will shift after a competitor launch.

Efficacy in specific populations is where this gets most interesting. Models trained on trial data, genomics, and real-world outcomes can point to the patients who benefit most, which is the idea behind precision medicine AI.

A model is only as good as its history, though. Biased or incomplete training data produces confident but skewed predictions, and regulators will ask how a model reached its output. In AI software development for pharma, most of the effort goes into feature selection, validation on data the model hasn’t seen, and monitoring for drift once it’s live.

Real-world evidence (RWE)

Real-world data comes from everyday care: electronic health records, claims, registries, and wearables. It becomes real-world evidence when it’s analyzed well enough to answer a clinical or regulatory question.

Regulators take it seriously. Since the 21st Century Cures Act of 2016, FDA has run a formal program for evaluating RWE. In 2021, it approved a new use of a transplant drug based on real-world evidence, using a non-interventional study built on national transplant registry data. Companies also use RWE for label expansion, external control arms, post-marketing safety, and market access.

Commercially, RWE can open doors that would otherwise need a new randomized trial: a new indication, stronger evidence for payers, answers to post-marketing commitments. The transplant approval above is a label expansion built on registry data.

The hard part is data quality. EHR data is full of free text, coded differently from one hospital to the next, and often exchanged through HL7 or FHIR interfaces that need careful mapping. On RWE projects, data quality management can cover that groundwork: linking records across sources, removing duplicates, de-identifying patients, and documenting every transformation, so the evidence holds up when a regulator or payer reviews it.

What structured pharma data looks like and the benefits it brings

So far, we’ve looked at what poor data costs. Here’s the other side. Structured data means every document becomes a record you can count, filter, and compare. Here’s how that works on an illustrative MSL field note, the kind of entry that lands in CRM every day.

One record like this isn’t much. A few thousand of them are. Now you can see how many cardiologists raised renal dosing this quarter, whether the number is growing, and which possible safety events still need a report. Nobody had to read every note to get there.

The same logic works for other sources:

• An adverse event email becomes a case with patient age and sex, suspect drug, event terms coded to MedDRA, seriousness, and onset date.
• A trial protocol becomes a list of inclusion and exclusion criteria, visit schedule, endpoints, and sample size you can compare across studies.
• A deviation report becomes a record with site, batch, root cause category, and CAPA status. Quality teams can then spot repeat issues across sites.

Benefits of structured data: six use cases

Once text becomes records like one showed above, the benefits show up quickly. We see the fastest payback in the following six areas. Where there’s solid data on the impact, we’ve added it under each one.

Medical affairs insights from field notes

MSLs log hundreds of conversations with healthcare professionals. Taken together, those notes show evidence gaps, competitor messages and repeat safety questions. In practice, few people have time to read them all. An AI layer groups the notes by theme, flags what’s new, and drafts the monthly insight report.

Most of these notes live in CRM, and that’s where the insights should show up too. In healthcare CRM software development, we add note templates that capture key fields at entry, connect the CRM to the AI layer and show themes, and flags right on the HCP record. Our AstraZeneca project below is a good example.

Impact: after we built an AI analytics system for AstraZeneca’s medical affairs teams, they now prepare medical materials 40% faster, with 25% fewer data errors.

Literature and evidence monitoring

Safety and medical teams have to scan new publications every week. A retrieval-based assistant can screen abstracts, pull out the relevant findings, and link each one to its source. Reviewers then work from a shortlist, which takes far less time than the full feed.

Impact: in a study of eight completed systematic reviews, machine learning-assisted screening could save 40% to 60% of screening effort and still catch 95% of eligible records.

Pharmacy prescriptions and dispensing records

Pharmacies still receive a lot of unstructured input: scanned and faxed prescriptions, handwritten notes from doctors, supplier invoices, and recall notices. Each one has to be read and re-entered before stock, dispensing, and billing systems can use it. Once the drug, dose, quantity and prescriber land in structured fields, the system can check interactions and stock levels on its own.

In pharmacy management software development, we connect that extraction layer to e-prescribing, inventory, and dispensing modules. A prescription then moves from intake to the stock check without anyone retyping it.

Impact: in a study of 12 community practices, prescription error rates fell from 43% to 7% within a year after providers switched to electronic prescriptions, while the group that kept writing by hand stayed at 38%.

Clinical trial documents

Protocols, amendments and site reports are long and full of cross-references. Comparing a draft protocol against past studies, or spotting inconsistencies across documents, can take days by hand. A well-tuned model gets the first pass done much faster.

Impact: McKinsey estimates gen AI could cut clinical study report drafting time almost in half and speed up responses to health authority queries by about 30%.

Procurement and pricing analytics

Buyers need current prices, stock levels and price history in seconds, especially during tender windows. A natural-language interface over that data removes the wait for an analyst to write SQL.

Impact: after we built natural-language analytics for a pharma procurement platform, its buyers get answers in under 15 seconds at the median and run 650+ self-service reports a month.

If you’re not sure which of these fits your data, start small. An AI audit looks at the data you have, the systems it lives in, and the processes around it, then ranks use cases by value and effort. You’ll come out with a shortlist and a realistic view of what each one needs.

How pharma companies turn unstructured data into decisions in 6 steps

So how do you get from scattered documents to the structured records shown above? On our pharma and healthcare projects, the path usually takes six steps. Here’s what happens at each one and where we come in.

Step 1. Start from the decision

Pick one decision that’s slow, expensive, or risky today. For example: “Which HCP concerns should shape next quarter’s medical plan?” A vague goal like “use AI on our documents” gives the team nothing to measure.

Then list the sources that feed that decision, who owns them, and what rules apply to each. Before anything gets built, a data strategy consulting engagement answers the practical questions: which sources hold the answer, how clean they are and what it would take to connect them. It usually takes two to four weeks and ends with a short roadmap, and a first use case.

Some teams aren’t sure the organization is ready for AI at all. Legal may have no policy yet, IT may not allow cloud models, and nobody clearly owns the data. An AI maturity assessment checks those things up front: data access, infrastructure, skills, governance, and the processes AI would touch. You’ll know where the gaps are before you pay for a build.

Step 2. Bring the sources together

Next, you connect the systems: CRM, EDC, LIMS, safety databases, document repositories, email exports, and scanned PDFs. Each one has its own format and access rules.

This part isn’t glamorous, but it decides how fresh your insights are. We set up data ingestion pipelines that pull documents from each source on a schedule or in real time, convert them to a common format, and record where every file came from and when. That lineage pays off later, when QA asks which version of a report an insight was based on.

CRM needs extra care. Field notes are often the richest source you have and the messiest: free text, abbreviations, mixed languages, notes attached to the wrong account. Here, CRM data integration means mapping accounts, and contacts correctly, syncing notes without duplicates and keeping consent and access rules intact as data moves out of the CRM.

Step 3. Pick the right storage layer

Unstructured and structured data have to live side by side. Warehouses are built for tables, and documents fit poorly there. That’s why most pharma teams end up with a lakehouse. Raw files sit in cheap storage, and extracted fields land in governed tables you can query.

If you’re weighing the options, data lakehouse consulting helps you choose before you commit. We compare a lakehouse with your current warehouse, estimate storage and compute costs, and design how raw files, extracted fields, and access rules fit together.

Step 4. Extract structure with AI

This is where text turns into fields. Depending on the source, the stack includes OCR for scans, LLMs for extraction and summarization, and entity linking to medical vocabularies like MedDRA or SNOMED CT.

Two patterns dominate. Retrieval-augmented generation keeps the model grounded in your own documents and cites the source for every answer, which compliance teams like. Fine-tuning helps when the language is very specific, as with MSL shorthand. We often combine both.

In pharma, RAG development is usually the first piece we build. It covers splitting documents into chunks that keep their context, choosing embeddings that handle medical terms, setting up the vector index and testing answers against questions your experts have already answered. That last part tells you whether the system is ready for real users.

Step 5. Clean, govern and protect the data

Patient and HCP data also need protection before they reach any model. We apply data anonymization at the pipeline level. Names, birth dates, addresses, and other identifiers are masked or replaced with tokens, and free-text fields are scanned for identifiers that slip into notes. De-identified records become the default, and only approved roles see the originals.

The next task is defining data ownership and access. Data governance sets the rules: who can see which data, how long you keep it, how definitions stay consistent across teams, and how every change is logged for GxP and audit purposes. Without it, two dashboards end up showing different numbers for the same thing.

Step 6. Deliver insights where decisions happen

The last step is the one people see. An insight buried in a data lake changes nothing.

Depending on the user, that could be a dashboard for leadership, a weekly summary for medical affairs, an alert for the safety team, or a chat assistant that answers questions in plain language.

For teams that want to ask their data questions directly, we build AI-powered data analytics tools. A medical lead types “Which cardiologists raised renal dosing this quarter?” and the system writes the query, runs it against governed tables, and shows the answer with the records behind it. Analysts stop being the bottleneck for routine questions.

Some tasks repeat every day: sorting incoming documents, routing possible safety events, preparing a first draft of a report. An AI agent can take over those steps, use the same tools your team uses, and hand anything unclear to a person. We keep agents narrow at first, with clear limits on what they can do on their own.

Real cases: how we turn pharma data into decisions

Two projects show the pattern from different ends of the value chain. One deals with free text in medical affairs. The other makes a large pharma dataset answerable in plain English.

AI-powered medical affairs analytics for AstraZeneca

Challenge. AstraZeneca’s medical affairs teams were processing a growing volume of complex medical data by hand. Reporting was slow, and insights reached decision-makers late. Any automation had to keep accuracy and compliance intact.

Data. Three kinds of unstructured text: clinical trial documentation (protocols, treatment results, safety observations), real-world evidence (physician feedback, patient stories, healthcare reports), and MSL field notes from conversations with healthcare providers.

Solution. We built an AI-powered CRM analysis system that collects data from several sources, checks it, and turns it into structured, compliant reports. The models, ChatGPT and Llama-2, were fine-tuned on medical affairs data and run on Google Cloud. Encryption, role-based access and GDPR- and HIPAA-aligned safeguards protect the data throughout.

Outcome.

• 35,000+ records processed automatically
• 40% faster preparation of medical materials
• 25% fewer data errors

Decision it enabled. Medical affairs leads now see themes from the field while they’re still current, which helps them plan evidence work and HCP engagement on time.

Natural-language analytics for a pharma procurement platform

Challenge. An Austrian healthtech company turns pharmaceutical supply-chain data into analytics for pharma buyers and procurement teams. As its customer base grew, dashboards froze and complex SQL requests stalled during peak bidding windows, exactly when buyers needed live prices and stock levels.

Data. Millions of transactional records a day, live price and inventory data, price history, pack sizes, and product identifiers such as PZN, spread across eight source systems.

Solution. We built a chatbot that turns plain-language questions into validated database queries, backed by a multi-agent setup that routes each question to the right workflow. The ingestion pipeline also vectorizes product data, so users can search by similarity instead of exact names. Every answer shows the generated SQL, which keeps the logic transparent.

Outcome.

• 8 source systems unified
• 650+ self-service reports a month
• Median response time under 15 seconds

Decision it enabled. Procurement teams check prices and stock during a tender without waiting for an analyst or a nightly batch job.

How Relevant Software can help

Here’s where we usually come in, depending on where you are today.

“We don’t know where to start”

We’ll map your decisions, sources, and constraints, then pick one use case that can show value in a quarter. At this stage, AI consulting is mostly workshops and data reviews. We talk to the people who’d use the system, check the data they’d rely on, and estimate the cost and timeline of a first release. You get a plan you can take to budget approval.

“We have the data, but nobody can use it”

Then the job is pipelines, storage, quality checks, and access rules. In data engineering consulting, the first step is almost always an audit of what you already have: which systems hold what, where data breaks, and what’s duplicated. From there, we design the target architecture and plan the migration in stages. Existing reports keep working while the data moves.

“We need AI on top of our data, and it has to be safe”

We’ll build the extraction, search, and assistant layers and set up the controls your QA team expects. AI integration is about fitting those tools into the systems you already run, such as CRM, EDC, and document management. Users get AI features inside the screens they know, and data stays under the same access rules.

We’ve worked with pharma companies, healthtech startups and healthcare providers, including AstraZeneca. Beyond data and AI, we build clinical trial software, EDC systems, pharmacovigilance tools and regulatory document management, and connect them to LIMS and other lab systems. That range is what pharma software development at Relevant covers. Tell us about your data, and we’ll suggest where to start.

FAQ