A regulator asks for the record behind a three-month-old AI response. Could your team produce it today? Relevant sets up LLM observability that records the prompt, sources, and model version behind every answer: traces your engineers can debug and your auditors can accept.
Relevant engineers record every model call, retrieval, and tool call as linked spans, with one span per step in workflows built with LLM orchestration.
Each trace records the passages your retriever returned, their relevance scores, and the ones the answer cited, and hands weak retrievals to RAG development for tuning.
Personal data leaves each prompt and response before storage: our engineers hash customer IDs so traces still link, replace names, and drop card and account numbers entirely.
Auditors can request any trace as a record: our team sets retention periods, role-based access, and the export formats your IT compliance reviews expect.
Any traced request can run again with the same prompt, context, and model version, so engineers find the exact step that changed an answer.
Your traces can live in the tools you already run, such as Datadog, Grafana, or Elastic, or in a self-hosted Langfuse, and connect to the routing layer we set up through our AI orchestration development services.
Our engineers list each feature that calls a model, the data it sends, and the regulation that applies to it. Each feature receives a risk tier, and the tier sets how much detail its traces keep.
One feature goes first. Relevant engineers add spans and redaction to it in staging, and your team reviews sample traces for anything missing or too revealing before production.
The feature with the highest regulatory exposure moves to production tracing first, such as loan pre-screening or patient-facing answers. Relevant then adds the remaining features in order of their risk tier.
Early traces often show recurring causes of weak answers, such as a retriever that returns outdated documents or a prompt that drops a rule. The report ranks them by frequency. Where the cause sits in the model or the retrieval design, the fix goes through LLM development.
Your compliance lead picks a past response at random. Relevant reconstructs it end to end from the trace: prompt, sources, versions, and who viewed the record. The drill shows how long a real request takes before an auditor sends one.
Every LLM feature now writes traces in production. Relevant hands over the instrumentation code, redaction rules, and runbooks, and each new feature starts on the same setup from its first release.
Certified under ISO 27001:2022, ready for HIPAA, aligned with GDPR. Relevant engineers work with your trace data under those controls, including access by role and logged sessions.
Delivery up to 50% faster, with up to 20% fewer bugs. Our engineers apply AI-assisted development to the repetitive parts of instrumentation, such as span code and redaction tests, within the Relevant AI Delivery Framework™. A senior engineer approves every change to what your traces capture before it merges.
More than 10 years in data engineering. Relevant data engineers design trace storage for volume, partitioned by date and feature, with retention jobs that remove expired records on schedule.
Five Claude Certified Architects. Before instrumentation starts, one of them reviews what each trace will capture for every model call.
Custom development is all Relevant sells. The backend choice for your traces stays open between open-source and commercial options.
Client satisfaction at 98%. Your tracing work runs under the same delivery process those clients rated.
Monitoring watches the whole system: latency, error rates, spend, and quality scores against set limits, with alerts when a limit breaks. Observability explains a single request: what the model received, what it retrieved, and which version answered. Most regulated teams need both, and the order depends on the question in front of them. If complaints or audit requests already need answers, observability comes first. If the priority is proof that the system stays within approved limits, monitoring leads. For fleets of agents that act across systems, AI agent monitoring and orchestration covers the coordination layer as well.
Yes, tracing can go into most applications as they stand. Calls made through common SDKs, such as the OpenAI or Anthropic clients and LangChain, pick up spans through OpenTelemetry auto-instrumentation. Custom retrieval code and in-house tools need spans added by hand, and our engineers write those. The harder cases sit in two places: a vendor wrapper around the model client, and streamed responses that arrive in pieces. Both need manual work at the seam. If the vendor gives you no access to the code at all, tracing can still run at the routing layer and capture prompts and responses, though retrieval steps stay out of view.
Four counts set the size of the work, and you can pull them from your provider dashboards before a first call:
Running cost follows the backend. A self-hosted setup costs storage and compute in your cloud. A commercial tool usually charges by trace or by span. If the counts are hard to find, the first step of the engagement produces them. Relevant delivers the tracing roadmap on average 14 days after kickoff. To size your setup, contact us with the counts you have.
The answer depends on the feature. The OpenTelemetry standard records only metadata by default: model name, token counts, and duration. Features whose answers a customer could dispute need the full prompt, sources, and response, with personal data removed before storage. Internal tools, such as a meeting summarizer, can run on metadata plus a sample of full traces. A quick test for your own list: mark every feature where a customer or regulator could ask why the system said what it said. Those features need full content. If the list is unclear, our engineers build it with you in the first step, and you need nothing prepared.
LLM traces often hold more personal data than the application database, because a single prompt can combine records from several systems. Access works best in three tiers:
Relevant sets up these roles in the backend and records each view, so your privacy officer can review who opened what.
Relevant builds traces your engineers can debug and your auditors can accept, so it does.
