Building an Enterprise AI Platform: Architecture, Guardrails & Lessons

The Challenge: Enterprise AI That Talks to Real Databases

In early 2026 I ran into a problem a lot of engineering teams are wrestling with right now: how do you bring generative AI inside a company without exposing its sensitive data?

The goal was clear: let non-technical users ask natural-language questions about their operational data — sales, inventory, financial metrics — and get accurate answers grounded in real information, not hallucinations. But with one non-negotiable requirement: database credentials never leave the customer’s network.

This post walks through the architectural decisions, the trade-offs, and the lessons I learned building this platform from scratch.

---

The Architecture: 5 Services with Clear Responsibilities

After evaluating a monolith, serverless, and several hybrids, I landed on an architecture of 5 distributed services, each with a single responsibility and a well-defined contract:

┌─────────────────────────────────────────────────────────────┐
│                        API Layer                             │
│   POST /api/v1/query       — Main AI inference               │
│   POST /api/v1/flows/exec  — Agent Flow execution           │
│   GET  /api/v1/health      — Health + provider status       │
└────────────────────────┬────────────────────────────────────┘
                         │
                ┌────────▼─────────┐
                │ QueryOrchestrator │
                └───────┬──────────┘
          ┌─────────────┼──────────────┐
          ▼             ▼              ▼
   ┌──────────┐  ┌───────────┐  ┌──────────┐
   │  RAG-SQL │  │ RAG-Docs  │  │   Chat   │
   │ Pipeline │  │ Pipeline  │  │ Pipeline │
   └──────────┘  └───────────┘  └──────────┘
          │             │              │
          └─────────────┼──────────────┘
                        ▼
              ┌───────────────────┐
              │ 3-Layer Guardrails│
              └───────────────────┘
                        ▼
              ┌───────────────────┐
              │    LLM Router     │
              │ OpenAI · Anthropic│
              │ Google · Cohere   │
              └───────────────────┘

Why This Split

This wasn’t an arbitrary call. Each service has a different lifecycle, scaling model, and risk profile:

1. AI Engine (FastAPI + Haystack 2.x + Qdrant) — The brain. It processes natural language, classifies intent, and runs the RAG pipelines. Python was the obvious choice given the ML ecosystem.

2. SaaS Backend (Express + Prisma + TypeScript) — The business layer. Auth, RBAC, onboarding, project management, the semantic layer, agent flows. TypeScript, because that’s where the product team lives.

3. Frontend (Next.js + MUI + @xyflow/react) — The interface. A chatbot UI plus a visual flow builder so non-technical users can design agent flows.

4. Gateway Agent (Node.js) — The on-premise bridge. It’s installed inside the customer’s network. Outbound HTTPS only.

5. Gateway Provisioner (Go) — The orchestrator. It manages the lifecycle of gateway agents as Docker containers.

---

The RAG-SQL Pipeline: From Natural Language to Precise SQL Queries

The most complex component was the RAG-SQL pipeline. Here’s the flow:

1. Semantic Schema Linking — A user asks, “How much did we sell in January?” The system searches Qdrant for the most relevant tables and columns using embeddings of the customer’s database schema.

2. LLM-Powered SQL Generation — With the schema context in hand, the LLM generates the SQL query. We use few-shot examples indexed in Qdrant to improve accuracy.

3. Validation and Execution — The query goes through scope validation (only SELECT allowed), runs against the customer’s database through the gateway agent, and the result is formatted back into natural language.

The Decision: Haystack 2.x over LangChain

I chose Haystack 2.x over LangChain for three reasons:

• Typed, composable components: Every pipeline step is a component with explicit inputs and outputs. No magic chains, no runtime type inference.
• Native providers: Haystack’s ChatGenerator classes for OpenAI, Anthropic, and Google are thin wrappers over the official SDKs — a smaller attack surface than generic third-party abstractions.
• Explicit pipelines: Connections between components are defined with Pipeline.connect(). When something breaks, the stack trace points to the exact component, not to a tangle of nested callbacks.

---

The Zero-Trust Gateway: Why We Didn’t Use a VPN

This was the most important architectural trade-off of the project. Most enterprise solutions handle connectivity to customer databases with site-to-site VPNs or SSH tunnels. We didn’t.

The Problem with VPNs

• Operational friction: Setting up a VPN requires the customer’s networking team to get involved. At Latin American companies, that can take weeks.
• Attack surface: A VPN tunnel exposes the entire network, not just the database.
• Maintenance: Expiring certificates, changing firewall rules, incompatibilities between vendors.

The Solution: A Gateway Agent with Outbound-Only HTTPS

The gateway agent is a lightweight service (Node.js) installed inside the customer’s network. Its security model is radical in its simplicity:

• Outbound connections only: The agent initiates every connection to the cloud. Zero inbound ports. Zero firewall changes.
• Read-only enforcement: At the application level (before the query ever touches the database), the agent blocks any statement that isn’t a SELECT. INSERT, UPDATE, DELETE, DROP — all blocked.
• Job polling: Instead of receiving queries (which would require inbound ports), the agent periodically asks the SaaS backend, “Any work for me?” When there’s a query, it runs it locally and sends back the result.
• Heartbeat with backoff: The agent reports its status (ONLINE/DEGRADED) every 30 seconds. If it loses connectivity, it uses exponential backoff so it doesn’t flood the network when reconnecting.

The Provisioner: Go for Docker Orchestration

Why Go for the provisioner? Because we needed a binary that could:

• Talk efficiently to the Docker socket
• Handle HTTP webhooks with HMAC authentication
• Implement a supervisor with dormancy/wake logic (if a gateway agent has no SQL activity for X minutes, it’s stopped; when a job arrives, it’s spun back up)

Go checks all three boxes with a ~15MB static binary — no runtime, no dependencies. The provisioner receives a webhook from the backend when a new gateway is created, downloads the configuration artifact, and starts a Docker container with the right environment variables.

---

3-Layer Guardrails: Defense in Depth Against Prompt Injection

LLMs are vulnerable to prompt injection. In a system that generates SQL against real databases, a successful attack could mean exfiltration of sensitive data. I implemented a 3-layer guardrail system:

Layer 1: Input Guardrail (Pre-processing)

Before the user’s question ever reaches the LLM:

• Prompt injection detection: Regex patterns plus embedding-based classification. If the injection confidence exceeds the threshold (configurable, default 0.8), the query is blocked.
• Input sanitization: Stripping control characters, Unicode normalization, truncation to a maximum length.

Layer 2: Context Guardrail (SQL Scope)

After SQL generation, before execution:

• Operation whitelist: Only SELECT is allowed. Any DDL or DML is rejected.
• Table scope: The query can only access tables the user is authorized for in the semantic layer.
• Row limit: A forced LIMIT to prevent queries that return massive datasets.

Layer 3: Output Guardrail (Post-processing)

Before the response goes back to the user:

• PII sanitization: Detecting and redacting ID numbers, emails, phone numbers, and other personal data that might show up in the results.
• Credential sanitization: Detecting patterns that look like API keys, tokens, or connection strings in the response.

The rules live in a YAML file, so you can tune thresholds, add custom patterns, and enable or disable layers without touching code.

---

Multi-LLM Routing: Don’t Bet Everything on a Single Provider

One early decision was to support multiple LLM providers with automatic fallback. The implementation:

1. You configure an ordered list of providers (e.g., OpenAI → Anthropic → Google)
2. If the primary provider fails (timeout, rate limit, 5xx error), the router tries the next one
3. Each provider uses its native Haystack ChatGenerator — no intermediate abstractions that lose functionality

Why This Matters in Production

• Availability: OpenAI had multiple outages in 2025. With fallback to Anthropic, the system kept running.
• Cost: Different queries have different requirements. A simple question can use a cheaper model; a complex SQL query needs the most capable one.
• Compliance: Some enterprise customers require that their data not pass through certain providers. Routing lets you exclude providers on a per-project basis.

---

Lessons Learned

1. Schema Linking Is 80% of the RAG-SQL Problem

Generating correct SQL depends almost entirely on the LLM understanding the database schema. We spent more time on the semantic indexer (which generates human-readable schema documentation and indexes it in Qdrant) than on the SQL generation prompt.

2. Guardrails Are Not Optional

It’s tempting to ship without guardrails and bolt them on later. Don’t. The first time a user discovers they can run DROP TABLE with a clever prompt, it’s already too late. Build guardrails in from day one.

3. Go for Infrastructure, Python for ML, TypeScript for Product

Every language has its sweet spot. Trying to do Docker orchestration in Python or ML pipelines in Go would have been a mistake. Accept the complexity of a polyglot stack as long as each piece is in the right language.

4. The On-Premise Gateway Is a Product in Its Own Right

The gateway agent looked like a minor component. In practice, it took more iteration than the AI engine: reconnection handling, connectivity diagnostics, TLS/SSL edge cases, audit logging, and a provisioning flow that doesn’t require any manual intervention from the customer.

5. Observability Is Non-Negotiable

In a distributed system with 5 services, if you can’t correlate logs across the frontend, the backend, the AI engine, and the gateway agent, you’re flying blind. Set up distributed tracing from the start — not once things are already breaking in production.

---

Final Tech Stack

• AI Engine: Python 3.11, FastAPI, Haystack 2.x, Qdrant, sentence-transformers
• SaaS Backend: Node.js, Express, Prisma, PostgreSQL, TypeScript
• Frontend: Next.js, React, MUI, @xyflow/react
• Gateway Agent: Node.js, TypeScript, pg/mysql2/mssql drivers
• Gateway Provisioner: Go, Docker SDK
• Infrastructure: Docker Compose, Qdrant (vector DB)
• LLM Providers: OpenAI, Anthropic, Google Gemini, Cohere

---

Conclusion

Building an enterprise AI platform isn’t just wiring an LLM up to a prompt. It’s designing a distributed system that solves real problems around security, connectivity, and reliability. The decisions that had the most impact weren’t about which model to use — they were about how to protect customer data, how to stay available when a provider goes down, and how to let a non-technical team connect their database in 10 minutes without filing tickets with the networking team.

If you’re thinking about building something similar, my advice is: start with the gateway and the guardrails. The LLM is the easiest part to swap out; the trust infrastructure is what determines whether your product survives its first enterprise customer.

---

Have you built enterprise AI platforms yourself? I’d love to hear about the challenges you’ve run into. Share them in the comments or find me on LinkedIn.