Integrating AI into Your Applications with OpenRouter

Here is a pattern we see constantly: a team spends two weeks building an AI chatbot, demos it to stakeholders, everyone is impressed, and then it goes to production and immediately starts hallucinating, running up costs, and responding in 8 seconds. The demo was great. The engineering was not.

The gap between "AI that works in a demo" and "AI that works in production" is enormous. We have shipped AI features into applications serving hundreds of thousands of users, and the model selection is maybe 10% of the work. The other 90% is everything around it: prompt engineering, error handling, cost management, latency optimization, and output validation.

OpenRouter has become our default gateway for LLM access, and here is why.

Why We Stopped Managing Multiple AI Provider Accounts

Eighteen months ago, we managed separate API keys for OpenAI, Anthropic, and Mistral across every client project. Each provider had different rate limits, different error formats, different pricing models, and different content policies. When OpenAI had an outage (which happened more often than you would expect), our clients' AI features went completely dark.

OpenRouter solved this by giving us one API endpoint, one authentication mechanism, and automatic fallback routing:

TYPESCRIPT
// lib/ai.ts - our production AI client
const AI_CONFIG = {
  baseUrl: 'https://openrouter.ai/api/v1',
  defaultModel: 'anthropic/claude-3-haiku',
  fallbackModel: 'mistralai/mistral-7b-instruct',
  maxRetries: 3,
  timeout: 30000,
} as const

export async function generateCompletion(
  messages: ChatMessage[],
  options: CompletionOptions = {}
): Promise<CompletionResult> {
  const model = options.model ?? AI_CONFIG.defaultModel

  for (let attempt = 0; attempt < AI_CONFIG.maxRetries; attempt++) {
    try {
      const response = await fetch(
        `${AI_CONFIG.baseUrl}/chat/completions`,
        {
          method: 'POST',
          headers: {
            Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
            'Content-Type': 'application/json',
            'X-Title': 'Fametoll Client App',
          },
          body: JSON.stringify({
            model: attempt === 0 ? model : AI_CONFIG.fallbackModel,
            messages,
            max_tokens: options.maxTokens ?? 1024,
            temperature: options.temperature ?? 0.7,
            stream: options.stream ?? false,
          }),
          signal: AbortSignal.timeout(AI_CONFIG.timeout),
        }
      )

      if (!response.ok) {
        const error = await response.json()
        // Rate limited - wait and retry
        if (response.status === 429) {
          await sleep(Math.pow(2, attempt) * 1000)
          continue
        }
        throw new AIError(error.error?.message ?? 'Unknown error', response.status)
      }

      return await response.json()
    } catch (error) {
      if (attempt === AI_CONFIG.maxRetries - 1) throw error
      // On failure, next attempt uses fallback model
    }
  }

  throw new AIError('All retry attempts exhausted', 503)
}

Notice the retry logic. On the first attempt, we use the requested model. On subsequent retries, we fall back to a cheaper, faster model. The user gets a slightly less capable response instead of an error page. In production, this fallback triggers roughly 2-3% of the time, and users rarely notice the difference.

The Prompt Engineering Practices Nobody Talks About

Most tutorials show you how to write a system prompt. Few discuss prompt versioning, A/B testing, or regression testing for prompts. Here is how we manage prompts in production:

TYPESCRIPT
// prompts/product-description.ts
export const PRODUCT_DESCRIPTION_PROMPT = {
  version: '2.3',
  systemMessage: `You are a product copywriter for an e-commerce platform.
Write compelling product descriptions that are:
- Exactly 2-3 sentences long
- Focused on benefits, not features
- Written in active voice
- Free of superlatives (no "best", "amazing", "incredible")

IMPORTANT: Return ONLY the description text. No quotes, no labels, no preamble.`,

  // Few-shot examples improve consistency dramatically
  examples: [
    {
      input: 'Wireless noise-canceling headphones, 30hr battery, Bluetooth 5.3',
      output: 'Block out the world and focus on what matters with 30 hours of uninterrupted listening. These headphones connect instantly to any device and deliver studio-quality sound whether you are on a call or deep in a playlist.'
    }
  ],

  // Validation function for output quality
  validate: (output: string): boolean => {
    const sentences = output.split(/[.!?]+/).filter(Boolean)
    if (sentences.length < 2 || sentences.length > 3) return false
    if (/best|amazing|incredible|revolutionary/i.test(output)) return false
    return true
  }
}

That validate function is critical. We run it on every LLM response before returning it to the user. If validation fails, we retry with a slightly modified prompt that includes the failure reason. This catches about 15% of responses that would otherwise be off-format.

Streaming: The UX Difference Between "Slow" and "Fast"

A non-streaming AI response that takes 4 seconds feels broken. A streaming response that takes 6 seconds feels fast. The psychology is simple: users are patient when they can see progress.

TYPESCRIPT
// app/api/chat/route.ts
export async function POST(request: Request) {
  const { messages } = await request.json()

  const response = await fetch(
    'https://openrouter.ai/api/v1/chat/completions',
    {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: 'anthropic/claude-3-haiku',
        messages,
        stream: true,
      }),
    }
  )

  // Forward the stream directly to the client
  return new Response(response.body, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
    },
  })
}

On the client side, we use a custom hook that handles the Server-Sent Events stream, manages loading states, and buffers tokens for smooth rendering. The difference in user satisfaction is measurable: our streaming implementations consistently achieve 40% higher task completion rates than request-response patterns.

Cost Management Is Engineering, Not Budgeting

AI costs can spiral fast. A single chatbot feature processing 10,000 messages per day with GPT-4 can cost $3,000-5,000 per month. The same feature with intelligent model routing costs $200-400.

Our approach:

1. Classify by complexity. Simple queries (FAQ, basic classification) go to Mistral 7B. Complex queries (code generation, analysis) go to Claude or GPT-4. We use a lightweight classifier to route automatically.
2. Cache aggressively. If someone asks "What are your business hours?" three times, we do not call the LLM three times. We hash the input and cache responses with a 24-hour TTL.
3. Set hard limits per tenant. Every SaaS customer gets a token budget. When they hit 80%, we notify them. At 100%, we downgrade to a cheaper model rather than cutting off service.

The Uncomfortable Truth About AI in Production

AI features are not fire-and-forget. They require monitoring, prompt tuning, and constant evaluation. We track three metrics for every AI feature we ship:

• Response quality score (human-rated sample of outputs, weekly)
• Latency p95 (must stay under 3 seconds for interactive features)
• Cost per interaction (must stay within the per-user budget)

If any of these drift, we investigate. Usually it is a prompt that needs refinement as the underlying model gets updated. Sometimes it is a new edge case the system was not designed for. Rarely is it a fundamental architecture problem, because we design for adaptability from the start.

The teams that treat AI integration as a one-time implementation task are the ones whose features degrade over time. The teams that treat it as an ongoing engineering discipline are the ones whose AI features get better every month.