Building a Multi-Model AI Upscaling API: 6 Models, One Endpoint, Under 60 Seconds

Different images need different treatment. A product photo needs sharp edges and clean detail. A portrait needs skin texture preservation. An old scan needs noise removal before upscaling. AI-generated video from Sora or Kling needs temporal consistency.

So we built UpRes with 14 public aliases behind a single API endpoint. Here's how it works and what I learned along the way.

The Core Problem

Most upscaling APIs expose one model. You send an image, you get back a bigger image. If the result looks wrong — too smooth, too sharp, artifacts on text — you're stuck. There's no knob to turn.

The fix seems obvious: offer multiple models. But that creates a new problem. Now the user has to know which model to pick. Most people don't know the difference between ESRGAN and Real-ESRGAN. They just want their image to look better.

The Routing Layer

Our solution is a thin routing layer that sits between the user and the model providers (Fal.ai and Replicate). It does three things:

1. Accepts any image or video and normalizes it (format conversion, color space, dimensions).
2. Routes to the right model based on user-selected mode (Flare, Prism, Lumen, Mirage, Motion, Motion X) — each tuned for a specific use case.
3. Handles failures gracefully — if one provider is down or rate-limited, it falls back to the other.

// Public aliases only. Vendor endpoints stay on the server.
const ALIASES = {
  flare: { maxScale: 4 },
  prism: { maxScale: 4 },
  lumen: { maxScale: 8 },
  mirage: { maxScale: 8 },
  motion: { maxScale: 4 },
  'motion-x': { maxScale: 4 },
} as const;

Each mode maps to a specific model on a specific provider. The user picks a mode by name — "Flare for product shots, Mirage for old photos" — and the routing layer handles the rest.

The 60-Second Constraint

The hardest part wasn't the routing. It was the latency budget.

Users expect results in under a minute. But AI upscaling is compute-heavy. A 4K upscale of a 1080p image can take 15-40 seconds depending on the model. Video is worse — a 10-second clip can take 2-5 minutes.

Our approach:

• Images: Process synchronously, stream progress events via SSE. Most images finish in 20-40 seconds.
• Video: Process asynchronously with a job queue. Return a job ID immediately, poll or webhook for completion.
• Batch: Parallel processing with a concurrency limit per user to prevent queue starvation.

// Image: synchronous with SSE progress
app.post('/api/upscale', upload.single('image'), async (req, res) => {
  const { mode, scale } = req.body;
  const config = ALIASES[mode];

  res.setHeader('Content-Type', 'text/event-stream');

  try {
    const result = await processImage({
      image: req.file.buffer,
      config,
      scale: parseInt(scale),
      onProgress: (pct) => {
        res.write(`data: ${JSON.stringify({ progress: pct })}\n\n`);
      },
    });

    res.write(`data: ${JSON.stringify({ url: result.url })}\n\n`);
    res.end();
  } catch (err) {
    res.write(`data: ${JSON.stringify({ error: err.message })}\n\n`);
    res.end();
  }
});

The MCP Server

One of the most interesting parts of building this was exposing it as an MCP server — so AI agents like Claude or Cursor can upscale images directly in their workflow.

The MCP server exposes five tools:

• upscale_image — single image upscale
• upscale_video — video upscale with job polling
• batch_upscale — folder processing
• list_models — available models and their strengths
• check_status — job status for async operations

server.tool('upscale_image', {
  description: 'Upscale an image to 4K or 8K using AI',
  inputSchema: {
    type: 'object',
    properties: {
      imageUrl: { type: 'string' },
      mode: { type: 'string', enum: ['flare', 'prism', 'lumen', 'mirage'] },
      scale: { type: 'number', enum: [2, 4, 8] },
    },
    required: ['imageUrl', 'mode', 'scale'],
  },
}, async (params) => {
  const result = await upscaleImage(params);
  return { content: [{ type: 'text', text: result.url }] };
});

This means a developer can say to Claude: "Take this screenshot, upscale it to 4K for the presentation" — and Claude calls the MCP tool directly. No API key management, no HTTP calls to write.

What I'd Do Differently

Three things I learned the hard way:

1. Don't build the routing layer first. Start with one model. Get it working end-to-end. Then add the second. The routing abstraction is easier to design when you have two concrete implementations to compare.

2. SSE beats WebSocket for progress updates. WebSockets give you bidirectional communication, but for one-way progress updates, SSE is simpler, works through proxies better, and doesn't need a connection upgrade.

3. Store the result before returning the URL. We initially returned a presigned URL from the AI provider. When the provider's CDN expired the link, users got 404s. Now we download the result to our own S3 bucket and return a permanent URL.

The CLI

The same API powers our CLI tools, available on npm and pip:

# npm
npx upres-cli upscale photo.jpg --mode flare --scale 4

# pip
pip install upres-cli
upres-cli upscale photo.jpg --mode flare --scale 4

Both CLIs use the same API key, so you can use them interchangeably. The npm package also works as a library:

import { upscale } from 'upres-cli';

const result = await upscale({
  imageUrl: 'https://example.com/photo.jpg',
  mode: 'flare',
  scale: 4,
});

Wrapping Up

The multi-model approach works because no single AI model is best at everything. By exposing six tuned models behind one API, users get the right result without needing to become AI upscaling experts.

If you want to try it, there's a free tier with 3 upscales per month — no credit card needed. The API docs and MCP server are both live.