Redis Cache Migration Analysis

Based on Database Schema Changes for MongoDB Prisma Models

Overview

With 58,000 videos in VideoMedia and the new multi-category support (categoryIds array), this document outlines all Redis cache key changes, memory implications, and new keys for optimal performance.

Database Changes Summary

Entity	Changes	Impact
Channel	Add categories & tags as JSON; add tagNames array	New denormalized data in cache
Category	Remove channelId; change tags to JSON; add tagNames	No direct dependency on Channel; tag relationships stored as JSON
Tag	Remove channelId; no Channel relationship	Simpler structure; only tied to Category
VideoMedia	`categoryId` → `categoryIds` (string[])	Videos can belong to multiple categories

PART 1: REDIS KEYS REQUIRING CHANGES

1.1 Channel Cache Keys

CHANGE: Remove implicit category/tag relationships

Current Key Structure:

app:channel:all → ChannelCachePayload[]

Current ChannelCachePayload:

{
  id: string;
  name: string;
  landingUrl: string;
  videoCdn?: string;
  coverCdn?: string;
  clientName?: string;
  clientNotice?: string;
  remark?: string;
}

Updated ChannelCachePayload:

{
  id: string;
  name: string;
  landingUrl: string;
  videoCdn?: string;
  coverCdn?: string;
  clientName?: string;
  clientNotice?: string;
  remark?: string;
  categories?: Array<{ id: string; name: string }>;  // NEW
  tags?: Array<{ id: string; name: string }>;        // NEW
  tagNames?: string[];                               // NEW: tag names only
}

Memory Impact: +20-40 bytes per channel (minimal, channels < 100)

1.2 Category Cache Keys

CHANGE: Remove channelId; change tags to JSON; add tagNames

Current Key Structure:

app:category:all → CategoryCachePayload[]
app:category:by-id:{categoryId} → CategoryCachePayload (single)
app:category:with-tags:{categoryId} → CategoryCachePayload with tags
app:tag:list:{categoryId} → Tag[] (JSON array in Redis list)

Current CategoryCachePayload:

{
  id: string;
  name: string;
  subtitle?: string | null;
  channelId: string;           // REMOVE THIS
  seq: number;
  tags: string[];              // Array of tag IDs or names
}

Updated CategoryCachePayload:

{
  id: string;
  name: string;
  subtitle?: string | null;
  seq: number;
  tags?: Array<{ id: string; name: string }>;  // Changed from string[] to JSON objects
  tagNames?: string[];                         // NEW: array of tag names only
  status?: number;                             // OPTIONAL: add status for filtering
}

Affected Keys:

app:category:all
app:category:by-id:{categoryId}
app:category:with-tags:{categoryId}

Memory Impact per Category:

Old: ~100 bytes (id + name + subtitle + channelId + seq + tag IDs)
New: ~150-200 bytes (includes tag objects + tagNames)
Increase: ~50-100 bytes per category (assuming ~1000-5000 categories)
Total increase: ~50-500 KB

1.3 Tag Cache Keys

CHANGE: Remove channelId; update TagCachePayload

Current Key Structure:

app:tag:all → TagCachePayload[]
app:tag:list:{categoryId} → Tag[] (Redis list of JSON objects)

Current TagCachePayload:

{
  id: string;
  name: string;
  channelId: string; // REMOVE THIS
  categoryId: string;
  seq: number;
}

Updated TagCachePayload:

{
  id: string;
  name: string;
  categoryId: string;
  seq: number;
  status?: number;       // OPTIONAL: add status
}

Affected Keys:

app:tag:all
app:tag:list:{categoryId} (no change to key name, but payload updated)

Memory Impact per Tag:

Old: ~80 bytes (id + name + channelId + categoryId + seq)
New: ~60 bytes (removed channelId)
Savings: ~20 bytes per tag (assuming ~10,000-50,000 tags)
Total savings: ~200-1000 KB

1.4 VideoMedia Cache Keys - CRITICAL CHANGE

MAJOR CHANGE: Single categoryId → Multiple categoryIds

Current VideoMedia fields in cache:

{
  id: string;
  title: string;
  // ... 50+ fields from provider ...
  categoryId: string;    // Single category reference (CHANGE THIS)
  tagIds: string[];      // Tag array
  tags: string[];        // Tag names array
  tagsFlat: string;      // Concatenated tag names for search
  listStatus: number;
}

Updated VideoMedia fields in cache:

{
  id: string;
  title: string;
  // ... 50+ fields from provider ...
  categoryIds: string[]; // Multiple category references (NEW)
  tagIds: string[];      // Tag array (unchanged)
  tags: string[];        // Tag names array (unchanged)
  tagsFlat: string;      // Concatenated tag names for search (unchanged)
  listStatus: number;
}

Affected Cache Keys:

app:video:detail:{videoId} - Each video detail JSON

Current Size per Video:

~500-800 bytes (approx)

New Size per Video:

~520-820 bytes (add ~20 bytes for extra category ID)

Total Memory Impact (58,000 videos):

Old: ~58,000 × 650 bytes = ~37.7 MB
New: ~58,000 × 670 bytes = ~38.9 MB
Increase: ~1.2 MB

1.5 Video List Cache Keys (No Changes Needed)

Keys remain the same:

app:video:category:list:{categoryId}      → List of video IDs (LRANGE, LIST type)
app:video:tag:list:{categoryId}:{tagId}   → List of video IDs (LRANGE, LIST type)
app:video:list:home:{channelId}:{section} → List of video IDs (ZSET or LIST)

However: New application logic needed to handle:

A video appearing in multiple category lists
Efficient querying of videos across multiple categoryIds

PART 2: NEW REDIS KEYS TO ADD

2.1 Video-by-Category Index (for multi-category support)

Purpose: Enable efficient lookup of all videos in a set of categories

Key Pattern:

app:video:category:index:{categoryId}:videos

Type: ZSET (sorted set with score = addedTime for sorting)

Value: Video IDs with score as timestamp

Use Case:

// Get all videos in multiple categories with pagination
const categoryIds = ['cat1', 'cat2', 'cat3'];
const keys = categoryIds.map((id) => `app:video:category:index:${id}:videos`);

// Use ZUNIONSTORE to combine multiple category indexes
await redis.zunionstore('temp:combined:videos', categoryIds.length, ...keys);

// Paginate: ZREVRANGE 'temp:combined:videos' 0 19

Memory Per Key: ~100 bytes + 30 bytes per video

Total Memory (assuming avg 200 videos per category, ~2000 categories):

Keys: 2000 × 100 = 200 KB
Values: 2000 × 200 × 30 = 12 MB
Total: ~12.2 MB

2.2 Category-to-Video Count Cache (for UI statistics)

Purpose: Quick lookup of video count per category without scanning

Key Pattern:

app:category:video:count:{categoryId}

Type: STRING (integer count)

Value: Integer count

Use Case:

// Get category with video count for UI display
const count = await redis.get(`app:category:video:count:${categoryId}`);

Memory Per Key: ~40 bytes (one per category)

Total Memory (~2000 categories):

~80 KB

2.3 Tag Search Index (for efficient tag filtering with names)

Purpose: Full-text tag search by partial name matching

Key Pattern:

app:tag:search:{prefix}

Type: SET (set of matching tag IDs)

Value: Tag IDs matching the prefix

Use Case:

// Find all tags starting with "act"
const tagIds = await redis.smembers('app:tag:search:act');

Note: Update on tag name changes or new tags created

Memory Per Prefix (assuming 2-3 char prefixes, ~500-1000 prefixes):

~500 KB - 1 MB

2.4 Video Multi-Category Lookup Index

Purpose: Fast lookup of categories for a specific video

Key Pattern:

app:video:categories:{videoId}

Type: SET (set of category IDs)

Value: Category IDs

Use Case:

// Get all categories a video belongs to
const categoryIds = await redis.smembers(`app:video:categories:${videoId}`);

Memory Per Key: ~40 bytes + 30 bytes per category

Assuming avg 2-3 categories per video:

Per key: ~40 + (3 × 30) = ~130 bytes
Total (58,000 videos): ~7.5 MB

2.5 Category Tagnames Flat Index (for quick name searches)

Purpose: Pre-computed concatenation of all tag names in a category for search

Key Pattern:

app:category:tagnames:flat:{categoryId}

Type: STRING

Value: Space-separated tag names, lowercase

Use Case:

// Check if category has tag by name (without JSON parsing)
const tagnames = await redis.get(`app:category:tagnames:flat:${categoryId}`);
const hasTag = tagnames?.includes('action'); // O(1) string search

Memory Per Key: ~100-300 bytes per category

Total (~2000 categories): ~200-600 KB

2.6 Channel Denormalization Cache (future - optional)

Key Pattern:

app:channel:categories:{channelId}
app:channel:tags:{channelId}

Type: SET or JSON

Note: If Channel.categories and Channel.tags JSON fields are meant to be denormalized, these keys cache pre-computed sets

Memory: ~1-2 MB (if implemented)

PART 3: CACHE INVALIDATION STRATEGY

3.1 On Category Update

When a category is updated:

// Invalidate
app:category:all
app:category:by-id:{categoryId}
app:category:with-tags:{categoryId}
app:category:video:count:{categoryId}
app:category:tagnames:flat:{categoryId}

3.2 On Tag Update

When a tag is updated:

// Invalidate
app: tag: all;
app: tag: list: {
  categoryId;
}
app: tag: search: {
  prefix;
} // All prefixes containing the tag
app: category: tagnames: flat: {
  categoryId;
}

3.3 On Video Update (CategoryIds Change)

When a video's categoryIds array changes:

// Invalidate
app:video:detail:{videoId}
app:video:category:index:{categoryId}:videos  // For each old & new category
app:video:categories:{videoId}
app:category:video:count:{categoryId}  // For each affected category

3.4 On Channel Update

When a channel is updated:

// Invalidate
app:channel:all
app:channel:by-id:{channelId}
app:channel:with-categories:{channelId}
app:channel:categories:{channelId}
app:channel:tags:{channelId}

PART 4: MEMORY ESTIMATION

Current Memory Usage (Before Changes)

Component	Count	Size/Unit	Total
Channels	~50	200 B	~10 KB
Categories	~2000	150 B	~300 KB
Tags	~10000	80 B	~800 KB
Video Details	58000	650 B	~37.7 MB
Video Lists	~2000	varies	~5-10 MB
Subtotal			~44-49 MB

New Memory Usage (After Changes)

Component	Count	Size/Unit	Total	Change
Channels	~50	250 B	~12.5 KB	+2.5 KB
Categories	~2000	200 B	~400 KB	+100 KB
Tags	~10000	60 B	~600 KB	-200 KB
Video Details	58000	670 B	~38.9 MB	+1.2 MB
Video Lists	~2000	varies	~5-10 MB	0
Category-Video Index	2000	varies	~12.2 MB	+12.2 MB
Video Categories Index	58000	130 B	~7.5 MB	+7.5 MB
Category Count Cache	2000	40 B	~80 KB	+80 KB
Tag Search Index	~500-1000	varies	~0.5-1 MB	+0.5-1 MB
Category Tagnames Flat	2000	200 B	~400 KB	+400 KB
Subtotal			~77-82 MB	+28-33 MB

PART 5: RECOMMENDED NEW REDIS KEYS CHECKLIST

For Multi-Category Support ✅

app:video:category:index:{categoryId}:videos - ZSET of video IDs by category
app:video:categories:{videoId} - SET of category IDs per video
app:category:video:count:{categoryId} - STRING count of videos

For Search Optimization ✅

app:tag:search:{prefix} - SET of matching tag IDs
app:category:tagnames:flat:{categoryId} - STRING of space-separated tag names

For Quick Metadata Lookups ✅

app:channel:with-categories:{channelId} - JSON with denormalized categories
app:channel:with-tags:{channelId} - JSON with denormalized tags

PART 6: TTL (Time-To-Live) RECOMMENDATIONS

Short-lived (refresh on every update)

app:category:video:count:{categoryId} - TTL: 3600 (1 hour)
app:video:category:index:{categoryId}:videos - TTL: 7200 (2 hours)
app:tag:search:{prefix} - TTL: 7200 (2 hours)

Medium-lived (cache for 24 hours)

app:video:detail:{videoId} - TTL: 86400 (24 hours)
app:video:categories:{videoId} - TTL: 86400 (24 hours)
app:category:tagnames:flat:{categoryId} - TTL: 86400 (24 hours)

Long-lived (cache for 7 days)

app:channel:all - TTL: 604800 (7 days)
app:category:all - TTL: 604800 (7 days)
app:tag:all - TTL: 604800 (7 days)

PART 7: IMPLEMENTATION PRIORITY

Phase 1 (Critical - Do First)

Update VideoMedia cache to use categoryIds[] instead of categoryId
Update video detail key serialization
Add database migration for categoryIds field

Phase 2 (High - Build Index Support)

Add app:video:category:index:{categoryId}:videos index builder
Add app:video:categories:{videoId} lookup cache
Add app:category:video:count:{categoryId} counter

Phase 3 (Medium - Optimization)

Add app:category:tagnames:flat:{categoryId} for search
Add app:tag:search:{prefix} for tag autocomplete
Update Channel/Category cache builders

Phase 4 (Optional - Polish)

Add denormalization caches for quick channel lookups
Implement cache warming strategies
Add monitoring/alerting for cache hit rates

PART 8: SAMPLE CACHE BUILDER UPDATES

Updated TagCacheBuilder

export interface TagCachePayload {
  id: string;
  name: string;
  categoryId: string;  // Changed: removed channelId
  seq: number;
  status?: number;     // New: optional
}

async buildAll(): Promise<void> {
  const tags = await this.mongoPrisma.tag.findMany({
    where: { status: 1 },
    orderBy: [{ seq: 'asc' }, { name: 'asc' }],
  });

  const payload: TagCachePayload[] = tags.map((tag) => ({
    id: tag.id,
    name: tag.name,
    categoryId: tag.categoryId,  // Only categoryId now
    seq: tag.seq,
    status: tag.status,
  }));

  await this.redis.setJson(CacheKeys.appTagAll, payload);
  this.logger.log(`Built ${payload.length} tags`);
}

Updated CategoryCacheBuilder

export interface CategoryCachePayload {
  id: string;
  name: string;
  subtitle?: string | null;
  seq: number;
  tags?: Array<{ id: string; name: string }>;  // Changed: JSON objects
  tagNames?: string[];                         // New: tag names only
  status?: number;                             // New: optional status
}

async buildAll(): Promise<void> {
  const categories = await this.mongoPrisma.category.findMany({
    where: { status: 1 },
    orderBy: [{ seq: 'asc' }, { name: 'asc' }],
  });

  const categoryIds = categories.map((c) => c.id);
  const tags = categoryIds.length
    ? await this.mongoPrisma.tag.findMany({
        where: { status: 1, categoryId: { in: categoryIds } },
        orderBy: [{ seq: 'asc' }, { name: 'asc' }],
      })
    : [];

  // Build tag lookup
  const tagsByCategory = new Map<string, Array<{ id: string; name: string }>>();
  const tagNamesByCategory = new Map<string, string[]>();

  for (const tag of tags) {
    // JSON objects
    const list = tagsByCategory.get(tag.categoryId) || [];
    list.push({ id: tag.id, name: tag.name });
    tagsByCategory.set(tag.categoryId, list);

    // Tag names only
    const nameList = tagNamesByCategory.get(tag.categoryId) || [];
    nameList.push(tag.name);
    tagNamesByCategory.set(tag.categoryId, nameList);
  }

  const payloads: CategoryCachePayload[] = categories.map((category) => ({
    id: category.id,
    name: category.name,
    subtitle: category.subtitle ?? null,
    seq: category.seq,
    tags: tagsByCategory.get(category.id),        // JSON objects
    tagNames: tagNamesByCategory.get(category.id), // Names only
    status: category.status,
  }));

  await this.redis.setJson(CacheKeys.appCategoryAll, payloads);
  this.logger.log(`Built ${payloads.length} categories with tags`);
}

New: VideoCategoryIndexBuilder

export class VideoCategoryIndexBuilder {
  async buildAll(): Promise<void> {
    const videos = await this.mongoPrisma.videoMedia.findMany({
      select: { id: true, categoryIds: true, addedTime: true },
      where: { listStatus: 1 },
    });

    // Group videos by each category
    const categoryIndex = new Map<
      string,
      Array<{ id: string; score: number }>
    >();

    for (const video of videos) {
      const score = video.addedTime?.getTime() || 0;
      for (const categoryId of video.categoryIds) {
        const list = categoryIndex.get(categoryId) || [];
        list.push({ id: video.id, score });
        categoryIndex.set(categoryId, list);
      }
    }

    // Store in Redis as ZSET
    for (const [categoryId, videoList] of categoryIndex.entries()) {
      const key = `app:video:category:index:${categoryId}:videos`;
      const members = videoList.map((v) => ({ member: v.id, score: v.score }));
      await this.redis.zadd(key, ...members.flat());
      await this.redis.expire(key, 7200); // 2 hours TTL
    }

    this.logger.log(
      `Built category video index for ${categoryIndex.size} categories`,
    );
  }
}

Summary Table: All Changes

Layer	Old	New	Impact
Database	Category has channelId	Category independent	DB schema cleaner
Database	VideoMedia.categoryId	VideoMedia.categoryIds	Support multi-category
Database	Tag has channelId	Tag no Channel link	Simplified relationships
Cache: Channel	Basic fields only	+ categories, tags as JSON + tagNames	+20-40B per channel
Cache: Category	+ channelId	- channelId; tags as JSON; + tagNames	-20B but +50-100B for JSON
Cache: Tag	+ channelId	- channelId	-20B per tag
Cache: Video Detail	categoryId	categoryIds	+20B per video; +1.2MB total
Cache: Indexes	N/A	+ category-video index (ZSET)	+12.2 MB
Cache: Indexes	N/A	+ video-categories lookup	+7.5 MB
Cache: Indexes	N/A	+ category count cache	+80 KB
Total Memory	~44-49 MB	~77-82 MB	+33 MB (66% increase)

Estimated Timeline & Effort

Phase	Tasks	Effort	Time
Phase 1	Update Prisma, migration, video detail cache	High	2-3 days
Phase 2	Build index builders, implement lookup caches	Medium	2-3 days
Phase 3	Add search indexes, flatname caches	Medium	1-2 days
Phase 4	Denormalization, monitoring, testing	Low	1 day
Total			6-9 days

Notes & Recommendations

Redis Memory: Ensure Redis instance can handle 80+ MB (consider Redis Cluster if needed)
Index Freshness: Implement background jobs to refresh category-video indexes every 2 hours
Monitoring: Add cache hit rate monitoring and alert on low hit rates
Testing: Load test with 58K+ videos to validate performance
Backward Compatibility: Ensure old cache keys are invalidated during migration
Documentation: Update CACHE_SEMANTICS.md with new key patterns and semantics

REDIS_CACHE_MIGRATION.md 21 KB История Исходник

Redis Cache Migration Analysis

Overview

Database Changes Summary

PART 1: REDIS KEYS REQUIRING CHANGES

1.1 Channel Cache Keys

CHANGE: Remove implicit category/tag relationships

1.2 Category Cache Keys

CHANGE: Remove channelId; change tags to JSON; add tagNames

1.3 Tag Cache Keys

CHANGE: Remove channelId; update TagCachePayload

1.4 VideoMedia Cache Keys - CRITICAL CHANGE

MAJOR CHANGE: Single categoryId → Multiple categoryIds

1.5 Video List Cache Keys (No Changes Needed)

PART 2: NEW REDIS KEYS TO ADD

2.1 Video-by-Category Index (for multi-category support)

2.2 Category-to-Video Count Cache (for UI statistics)

2.3 Tag Search Index (for efficient tag filtering with names)

2.4 Video Multi-Category Lookup Index

2.5 Category Tagnames Flat Index (for quick name searches)

2.6 Channel Denormalization Cache (future - optional)

PART 3: CACHE INVALIDATION STRATEGY

3.1 On Category Update

3.2 On Tag Update

3.3 On Video Update (CategoryIds Change)

3.4 On Channel Update

PART 4: MEMORY ESTIMATION

Current Memory Usage (Before Changes)

New Memory Usage (After Changes)

PART 5: RECOMMENDED NEW REDIS KEYS CHECKLIST

For Multi-Category Support ✅

For Search Optimization ✅

For Quick Metadata Lookups ✅

PART 6: TTL (Time-To-Live) RECOMMENDATIONS

Short-lived (refresh on every update)

Medium-lived (cache for 24 hours)

Long-lived (cache for 7 days)

PART 7: IMPLEMENTATION PRIORITY

Phase 1 (Critical - Do First)

Phase 2 (High - Build Index Support)

Phase 3 (Medium - Optimization)

Phase 4 (Optional - Polish)

PART 8: SAMPLE CACHE BUILDER UPDATES

Updated TagCacheBuilder

Updated CategoryCacheBuilder

New: VideoCategoryIndexBuilder

Summary Table: All Changes

Estimated Timeline & Effort

Notes & Recommendations

REDIS_CACHE_MIGRATION.md 21 KB

История Исходник