# Redis Cache Architecture - Visual Reference ## Current Architecture (Before Changes) ``` ┌─────────────────────────────────────────────────────────────────┐ │ MongoDB │ ├─────────────────────────────────────────────────────────────────┤ │ Channel Category Tag │ │ ├─ id ├─ id ├─ id │ │ ├─ name ├─ name ├─ name │ │ ├─ landingUrl ├─ subtitle ├─ categoryId ✓ │ │ └─ ... ├─ channelId ✗ ├─ seq │ │ ├─ seq └─ status │ │ └─ status │ │ │ │ VideoMedia │ │ ├─ id │ │ ├─ title │ │ ├─ categoryId ✗ (single - PROBLEM) │ │ ├─ tagIds (array) │ │ ├─ tags (names only) │ │ └─ ...50+ provider fields │ └─────────────────────────────────────────────────────────────────┘ ↓ (builders) ↓ ┌─────────────────────────────────────────────────────────────────┐ │ Redis Cache │ ├─────────────────────────────────────────────────────────────────┤ │ CHANNELS: app:channel:all ~50 channels │ │ CATEGORIES: app:category:all ~2000 categories │ │ TAGS: app:tag:all ~10000 tags │ │ VIDEO DETAILS: app:video:detail:{id} ~58000 videos │ │ VIDEO LISTS: app:video:category:list:{id} ~2000 lists │ │ app:video:tag:list:{id}:{id} ~varies │ │ │ │ MEMORY: ~45 MB (efficient for current structure) │ └─────────────────────────────────────────────────────────────────┘ ↓ (Redis queries) ↓ Application Layer ``` ### Current Issues ❌ - Video tied to single category (limiting) - Category has channelId (redundant dependency) - Tag has channelId (unnecessary) - No efficient multi-category queries --- ## New Architecture (After Changes) ``` ┌─────────────────────────────────────────────────────────────────┐ │ MongoDB │ ├─────────────────────────────────────────────────────────────────┤ │ Channel Category Tag │ │ ├─ id ├─ id ├─ id │ │ ├─ name ├─ name ├─ name │ │ ├─ landingUrl ├─ subtitle ├─ categoryId ✓ │ │ └─ ... ├─ seq ├─ seq │ │ ├─ status └─ status │ │ └─ (no channelId) │ │ │ │ VideoMedia │ │ ├─ id │ │ ├─ title │ │ ├─ categoryIds: string[] ✅ (MULTIPLE - NEW) │ │ ├─ tagIds (array) │ │ ├─ tags (names only) │ │ └─ ...50+ provider fields │ └─────────────────────────────────────────────────────────────────┘ ↓ (updated builders) ↓ ┌─────────────────────────────────────────────────────────────────┐ │ Redis Cache │ ├─────────────────────────────────────────────────────────────────┤ │ CHANNELS: app:channel:all │ │ └─ ChannelCachePayload[] (now with denormalized │ │ categories, tags, tagNames) │ │ │ │ CATEGORIES: app:category:all │ │ └─ CategoryCachePayload[] (no channelId; │ │ tags as JSON {id,name}; tagNames array) │ │ │ │ TAGS: app:tag:all │ │ └─ TagCachePayload[] (no channelId - simplified) │ │ │ │ VIDEO DETAILS: app:video:detail:{id} │ │ └─ videoDetail with categoryIds[] (CHANGED) │ │ │ │ VIDEO LISTS: app:video:category:list:{id} (unchanged) │ │ app:video:tag:list:{id}:{id} (unchanged) │ │ │ │ ┌─ NEW INDEXES FOR MULTI-CATEGORY SUPPORT ─────────────────┐ │ │ │ app:video:category:index:{id}:videos (ZSET) │ │ │ │ → Efficient multi-category video queries │ │ │ │ │ │ │ │ app:video:categories:{id} (SET) │ │ │ │ → Quick lookup of all categories for a video │ │ │ │ │ │ │ │ app:category:video:count:{id} (STRING) │ │ │ │ → Video count per category for UI │ │ │ │ │ │ │ │ app:tag:search:{prefix} (SET) │ │ │ │ → Tag autocomplete/search index │ │ │ │ │ │ │ │ app:category:tagnames:flat:{id} (STRING) │ │ │ │ → Flat tag names for efficient search │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ │ MEMORY: ~80 MB (includes new indexes & denormalization) │ │ (+33 MB increase, acceptable) │ └─────────────────────────────────────────────────────────────────┘ ↓ (better queries) ↓ Application Layer ``` ### New Capabilities ✅ - Videos support multiple categories - Categories independent (no channelId) - Tags simplified (no redundant channelId) - Efficient multi-category queries (ZUNIONSTORE) - Better denormalization for quick lookups --- ## Key Changes Matrix ``` ┌──────────────────┬──────────────────┬──────────────────┬──────────────────┐ │ Entity │ Current │ New │ Impact │ ├──────────────────┼──────────────────┼──────────────────┼──────────────────┤ │ VideoMedia │ categoryId: │ categoryIds: │ Support multi- │ │ │ string (single) │ string[] │ category; cache │ │ │ │ (multiple) │ +1.2MB │ ├──────────────────┼──────────────────┼──────────────────┼──────────────────┤ │ Category │ channelId: │ (removed) │ Independent; │ │ │ string (FK to │ │ simpler; cache │ │ │ Channel) │ tags: JSON[] │ -20B but +100B │ │ │ tags: string[] │ (objects) │ for denormalized │ │ │ (names only) │ tagNames: str[] │ tags │ ├──────────────────┼──────────────────┼──────────────────┼──────────────────┤ │ Tag │ channelId: │ (removed) │ Simplified; only │ │ │ string (FK) │ │ tied to Category │ │ │ categoryId: │ categoryId: │ Cache -20B per │ │ │ string (FK) │ string (FK) │ tag │ ├──────────────────┼──────────────────┼──────────────────┼──────────────────┤ │ Channel │ Basic fields │ + categories: │ Denormalized for │ │ │ only │ JSON[] │ quick lookup; │ │ │ │ + tags: JSON[] │ cache +20-40B │ │ │ │ + tagNames: [] │ │ └──────────────────┴──────────────────┴──────────────────┴──────────────────┘ ``` --- ## Cache Key Relationships ### Old Relationships ``` Channel (1) ──── (M) Category ──── (M) Tag │ └─────────────────────────────────────┘ (implied for Tag) Video (N) ──── (1) Category (single!) ``` ### New Relationships ``` Channel (1) ──── (M) Category ──── (M) Tag (independent) Video (N) ──── (M) Category (multiple!) (categoryIds array) ``` --- ## Redis Memory Layout ### Before (45 MB) ``` ┌─────────────────────────────────────────────┐ │ app:channel:all 10 KB │ │ app:category:all 300 KB │ │ app:tag:all 800 KB │ │ app:video:detail:* 37.7 MB (58K videos)│ │ app:video:*:list:* 5-10 MB │ │ (other keys) ~1-2 MB │ ├─────────────────────────────────────────────┤ │ TOTAL: ~45 MB │ └─────────────────────────────────────────────┘ ``` ### After (80 MB) ``` ┌─────────────────────────────────────────────┐ │ EXISTING KEYS (modified content): │ │ app:channel:all 12.5 KB (+2.5) │ │ app:category:all 400 KB (+100) │ │ app:tag:all 600 KB (-200) │ │ app:video:detail:* 38.9 MB (+1.2) │ │ app:video:*:list:* 5-10 MB (same) │ │ │ │ NEW INDEX KEYS: │ │ app:video:category:index:* 12.2 MB ✨ │ │ app:video:categories:* 7.5 MB ✨ │ │ app:category:video:count:* 80 KB ✨ │ │ app:tag:search:* 0.5-1 MB ✨ │ │ app:category:tagnames:flat:* 400 KB ✨ │ │ │ │ (other keys) ~1-2 MB │ ├─────────────────────────────────────────────┤ │ TOTAL: ~80 MB │ │ INCREASE: +35 MB (77%) │ │ HEADROOM NEEDED: 100+ MB total │ └─────────────────────────────────────────────┘ ``` --- ## Query Pattern Evolution ### Multi-Category Video Query (Example) #### Before ❌ (Inefficient) ```typescript // Video can only belong to 1 category const categoryId = 'cat-123'; const videoIds = await redis.lrange( `app:video:category:list:${categoryId}`, 0, -1, ); // Want videos from multiple categories? Must do separate queries const videos1 = await redis.lrange(`app:video:category:list:cat1`, 0, -1); const videos2 = await redis.lrange(`app:video:category:list:cat2`, 0, -1); const allVideos = [...new Set([...videos1, ...videos2])]; // Manual union ``` #### After ✅ (Efficient) ```typescript // Use new ZSET index for efficient union const categoryIds = ['cat-1', 'cat-2', 'cat-3']; const keys = categoryIds.map((id) => `app:video:category:index:${id}:videos`); // ZUNIONSTORE combines all category indexes await redis.zunionstore('temp:combined', keys.length, ...keys); // Get paginated results with scores (timestamps) const videoIds = await redis.zrevrange('temp:combined', 0, 19); // Or use direct lookup const videoCats = await redis.smembers(`app:video:categories:video-123`); ``` --- ## Builder Execution Order ``` Start Cache Rebuild │ ├─→ ChannelCacheBuilder.buildAll() │ └─ Populates: app:channel:all │ ├─→ CategoryCacheBuilder.buildAll() │ ├─ Populates: app:category:all │ ├─ Populates: app:category:by-id:* │ └─ Populates: app:category:tagnames:flat:* │ ├─→ TagCacheBuilder.buildAll() │ ├─ Populates: app:tag:all │ └─ Populates: app:tag:list:* │ ├─→ VideoCategoryIndexBuilder.buildAll() ✨ NEW │ └─ Populates: app:video:category:index:*:videos │ ├─→ VideoCategoriesLookupBuilder.buildAll() ✨ NEW │ └─ Populates: app:video:categories:* │ ├─→ CategoryVideoCountBuilder.buildAll() ✨ NEW │ └─ Populates: app:category:video:count:* │ └─→ (Other builders...) └─ Populate remaining keys Total Time: ~60 seconds for 58K videos ``` --- ## Invalidation Flow ### When Video Updates CategoryIds ``` VideoMediaUpdated Event │ ├─→ Delete: app:video:detail:{videoId} │ ├─→ Delete: app:video:categories:{videoId} │ ├─→ For each OLD categoryId: │ ├─ Delete: app:video:category:index:{catId}:videos │ └─ Delete: app:category:video:count:{catId} │ └─→ For each NEW categoryId: ├─ Delete: app:video:category:index:{catId}:videos └─ Delete: app:category:video:count:{catId} (Keys get rebuilt on next cache rebuild or lazy load) ``` --- ## TTL Lifecycle ``` ┌─────────────────────────────────────────────────────────────┐ │ Cache Key Lifecycle (TTL Management) │ ├─────────────────────────────────────────────────────────────┤ │ │ │ app:tag:all [7 days ━━━━━━━━━━━━] │ │ app:category:all [7 days ━━━━━━━━━━━━] │ │ app:channel:all [7 days ━━━━━━━━━━━━] │ │ │ │ app:video:detail:* [24h ━━━━━━━━] │ │ app:video:categories:* [24h ━━━━━━━━] │ │ app:category:tagnames:flat:* [24h ━━━━━━━━] │ │ │ │ app:video:category:index:* [2h ━━━] │ │ app:category:video:count:* [1h ━] │ │ app:tag:search:* [2h ━━━] │ │ │ │ ↑ ↑ ↑ │ │ Long-lived Medium-lived Short-lived │ │ (7 days) (24 hours) (1-2 hours) │ │ Metadata Content Indexes │ └─────────────────────────────────────────────────────────────┘ ``` --- ## Implementation Phase Timeline ``` ┌─────────────────────────────────────────────────────────────┐ │ PHASE 1: Database Schema (2-3 days) │ ├─────────────────────────────────────────────────────────────┤ │ ✓ Update Prisma schema │ │ ✓ Create migration │ │ ✓ Run migration │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ PHASE 2: Core Builders (2-3 days) │ ├─────────────────────────────────────────────────────────────┤ │ ✓ Update TagCacheBuilder │ │ ✓ Update CategoryCacheBuilder │ │ ✓ Update ChannelCacheBuilder │ │ ✓ Update VideoDetail builder │ │ ✓ Integration test │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ PHASE 3: Index Builders (1-2 days) │ ├─────────────────────────────────────────────────────────────┤ │ ✓ Create VideoCategoryIndexBuilder │ │ ✓ Create VideoCategoriesLookupBuilder │ │ ✓ Create CategoryVideoCountBuilder │ │ ✓ Unit tests │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ PHASE 4: Integration (1 day) │ ├─────────────────────────────────────────────────────────────┤ │ ✓ Update CacheSyncService │ │ ✓ Add invalidation logic │ │ ✓ Integration tests │ └─────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────┐ │ PHASE 5: Testing & Deploy (2-3 days) │ ├─────────────────────────────────────────────────────────────┤ │ ✓ Load test (58K+ videos) │ │ ✓ Performance benchmarks │ │ ✓ Staging deployment │ │ ✓ Production deployment │ │ ✓ Monitoring & alerts │ └─────────────────────────────────────────────────────────────┘ Total: 8-12 days | Team: 1-2 engineers | Risk: Medium ``` --- **Visual Summary:** New architecture supports flexible multi-category organization while maintaining performance through strategic denormalization and indexing.