REDIS_CACHE_VISUAL_GUIDE.md 24 KB
Redis Cache Architecture - Visual Reference
Current Architecture (Before Changes)
┌─────────────────────────────────────────────────────────────────┐
│ MongoDB │
├─────────────────────────────────────────────────────────────────┤
│ Channel Category Tag │
│ ├─ id ├─ id ├─ id │
│ ├─ name ├─ name ├─ name │
│ ├─ landingUrl ├─ subtitle ├─ categoryId ✓ │
│ └─ ... ├─ channelId ✗ ├─ seq │
│ ├─ seq └─ status │
│ └─ status │
│ │
│ VideoMedia │
│ ├─ id │
│ ├─ title │
│ ├─ categoryId ✗ (single - PROBLEM) │
│ ├─ tagIds (array) │
│ ├─ tags (names only) │
│ └─ ...50+ provider fields │
└─────────────────────────────────────────────────────────────────┘
↓
(builders)
↓
┌─────────────────────────────────────────────────────────────────┐
│ Redis Cache │
├─────────────────────────────────────────────────────────────────┤
│ CHANNELS: app:channel:all ~50 channels │
│ CATEGORIES: app:category:all ~2000 categories │
│ TAGS: app:tag:all ~10000 tags │
│ VIDEO DETAILS: app:video:detail:{id} ~58000 videos │
│ VIDEO LISTS: app:video:category:list:{id} ~2000 lists │
│ app:video:tag:list:{id}:{id} ~varies │
│ │
│ MEMORY: ~45 MB (efficient for current structure) │
└─────────────────────────────────────────────────────────────────┘
↓
(Redis queries)
↓
Application Layer
Current Issues ❌
- Video tied to single category (limiting)
- Category has channelId (redundant dependency)
- Tag has channelId (unnecessary)
- No efficient multi-category queries
New Architecture (After Changes)
┌─────────────────────────────────────────────────────────────────┐
│ MongoDB │
├─────────────────────────────────────────────────────────────────┤
│ Channel Category Tag │
│ ├─ id ├─ id ├─ id │
│ ├─ name ├─ name ├─ name │
│ ├─ landingUrl ├─ subtitle ├─ categoryId ✓ │
│ └─ ... ├─ seq ├─ seq │
│ ├─ status └─ status │
│ └─ (no channelId) │
│ │
│ VideoMedia │
│ ├─ id │
│ ├─ title │
│ ├─ categoryIds: string[] ✅ (MULTIPLE - NEW) │
│ ├─ tagIds (array) │
│ ├─ tags (names only) │
│ └─ ...50+ provider fields │
└─────────────────────────────────────────────────────────────────┘
↓
(updated builders)
↓
┌─────────────────────────────────────────────────────────────────┐
│ Redis Cache │
├─────────────────────────────────────────────────────────────────┤
│ CHANNELS: app:channel:all │
│ └─ ChannelCachePayload[] (now with denormalized │
│ categories, tags, tagNames) │
│ │
│ CATEGORIES: app:category:all │
│ └─ CategoryCachePayload[] (no channelId; │
│ tags as JSON {id,name}; tagNames array) │
│ │
│ TAGS: app:tag:all │
│ └─ TagCachePayload[] (no channelId - simplified) │
│ │
│ VIDEO DETAILS: app:video:detail:{id} │
│ └─ videoDetail with categoryIds[] (CHANGED) │
│ │
│ VIDEO LISTS: app:video:category:list:{id} (unchanged) │
│ app:video:tag:list:{id}:{id} (unchanged) │
│ │
│ ┌─ NEW INDEXES FOR MULTI-CATEGORY SUPPORT ─────────────────┐ │
│ │ app:video:category:index:{id}:videos (ZSET) │ │
│ │ → Efficient multi-category video queries │ │
│ │ │ │
│ │ app:video:categories:{id} (SET) │ │
│ │ → Quick lookup of all categories for a video │ │
│ │ │ │
│ │ app:category:video:count:{id} (STRING) │ │
│ │ → Video count per category for UI │ │
│ │ │ │
│ │ app:tag:search:{prefix} (SET) │ │
│ │ → Tag autocomplete/search index │ │
│ │ │ │
│ │ app:category:tagnames:flat:{id} (STRING) │ │
│ │ → Flat tag names for efficient search │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ MEMORY: ~80 MB (includes new indexes & denormalization) │
│ (+33 MB increase, acceptable) │
└─────────────────────────────────────────────────────────────────┘
↓
(better queries)
↓
Application Layer
New Capabilities ✅
- Videos support multiple categories
- Categories independent (no channelId)
- Tags simplified (no redundant channelId)
- Efficient multi-category queries (ZUNIONSTORE)
- Better denormalization for quick lookups
Key Changes Matrix
┌──────────────────┬──────────────────┬──────────────────┬──────────────────┐
│ Entity │ Current │ New │ Impact │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ VideoMedia │ categoryId: │ categoryIds: │ Support multi- │
│ │ string (single) │ string[] │ category; cache │
│ │ │ (multiple) │ +1.2MB │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ Category │ channelId: │ (removed) │ Independent; │
│ │ string (FK to │ │ simpler; cache │
│ │ Channel) │ tags: JSON[] │ -20B but +100B │
│ │ tags: string[] │ (objects) │ for denormalized │
│ │ (names only) │ tagNames: str[] │ tags │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ Tag │ channelId: │ (removed) │ Simplified; only │
│ │ string (FK) │ │ tied to Category │
│ │ categoryId: │ categoryId: │ Cache -20B per │
│ │ string (FK) │ string (FK) │ tag │
├──────────────────┼──────────────────┼──────────────────┼──────────────────┤
│ Channel │ Basic fields │ + categories: │ Denormalized for │
│ │ only │ JSON[] │ quick lookup; │
│ │ │ + tags: JSON[] │ cache +20-40B │
│ │ │ + tagNames: [] │ │
└──────────────────┴──────────────────┴──────────────────┴──────────────────┘
Cache Key Relationships
Old Relationships
Channel (1) ──── (M) Category ──── (M) Tag
│
└─────────────────────────────────────┘
(implied for Tag)
Video (N) ──── (1) Category (single!)
New Relationships
Channel (1) ──── (M) Category ──── (M) Tag
(independent)
Video (N) ──── (M) Category (multiple!)
(categoryIds array)
Redis Memory Layout
Before (45 MB)
┌─────────────────────────────────────────────┐
│ app:channel:all 10 KB │
│ app:category:all 300 KB │
│ app:tag:all 800 KB │
│ app:video:detail:* 37.7 MB (58K videos)│
│ app:video:*:list:* 5-10 MB │
│ (other keys) ~1-2 MB │
├─────────────────────────────────────────────┤
│ TOTAL: ~45 MB │
└─────────────────────────────────────────────┘
After (80 MB)
┌─────────────────────────────────────────────┐
│ EXISTING KEYS (modified content): │
│ app:channel:all 12.5 KB (+2.5) │
│ app:category:all 400 KB (+100) │
│ app:tag:all 600 KB (-200) │
│ app:video:detail:* 38.9 MB (+1.2) │
│ app:video:*:list:* 5-10 MB (same) │
│ │
│ NEW INDEX KEYS: │
│ app:video:category:index:* 12.2 MB ✨ │
│ app:video:categories:* 7.5 MB ✨ │
│ app:category:video:count:* 80 KB ✨ │
│ app:tag:search:* 0.5-1 MB ✨ │
│ app:category:tagnames:flat:* 400 KB ✨ │
│ │
│ (other keys) ~1-2 MB │
├─────────────────────────────────────────────┤
│ TOTAL: ~80 MB │
│ INCREASE: +35 MB (77%) │
│ HEADROOM NEEDED: 100+ MB total │
└─────────────────────────────────────────────┘
Query Pattern Evolution
Multi-Category Video Query (Example)
Before ❌ (Inefficient)
// Video can only belong to 1 category
const categoryId = 'cat-123';
const videoIds = await redis.lrange(
`app:video:category:list:${categoryId}`,
0,
-1,
);
// Want videos from multiple categories? Must do separate queries
const videos1 = await redis.lrange(`app:video:category:list:cat1`, 0, -1);
const videos2 = await redis.lrange(`app:video:category:list:cat2`, 0, -1);
const allVideos = [...new Set([...videos1, ...videos2])]; // Manual union
After ✅ (Efficient)
// Use new ZSET index for efficient union
const categoryIds = ['cat-1', 'cat-2', 'cat-3'];
const keys = categoryIds.map((id) => `app:video:category:index:${id}:videos`);
// ZUNIONSTORE combines all category indexes
await redis.zunionstore('temp:combined', keys.length, ...keys);
// Get paginated results with scores (timestamps)
const videoIds = await redis.zrevrange('temp:combined', 0, 19);
// Or use direct lookup
const videoCats = await redis.smembers(`app:video:categories:video-123`);
Builder Execution Order
Start Cache Rebuild
│
├─→ ChannelCacheBuilder.buildAll()
│ └─ Populates: app:channel:all
│
├─→ CategoryCacheBuilder.buildAll()
│ ├─ Populates: app:category:all
│ ├─ Populates: app:category:by-id:*
│ └─ Populates: app:category:tagnames:flat:*
│
├─→ TagCacheBuilder.buildAll()
│ ├─ Populates: app:tag:all
│ └─ Populates: app:tag:list:*
│
├─→ VideoCategoryIndexBuilder.buildAll() ✨ NEW
│ └─ Populates: app:video:category:index:*:videos
│
├─→ VideoCategoriesLookupBuilder.buildAll() ✨ NEW
│ └─ Populates: app:video:categories:*
│
├─→ CategoryVideoCountBuilder.buildAll() ✨ NEW
│ └─ Populates: app:category:video:count:*
│
└─→ (Other builders...)
└─ Populate remaining keys
Total Time: ~60 seconds for 58K videos
Invalidation Flow
When Video Updates CategoryIds
VideoMediaUpdated Event
│
├─→ Delete: app:video:detail:{videoId}
│
├─→ Delete: app:video:categories:{videoId}
│
├─→ For each OLD categoryId:
│ ├─ Delete: app:video:category:index:{catId}:videos
│ └─ Delete: app:category:video:count:{catId}
│
└─→ For each NEW categoryId:
├─ Delete: app:video:category:index:{catId}:videos
└─ Delete: app:category:video:count:{catId}
(Keys get rebuilt on next cache rebuild or lazy load)
TTL Lifecycle
┌─────────────────────────────────────────────────────────────┐
│ Cache Key Lifecycle (TTL Management) │
├─────────────────────────────────────────────────────────────┤
│ │
│ app:tag:all [7 days ━━━━━━━━━━━━] │
│ app:category:all [7 days ━━━━━━━━━━━━] │
│ app:channel:all [7 days ━━━━━━━━━━━━] │
│ │
│ app:video:detail:* [24h ━━━━━━━━] │
│ app:video:categories:* [24h ━━━━━━━━] │
│ app:category:tagnames:flat:* [24h ━━━━━━━━] │
│ │
│ app:video:category:index:* [2h ━━━] │
│ app:category:video:count:* [1h ━] │
│ app:tag:search:* [2h ━━━] │
│ │
│ ↑ ↑ ↑ │
│ Long-lived Medium-lived Short-lived │
│ (7 days) (24 hours) (1-2 hours) │
│ Metadata Content Indexes │
└─────────────────────────────────────────────────────────────┘
Implementation Phase Timeline
┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: Database Schema (2-3 days) │
├─────────────────────────────────────────────────────────────┤
│ ✓ Update Prisma schema │
│ ✓ Create migration │
│ ✓ Run migration │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: Core Builders (2-3 days) │
├─────────────────────────────────────────────────────────────┤
│ ✓ Update TagCacheBuilder │
│ ✓ Update CategoryCacheBuilder │
│ ✓ Update ChannelCacheBuilder │
│ ✓ Update VideoDetail builder │
│ ✓ Integration test │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ PHASE 3: Index Builders (1-2 days) │
├─────────────────────────────────────────────────────────────┤
│ ✓ Create VideoCategoryIndexBuilder │
│ ✓ Create VideoCategoriesLookupBuilder │
│ ✓ Create CategoryVideoCountBuilder │
│ ✓ Unit tests │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ PHASE 4: Integration (1 day) │
├─────────────────────────────────────────────────────────────┤
│ ✓ Update CacheSyncService │
│ ✓ Add invalidation logic │
│ ✓ Integration tests │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ PHASE 5: Testing & Deploy (2-3 days) │
├─────────────────────────────────────────────────────────────┤
│ ✓ Load test (58K+ videos) │
│ ✓ Performance benchmarks │
│ ✓ Staging deployment │
│ ✓ Production deployment │
│ ✓ Monitoring & alerts │
└─────────────────────────────────────────────────────────────┘
Total: 8-12 days | Team: 1-2 engineers | Risk: Medium
Visual Summary: New architecture supports flexible multi-category organization while maintaining performance through strategic denormalization and indexing.