Case-Design-Gaming-Leaderboard

Case Study: Design a Real-time Gaming Leaderboard

“Leaderboard giong nhu bang xep hang giai dau the thao: hang trieu van dong vien, diem thay doi lien tuc, ai cung muon biet thu hang cua minh — ngay lap tuc, khong phai doi 5 phut.”

Tags: system-design leaderboard redis sorted-set real-time gaming alex-xu-vol2 case-study Student: Hieu Source: Alex Xu — System Design Interview Volume 2, Chapter 10 Prerequisite: Tuan-01-Scale-From-Zero-To-Millions · Tuan-02-Back-of-the-envelope · Tuan-06-Cache-Strategy Lien quan: Tuan-09-Rate-Limiter · Tuan-10-Consistent-Hashing · Tuan-08-Message-Queue · Tuan-07-Database-Sharding-Replication

---

1. Context & Why — Tai sao Gaming Leaderboard quan trong?

1.1 Analogy: Bang xep hang giai dau the thao

Hieu, em tuong tuong minh dang to chuc mot giai dau the thao quoc te — giong nhu Olympic hoac World Cup. Co hang trieu van dong vien tham gia, diem so thay doi lien tuc sau moi tran dau, va khac gia tren toan the gioi deu muon biet:

1. Top 10 ai? — Ai dang dan dau bang xep hang?
2. Toi dang o dau? — Thu hang cua toi la bao nhieu trong so hang trieu nguoi?
3. Ai o quanh toi? — Nhung nguoi co thu hang gan toi la ai?
4. Cap nhat ngay! — Khi toi ghi ban, bang xep hang phai cap nhat lien tuc, khong phai doi den cuoi ngay

Day chinh xac la bai toan cua gaming leaderboard. Trong game, moi khi nguoi choi ket thuc 1 tran, diem so thay doi va bang xep hang phai cap nhat real-time. Voi hang trieu nguoi choi dong thoi, day khong phai bai toan don gian.

1.2 Gaming Leaderboard trong thuc te

Cac he thong leaderboard noi tieng: Fortnite, League of Legends, PUBG, Clash Royale, Among Us, Candy Crush. Moi game co hang chuc trieu nguoi choi, va leaderboard la feature khong the thieu.

Thong so	Gia tri dien hinh
DAU (Daily Active Users)	5-50 trieu
Score updates/ngay	25-250 trieu
Leaderboard views/ngay	50-500 trieu
Latency yeu cau	< 100ms cho moi thao tac
Freshness	Real-time hoac near-real-time (< 5 giay)

1.3 Tai sao day la bai toan kho?

Thach thuc	Giai thich
Scale	25 trieu score updates/ngay, 5 trieu DAU — khong the dung brute-force
Real-time ranking	Khi 1 nguoi thay doi diem, thu hang cua hang trieu nguoi khac cung thay doi
Top K query	”Top 10 ai?” phai tra loi trong < 100ms
User rank query	”Toi xep thu bao nhieu trong 25 trieu nguoi?” — cau hoi nay kho hon nhieu
Tie-breaking	2 nguoi cung diem — ai xep truoc?
Monthly reset	Moi thang reset leaderboard — nhung phai luu lich su
Multiple leaderboards	Per-game, per-region, per-friend-group — hang ngan leaderboard dong thoi

1.4 Tai sao Backend Dev can hieu Leaderboard?

Ly do	Giai thich
Moi san pham deu can ranking	Khong chi game — e-commerce (top sellers), social media (trending), education (student ranking), fitness (step count)
Redis Sorted Set la pattern co ban	Hieu Sorted Set = hieu duoc hang chuc bai toan ranking khac
Interview favorite	Leaderboard la bai interview pho bien vi no kiem tra kha nang chon dung data structure
Trade-off dien hinh	Real-time vs near-real-time, memory vs speed, single node vs sharded — moi thu deu co trade-off

Aha Moment: Leaderboard trong nhin co ve don gian — chi la sap xep diem so. Nhung khi scale len hang trieu nguoi, “sap xep” tro thanh bai toan O(n log n) moi lan update, va “tim thu hang” tro thanh O(n) — hoan toan khong chap nhan duoc. Day la ly do ta can data structure dac biet.

1.5 Leaderboard vs Sort — Su khac biet co ban

Khia canh	Sort thong thuong	Leaderboard system
Tan suat update	Mot lan (batch)	Lien tuc (real-time)
Tan suat query	Mot lan	Hang trieu lan/ngay
Du lieu	Static	Dynamic — thay doi moi giay
Kich thuoc	Nho (van phan tu)	Lon (hang trieu phan tu)
Yeu cau	Sap xep 1 lan	Insert, update, rank query, range query — tat ca O(log N)
Giải phap	Quick sort, merge sort	Redis Sorted Set (skip list + hash table)

---

2. Deep Dive — Alex Xu 4-Step Framework

Step 1: Requirements — Hieu va gioi han bai toan

2.1.1 Functional Requirements

Chuc nang	Mo ta chi tiet
Display top 10	Hien thi bang xep hang top 10 nguoi choi co diem cao nhat
Show user rank	Nguoi choi xem duoc thu hang cua minh trong toan bo leaderboard
Real-time update	Khi nguoi choi ket thuc tran dau va diem thay doi, leaderboard cap nhat ngay
Score update	He thong cap nhat diem khi nguoi choi thang/thua tran
Monthly reset	Leaderboard reset vao dau moi thang, bat dau lai tu 0
Relative ranking	Hien thi nhung nguoi choi co thu hang gan voi minh (“users near me”)

2.1.2 Non-Functional Requirements

Yeu cau	Muc tieu	Ly do
Latency	P99 < 100ms cho moi thao tac	User experience — game player khong chiu doi
Throughput	25 trieu score updates/ngay (~290 updates/sec avg, ~1,000/sec peak)	Game match results tu 5M DAU
Availability	99.9% uptime	Leaderboard down = player mat dong luc choi
Scalability	5 trieu DAU, 25 trieu scores/ngay	Scale theo so luong player
Freshness	Real-time (< 1 giay) cho score update	Player muon thay ket qua ngay sau tran dau
Consistency	Eventually consistent ok	Sai lech vai giay chap nhan duoc

2.1.3 Clarifying Questions

Trong interview, Hieu nen hoi nhung cau nay truoc khi thiet ke:

Cau hoi	Tra loi	Ghi chu
Score tinh the nao?	Moi tran thang +1 diem	Don gian, khong co weighted score
Co bao nhieu nguoi choi?	5 trieu DAU, 25 trieu tran dau/ngay	Moi nguoi choi trung binh 5 tran/ngay
Leaderboard reset chu ky nao?	Hang thang	Dau thang reset ve 0
Can xem lich su leaderboard khong?	Co — xem bang xep hang thang truoc	Archive leaderboard cu
Real-time hay near-real-time?	Real-time cho score update	Player thay ket qua ngay
Co bao nhieu loai leaderboard?	Global + per-region + per-friend	Nhieu leaderboard dong thoi
Tie-breaking rule?	Cung diem — ai dat diem truoc xep truoc	Time-based tie-breaking

2.1.4 Scale Summary

Thong so	Gia tri
DAU	5,000,000
Tran dau/ngay	25,000,000
Score updates/ngay	25,000,000
Leaderboard views/ngay	~50,000,000 (moi nguoi xem ~10 lan/ngay)
Tong so nguoi choi co diem	~25,000,000 (1 thang)
Reset cycle	Monthly

---

Step 2: High-Level Design

2.2.1 Kien truc tong quan

flowchart TB
    subgraph "Client Layer"
        PLAYER[Game Client<br/>Mobile / PC / Console]
    end

    subgraph "API Gateway"
        GW[API Gateway<br/>Auth, Rate Limiting, Routing]
    end

    subgraph "Game Services"
        GS[Game Service<br/>Match Result Processing]
    end

    subgraph "Leaderboard System"
        LS[Leaderboard Service<br/>Score Update & Query]
        REDIS[(Redis Cluster<br/>Sorted Sets)]
    end

    subgraph "Data Store"
        DB[(MySQL / PostgreSQL<br/>Player Profiles<br/>Match History)]
        MQ[Message Queue<br/>Kafka]
    end

    PLAYER --> GW
    GW --> GS
    GS -->|"Score Update"| LS
    GS -->|"Match Result"| DB
    GS -->|"Event"| MQ
    LS <-->|"ZADD, ZREVRANGE,<br/>ZREVRANK, ZSCORE"| REDIS
    LS -->|"Leaderboard Data"| GW
    GW -->|"Top 10, User Rank"| PLAYER
    MQ -->|"Analytics, Archive"| DB

    style REDIS fill:#e53935,color:#fff
    style LS fill:#1e88e5,color:#fff
    style GS fill:#43a047,color:#fff

2.2.2 Luong xu ly diem so (Score Update Flow)

sequenceDiagram
    participant Player
    participant Gateway as API Gateway
    participant GS as Game Service
    participant LS as Leaderboard Service
    participant Redis as Redis Sorted Set
    participant DB as Database

    Player->>Gateway: Match Completed (result: WIN)
    Gateway->>GS: Process Match Result
    GS->>GS: Validate Match Result
    GS->>DB: Save Match History
    GS->>LS: Update Score (user_id, +1)

    LS->>Redis: ZINCRBY leaderboard:2026-03 1 user_id
    Redis-->>LS: New Score: 42

    LS-->>GS: Score Updated
    GS-->>Gateway: Match Result + New Rank
    Gateway-->>Player: "You won! New score: 42, Rank: #1,523"

2.2.3 Luong truy van leaderboard (Leaderboard Query Flow)

sequenceDiagram
    participant Player
    participant Gateway as API Gateway
    participant LS as Leaderboard Service
    participant Redis as Redis Sorted Set
    participant DB as Database

    Player->>Gateway: Get Top 10
    Gateway->>LS: getTopK(10)
    LS->>Redis: ZREVRANGE leaderboard:2026-03 0 9 WITHSCORES
    Redis-->>LS: [(user_A, 150), (user_B, 148), ..., (user_J, 120)]
    LS->>DB: Get player profiles for top 10 user IDs
    DB-->>LS: [{name: "ProGamer", avatar: "..."}, ...]
    LS-->>Gateway: Top 10 with profiles
    Gateway-->>Player: Display Leaderboard

    Player->>Gateway: Get My Rank
    Gateway->>LS: getUserRank(user_id)
    LS->>Redis: ZREVRANK leaderboard:2026-03 user_id
    Redis-->>LS: Rank: 1522 (0-indexed)
    LS->>Redis: ZSCORE leaderboard:2026-03 user_id
    Redis-->>LS: Score: 42
    LS-->>Gateway: {rank: 1523, score: 42}
    Gateway-->>Player: "Your rank: #1,523 | Score: 42"

2.2.4 Cac component chinh va vai tro

Component	Vai tro	Dac diem
Game Client	Gui ket qua tran dau, hien thi leaderboard	Mobile, PC, Console — da nen tang
API Gateway	Xac thuc, rate limiting, routing	Ngan chan abuse, phan phoi tai
Game Service	Xu ly ket qua tran dau, tinh diem	Business logic, validation
Leaderboard Service	Cap nhat diem va truy van thu hang	Interface giua business logic va Redis
Redis Sorted Set	Luu tru va sap xep diem so	O(log N) cho moi thao tac — trai tim cua he thong
Database (MySQL/PostgreSQL)	Luu player profile, match history	Persistent storage, khong dung cho real-time ranking
Message Queue (Kafka)	Event streaming cho analytics, archive	Async processing, decouple services

Key Insight: Redis Sorted Set la trai tim cua he thong. Moi quyet dinh thiet ke xoay quanh viec toi uu hoa cach su dung Sorted Set. Database chi luu du lieu phu (profile, history) — khong bao gio dung database cho ranking query.

---

Step 3: Deep Dive — Chi tiet tung component

2.3.1 Naive Approach — Tai sao SQL khong dung duoc?

Truoc khi di vao giai phap, hay hieu tai sao cach tiep can “don gian” khong hoat dong.

Cach don gian nhat: Luu score trong SQL table va query.

Column	Type
user_id	BIGINT (PK)
score	INT
updated_at	TIMESTAMP

Query lay thu hang cua 1 user:

SELECT COUNT(*) FROM leaderboard WHERE score > (SELECT score FROM leaderboard WHERE user_id = ?)

Van de:

Van de	Giai thich
O(n) per query	Moi lan hoi thu hang, database phai scan toan bo bang de dem co bao nhieu nguoi diem cao hon
25 trieu rows	Voi 25 trieu nguoi choi, moi query mat vai giay
50 trieu queries/ngay	50 trieu leaderboard views × O(n) query = database chet
Lock contention	Score update (write) va rank query (read) deu can truy cap cung bang — deadlock, slow
No real-time	Index update sau moi INSERT/UPDATE mat thoi gian, query result khong real-time

Them index co giup duoc khong?

Index tren score giup ORDER BY score DESC LIMIT 10 (top K) chay nhanh — O(log N). Nhung query “rank cua 1 user” van la COUNT(*) — database phai scan toan bo index de dem. Index B-tree khong ho tro “dem so phan tu lon hon X” mot cach hieu qua.

Aha Moment: SQL B-tree index duoc thiet ke cho point query (tim 1 gia tri) va range scan (lay 1 khoang). No khong duoc thiet ke cho rank query (dem so phan tu > X). Day la ly do can data structure dac biet nhu skip list.

Nguong gioi han cua SQL approach:

Kich thuoc	SQL Performance	Chap nhan duoc?
10,000 users	~5ms	Co
100,000 users	~50ms	Tam on
1,000,000 users	~500ms	Cham
25,000,000 users	~5-10 giay	KHONG

Ket luan: SQL chi hoat dong tot voi leaderboard nho (< 100K users). Voi 25M users, can giai phap khac.

---

2.3.2 Redis Sorted Set — Giai phap hoan hao

Redis Sorted Set la gi?

Redis Sorted Set la data structure luu tru cac cap (member, score), tu dong sap xep theo score. Moi member la duy nhat (giong Set), nhung moi member co 1 gia tri score di kem.

4 operations co ban cho leaderboard:

Operation	Mo ta	Complexity	Leaderboard Use Case
ZADD	Them hoac cap nhat member voi score	O(log N)	Cap nhat diem nguoi choi
ZINCRBY	Tang score cua member them 1 gia tri	O(log N)	Cong diem sau moi tran thang
ZREVRANGE	Lay top K members (score giam dan)	O(log N + K)	Hien thi top 10 leaderboard
ZREVRANK	Lay thu hang cua 1 member (0-indexed, score giam dan)	O(log N)	“Toi xep thu bao nhieu?”
ZSCORE	Lay score cua 1 member	O(1)	Hien thi diem hien tai cua nguoi choi
ZREVRANGEBYSCORE	Lay members trong khoang score	O(log N + K)	Tim nguoi choi co diem trong khoang
ZCARD	Dem tong so members	O(1)	Tong so nguoi choi tren leaderboard
ZREM	Xoa member	O(log N)	Xoa nguoi choi bi ban

Tai sao Redis Sorted Set hoan hao cho leaderboard?

Dac diem	Giai thich
O(log N) cho moi thao tac	Voi 25M members, log(25M) ~ 24 buoc. Sieu nhanh.
In-memory	Khong co disk I/O, moi thao tac mat vai microseconds
Atomic operations	ZADD, ZINCRBY la atomic — khong can lock, khong co race condition
Built-in ranking	ZREVRANK tra ve thu hang ngay — khong can tinh toan them
Range query	ZREVRANGE lay top K hieu qua — O(log N + K)
Single data structure	1 Sorted Set giai quyet 100% cac use case cua leaderboard
Persistence	RDB snapshot + AOF log dam bao khong mat du lieu khi restart

Aha Moment: Redis Sorted Set khong phai la “hack” hay “workaround”. No duoc thiet ke chinh xac cho bai toan nay. Moi operation em can (add score, get rank, get top K) deu co san va deu O(log N). Day la vi du hoan hao cua viec chon dung data structure giai quyet bai toan.

So sanh SQL vs Redis Sorted Set:

Thao tac	SQL (25M rows)	Redis Sorted Set (25M members)
Update score	~10ms (UPDATE + index rebuild)	~0.01ms (ZADD/ZINCRBY)
Get top 10	~5ms (ORDER BY LIMIT)	~0.01ms (ZREVRANGE)
Get user rank	~5,000ms (COUNT WHERE)	~0.01ms (ZREVRANK)
Get users near me	~5,000ms (complex query)	~0.01ms (ZREVRANGE with offset)

Chenh lech: 100,000 lan cho rank query. Day khong phai toi uu — day la su khac biet giua duoc va khong duoc.

---

2.3.3 Redis Sorted Set Internals — Skip List + Hash Table

Ben trong Redis Sorted Set co gi?

Redis Sorted Set su dung 2 data structure ket hop:

Data Structure	Vai tro	Operations
Skip List	Luu members da sap xep theo score	ZREVRANGE, ZREVRANK, range queries
Hash Table	Map member → score (O(1) lookup)	ZSCORE, kiem tra member ton tai

Hai data structure nay dong bo voi nhau — khi ZADD duoc goi, Redis cap nhat ca skip list va hash table.

Skip List la gi?

Skip list la mot cau truc du lieu xac suat (probabilistic) duoc phat minh boi William Pugh nam 1990. No giong nhu linked list nhieu tang (multi-level linked list):

• Level 0 (bottom): Chua TAT CA cac phan tu, sap xep theo score
• Level 1: Chua ~50% phan tu (ngau nhien)
• Level 2: Chua ~25% phan tu
• Level k: Chua ~1/2^k phan tu
• Top level: Chi co vai phan tu

Vi du Skip List voi scores [10, 20, 30, 40, 50, 60, 70, 80]:

Level 3:  HEAD ─────────────────────────────── 40 ─────────────────────────────── NIL
Level 2:  HEAD ──────────── 20 ──────────────── 40 ──────────── 60 ──────────── NIL
Level 1:  HEAD ──── 10 ──── 20 ──── 30 ──── 40 ──── 50 ──── 60 ──── 70 ──── NIL
Level 0:  HEAD ── 10 ── 20 ── 30 ── 40 ── 50 ── 60 ── 70 ── 80 ── NIL

Tim kiem trong skip list (vi du: tim 50):

1. Bat dau tu Level 3, HEAD → 40 (40 < 50, di tiep) → NIL (qua lon, xuong level)
2. Level 2: 40 → 60 (60 > 50, xuong level)
3. Level 1: 40 → 50 (TIM THAY!)

So buoc: 3-4 buoc thay vi 5 buoc (linear scan). Voi 25M phan tu, skip list chi can ~24 buoc (log2(25M)).

Tai sao skip list ma khong phai balanced BST (Red-Black Tree, AVL)?

Tieu chi	Skip List	Balanced BST (Red-Black Tree)
Implementation	Don gian hon nhieu	Phuc tap — rotation, recoloring
Concurrent access	Lock-free algorithms de hon	Lock toan bo cay khi rebalance
Range query	Duyet linked list — cache-friendly	In-order traversal — pointer chasing
Memory overhead	~1.33 pointers/node (average)	3 pointers/node (left, right, parent) + color
Insert/Delete	Update pointers o cac level — local operation	Co the trigger rebalance cascade
Complexity	O(log N) expected	O(log N) worst-case
Debug	De visualize, de debug	Kho debug rotation/recoloring

Aha Moment: Skip list la mot trong nhung data structure “elegant” nhat trong computer science. No dat duoc O(log N) performance chi bang cach dung random — khong can rotation, khong can rebalance, khong can phuc tap. Redis chon skip list vi no don gian, nhanh, va than thien voi concurrent access. Antirez (tac gia Redis) da noi: “Skip lists are simpler to implement, and allow me to do things I could not do with balanced trees.”

Redis Skip List dac biet o cho nao?

Redis skip list co 2 dac diem khac voi skip list chuan:

1. Backward pointer: Moi node co pointer nguoc ve node truoc — cho phep duyet nguoc (ZREVRANGE)
2. Span field: Moi forward pointer luu “khoang cach” den node tiep theo o level do — cho phep tinh rank ma khong can dem

Span field la bi quyet de ZREVRANK dat O(log N). Khi duyet tu head den target node, cong tat ca span values = rank cua node do. Khong can dem tung phan tu.

---

2.3.4 Handling Ties — Xu ly trung diem

Van de: 2 nguoi choi cung 42 diem — ai xep truoc?

Rule: Nguoi dat 42 diem truoc (thoi gian som hon) xep truoc. Day la quy tac cong bang — ai dat thanh tich truoc thi duoc uu tien.

Giai phap: Composite Score

Thay vi luu score thong thuong, ta tao composite score ket hop diem so va thoi gian:

$$composite\_score=score\times 10^{13}+(MAX\_TIMESTAMP-actual\_timestamp)$$

Trong do:

• score: Diem thuc te cua nguoi choi (vi du: 42)
• MAX_TIMESTAMP: Gia tri timestamp lon nhat co the (vi du: 9,999,999,999,999 — 13 chu so)
• actual_timestamp: Thoi diem nguoi choi dat duoc diem nay (Unix timestamp milliseconds)

Vi du cu the:

Player	Score	Timestamp (ms)	Composite Score
Alice	42	1,710,700,800,000	42 × 10^13 + (9,999,999,999,999 - 1,710,700,800,000) = 428,289,299,199,999
Bob	42	1,710,700,900,000	42 × 10^13 + (9,999,999,999,999 - 1,710,700,900,000) = 428,289,299,099,999

Alice co composite score lon hon Bob (vi Alice dat diem truoc — timestamp nho hon → MAX_TIMESTAMP - timestamp lon hon). Redis sap xep score giam dan → Alice xep truoc Bob.

Tai sao dung MAX_TIMESTAMP - actual_timestamp?

Cach	Ket qua	Dung/Sai
score * 10^13 + timestamp	Timestamp lon hon = score lon hon → nguoi dat diem sau xep truoc	SAI — nguoc logic
score * 10^13 + (MAX - timestamp)	Timestamp nho hon = bonus lon hon → nguoi dat diem truoc xep truoc	DUNG

Tai sao 10^13?

Unix timestamp milliseconds hien tai co 13 chu so (vi du: 1,710,700,800,000). Ta can 10^13 de dam bao phan score va phan timestamp khong chong lan nhau.

$$\text{Diem thuc te}=\lfloor \frac{composite\_score}{10^{13}}\rfloor$$

Aha Moment: Composite score la ky thuat packing 2 gia tri vao 1 number. Day la pattern pho bien trong system design — khi data structure chi ho tro 1 truong sap xep, ta “nhet” nhieu truong vao 1 bang cach su dung he so. Tuong tu nhu cach ma composite key hoat dong trong database.

Luu y ve do chinh xac:

Redis luu score dang double-precision floating point (64-bit IEEE 754). Double co ~15-17 chu so thap phan chinh xac. Composite score cua ta co toi da:

$$\text{So chu so}=\text{score digits}+13=2+13=15\text{ chu so (voi score max 99)}$$

15 chu so nam trong gioi han chinh xac cua double. Nhung neu score co the len hang trieu (6 chu so), tong la 19 chu so — vuot qua gioi han cua double. Trong truong hop do, can dung cach khac (vi du: 2 Sorted Sets, hoac su dung string encoding).

---

2.3.5 Relative Ranking — “Users Near Me”

Use case: Nguoi choi muon thay nhung nguoi co thu hang gan minh — vi du: nguoi xep truoc 5 bac va sau 5 bac.

Giai phap: Su dung ZREVRANGE voi offset.

Flow:

1. Lay rank cua user: ZREVRANK leaderboard:2026-03 user_id → rank = 1522
2. Lay users xung quanh: ZREVRANGE leaderboard:2026-03 1517 1527 WITHSCORES (rank 1518 den 1528)

Ket qua:

Rank	Player	Score
#1518	player_A	43
#1519	player_B	43
#1520	player_C	42
#1521	player_D	42
#1522	player_E	42
#1523	YOU	42
#1524	player_F	42
#1525	player_G	41
#1526	player_H	41
#1527	player_I	41
#1528	player_J	40

Complexity: O(log N) cho ZREVRANK + O(log N + K) cho ZREVRANGE = O(log N + K) tong cong. Voi 25M users va K = 11: ~24 + 11 = 35 operations. Sieu nhanh.

UX considerations:

• Hien thi avatar, username, va score cua moi nguoi — can query them database cho profile info
• Highlight dong cua user hien tai
• Cho phep tap vao profile nguoi khac de xem chi tiet
• Hien thi trend (len bac, xuong bac so voi ngay hom qua)

---

2.3.6 Sharded Leaderboard — Khi 1 Redis khong du

Khi nao can shard?

Tinh huong	Can shard?
25M users, ~1.5GB memory	Khong — 1 Redis node du suc (Redis ho tro hang chuc GB)
500M users, ~30GB memory	Co the — memory van ok nhung throughput co the la bottleneck
5B users, ~300GB memory	Chac chan — vuot qua memory cua 1 node
Write throughput > 100K/sec	Co the — 1 Redis node co gioi han write throughput

Quan trong: Voi 25M users va 290 writes/sec trung binh, 1 Redis node hoan toan du. Sharding chi can khi scale vuot qua gioi han cua single node. Trong interview, hay noi ro dieu nay truoc khi thao luan sharding.

Cach shard: Score Range Partitioning

Chia leaderboard thanh N shard theo khoang diem (score range):

Shard	Score Range	So luong users (uoc tinh)
Shard 0	0 - 99	10M (majority la low scores)
Shard 1	100 - 199	8M
Shard 2	200 - 299	4M
Shard 3	300 - 399	2M
Shard 4	400+	1M

Van de voi score range partitioning: Data khong deu — shard 0 (low scores) co nhieu user hon shard 4 (high scores). Day la data skew.

Cach khac: Hash Partitioning

Shard theo hash(user_id) % N. Data deu nhung khong the lay top K global — vi top K co the nam o bat ky shard nao.

Giai phap cho hash partitioning — Merge Top K:

flowchart TB
    subgraph "Client Request"
        REQ["Get Global Top 10"]
    end

    subgraph "Leaderboard Service"
        LS[Scatter-Gather<br/>Coordinator]
    end

    subgraph "Redis Shards"
        S1["Shard 1<br/>ZREVRANGE 0 9"]
        S2["Shard 2<br/>ZREVRANGE 0 9"]
        S3["Shard 3<br/>ZREVRANGE 0 9"]
        S4["Shard 4<br/>ZREVRANGE 0 9"]
    end

    subgraph "Merge"
        MRG["Merge Sort<br/>Top 10 from 4×10 = 40 results"]
    end

    REQ --> LS
    LS -->|"Parallel query"| S1 & S2 & S3 & S4
    S1 & S2 & S3 & S4 --> MRG
    MRG -->|"Global Top 10"| LS
    LS --> REQ

    style MRG fill:#e53935,color:#fff
    style LS fill:#1e88e5,color:#fff

Scatter-Gather pattern:

1. Gui ZREVRANGE 0 9 den tat ca N shards (song song)
2. Moi shard tra ve top 10 cua no
3. Coordinator merge N × 10 results va lay top 10 global

Complexity: O(N × log M + N × K × log(N × K)) trong do N = so shards, M = so members/shard, K = so ket qua mong muon.

Van de cua sharded leaderboard cho rank query:

“User X xep thu bao nhieu globally?” — day la bai toan kho nhat cua sharding.

Approach	Mo ta	Do chinh xac
Exact rank	Query ZREVRANK tren tat ca shards, cong rank lai	Chinh xac 100% nhung cham (query N shards)
Approximate rank	Dung sampling hoac statistics	Nhanh nhung khong chinh xac
Score range partition	Rank = tong so users o tat ca shard diem cao hon + rank trong shard hien tai	Chinh xac, nhung can luu cardinality moi shard

Exact rank voi score range partitioning:

$$global\_rank=\sum _{i\in \text{higher shards}}ZCARD(shar{d}_i)+ZREVRANK(current\_shard,user)$$

Vi du: User co score 250 nam o Shard 2 (200-299). Rank cua user:

• ZCARD(Shard 3) = 2,000,000 (users co 300-399 diem)
• ZCARD(Shard 4) = 1,000,000 (users co 400+ diem)
• ZREVRANK(Shard 2, user) = 500 (rank 500 trong Shard 2)
• Global rank = 2,000,000 + 1,000,000 + 500 = 3,000,500

Aha Moment: Sharding leaderboard kho hon rat nhieu so voi single node. Top K query can scatter-gather, rank query can cong rank tu nhieu shard, va data skew la van de thuong truc. Day la ly do tai sao interview thuong bat dau voi single node va chi chuyen sang sharding khi interviewer hoi “what if you need to scale beyond single node?“. Luon uu tien single node neu co the.

---

2.3.7 Real-time vs Near-real-time — Chon dung cho dung bai toan

Hai cach tiep can:

Cach	Mo ta	Khi nao dung
Real-time (Direct Write)	Game Service goi truc tiep Redis ZADD/ZINCRBY	Write throughput thap-trung binh (< 10K/sec), can real-time update
Near-real-time (Batch via MQ)	Game Service gui event vao Kafka, consumer batch write vao Redis	Write throughput cao (> 10K/sec), chap nhan delay 1-5 giay

Real-time approach:

flowchart LR
    GS[Game Service] -->|"ZINCRBY"| REDIS[(Redis<br/>Sorted Set)]

    style REDIS fill:#e53935,color:#fff

• Uu diem: Diem cap nhat ngay, nguoi choi thay ket qua lien tuc
• Nhuoc diem: Redis chiu toan bo write load truc tiep; neu burst traffic, co the qua tai

Near-real-time approach:

flowchart LR
    GS[Game Service] -->|"Produce"| KAFKA[Kafka<br/>Message Queue]
    KAFKA -->|"Consume batch"| CONSUMER[Score Consumer<br/>Batch ZADD]
    CONSUMER -->|"Batch ZINCRBY"| REDIS[(Redis<br/>Sorted Set)]

    style KAFKA fill:#43a047,color:#fff
    style REDIS fill:#e53935,color:#fff

• Uu diem: Kafka lam buffer, Redis khong bi burst; consumer co the batch nhieu update thanh 1 pipeline command
• Nhuoc diem: Delay 1-5 giay giua khi tran dau ket thuc va khi leaderboard cap nhat

Khi nao chon cach nao?

Tieu chi	Real-time	Near-real-time
Write QPS	< 10,000/sec	> 10,000/sec
Latency requirement	< 1 giay	1-5 giay ok
Burst traffic	Khong co hoac it	Co burst lon (vi du: event dac biet)
Complexity	Don gian hon	Phuc tap hon (them Kafka, consumer)

Cho bai toan nay (290 avg, 1000 peak): Real-time la du. 1,000 writes/sec hoan toan trong kha nang cua 1 Redis node (Redis co the xu ly 100K+ commands/sec). Chi can near-real-time khi scale len 100K+ writes/sec.

Key Insight: Dung over-engineer. 1,000 writes/sec la con so nho voi Redis. Them Kafka vao chi tang complexity ma khong mang lai loi ich ro rang. KISS — Keep It Simple, Stupid.

---

2.3.8 Monthly Reset — Reset leaderboard hang thang

Strategy: Sorted Set per month

Moi thang, tao 1 Sorted Set moi voi key co ten thang:

Thang	Redis Key
Thang 1/2026	leaderboard:2026-01
Thang 2/2026	leaderboard:2026-02
Thang 3/2026	leaderboard:2026-03

Flow dau thang moi:

1. Khi thang moi bat dau (vi du: 2026-04-01 00:00:00 UTC), Leaderboard Service tu dong chuyen sang key leaderboard:2026-04
2. Key leaderboard:2026-03 van ton tai — nguoi choi van xem duoc bang xep hang thang truoc
3. Dat TTL cho key cu: EXPIRE leaderboard:2026-01 7776000 (90 ngay = 3 thang luu tru)
4. Sau 3 thang, Redis tu dong xoa key cu — khong can manual cleanup

Automation workflow:

Buoc	Thao tac	Thoi diem
1	Leaderboard Service detect thang moi (tu config hoac cron)	00:00:00 UTC ngay 1
2	Chuyen current_leaderboard pointer sang key moi	00:00:01 UTC
3	Archive key cu vao persistent storage (S3/database) neu can	00:01:00 UTC (async)
4	Dat TTL cho key cu tren Redis	00:01:01 UTC
5	Gui notification cho players: “New season started!“	00:05:00 UTC

Edge case — tran dau bat dau thang cu, ket thuc thang moi:

Tinh huong	Giai phap
Tran bat dau 23:58 thang 3, ket thuc 00:02 thang 4	Dung match_start_time de quyet dinh ghi vao leaderboard nao
Hoac: dung match_end_time	Ghi vao leaderboard cua thang ma tran ket thuc
Race condition khi chuyen thang	Dung atomic flag hoac distributed lock de dam bao chi chuyen 1 lan

Aha Moment: Strategy “1 Sorted Set per month” don gian nhung hieu qua. Khong can DEL (xoa toan bo key — O(N), block Redis). Khong can ZREMRANGEBYSCORE (xoa tung phan tu). Chi can tao key moi va de key cu tu het han. Day la pattern immutable data — thay vi sua du lieu cu, tao du lieu moi.

---

2.3.9 Multiple Leaderboards — Per-game, Per-region, Per-friend

Use cases:

Loai leaderboard	Mo ta	Vi du key
Global	Toan bo nguoi choi	lb:global:2026-03
Per-game	Theo tung game mode	lb:mode:ranked:2026-03
Per-region	Theo vung dia ly	lb:region:asia:2026-03
Per-friend	Chi trong danh sach ban be	lb:friends:{user_id}:2026-03
Per-clan	Theo clan/guild	lb:clan:{clan_id}:2026-03
Weekly	Reset hang tuan	lb:weekly:2026-W12
All-time	Khong bao gio reset	lb:alltime

Namespaced keys:

Pattern key: lb:{scope}:{identifier}:{period}

Phan	Y nghia	Vi du
lb	Prefix cho leaderboard	Co dinh
{scope}	Loai leaderboard	global, region, friends, clan
{identifier}	ID cu the	asia, user_123, clan_456
{period}	Ky reset	2026-03, 2026-W12, alltime

Friend leaderboard — dac biet kho:

Thach thuc	Giai thich
Moi user co danh sach ban be khac nhau	Khong the dung 1 Sorted Set chung
So luong leaderboard = so luong users	5M users = 5M friend leaderboards? KHONG
Ban be thay doi (add/remove friend)	Leaderboard phai cap nhat theo

Giai phap cho friend leaderboard:

Approach	Mo ta	Uu/Nhuoc
On-demand computation	Khi user xem friend leaderboard, lay score cua tat ca friends tu global leaderboard, sort client-side	Don gian, khong ton memory. Nhung cham neu co 500+ friends
Fan-out on write	Khi user update score, ghi vao friend leaderboard cua tat ca ban be	Nhanh khi doc, nhung write amplification lon (500 friends = 500 writes)
Hybrid	Cache friend leaderboard voi short TTL (30 giay), invalidate khi co update	Can bang giua read speed va write cost

Recommendation: Voi game thong thuong (100-200 friends), on-demand computation la du tot. Lay 200 scores tu Redis (200 × ZSCORE) mat ~2ms. Sort 200 items mat ~0.001ms. Tong: ~2ms — hoan toan chap nhan duoc.

---

2.3.10 Pagination — Phan trang leaderboard

Van de: Leaderboard co 25M users. Client chi hien thi 20 users/trang. User muon cuon xuong xem nhieu hon.

Giai phap: Cursor-based pagination dung ZREVRANGE voi LIMIT

Trang	Redis Command	Giai thich
Trang 1 (rank 1-20)	ZREVRANGE lb:2026-03 0 19 WITHSCORES	Lay 20 users dau tien
Trang 2 (rank 21-40)	ZREVRANGE lb:2026-03 20 39 WITHSCORES	Lay 20 users tiep theo
Trang N	ZREVRANGE lb:2026-03 (N-1)*20 N*20-1 WITHSCORES	Lay trang thu N

Complexity: O(log N + K) cho moi trang, trong do K = page size (20). Fast cho bat ky trang nao — ke ca trang 1,000,000.

Cursor-based vs Offset-based:

Cach	Mo ta	Van de
Offset-based	ZREVRANGE ... start stop	Neu leaderboard thay doi giua 2 request, user co the thay duplicate hoac miss entries
Score-based cursor	”Lay 20 users co score < last_score_seen”	Chinh xac hon, nhung kho xu ly ties (nhieu nguoi cung score)
Rank-based cursor	”Lay rank 21-40”	Giong offset-based, co the shift khi data thay doi

Thuc te: Offset-based pagination (ZREVRANGE start stop) la du tot cho leaderboard. Ly do:

• Leaderboard thay doi cham (vai giay giua cac update)
• User hiem khi cuon qua trang 10 (~rank 200)
• Duplicate/miss 1-2 entries khong anh huong trai nghiem game

Key Insight: Don’t over-engineer pagination. ZREVRANGE voi start/stop la cach don gian nhat va du tot cho 99% use cases. Chi can cursor phuc tap khi leaderboard thay doi cuc ky nhanh (hang ngan updates/giay) va user cuon sau (trang 1000+).

---

2.3.11 Caching Top K — Giam tai cho Redis

Van de: Top 10 leaderboard la hot data — hang trieu nguoi xem moi phut. Du Redis nhanh, nhung hang trieu request/phut cho cung 1 query van tao load khong can thiet.

Giai phap: Cache top K voi short TTL

Tang	Du lieu	TTL	Nguoi dung
Application cache (local)	Top 10	5 giay	Giam load xuong Redis
Redis Sorted Set	Toan bo 25M users	Khong TTL (persistent)	Source of truth

Flow voi cache:

1. Client request “Get Top 10”
2. Leaderboard Service kiem tra local cache
3. Neu cache hit (< 5 giay) → tra ve ngay
4. Neu cache miss → query Redis ZREVRANGE ... 0 9 WITHSCORES → luu vao local cache voi TTL 5s → tra ve

Tai sao TTL 5 giay?

TTL	Freshness	Cache hit rate	Load giam
1 giay	Gan nhu real-time	~50%	~50%
5 giay	Chap nhan duoc cho game	~90%	~90%
30 giay	Hoi cu	~98%	~98%
5 phut	Cu	~99.9%	~99.9%

5 giay la sweet spot: giam 90% load trong khi data chi cu toi da 5 giay — hoan toan chap nhan duoc cho game leaderboard.

Cache gi ngoai top 10?

Data	Cache	TTL	Ly do
Top 10	Co	5s	Hot data — ai cung xem
Top 100	Co	10s	Pho bien, nhieu nguoi cuon xuong
User rank (individual)	Co	3s	Moi user chi care rank cua minh
Friend leaderboard	Co	30s	It thay doi, it nguoi xem lien tuc

Aha Moment: Cache o day khong phai de Redis nhanh hon — Redis da du nhanh (0.01ms/query). Cache la de giam so luong request den Redis, ngan Redis bi overload khi co hang trieu concurrent users. Day la su khac biet giua “caching for speed” va “caching for load reduction”.

---

3. Estimation — Uoc luong he thong

3.1 Score Update Throughput

Assumptions:

Thong so	Gia tri	Giai thich
DAU	5,000,000	5 trieu nguoi choi hoat dong moi ngay
Tran dau/nguoi/ngay	5	Trung binh moi nguoi choi 5 tran
Tong tran dau/ngay	25,000,000	5M × 5
Active hours/ngay	24 gio	Game online 24/7
Peak/Average ratio	3.5x	Gio cao diem (toi)

$$AverageScoreUpdates/sec=\frac{25,000,000}{24\times 3,600}=\frac{25,000,000}{86,400}\approx 290updates/sec$$ $$PeakScoreUpdates/sec=290\times 3.5\approx 1,015updates/sec$$ $$DesignTarget=5,000updates/sec(duheadroom5x)$$

Redis co the xu ly 100,000+ commands/sec tren 1 node. 5,000 writes/sec chi chiem 5% capacity cua Redis. Single node hoan toan du.

3.2 Leaderboard Read QPS

Assumptions:

Thong so	Gia tri	Giai thich
Leaderboard views/nguoi/ngay	10	Xem top 10, xem rank, xem friends
Tong views/ngay	50,000,000	5M × 10
Read/Write ratio	2:1	Xem nhieu hon cap nhat

$$AverageReadQPS=\frac{50,000,000}{86,400}\approx 579reads/sec$$ $$PeakReadQPS=579\times 3.5\approx 2,027reads/sec$$ $$TotalPeakQPS(Read+Write)=2,027+1,015=3,042commands/sec$$

3,042 commands/sec la ~3% capacity cua Redis (100K+/sec). Single node chac chan du.

3.3 Memory Estimation

Redis Sorted Set memory per entry:

Thanh phan	Kich thuoc	Giai thich
Member (user_id)	~20 bytes	String “user_12345678”
Score	8 bytes	Double-precision float
Skip list node	~40 bytes	Pointers (forward, backward, span) — trung binh ~2.67 levels
Hash table entry	~56 bytes	Key + value + metadata cua Redis dict
Redis overhead	~16 bytes	Object header, encoding

$$Memoryperentry\approx 20+8+40+56+16=140bytes$$

Luu y: Day la uoc tinh. Thuc te co the dao dong 100-200 bytes/entry tuy vao do dai cua member string va so levels trong skip list.

$$Totalmemory(25Musers)=25,000,000\times 140bytes=3,500,000,000bytes\approx 3.5GB$$

Nhung ta co nhieu leaderboards:

Leaderboard	So entries	Memory
Global thang hien tai	25,000,000	3.5 GB
Global thang truoc (archive)	25,000,000	3.5 GB
Per-region (5 regions)	5 × 5,000,000	3.5 GB
Per-game-mode (3 modes)	3 × 25,000,000	10.5 GB
Weekly	10,000,000	1.4 GB

$$TotalRedisMemory\approx 3.5+3.5+3.5+10.5+1.4=22.4GB$$ $$DesignTarget(voibuffer2x)=22.4\times 2=44.8GB\approx 50GB$$

Redis server voi 64GB RAM co the handle thoai mai. Khong can cluster cho memory — chi can cluster cho high availability (replication).

3.4 Network Bandwidth Estimation

Write bandwidth:

$$Writebandwidth=1,015updates/sec\times 100bytes/command\approx 100KB/sec$$

Read bandwidth (moi response tra ve top 10 voi scores):

$$Readresponsesize=10\times (20+8)bytes=280bytes\approx 300bytes$$ $$Readbandwidth=2,027reads/sec\times 300bytes\approx 600KB/sec$$ $$Totalbandwidth=100+600=700KB/sec\approx 1MB/sec$$

1 MB/sec la con so rat nho. Network khong phai bottleneck.

3.5 Tom tat Estimation

Metric	Average	Peak	Design Target
Write QPS	290/sec	1,015/sec	5,000/sec
Read QPS	579/sec	2,027/sec	10,000/sec
Total QPS	869/sec	3,042/sec	15,000/sec
Memory	22.4 GB	—	50 GB
Bandwidth	~700 KB/sec	~2.5 MB/sec	10 MB/sec

Ket luan tu estimation:

• Single Redis node du suc handle toan bo workload
• Memory, QPS, bandwidth deu nam thoai mai trong gioi han cua 1 Redis instance
• Chi can Redis replication (1 master + 2 replicas) cho high availability, khong can sharding
• Over-engineering (sharding, Kafka) la khong can thiet cho scale nay

---

4. Security — Bao ve he thong leaderboard

4.1 Score Manipulation Prevention — Chong gian lan diem

Nguyen tac	Mo ta
Server-side validation only	TUYET DOI khong cho client gui score truc tiep. Chi server moi duoc cap nhat score sau khi validate match result
Server authority	Game server la nguon duy nhat cua truth. Client chi gui actions (di chuyen, ban, skill), server tinh toan ket qua
Match result verification	Server kiem tra match result co hop ly khong truoc khi cap nhat score
No client-side score	API khong co endpoint “set my score to X”. Chi co “match completed, here’s the result”

Flow an toan:

DUNG: Game Server tinh toan ket qua → Validate → Cap nhat score
SAI:  Client gui "toi duoc 100 diem" → Server tin va cap nhat

4.2 Anti-Cheat — Phat hien gian lan

Loai gian lan	Mo ta	Cach phat hien
Score injection	Hack API de gui score gia	Server-side validation — khong chap nhan score tu client
Speed hack	Choi nhanh bat thuong (10 tran/phut)	Velocity check — gioi han so tran/thoi gian
Win trading	2 nguoi thoa thuan — 1 nguoi luon thang	Pattern detection — cung 2 nguoi choi gap nhau lien tuc
Bot/automation	Dung bot de choi tu dong	Behavioral analysis — thao tac qua chinh xac, khong co variance
Account boosting	Nguoi gioi choi ho tai khoan nguoi yeu	IP/device fingerprint thay doi bat thuong
Exploit abuse	Loi dung bug de ghi diem bat thuong	Score anomaly detection — diem tang bat thuong

Anomaly Detection rules:

Rule	Dieu kien	Hanh dong
Velocity check	> 20 tran/gio	Flag account, yeu cau review
Win rate anomaly	Win rate > 95% trong 50+ tran	Flag, kiem tra match history
Score spike	Tang > 50 diem trong 1 gio	Alert, manual review
Impossible score	Score vuot qua gia tri ly thuyet toi da	Auto-reject, ban account
Time anomaly	Tran dau ket thuc trong < 10 giay (minimum match duration)	Reject match result

4.3 Rate Limiting — Gioi han tan suat

Endpoint	Rate Limit	Ly do
Score update	10 requests/phut/user	Moi tran mat it nhat 5 phut, 10 requests/phut la du
Get leaderboard (top K)	30 requests/phut/user	Ngan polling lien tuc
Get my rank	30 requests/phut/user	Tuong tu top K
Get friend leaderboard	10 requests/phut/user	It xem hon global

Tham chieu: Tuan-09-Rate-Limiter — su dung Token Bucket hoac Sliding Window algorithm.

4.4 Sybil Attack Prevention — Chong tao tai khoan gia

Tan cong	Mo ta	Phong chong
Sybil attack	Tao hang ngan tai khoan gia de chiem top leaderboard	Phone verification, email verification, minimum level requirement
Farm accounts	Tai khoan gia “thua” cho tai khoan chinh thang	Match-making co kiem tra: khong cho tai khoan moi gap nhau lien tuc
Multi-accounting	1 nguoi co nhieu tai khoan tren leaderboard	Device fingerprinting, IP tracking, behavioral analysis

Minimum eligibility: Chi hien thi tren leaderboard khi:

• Account level >= 10 (choi du 20+ tran)
• Email verified
• Khong bi flag vi gian lan

4.5 Data Integrity

Bao ve	Mo ta
Audit log	Moi score change duoc log: who, when, why, old_score, new_score
Immutable match history	Match results luu trong database — khong the sua doi
Reconciliation	Dinh ky kiem tra: tong score trong Redis = tong wins trong match history
Backup	Redis RDB snapshot + AOF — dam bao khong mat du lieu

---

5. DevOps — Van hanh he thong leaderboard

5.1 Redis Monitoring — Giam sat Redis

Cac metric quan trong

Metric	Mo ta	Nguong canh bao
used_memory	Tong memory dang su dung	> 80% maxmemory
connected_clients	So client dang ket noi	> 10,000
instantaneous_ops_per_sec	So commands/sec hien tai	> 80,000 (80% capacity)
keyspace_hits / keyspace_misses	Hit rate cua key lookup	Miss rate > 10%
rdb_last_bgsave_status	Trang thai RDB snapshot gan nhat	Khac “ok”
aof_last_bgrewrite_status	Trang thai AOF rewrite gan nhat	Khac “ok”
master_link_status	Trang thai ket noi den master (replica)	Khac “up”
master_last_io_seconds_ago	Thoi gian tu lan dong bo gan nhat	> 10 giay

5.2 Sorted Set Monitoring

Metric	Command	Nguong canh bao
Cardinality	ZCARD leaderboard:2026-03	Bat thuong neu tang/giam dot ngot
Memory cua 1 key	MEMORY USAGE leaderboard:2026-03	> 5GB/key
Slow log	SLOWLOG GET 10	Co command > 10ms

5.3 Response Latency Monitoring

Operation	Latency mong doi	Alert threshold
ZADD / ZINCRBY	< 0.1ms	> 1ms
ZREVRANGE (top 10)	< 0.1ms	> 1ms
ZREVRANK	< 0.1ms	> 1ms
ZSCORE	< 0.05ms	> 0.5ms

Cach do latency:

• Su dung redis-cli --latency de do round-trip time
• Application-side: do thoi gian tu gui command den nhan response
• Do P50, P95, P99 — khong chi average

5.4 Replication Lag Monitoring

Metric	Mo ta	Nguong canh bao
repl_backlog_active	Replication backlog co dang hoat dong	Phai luon = 1
master_repl_offset	Offset hien tai cua master	—
slave_repl_offset	Offset hien tai cua replica	Chenh lech voi master > 1MB
replication_lag_seconds	Do tre giua master va replica	> 5 giay

Tai sao replication lag quan trong cho leaderboard?

Neu read requests duoc gui den replica (read replica pattern), replication lag = stale data. Nguoi choi thay rank cu. Voi lag > 5 giay, nguoi choi co the thay minh van o rank cu du da thang tran moi.

Giai phap:

• Read writes (score update) tu master
• Reads co the tu replica, nhung cau hinh max lag tolerance
• Neu lag > threshold, chuyen read ve master

5.5 Monthly Reset Automation

Buoc	Tool	Cron/Trigger
Detect thang moi	Cron job	0 0 1 * * (00:00 ngay 1 moi thang)
Create new Sorted Set	Leaderboard Service	Automatic khi detect thang moi
Archive old leaderboard	Batch job	Export to S3/database
Set TTL for old Redis key	Script	EXPIRE leaderboard:old 7776000
Send notification	Notification Service	”New season has started!”
Verify new leaderboard	Health check	Kiem tra ZCARD = 0 cho key moi

Runbook cho monthly reset:

1. Pre-reset check: Verify backup cua leaderboard hien tai da hoan tat
2. Execute reset: Chuyen pointer sang key moi
3. Post-reset verification:
   • Key moi ton tai va rong (ZCARD = 0)
   • Key cu van truy cap duoc (archive)
   • Score updates ghi vao key moi (khong ghi vao key cu)
   • API tra ve data tu key moi
4. Rollback plan: Neu co loi, chuyen pointer ve key cu

5.6 Alerting Rules

Alert	Dieu kien	Severity	Hanh dong
Redis memory high	used_memory > 80% maxmemory	WARNING	Kiem tra co leak khong, tang maxmemory
Redis memory critical	used_memory > 95% maxmemory	CRITICAL	Eviction bat dau — co the mat data!
Redis down	Connection refused	CRITICAL	Failover sang replica
Replication broken	master_link_status = down	HIGH	Kiem tra network, restart replica
Slow commands	Slowlog entries > 10/phut	WARNING	Kiem tra co command O(N) khong
Score update failure rate	> 1% failures	HIGH	Kiem tra Redis connection, network
Leaderboard read latency	P99 > 100ms	WARNING	Kiem tra Redis load, network latency

5.7 Disaster Recovery

Tinh huong	Giai phap
Redis master down	Sentinel/Cluster auto-failover sang replica
Data corruption	Restore tu RDB snapshot + replay AOF
Datacenter failure	Cross-region replica, failover sang region khac
Accidental DEL	RDB snapshot backup hang gio, AOF cho point-in-time recovery

Redis high availability setup:

Config	Gia tri	Ly do
Topology	1 Master + 2 Replicas	Majority vote cho failover
Sentinel	3 Sentinel instances	Monitor va auto-failover
Persistence	RDB (moi 1 gio) + AOF (everysec)	Balance giua performance va durability
maxmemory-policy	noeviction	Khong tu dong xoa data — bao loi thay vi mat data

---

6. Mermaid Diagrams — Tong hop kien truc

6.1 High-Level Architecture (Chi tiet)

flowchart TB
    subgraph "Client Layer"
        MC[Mobile Client]
        PC[PC Client]
        WC[Web Client]
    end

    subgraph "Load Balancer"
        LB[Load Balancer<br/>Nginx / AWS ALB]
    end

    subgraph "API Gateway"
        GW[API Gateway<br/>Authentication<br/>Rate Limiting<br/>Request Routing]
    end

    subgraph "Application Services"
        GS[Game Service<br/>Match Processing<br/>Score Calculation]
        LS[Leaderboard Service<br/>Score Update<br/>Rank Query<br/>Top K Query]
        PS[Player Service<br/>Profile Management]
        NS[Notification Service<br/>Push Notifications]
    end

    subgraph "Cache Layer"
        LC[Local Cache<br/>Top 10/100<br/>TTL: 5s]
    end

    subgraph "Redis Cluster"
        RM[Redis Master<br/>Sorted Sets<br/>Write Operations]
        RR1[Redis Replica 1<br/>Read Operations]
        RR2[Redis Replica 2<br/>Read Operations]
        RS[Redis Sentinel<br/>Auto-Failover]
    end

    subgraph "Persistent Storage"
        DB[(MySQL<br/>Player Profiles<br/>Match History)]
        S3[(S3 / Object Storage<br/>Leaderboard Archives)]
    end

    subgraph "Monitoring"
        MON[Prometheus + Grafana<br/>Redis Metrics<br/>Latency Dashboards]
    end

    MC & PC & WC --> LB
    LB --> GW
    GW --> GS & LS & PS
    GS -->|"Score Update"| LS
    LS --> LC
    LS -->|"Write"| RM
    LS -->|"Read"| RR1 & RR2
    RM -.->|"Replication"| RR1 & RR2
    RS -.->|"Monitor"| RM & RR1 & RR2
    GS --> DB
    LS --> PS
    PS --> DB
    GS --> NS
    LS -->|"Archive"| S3
    RM & RR1 & RR2 -.-> MON

    style RM fill:#e53935,color:#fff
    style LS fill:#1e88e5,color:#fff
    style GS fill:#43a047,color:#fff
    style LC fill:#f9a825,color:#000

6.2 Redis Sorted Set Operations Flow

flowchart LR
    subgraph "Score Update"
        A1[Player wins match] --> A2["ZINCRBY lb:2026-03 1 user_123"]
        A2 --> A3[Skip List: reposition node<br/>Hash Table: update score]
        A3 --> A4["New score: 42<br/>New rank: auto-calculated"]
    end

    subgraph "Get Top 10"
        B1[Client: Show top 10] --> B2["ZREVRANGE lb:2026-03 0 9 WITHSCORES"]
        B2 --> B3[Skip List: traverse from tail<br/>collect 10 highest nodes]
        B3 --> B4["Result: [(Alice,150), (Bob,148), ...]"]
    end

    subgraph "Get My Rank"
        C1[Client: What's my rank?] --> C2["ZREVRANK lb:2026-03 user_123"]
        C2 --> C3[Hash Table: find node by member<br/>Skip List: sum spans to head]
        C3 --> C4["Rank: 1522 (0-indexed)"]
    end

    subgraph "Get My Score"
        D1[Client: What's my score?] --> D2["ZSCORE lb:2026-03 user_123"]
        D2 --> D3[Hash Table: direct O(1) lookup]
        D3 --> D4["Score: 42"]
    end

    style A2 fill:#e53935,color:#fff
    style B2 fill:#1e88e5,color:#fff
    style C2 fill:#43a047,color:#fff
    style D2 fill:#f9a825,color:#000

6.3 Sharded Leaderboard Merge Flow

flowchart TB
    subgraph "Request"
        CLIENT["Client: Get Global Top 10"]
    end

    subgraph "Coordinator"
        COORD[Leaderboard Service<br/>Scatter-Gather Coordinator]
    end

    subgraph "Shard Layer"
        S1["Shard 1 (hash 0-24%)<br/>ZREVRANGE 0 9<br/>→ [A:95, B:90, C:88, ...]"]
        S2["Shard 2 (hash 25-49%)<br/>ZREVRANGE 0 9<br/>→ [D:97, E:85, F:82, ...]"]
        S3["Shard 3 (hash 50-74%)<br/>ZREVRANGE 0 9<br/>→ [G:150, H:92, I:80, ...]"]
        S4["Shard 4 (hash 75-99%)<br/>ZREVRANGE 0 9<br/>→ [J:120, K:88, L:75, ...]"]
    end

    subgraph "Merge"
        MERGE["K-way Merge Sort<br/>Input: 4 × 10 = 40 entries<br/>Output: Global Top 10<br/>→ [G:150, J:120, D:97, A:95,<br/>H:92, B:90, C:88, K:88, E:85, F:82]"]
    end

    CLIENT --> COORD
    COORD -->|"Parallel"| S1 & S2 & S3 & S4
    S1 & S2 & S3 & S4 --> MERGE
    MERGE --> COORD
    COORD --> CLIENT

    style MERGE fill:#e53935,color:#fff
    style COORD fill:#1e88e5,color:#fff

6.4 Score Update Pipeline (End-to-End)

flowchart TB
    subgraph "Step 1: Match Ends"
        MATCH["Game Server detects<br/>match result: Player A wins"]
    end

    subgraph "Step 2: Validate"
        VAL["Validation<br/>- Match ID exists?<br/>- Result consistent?<br/>- No duplicate submission?<br/>- Anti-cheat checks passed?"]
    end

    subgraph "Step 3: Persist"
        PERSIST["Save to MySQL<br/>- Match history record<br/>- Idempotency key"]
    end

    subgraph "Step 4: Update Leaderboard"
        UPDATE["Redis ZINCRBY<br/>leaderboard:2026-03 1 player_A<br/><br/>Also update:<br/>- Per-region leaderboard<br/>- Per-game-mode leaderboard<br/>- Weekly leaderboard<br/>- All-time leaderboard"]
    end

    subgraph "Step 5: Response"
        RESP["Return to client:<br/>- New score: 42<br/>- New rank: #1,523<br/>- Rank change: ↑ 5"]
    end

    subgraph "Step 6: Async"
        ASYNC["Async tasks:<br/>- Push notification if milestone<br/>- Analytics event<br/>- Achievement check"]
    end

    MATCH --> VAL
    VAL -->|"Valid"| PERSIST
    VAL -->|"Invalid"| REJECT["Reject<br/>Log suspicious activity"]
    PERSIST --> UPDATE
    UPDATE --> RESP
    UPDATE -.-> ASYNC

    style UPDATE fill:#e53935,color:#fff
    style VAL fill:#f9a825,color:#000
    style REJECT fill:#757575,color:#fff

6.5 Monthly Reset Flow

flowchart TB
    subgraph "Trigger"
        CRON["Cron Job<br/>0 0 1 * * (1st of month, 00:00 UTC)"]
    end

    subgraph "Pre-check"
        CHECK["Pre-reset Verification<br/>- Backup current leaderboard?<br/>- Archive to S3?<br/>- New key name ready?"]
    end

    subgraph "Archive"
        ARCHIVE["Archive Old Leaderboard<br/>1. Export lb:2026-02 to S3 (JSON)<br/>2. Save top 100 to MySQL<br/>3. Generate season report"]
    end

    subgraph "Switch"
        SWITCH["Atomic Switch<br/>1. Set current_month = 2026-03<br/>2. New writes → lb:2026-03<br/>3. EXPIRE lb:2026-02 7776000"]
    end

    subgraph "Verify"
        VERIFY["Post-reset Verification<br/>- ZCARD lb:2026-03 = 0?<br/>- New scores go to new key?<br/>- Old key still readable?<br/>- API returns correct data?"]
    end

    subgraph "Notify"
        NOTIFY["Notifications<br/>- Push: 'New season started!'<br/>- Email: Season 2 summary<br/>- In-game: Season rewards"]
    end

    CRON --> CHECK
    CHECK -->|"OK"| ARCHIVE
    ARCHIVE --> SWITCH
    SWITCH --> VERIFY
    VERIFY -->|"Pass"| NOTIFY
    VERIFY -->|"Fail"| ROLLBACK["Rollback<br/>Switch back to old key<br/>Alert on-call engineer"]

    style SWITCH fill:#e53935,color:#fff
    style ROLLBACK fill:#757575,color:#fff

---

7. Aha Moments & Pitfalls

7.1 Aha Moments — Nhung insight quan trong nhat

Insight #1: Redis Sorted Set la THE Answer cho Leaderboard

“Khong co data structure nao khac phu hop hon Redis Sorted Set cho bai toan leaderboard. No giong nhu data structure duoc sinh ra de giai quyet van de nay.”

Moi operation cua leaderboard deu map 1:1 voi 1 Redis command:

• Cap nhat diem → ZINCRBY
• Top 10 → ZREVRANGE
• Rank cua toi → ZREVRANK
• Diem cua toi → ZSCORE
• Nguoi quanh toi → ZREVRANGE voi offset
• Tong so nguoi choi → ZCARD

Tat ca deu O(log N). Khong can viet them logic. Khong can custom data structure. Redis Sorted Set la complete solution.

Bai hoc cho Hieu: Trong system design interview, viec chon dung data structure co the bien bai toan phuc tap thanh bai toan don gian. Khi thay bai toan leaderboard/ranking, nghi ngay den Redis Sorted Set.

Insight #2: Skip List la Data Structure “Elegant”

“Skip list dat O(log N) chi bang random — khong can rotation, khong can rebalance. Elegant nhu mot bai tho.”

So sanh	Balanced BST	Skip List
Insert	Rotate/rebalance (phuc tap)	Flip coin de chon level (don gian)
Delete	Rotate/rebalance	Update pointers (don gian)
Range query	In-order traversal (pointer chasing)	Follow linked list (cache-friendly)
Concurrent	Can lock toan bo subtree	Lock-free algorithms de implement
Implementation	200+ dong code	100 dong code

Skip list la minh chung rang don gian co the manh me. Antirez chon skip list cho Redis vi no de implement, de debug, va du nhanh.

Bai hoc cho Hieu: Khong phai luc nao data structure “hoc thuat” nhat (Red-Black Tree, B+ Tree) cung la lua chon tot nhat. Doi khi data structure don gian hon (skip list, hash table) lai thuc te hon vi de implement, de debug, va de maintain.

Insight #3: Sharding Leaderboard kho hon nhieu so voi Single Node

“Single Redis node giai quyet bai toan leaderboard trong 5 phut. Sharded leaderboard mat 5 ngay.”

Thao tac	Single Node	Sharded
Top 10	1 command	N commands + merge
User rank	1 command	N commands + sum
Update score	1 command	1 command (nhung can routing)
Monthly reset	1 key	N keys
Consistency	Guaranteed	Eventual (cross-shard)

Sharding bien moi operation tu 1 buoc thanh N buoc. Complexity tang tuyen tinh voi so shards. Day la ly do tai sao luon bat dau voi single node va chi shard khi thuc su can.

Bai hoc cho Hieu: Trong interview, dung nhay vao sharding ngay. Trinh bay single node truoc, lam estimation de chung minh single node du (hoac khong du), roi moi thao luan sharding. Interviewer muon thay em hieu khi nao can shard, khong chi cach shard.

Insight #4: Estimation Quyet Dinh Kien Truc

“290 writes/sec va 3.5GB memory — nhung con so nay cho thay single Redis node la qua du. Estimation cuu em khoi over-engineering.”

Neu khong lam estimation:

• Co the them Kafka “cho an toan” → tang complexity khong can thiet
• Co the shard Redis “phong truong hop” → tang cost va complexity
• Co the dung distributed cache → overkill

Estimation cho thay:

• Redis xu ly 100K+ ops/sec, ta chi can 3K → 3% capacity
• Redis ho tro hang chuc GB, ta chi can 3.5GB → 10% capacity
• Khong can lam gi phuc tap — single node + replication la du

Bai hoc cho Hieu: Luon lam estimation truoc khi quyet dinh kien truc. Con so khong noi doi. Tham chieu Tuan-02-Back-of-the-envelope cho cac ky thuat estimation.

Insight #5: Tie-breaking Can Thiet Ke Can Than

“2 nguoi cung 42 diem — ai xep truoc? Cau hoi tuong don gian nhung giai phap (composite score) cho thay do sau cua thiet ke.”

Composite score (score * 10^13 + (MAX_TIMESTAMP - timestamp)) la ky thuat packing 2 values into 1 number. Day la pattern reusable cho nhieu bai toan:

• Leaderboard: score + time
• Task priority: priority level + creation time
• Version ordering: major version * 1000 + minor version

Bai hoc cho Hieu: Khi data structure chi ho tro sort theo 1 field, hay nghi den composite key/composite score. Day la trick nho nhung huu ich trong nhieu tinh huong thuc te.

7.2 Pitfalls — Nhung cai bay thuong gap

Pitfall #1: Dung SQL cho real-time ranking

Sai: SELECT COUNT(*) FROM scores WHERE score > ? cho moi rank query. Dung: Redis Sorted Set — ZREVRANK tra ve rank trong O(log N).

Ly do: SQL COUNT(*) la O(n) scan. Voi 25M rows, moi query mat vai giay. Nhan voi 50M queries/ngay = database chet.

Pitfall #2: Client gui score truc tiep

Sai: Client gui API request “set my score to 100”. Dung: Game server tinh toan score va gui ket qua tu server-side.

Ly do: Client co the bi hack, modified, hoac intercept. Bat ky data nao tu client deu khong dang tin cay. Never trust the client.

Pitfall #3: Over-engineer voi sharding tu dau

Sai: “25M users — can shard Redis thanh 10 nodes!” Dung: Lam estimation truoc. 25M entries × 140 bytes = 3.5GB. 1 Redis node (64GB) du suc.

Ly do: Sharding tang complexity O(N) cho moi operation. Chi shard khi estimation chung minh single node khong du.

Pitfall #4: Khong xu ly tie-breaking

Sai: 2 nguoi cung 42 diem — Redis tra ve thu tu ngau nhien. Dung: Composite score dam bao thu tu deterministic.

Ly do: Redis Sorted Set sap xep member co cung score theo lexicographic order cua member name. Dieu nay khong cong bang va khong predictable. Composite score giai quyet van de nay.

Pitfall #5: Dung DEL de reset leaderboard

Sai: DEL leaderboard:2026-02 (xoa toan bo Sorted Set). Dung: Tao key moi leaderboard:2026-03, de key cu tu het han bang TTL.

Ly do: DEL tren Sorted Set 25M entries la O(N) operation — block Redis hang giay. Trong thoi gian do, moi operation khac bi queue lai. Dung UNLINK (async delete) neu bat buoc phai xoa, hoac tot hon — dung TTL.

Pitfall #6: Khong cache top K

Sai: Moi request “Get Top 10” deu query Redis. Dung: Cache top 10 voi TTL 5 giay.

Ly do: Top 10 la hot data — hang trieu nguoi xem. Du Redis nhanh, hang trieu concurrent requests van tao load khong can thiet. Cache 5 giay giam 90% load ma data chi cu toi da 5 giay.

Pitfall #7: Quen archive leaderboard cu truoc khi reset

Sai: Reset leaderboard ma khong luu ket qua thang truoc. Dung: Archive ra persistent storage (S3, database) truoc khi set TTL cho key cu.

Ly do: Nguoi choi muon xem lai ket qua thang truoc. Rewards, badges, certificates deu can du lieu lich su. Mat data lich su = mat long tin nguoi choi.

---

8. Internal Links — Lien ket voi cac tuan khac

Tuan	Lien ket	Ap dung trong Gaming Leaderboard
Tuan-06-Cache-Strategy	Cache strategy	Cache top K leaderboard voi short TTL (5s) de giam load len Redis. Application-level cache cho hot data. Cache invalidation strategy cho leaderboard data.
Tuan-09-Rate-Limiter	Rate Limiting	Rate limit score update endpoints (10 req/phut/user) va leaderboard view endpoints (30 req/phut/user). Ngan abuse va DDoS. Token Bucket hoac Sliding Window algorithm.
Tuan-10-Consistent-Hashing	Consistent Hashing	Khi shard leaderboard, consistent hashing giup phan phoi users deu giua cac shards va giam data migration khi them/bot shard.
Tuan-02-Back-of-the-envelope	Estimation	Estimation cho QPS, memory, bandwidth — chung minh single Redis node du cho 25M users. Estimation la co so de quyet dinh kien truc (single vs sharded).
Tuan-07-Database-Sharding-Replication	Sharding & Replication	Redis replication cho high availability (1 master + 2 replicas). Sharding strategy khi vuot qua single node. Score range partitioning vs hash partitioning.
Tuan-08-Message-Queue	Message Queue	Kafka cho near-real-time approach (batch score updates). Event streaming cho analytics va archive. Decouple game service va leaderboard service.
Tuan-01-Scale-From-Zero-To-Millions	Scaling fundamentals	Bat dau voi single Redis node, scale len replication, roi moi sharding. Vertical scaling (bigger Redis) truoc horizontal scaling (more shards).

---

9. Glossary — Tu dien thuat ngu

Thuat ngu	Tieng Viet	Mo ta
Leaderboard	Bang xep hang	Danh sach nguoi choi sap xep theo diem
Sorted Set	Tap hop co thu tu	Data structure cua Redis luu (member, score) da sap xep
Skip List	Danh sach nhay	Probabilistic data structure — multi-level linked list
ZADD	—	Redis command them/cap nhat member trong Sorted Set
ZINCRBY	—	Redis command tang score cua member
ZREVRANGE	—	Redis command lay members theo thu tu score giam dan
ZREVRANK	—	Redis command lay rank cua member (score giam dan, 0-indexed)
ZSCORE	—	Redis command lay score cua member
ZCARD	—	Redis command dem tong so members trong Sorted Set
Composite Score	Diem ket hop	Ky thuat pack nhieu gia tri (score + timestamp) vao 1 number
Tie-breaking	Xu ly trung diem	Quy tac xep hang khi 2 nguoi cung diem
Scatter-Gather	Phan tan — thu thap	Pattern gui request den nhieu shards roi gop ket qua
DAU	Nguoi dung hoat dong hang ngay	Daily Active Users
QPS	Truy van moi giay	Queries Per Second
TTL	Thoi gian song	Time To Live — thoi gian key ton tai truoc khi bi xoa
Sentinel	Linh canh	Redis Sentinel — he thong giam sat va auto-failover
RDB	—	Redis Database Backup — snapshot toan bo du lieu
AOF	—	Append-Only File — log moi write command
Sybil Attack	Tan cong Sybil	Tao nhieu tai khoan gia de thao tung he thong

---

10. Tong ket — Nhung dieu Hieu can nho

Top 5 Takeaways

1. Redis Sorted Set la THE answer cho leaderboard — Moi operation (add, rank, top K, score) deu co san va deu O(log N). Khong can custom data structure, khong can phuc tap.

2. Skip list la data structure elegant — Dat O(log N) chi bang random (coin flip). Don gian hon balanced BST, than thien voi concurrent access, de implement va debug. Redis chon skip list la co ly do.

3. Estimation quyet dinh kien truc — 290 writes/sec, 3.5GB memory → single Redis node du thoai mai. Khong can Kafka, khong can sharding, khong can over-engineering. Lam estimation truoc, thiet ke sau.

4. Sharding leaderboard kho hon rat nhieu so voi single node — Top K can scatter-gather, rank can sum across shards, consistency kho dam bao. Luon uu tien single node. Chi shard khi estimation chung minh bat buoc.

5. Tie-breaking can thiet ke tu dau — Composite score (score * 10^13 + MAX_TIMESTAMP - timestamp) dam bao thu tu deterministic khi 2 nguoi cung diem. Day la chi tiet nho nhung quan trong.

So sanh Gaming Leaderboard voi cac he thong khac

Khia canh	Leaderboard	Stock Exchange	Payment System	Chat System
Data structure chinh	Redis Sorted Set	In-memory Order Book	Database + Ledger	Message Queue + DB
Latency target	< 100ms	< 100 microseconds	< 1 giay	< 500ms
Consistency	Eventually consistent	Strict ordering	Strong consistency	Eventually consistent
Scaling strategy	Single node → Shard	Vertical (1 node)	Horizontal (microservices)	Horizontal (shard by user)
Data structure insight	Skip list	FIFO queue per price level	Double-entry ledger	Append-only message log
Core challenge	O(log N) rank query at scale	Microsecond matching	Exactly-once payment	Message ordering + delivery

Cau hoi tu kiem tra cho Hieu

1. Tai sao SQL COUNT(*) WHERE score > X khong scale cho leaderboard?
2. Redis Sorted Set dung data structure gi ben trong? Tai sao khong dung Red-Black Tree?
3. Giai thich cach composite score xu ly tie-breaking. Tai sao dung MAX_TIMESTAMP - timestamp thay vi chi timestamp?
4. Khi nao nen dung real-time (direct Redis write) va khi nao nen dung near-real-time (Kafka → Redis)?
5. Tai sao nen bat dau voi single Redis node thay vi sharding tu dau?
6. Giai thich scatter-gather pattern cho sharded leaderboard top K query.
7. DEL tren Sorted Set 25M entries co van de gi? Giai phap thay the la gi?
8. Lam estimation: voi 100M DAU va 500M scores/ngay, co can shard Redis khong? Memory va QPS la bao nhieu?

---

“Bai toan leaderboard day Hieu mot bai hoc quan trong: chon dung data structure co the bien bai toan O(n) thanh O(log n), bien he thong khong chay duoc thanh he thong chay muot ma. Redis Sorted Set khong chi la tool — no la minh chung cho suc manh cua computer science fundamentals.”