Using Cassandra for DVR Scheduling at Comcast 04/11/2023

VERSION

DATE

AUTHOR

1

Using Cassandra for DVR Scheduling at Comcast

1.0

3/20/213

John McCann | @dangermccann

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 2

The X1 Platform

More than a back button See the last nine shows and channels you've watched with a single tap.

XFINITY On DemandChoose from thousands of movies and shows.

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 3

The X1 Platform

Apps right on your TVNow your TV is for more than just watching.

The ultimate scoreboardTrack multiple games at once and check the latest scores, standings, and schedules.

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 4

The X1 Platform

• Next generation of the XFINITY TV experience• New line of set-top boxes and technologies that lets XFINITY

TV run in the cloud• Premium content viewing experience, with access to XFINITY

On Demand, DVR and Pay-Per-View video• De-coupled, service-based architecture where services are

developed and deployed independently

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 5

What Comcast Stores in Cassandra

• Customer viewing history• Resume points for DVR and XFINITY On Demand content• Device and account preferences• Pay-Per-View purchases• DVR recordings and scheduling instructions

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 6

Why Comcast Uses Cassandra

• Multi-data center (active / active)• Performance of write-heavy use cases

– DVR status updates

– Remembering resume points during content playback

• Developers (not DBAs) model and manage the data• Open source encourages in-house expertise

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 7

Redesigned DVR

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 8

DVR System Architecture

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

East Data Center

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

West Data Center

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 9

Cassandra Configuration

• 4-node rings in 2 data centers• NetworkTopologyStrategy replication strategy• Replication factor is 3 (per data center)• LOCAL_QUORUM consistency level for most operations

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 10

https://github.com/Comcast/cmb

Comcast Message Bus

• Horizontally scalable queuing and notification service

• Compatible with Amazon SQS and SNS• Cassandra used to persist messages• Redis used as caching layer• Open source (Apache license)

Cassandra

CNS Endpoint

CQS Endpoint

CNS Worker

Redis

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 11

Use Case Exploration

Scheduling a Series Recording

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 12

User Interaction

UI Server

Set-top box

User

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 13

User Interaction

UI Server

Set-top box

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Memcached

User

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 14

Recording Schedule Update

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 15

Recording Schedule Update

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Set-top box

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 16

Recording Schedule Update

Set-top box

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 17

UI Update

UI Server

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 18

UI Update

UI Server

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Memcached

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 19

System Diagram

UI Server

Set-top box

Cassandra

API Server

API Server

Worker Q

ueue

Worker

Worker

API Server Worker

Memcached

User

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 20

Data Model Analysis

1. Recording Instructions

2. Recording Schedule

3. Completed Recordings

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 21

Recording Instructions Schema

ScheduleInstruction

AccountID/DeviceID TimeUUID : [instruction] TimeUUID : [instruction] …

Composite row keyInstructions sorted by creation time using TimeUUID as the column name. The column value contains the instruction data.

• Stores the instructions used to produce the recording schedule in the order that the instructions were generated.

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 22

Recording Schedule Schema

DvrSchedule

AccountID/DeviceID

• Stores the data for the most recent recording schedule that has been delivered to the Recorder on the STB.

Properties of each recording stored as separate columns usingcomposite column names: recording_ID/[property_name]

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 23

Completed Recordings Schema

Recording

AccountID/DeviceID

• Stores the data for all of the completed recordings on the STB.

Properties of each recording stored as separate columns usingcomposite column names: recording_ID/[property_name]

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 24

• HP DL360 G8• 64GB RAM• 2 x 600 GB SATA Hard Drive• RAID 1 for OS Partition• LSI controller• 2 x Samsung 840 512GB SSD (room for up to 4 SSDs)

Hardware Configuration

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 25

Samsung 840 512GB

• 50% cost increase per node• 400% capacity increase per node• Smaller rings, better read performance• Read performance consistency (flat 99th percentiles for reads)

Solid State Drives

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 26

Lessons Learned

• If you're using Hector, pay close attention to CassandraHostConfigurator.maxActive.

• Don’t enable the row cache if you need to perform slice queries.

• Don’t delete a row if you plan to write columns to the same row later (better in 1.2).

• Don’t run Cassandra on shared storage if you can avoid it.

Using Cassandra for DVR Scheduling at Comcast 04/11/2023 27

Q&A

Thanks for your attention!

http://xfinity.comcast.net/x1

John McCann

@dangermccann