![Page 1: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/1.jpg)
Peer-to-Peer Video Systems: Storage
ManagementCS587x Lecture
Department of Computer ScienceIowa State University
![Page 2: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/2.jpg)
OutlineIntroduction on P2P video systems What and whyResearch issues Storage management Service scheduling
![Page 3: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/3.jpg)
P2P Video SystemsDefinition A P2P video system is a logical system
consisting a number of hosts collaborating for the purpose of video services
Features Resource sharing
A host can be both a service requester and a service provider
There is no central server and no host guarantees the existence of any video a host can introduce a video to the system and
then delete it quickly The participation is fully volunteer
A seeding host might quit the service at a later time
![Page 4: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/4.jpg)
Why not ALMALM relies on client forwarding to improve the scalability of the original server
High demand on client bandwidth In ALM, a client needs to have a
bandwidth at least 2x of the video playback rate
In P2P video systems, a client can download a video for its own playback first and then provide it back to serve others
Limited client participation In ALM, a client can contribute its
resource only when it is being served
In P2P video systems, a client can become a service provider of a
video by keeping a copy of the video can participate in video services
even if it does not request video itself
s
c1 c3
c4c2 c5
c6
![Page 5: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/5.jpg)
Building a P2P Video System
A P2P video system can be built on top of a P2P file sharing system
by extending its functionality to support video servicesChoosing a P2P file sharing system
Napster-type? Single point of failure, scalability issue, etc..
Gnutella-like? Great for its simplicity, but searching cost could be high
Freenet-like? Great for its search efficiency, but such efficiency is
achieved by proactive replication, which may be inappropriate
Replicating a video over its query path may dramatically increase service latency
Streaming a video through its query path is untenable
![Page 6: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/6.jpg)
A hybrid of Gnutella and Freenet
Adopt Freenet’s routing-table and depth-first search strategy, but replicate only the file pointer
Why does Freenet replicate a file, not its pointer? What are the drawbacks of using routing-table?
Adopt Gnutella’s download mechanism When a client find a video, it downloads the file from
the source directlyA B
CDE
(V, E)
(V, E)
![Page 7: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/7.jpg)
Video Management
Why this becomes a problem Videos are large
Statistics show that a majority of clients contribute less than 1 GB of disk space
streaming out of a video takes a substantial communication bandwidth This seems to be more concerned by users
If we allow a client to cache a video in its all or not at all, we may end up only a few clients can contribute This will again create the problem of server
bottleneck
![Page 8: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/8.jpg)
Challenges of Caching Partial Videos
Caching collaboration Given a fixed amount of disk space, what
part of a video should a client cache? How can we find a set of video segments that
are cached by many different hosts?
![Page 9: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/9.jpg)
Challenges of Caching Partial Videos
Caching collaboration Given a fixed amount of disk space, what
part of a video should a client cache? How can we find a set of video segments that
are cached by many different hosts?Cache allocation Should a client cache a newly download
video? If yes, how much cache space should be
allocated for this video? If the cache is full, what existing data should
be expunged to make room?
![Page 10: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/10.jpg)
Caching Collaboration: CellCell concept
A cell is a cluster of hosts that together can supply a complete video Each host in a cell caches a number of video segments A CacheTable is used to track who caches what
H1
S1
H2
H3 H5
H4
H6
cell1
H7
caching host
File segmentsnon-caching host
CacheTable for cell2
cell2
S2 S3 S4
S1 S3 S4
S1
S2 S4
H2 S1
H3 S3H3 S4H5 S2H5 S4
Host Segment
H3 S1
![Page 11: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/11.jpg)
Cell AdvantagesReducing cache redundancy Caching and deletion of file segments can be
coordinated at cell level The cost of such caching collaboration can be
minimal since each cell typically contain only a small number of member A cell is split whenever possible
Reducing search cost Looking for a set of video segments now
becomes looking for any single segment Search scope is dramatically reduced: determined
by the distance to the nearest caching host instead of the farthest caching host
![Page 12: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/12.jpg)
Cache OperationCache procedure Retrieve CacheTable from a cell that provides
service Cache the least redundant segments in
CacheTable Modify CacheTable and update other cell
members
![Page 13: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/13.jpg)
Split OperationWhy split Keep the cell size small and reduce
maintenance overheadExhaustive search is impractical To split k hosts into two separate groups,
there are O(2k) different combinations Proposed split algorithm Can split a cell whenever possible Takes only O(k2) computation
![Page 14: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/14.jpg)
Split OperationSplit procedure
Put all members in OldSet and create an empty NewSet
Find a host in OldSet and move it to NewSet which satisfies two conditions: Without this host OldSet can still supply a complete
file This host can provide maximum number of
segments that are currently not in NewSet Repeat this procedure until no such host can be found If NewSet can supply a complete file then the original
cell can be split into two cells
![Page 15: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/15.jpg)
Split Example
cell1
S3
H3
S1 S4
CacheTable for cell2
H3 S1
H3 S4H4 S2H4 S4
Host Segment
H3 S3
CacheTable for cell1
H1 S2
H1 S4H2 S1
Host Segment
H1 S3
H1
S2 S3 S4 H2
S1
cell2
H1
S2 S3 S4
H2
S1
S3
H3
S1 S4
S4
H4
S2
S4
H4
S2
CacheTable for cell1
H1 S2
H1 S4H2 S1
Host Segment
H1 S3
H3 S1
H3S2
H4 S4
H3 S3
H4S4
cell1
Split
![Page 16: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/16.jpg)
Delete OperationDeletion happens when a host needs to recycle its cache spaceDelete procedure Delete most redundant segments in CacheTable Modify CacheTable and update other cell members
![Page 17: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/17.jpg)
Merge OperationMerge happens when a cell is broken Cell members delete non-redundant segments Cell members found off-line
Merge procedure Find a list of cells For each member in the broken cell, select a cell
from the list and add it into this cell Split the resulting cell if possible
Select procedure A host should join a cell whose least redundant
segments can be maximally reinforced
![Page 18: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/18.jpg)
Implementation IssuesCacheTable consistency
Using mutual exclusion mechanism such as majority quorum algorithm to avoid concurrent updates on CacheTable
Using optimistic concurrency control mechanism. When concurrent Delete operations happen and a cell is broken, Merge operation will be invoked to reorganize the cell memebers
Cell data redundancy control Increasing the data redundancy level in a cell to avoid
frequent Split and Merge operations
![Page 19: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/19.jpg)
Cache AllocationGiven a newly downloaded video, should it be cached? If yes, three questions remain: How much cache space should be allocated
for the new file Which part of the file should cached
This is solved by Cell technique If necessary, what data in the cache should
be expunged to make space for the new file
![Page 20: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/20.jpg)
Traditional SchemesLeast Recently Used (LRU) and Least Frequently Used (LFU) Keep popular data items in the cache to
reduce the service cost of popular items Unpopular files might be quickly deleted Suitable for systems with a central server but
might cause unpopular files to vanish in a decentralized P2P system
![Page 21: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/21.jpg)
Uniform and Proportional Allocation
Uniform allocation Maintains equal number of copies in the
system for all files Protects unpopular files from being extinct
but suffers from load balancing problemProportional allocation Allocates cache space based on the file
popularity Achieves optimal load balancing but
unpopular files might become extinct quickly
![Page 22: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/22.jpg)
Controlled Inverse Proportional (CIP) Cache Allocation
1. Determine if a newly downloaded file should be cached
2. Calculate the amount of cache space to be allocated for the new file
Try to allocate equal cache space for all videos3. If there is no empty space for a new file, some
redundant segments of cached videos are deleted
Look at the CacheTable of each video4. If there are no redundant segments, a cached
video will be deleted to make room for the new video segments
Which segments should be cached is determined by Cell
![Page 23: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/23.jpg)
CIP-HA heuristic solution Compare the popularity of the new video with
the popularity of each cached video Among all cached videos that have higher
popularity than that of the new video, choose the one with the lowest popularity to replace
Advantages Files with lower popularity then the new video
will be retained Popular videos are replaced dynamically and
most popular videos are also retained
![Page 24: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/24.jpg)
CIP-αA solution based on analysis model Define Caching Probability of video i as
Pi=(pmin/pi)a pmin: the popularity of the least popular video pi: the popularity of video i α : an adjustable parameter
By adjusting α, the caching effect of CIP-α can vary between uniform and proportional allocation When α= 0, Pi =1, CIP-α becomes proportional allocation When α= 1, Pi=pmin/pi, CIP-α becomes uniform allocation
![Page 25: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/25.jpg)
Performance MetricsAccessibility Minimum number of hops that a query
message needs to be delivered in order to find a complete file
Measures the cost of searching a large fileAvailability Number of distinct copies of the file in the
whole system Measures the cache space allocated for a file
in the system
![Page 26: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/26.jpg)
Schemes for ComparisonCaching collaboration Cell vs. Random
Random: a host randomly chooses segments to cache after downloading a video
Cache allocation CIP-h vs. CIP-a vs. Uniform vs. Proportional
Uniform: all videos are allocated equal cache space in the whole system
Proportional: the total cache space allocated to a video in the whole system is proportional to its popularity
![Page 27: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/27.jpg)
Simulation SetupSimulated in a Gnutella-like P2P network of 5000 hosts with the average distance between two hosts of 7 hopsThe video popularity follows Zipf-like distribution with skew set to 1The cache space among hosts also follows Zipf-like distribution with skew set to 1100 videos are shared. In the beginning each file has one seeding host. Then hosts begin to download files and cache files. When the network caching capacity has been sufficiently utilized we collect the performance data
![Page 28: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/28.jpg)
Effect of Cell on Video Accessibility
Results in Cell and Random show some similaritiesThe file accessibility in Cell is much better than that in Random
![Page 29: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/29.jpg)
Effect of CIP on Video Availability
CIP provides balance between Uniform and ProportionalCIP-αprovides more fine tuned control over the file availability
![Page 30: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/30.jpg)
Effect of Cell/CIP on Video Accessibility
In all cases the accessibility in Cell/CIP is better than that in Random/Uniform and Random/Proportional
![Page 31: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/31.jpg)
Effect of Cell/CIP on Video Availability
The curves of Cell/CIP fall between Random/Uniform and Random/ProportionalAdjusting αcan change CIP- αbetween Uniform and Proportional
![Page 32: Peer-to-Peer Video Systems: Storage Management CS587x Lecture Department of Computer Science Iowa State University](https://reader034.vdocuments.site/reader034/viewer/2022052515/5a4d1b5a7f8b9ab0599aa9fb/html5/thumbnails/32.jpg)
ConclusionsCaching collaboration
Cell caching can effectively balance the data redundancy and reduce the search cost for large files
Cache allocation CIP cache allocation can achieve loading balancing and
maintaining unpopular files in the system