Python Experience in KKBOX Group

Download Python Experience in KKBOX Group

Post on 13-Apr-2017

5.554 views

Category:

Technology

8 download

TRANSCRIPT

  • Python Experience in KKBOX Group

    Shuen-Huei (Drake) Guan

    #Taipei.py (It was originally planned for PyCon TW)

    04/21/2016

  • Lets conduct some surveys first.pyKKBOX? pyUIA? Mass?

  • @drakeguan

    - Technical Director, KKStream

    - Sr. Principal Engineer, KKBOX

    - Volunteer, PyCon Taiwan

    https://twitter.com/drakeguanhttps://twitter.com/drakeguan

  • This talk is about a compilation of tasks and projects, done in Python, in KKBOX Group. Each project has its own challenges and context to deal with. What you can take home are some stories and experience.

    Warning: its just our own experience, not suitable to yours!

  • pyKKBOX, 2011

  • KKBOX

  • KKBOX

    An user can broadcast what he/she is listening to others.

    Multiple users are then able to subscribe to his/her station, listening to the

    same song simultaneously.

    A major/highlighted KKBOX feature developed in 2011.

  • Challenges by

    iOS (client) co-works with Windows (broadcaster) only.

    There is even no broadcasters to listen to!

    In academia, we may call this ill-posed (optimization) problem.

  • pyKKBOX, https://github.com/KKBOX/pyKKBOX

    https://github.com/KKBOX/pyKKBOX

  • pyKKBOX

    It is a lite-weight, not fully functional client.

    It is initially just a bot, which can login, and do broadcasting.

    It should be easy because we are iOS client team, not server team.

    Its a proof of concept.

  • API Testing by pyKKBOX

    Now we have bots: A tool to do stress testing.

    A tool to provide radio-like automatic service.

    And then, more APIs are in.

    Test cases for KKBOX API!

    Hourly testing on Jenkins CI.

  • Playlist Auto-Sync

    CPL (Cloud PlayList) synchronization.

    The sync is not done based on data (playlist), but JSON-based commands!

    In 3 months, the APIs version changed 5 times.

    n-to-m synchronization makes it even worse.

    pyKKBOX is, again, used to prototype even though technical spec is not

    finalized.

  • Toy Project: Photomosaic by pyKKBOX

  • Album covers by pyKKBOX

    Yes. Thats me.

  • Photomosaic

    Its based on KKBOX 2014 workshop group photo.

    1,500 album covers are used.

    Final photomosaic resolution: 14k x 6k.

    I didnt read the official KKBOX API yet. Just self-introspection of pyKKBOX in

    IPython is enough.

    Coding time: less than half hour.

  • pyKKBOX facts

    Initiated in 2011, to ~2013.

    Two main contributors (Welly Liu and @zonble), but with ~10 contributors then.

    Just the builtin Python modules (including PyObjC) are used.

    Another weapon for app developers;

    prototyping, or even trash-able code, is a great fit here;

    being a glue, a common language among diversified developers if youre doing

    multi-platform product development;

    Python speaks plain language.

    Hmm, we have no further plans yet. :p

  • pyUIA, 2013

  • Testing Goals & Reports

  • Testing for Playlist Auto-Sync

    Easy to manage and organize playlists across various devices.

    Multiple devices, mostly in different platforms, are involed.

  • One Test Case (iOS > Android)

    1. On iOS, create a new playlist.

    2. On Android, check if the playlist

    created in step 1 appears.

  • A (Pseudo) Test Script that Works!

  • Abstraction of Layers for Testing

  • pyUIA, https://github.com/imsardine/pyuia

    https://github.com/imsardine/pyuia

  • pyUIA, https://github.com/imsardine/pyuia

    https://github.com/imsardine/pyuia

  • pyUIA facts

    Initiated in 2014 and it keeps evolving.

    It doesnt take much time because its just simply works. v0.3 is released on

    June 2015. http://www.slideshare.net/jeremykao92/pyuia-03

    3 developers contribute their talents.

    Python packages: selenium, robotframework and Appium-Python-Client.

    Somehow, we just want to implement automated tests in Pythonic way!

    120 Release Acceptance Test (RAT) are on pyUIA, ~20% cases.

    http://www.slideshare.net/jeremykao92/pyuia-03

  • Video Encoding System, 2014

  • Video Streaming Service

    KKBOX forms a team, Video Development, to build video streaming services

    in Japan. They are Videopass and Animepass.

    More than 1,000,000 subscription users.

    More than 30,000 videos, not that many but resource-consuming.

  • Challenges for Video Encoding System

    Given a couple of thousands of videos;

    given ~150 videos per day;

    given a scalable number of encoders on EC2 instances;

    assuming the workflow for each video can be different;

    how to make a robust system to handle this challenge in consideration of just 1.5 developer(s)

    everything can be broken

    computing resource is expensive (c3.8xlarge, c4.8xlarge, )

    its a vivid world~

  • One Simplified Audio Encoding Workflow

  • Another Simplified Encoding Workflow

  • Render Farm Idea from VFX/Animation Industry

    Ed Catmull, Pixar

    Pixars Alfred

  • http://bit.ly/1rnR0Ht

    http://bit.ly/1rnR0Hthttp://bit.ly/1rnR0Ht

  • Mass, https://github.com/kkbox/mass

    https://github.com/kkbox/mass

  • Mass, https://github.com/kkbox/mass

    Job script as the central data to control the workflow, instead of fixed coding

    logics in workers.

    Focus on data, the workflow and real work, than underneath system or

    infrastructure.

    Leverage AWS SWF (or Apache Kafka, Mesos, or other message brokers) to

    provide robust, fault-tolerant and fully automatic distributed system.

    https://github.com/kkbox/mass

  • Mass facts

    Initiated in 2014, and it keeps rocking~

    Just 2 mid-level Python developers (they both have presented in PyCon TW).

    Python packages: arrow, awscli, boto, click, clint, flask, filechunkio, flask,

    fluent-logger, isodate, logbook, parmap, requests, sh.

    We hope to polish the open-sourced Mass much sexy and useful in near

    future.

    3,000,000-minute videos got encoded through it.

    Mass once finished 3,000 videos encoding within one day without any issue

    while 800 encoders on 100 c3.8xlarge instances were on duty.

  • RDC Toolbox, 2015

  • KKBOXers demands more data-driven facts/info

  • One example of request ticket from production team

  • Someone can learn from others queries!

  • Python, ,

    --- RDC Toolbox main developer

  • RDC Toolbox facts

    Initiated in Oct 2015, still quite young. 1.5 developer(s) with many contributes then. Python packages: flask, pandas, psycopg2, redis, requests, slacker, sqlparse.

    python flask pandas log slacker virtualenvpython, prototype

    10,000,000,000 metering data to deal with. 10,000 queries conducted by (mostly) non-developers. Lots of future plan. We are building music streaming specific BI tools.

  • AWS Lambda CI/CDIt would get shared in PyCon TW 2016

  • Finally, lets do some surveys again, but in KKBOX Group (just for fun).

  • What programming languages have you used to contribute your talents in KKBOX Group?

  • What programming languages are you HAPPY to work with in KKBOX Group?

  • What programming languages are you planning to learn for future challenges in KKBOX Group?

  • Thank you!