splunklive london 2014 developer presentation
DESCRIPTION
Developer Presentation by Jon Rooney and Damien Dallimore at SplunkLive London 2014TRANSCRIPT
![Page 1: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Developer PlatformJon RooneyDirector of Developer Marketing
Damien DallimoreDeveloper Evangelist
![Page 2: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/2.jpg)
Who is Jon Rooney?Director of Developer Marketing Splunker since April 2012 Based in Splunk’s Seattle OfficeVeteran of Microsoft, start-ups, Accenture
2
![Page 3: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/3.jpg)
What Does Splunk Have to Do with Developers?
3
![Page 4: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/4.jpg)
Application Development Challenges
4
Build DeploymentServer
QA / Test
Staging Environment
Continuous Integration / Build Server
SourceRepository
Task Tracking
Local Build
![Page 5: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/5.jpg)
Application Development Challenges
5
Build DeploymentServer
QA / Test
Staging Environment
Continuous Integration / Build Server
SourceRepository
Task Tracking
Local Build
Lack of visibility across the product development lifecycle
Slows down the ability to detect and troubleshoot errors
Limited visibility into application usage and performance
![Page 6: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/6.jpg)
6
CI / Build Servers
Project and Issue Tracking
Code Repository
QA / Testing Tools
Splunk can solve these problems
Deployment Servers
![Page 7: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/7.jpg)
Splunk for Application Lifecycle Intelligence
7
Reduce Time to Market
Resolve issues faster
Gain Agility
Improve Code Quality and Lifecycle Visibility
Generate Application Insights
Instrument Analytics
![Page 8: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/8.jpg)
Demo
![Page 9: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/9.jpg)
Customer Successes
9
![Page 10: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/10.jpg)
Touring the Splunk Developer Platform
![Page 11: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/11.jpg)
Evolving the Splunk Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other Content
SDKs
Operational Intelligence Platform
Content
Core Engine
User and Developer Interfaces
Web Framework
REST API
![Page 12: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/12.jpg)
Powerful Platform for Enterprise Developers
12
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web Framework
JavaJavaScriptPython
RubyC#PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
![Page 13: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/13.jpg)
Splunk Web Framework
13
Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js
Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey
charts, heat maps and bubble charts - SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web Framework
![Page 14: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/14.jpg)
Log directly to Splunk via TCP,
UDP, HTTP
Integrate search results with other applications using
custom visualizations
Create and run searches from
other applications
The REST API and SDKs
14
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index
![Page 15: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/15.jpg)
Let’s dig a bit deeper
![Page 16: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/16.jpg)
Who is Damien Dallimore
16
Worldwide Developer Evangelist @ SplunkI developI talk about developingHelping to build the Splunk developer ecosystemCame from the Splunk CommunityOnce was a customerCoder
![Page 17: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/17.jpg)
I develop
17
All 100% free and open sourced , published to Splunk Apps , source code on Github
![Page 18: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/18.jpg)
An Open Platform for Developers
18
Splunk is an open and extensible platform at numerous different touchpoints for developers
Extensibility creates ecosystems
![Page 19: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/19.jpg)
A Developer’s Smorgasboard
19
CLIREST API6 language SDKsSplunkbase Apps / Add-onsCustom search commandsScripted InputsModular InputsThe Web Framework Standard HTML/Javascript/CSSData Models
External scripted lookupsCustom REST EndpointsTools , utilitys and librarysIntegrations with other software frameworks (Spring)Hadoop dev with HUNKCustom user interfaces / visualizationsMobile with BugsenseCustom data connectors (ODBC)Custom authentication handlers
![Page 20: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/20.jpg)
Which Splunk Product for Devs ?
20
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.
Splunk> AMIs : BYOL versions for Amazon AWS Cloud.
Hunk> : Splunk for data in Hadoop clusters. Same platform, same easy apps.
Splunk> Storm : Free to use, cloud hosted. 20GB/30days.
![Page 21: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/21.jpg)
Modular Inputs
![Page 22: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/22.jpg)
22
Modular Inputs• Extend the Splunk framework to define a custom input capability, just like the standard inputs you are
familiar with (TCP/UDP/File etc…)
• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally integrated first class citizen objects in Splunk
• Users interactively create and update your custom inputs using Splunk manager, just as they do for native inputs.
• When deploying without a UI , you push out the inputs.conf file.
• All the properties are fully manageable via the REST API
• Version 5.0 +
![Page 23: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/23.jpg)
23
![Page 24: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/24.jpg)
Developing
24
• My preference is to use Python, however any language can be used.
• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
• There is a certain amount of “plumbing” to put in place , so I like utilities that take care of this for you, so you can just focus on the business logic.
• Java,Python,C# SDKs also have Modular Input APIs
• Eclipse plugin has a wizard for creating Modular Inputs in Java
![Page 25: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/25.jpg)
REST : The Data Potential
25
Twitter FoursquareLinkedIn Facebook Fitbit Amazon Yahoo Reddit YouTube Flickr Wikipedia GNIP Box
Okta Datasift Google APIs Weather Services Seismic monitoringPublicly available socio-economic dataTraffic data Stock monitoring Security service providers Proprietary systems and platforms Other “data related” software productsThe REST “dataverse” is vast , but I think you get the point.
There is a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and where your data might take you.
![Page 26: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/26.jpg)
You are only limited by your own “data imagination”
26
![Page 27: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/27.jpg)
Demo
![Page 28: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/28.jpg)
Custom Search Commands
![Page 29: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/29.jpg)
Splunk Search
29
Splunk’s search and querying language is called SPLAllows you to search, analyze and manipulate your data.Designed with the “unix pipeline” in mind – a “search pipeline”
From a (simplistic) mental point of view:– Series of commands– Each commands takes the input of the previous one– Each command outputs a sparse table
![Page 30: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/30.jpg)
Splunk Search - Example 1:
30
“index=_internal foo | eval bar=... | stats count by bar”Initial command fetches result from index/raw data store, and outputs a table, with a row for each event, and a column for each field (not all rows have all columns)Second command adds a new column to each row, “bar”Third command looks at all the unique values of “bar”, and counts how many rows each value has.Note that the last command completely transformed the table, hence it is a “transforming” or “non-streaming” command.The second command was merely additive, known as a “streaming” command.You can also have “generating” commands ie: inputlookup
![Page 31: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/31.jpg)
Custom Search Commands
31
Just like the “eval” or “stats” commands, you can write your own “search commands”.Python scripts which take data on stdin, and output data on stdoutData comes in/out as CSV (with special handling for MV fields)Many of Splunk’s builtin commands are written as Python scripts (e.g. head, return, transpose)
![Page 32: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/32.jpg)
Custom Search Commands: Building Blocks
32
Custom search commands are built in two parts:
A Python script containing the implementation
An entry in commands.conf declaring configuring the command
Splunk SDK for Python has librarys and examples for creating custom search commands
![Page 33: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/33.jpg)
Source Code !
![Page 34: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/34.jpg)
Web Framework
![Page 35: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/35.jpg)
Splunk Web Framework
35
Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js
Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey
charts, heat maps and bubble charts - SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web Framework
![Page 36: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/36.jpg)
Splunk JS Stack & Django Bindings Concepts
36
Managers• Search Query Wrappers• SearchManager and SaveSearch
Manager• Include Search Parameters• Available within Django or
JavaScript
![Page 37: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/37.jpg)
Splunk JS Stack & Django Bindings Concepts
37
Splunk Views• UI widgets• Designed to work with Search
Managers• Charts, Maps, Inputs, Timeline,
TimePicker, etc.
![Page 38: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/38.jpg)
Splunk JS Stack & Django Bindings Concepts
38
URL Maps & Django View• URL Maps enable custom routes• Django Views, enable custom logic
to provide to templates
![Page 39: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/39.jpg)
Splunk JS Stack & Django Bindings Concepts
39
Templates & Template Tags• Templates enable quick layout
options • Templates support inheritance of
other templates• Tags call a Python function can be
used for:– Text manipulation– Flow control– Load external information– … and more
![Page 40: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/40.jpg)
Splunk JS Stack & Django Bindings Concepts
40
Data Binding using Tokens• Token based data-binding
mechanism that keep shared data in sync.
• Enables in-page interactivity
![Page 41: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/41.jpg)
Web Framework Toolkit App
41
• Improved productivity for developing your own Web Framework based apps
• Reusable Visualization• Packaged to lower the effort of adding cool
visualizations• Improved Developer focused Command Line
Tools• Getting started templates• Automates common tasks
• Includes examples pages demonstrating advanced concepts.
![Page 42: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/42.jpg)
SimpleXML converted to HTML
42
Transition a Simple XML page to HTML/JS Option to overwrite/edit existing dashboard or create newFull Splunk JS Stack is available (Add custom viz, tokens, etc.)Does not include Django backendNo support for visual dashboard editor or PDF printing
![Page 43: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/43.jpg)
Simple XML with JS Import
43
![Page 44: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/44.jpg)
Splunk 6 Dashboard Examples App
44
![Page 45: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/45.jpg)
Custom Visualizations
![Page 46: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/46.jpg)
![Page 47: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/47.jpg)
![Page 48: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/48.jpg)
![Page 49: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/49.jpg)
![Page 50: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/50.jpg)
50
My Guiding Viz PrincipleThe visualization must be simple and intuitive to understand and derive meaning from at a glance.
Cool viz , but what are you telling me ?
![Page 51: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/51.jpg)
So many options , which one for me ?Splunk Web Form Editor– If you are not a coder , not familiar with Simple XML
Edit Simple XML– Familiar with Simple XML , what to customize more
Convert Simple XML to HTML/JS – Coder , want to see the underlying JS/HTML , want custom UI behavior above Simple
XML, want to use some other JS/CSS
Simple XML JS Import– Closely related to the above , perhaps you still want dashboard editing / PDF export
Django– Previous benefits + want to leverage Django tags , want custom server side processing in
Django views51
![Page 52: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/52.jpg)
Splunk 6.1 Features For Building Apps
52
![Page 53: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/53.jpg)
Mobile Developers
![Page 54: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/54.jpg)
Company Overview
The right toolset for analyzing and troubleshooting mobile apps in real-time
54
Deliver high quality, engaging apps.Splunk BugSense
![Page 55: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/55.jpg)
55
Capabilities
Mobile Data
HTML5 dashboard• Actionable reports• Easy to use
How It Works
Cross Platform SDKs • Install in < 5 min• Crashes/Errors• Events• Sessions• Transactions
Bug Sense Cloud
• Highly Scalable• Secure• Cloud Service• Highly available• Cost-effective
![Page 56: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/56.jpg)
Integration (Android)1. Add the .jar (download or use Maven/Gradle) & import
2. Add the permissions
3. Initialize:
![Page 57: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/57.jpg)
57
![Page 58: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/58.jpg)
58
![Page 59: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/59.jpg)
59
![Page 60: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/60.jpg)
Why Develop
![Page 61: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/61.jpg)
WHY should you develop ?
61
Make money , Promote your company, Make sales !Community and collaborationShare / Give BackGet a job / Build a careerLearn new skills / Educate yourself and othersHadoop productivityDo goodOpen up new data sources for others to collaborate on
We talk a lot about the how , what , where and who ….. but what about the WHY
![Page 62: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/62.jpg)
apps.splunk.com
62
![Page 63: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/63.jpg)
Wrap Up and Questions
![Page 64: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/64.jpg)
Splunk Developer License
64
![Page 65: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/65.jpg)
Where to go for More InfoTutorials, Code Samples, Getting Started, Downloads– http://dev.splunk.com/
Splunk Apps– https://apps.splunk.com
GitHub– https://github.com/splunk/
Twitter– https://twitter.com/splunkdev
Blogs– http://blogs.splunk.com/dev/
65
![Page 66: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/66.jpg)
The 5th Annual Splunk WWUC
• 50+ Customer speakers• 30+ Apps in Splunk Apps
Showcase• 30+ Technology Partners• Ask The Experts• Sales Meetings• Business Value ROI booth
conf.splunk.com
Las Vegas: Oct 6-9, 2014 The MGM Grand Hotel4000+ IT and Business Professionals3 days of content, 130+ sessions3 days of Splunk UniversityGet Certified!
![Page 67: SplunkLive London 2014 Developer Presentation](https://reader034.vdocuments.site/reader034/viewer/2022052222/53ed9a338d7f7289708b5da3/html5/thumbnails/67.jpg)
Thank you