develop modern apps using spring ecosystem at time of bigdata
TRANSCRIPT
![Page 1: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/1.jpg)
Develop modern apps using Spring ecosystem at BigData era
![Page 2: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/2.jpg)
LEAD SOFTWARE ENGINEER AT EPAM SYSTEMS. PATIONATE DEVELOPER, SPEAKER, ACTIVE MEMBER OF KIEV JUG. PARTICIPATE IN DIFFERENT EDUCATIONAL INITIATIVES, ENGINEERING EVENTS AND JCP/ADOPTJSR PROGRAMS.
OLEG TSAL-TSALKO
ABOUT ME
![Page 3: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/3.jpg)
Spring for a long Mme no more than
just a Framework
![Page 4: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/4.jpg)
![Page 5: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/5.jpg)
![Page 6: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/6.jpg)
Spring Boot • General purpose plaRorm for Web app and client app development which makes complex apps development look easy
• Rapid development of diff type applicaMons based on solid Spring foundaMon
• No boilerplate code. You write only needfull/funcMonal code. Spring knows beVer how make it all work.
• Nevertheless easy to customize and extend • Simple and robust dependency management
![Page 7: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/7.jpg)
[DEMO:] Tweets Boot Data REST app 1. Basic 5 mins Boot Data Rest app which simply works! 2. Lets add logging and external properMes support 3. Lets add integraMon tests 4. Lets add Tweet lifecycle event listeners 5. Lets add Actuator monitoring support 6. Lets add much more fields to Tweet enMty 7. Lets add Tweet compact projecMon 8. Lets add validaMon 9. Lets add addiMonal stuff to Tweet HATEOUS json view 10. Lets add more custom queries 11. But beVer do not try to make Java 8 data and Mme
types work))
![Page 8: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/8.jpg)
![Page 9: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/9.jpg)
Spring XD • General purpose plaRorm for developing BigData apps for data ingesMon, data stream and batch processing, data export and data analyMcs
• Scalable and fault tolerant plaRorm in your service • DeclaraMve processing workflow development, deployment and management
• Huge number of predefined available modules that can be used straight away to bring your processing pipelines in life in seconds
• Easy to extend plaRorm and pluggable custom modules
• Build on top of rich and robust Spring foundaMon
![Page 10: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/10.jpg)
![Page 11: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/11.jpg)
![Page 12: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/12.jpg)
![Page 13: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/13.jpg)
![Page 14: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/14.jpg)
![Page 15: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/15.jpg)
Real world example for XD
Raw data is captured from mobile phones in JSON format by a Spring XD cluster where several processes are performed. The data is then stored in an HDFS cluster where Spring XD batch jobs use SQL via the HAWQ interface to HDFS and store the calculated reports in Redis. Spring Boot is then used with Angular.js and D3.js to show analyMcs to end users.
![Page 16: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/16.jpg)
[DEMO:] Spring XD toy. Play with me!
1. Show XD bootstrap in singlenode mode and deploy simple streams. Show XD shell and XD UI usages.
2. Show XD distributed mode configuraMon with external Zookeeper, DB and Redis. Show XD meta data in Zookeeper. Deploy same simple streams on cluster and show XD cluster fault tolerance and recovery.
3. Show example of predefined XD module configuraMon (e.g. hdfs)
4. Create custom module and upload it into XD 5. Show DEMO for stream reading tweets from TwiVer and
posMng them into REST client app developed before via HTTP and wriMng them into HDFS in parallel
![Page 17: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/17.jpg)
![Page 18: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/18.jpg)
Spring XD (out of scope…) • Show XD job running over Hadoop • Show RabbitMQ integraMon • Show DEMO on running either MapReduce job using Cascading or Pig over HDFS tweets data OR running Spark app over HDFS tweets data
![Page 19: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/19.jpg)
Spring Boot/XD enable twelve factor apps I. Codebase One codebase tracked in revision control, many deploys II. Dependencies Explicitly declare and isolate dependencies III. Config Store config in the environment IV. Backing Services Treat backing services as aVached resources V. Build, release, run Strictly separate build and run stages VI. Processes Execute the app as one or more stateless processes VII. Port binding Export services via port binding VIII. Concurrency Scale out via the process model IX. Disposability Maximize robustness with fast startup and graceful shutdown X. Dev/prod parity Keep development, staging, and producMon as similar as possible XI. Logs Treat logs as event streams XII. Admin processes Run admin/management tasks as one-‐off processes
![Page 20: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/20.jpg)
Everything wasn’t so smooth though… • SPEL doesn’t work on filter with JSON elements access implicitly as it works with transformer
module. To make filter work you need either to convert JSON to Tuple or to use JSON Xpath SPEL syntax
• When transform JSON into Tuple and back WARNings are thrown on Jackson transformaMon??? • In order implicit converMon to Tuple works, you need to specify contentType MIME header in HTTP
request otherwise it can’t automaMcally cast text/plain MIME message to Tuple • If you deploy hVp module on port which is occupied it won’t fail on deploy but your stream will
simply silently not work • SomeMmes container gets broken and smth stops to work like Taps in my case for example and no
errors in logs. Only container restart helps. • Spring data rest doesn’t expose enMty IDs in JSON body by default • When Mongo enMty save operaMon failing as a result of HTTP call no excepMons visible in logs, just
500 ERROR response to client without proper stack trace, however if you call MongoTemplate directly you’ll get proper stack trace.
• If HTTP GET request contentType=applicaMon/json then response will be rendered as applicaMon/x-‐spring-‐data-‐compact+json, when by default it will be HAL document
• By default HVp POST request set contentType=text/plain which failed to be transformed into enMty • hVp://wiki.fasterxml.com/JacksonFAQDateHandling • hVps://github.com/FasterXML/jackson-‐datatype-‐jsr310 • Mongo doesn’t support Java 8 Date and Time types naMvely • hVps://jira.spring.io/browse/DATAREST-‐524 • hVp://stackoverflow.com/quesMons/25072967/spring-‐xd-‐redis-‐message-‐bus-‐removing-‐headers-‐
from-‐the-‐message issue with headers passing in Redis • SpringXD doesn’t support packages starMng with ‘java’))!!!
![Page 21: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/21.jpg)
Links
• Spring -‐ hVp://spring.io • Zookeeper -‐
hVp://zookeeper.apache.org/doc/trunk/zookeeperStarted.html • Hadoop -‐
hVp://hadoop.apache.org/docs/r2.7.1/hadoop-‐project-‐dist/hadoop-‐common/SingleCluster.html
• XD -‐ hVp://docs.spring.io/spring-‐xd/docs/current-‐SNAPSHOT/reference/html
• Boot -‐ hVp://docs.spring.io/autorepo/docs/spring-‐boot/1.2.0.M2/reference/htmlsingle/
• DataRest -‐ hVp://docs.spring.io/spring-‐data/rest/docs/2.3.2.RELEASE/reference/html/
• Redis -‐ hVp://redis.io/documentaMon • RestTemplate -‐
hVps://spring.io/blog/2009/03/27/rest-‐in-‐spring-‐3-‐resVemplate
![Page 22: Develop modern apps using Spring ecosystem at time of BigData](https://reader033.vdocuments.site/reader033/viewer/2022050614/589bf3271a28ab40348b5b29/html5/thumbnails/22.jpg)
Thank you!
TwiVer: @tsaltsol Skype: oleg.tsalko GitHub: olegts Code shown: hVps://github.com/olegts/SpringXDCustomTweetsTransformer hVps://github.com/olegts/TweetsBootDataRestApp