Tackling big data with hadoop and open source integration
11 likesā¢2,515 views
The document discusses Talend's goal of democratizing integration and big data. It describes how big data involves transactions, interactions and observations from diverse sources, requiring a different approach than traditional data integration. Talend aims to make big data accessible to everyone with its open source Talend Open Studio for Big Data, which improves the efficiency of designing big data jobs with intuitive interfaces and generates code to run transforms within Hadoop. Poor data quality in big data projects can magnify problems, so Talend recommends incorporating data quality checks into loading processes or via separate map reduce jobs.
Tackling big data with hadoop and open source integration
- 1. Tackling Big Data with Hadoop and Open Source Integration Ciaran Dynes Remy Dubois
- 2. Agenda 1.āÆTalendās Goal: Democratizing Integration 2.āÆWhat is Big Data (integration)? 3.āÆBig Data for the Masses: Talendās strategy and vision Ā© Talend 2011 2
- 3. Our goal
- 4. Talend ā The Market Leading Unified Integration Platform Talend Enterprise Data Data MDM ESB BPM Quality Integration Ā¾ļ¾āÆ Commercial license Ā¾ļ¾āÆ Subscription model Studio Repository Deployment Execution Monitoring Ā¾ļ¾āÆ Open source license Talend Open Studio for Ā¾ļ¾āÆ Free of charge Ā¾ļ¾āÆ Optional support Data Data Quality Integration MDM ESB Recognized as the open source leader in each of its market category by all industry analysts Ā© Talend 2011 4
- 5. Who uses Talend? A high adoption rate Ā§ļ§āÆ20 million downloads Ā§ļ§āÆ950,000 users Ā§ļ§āÆ3,500 customers 1 product download 150 new customers every 30 seconds per month Ā© Talend 2011 5
- 6. Trying to get from thisā¦ Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 6
- 7. to thisā¦ Why Talendā¦ ONLY Talend generates code that is executed within map reduce. This open approach removes the limitation of a proprietary āengineā to provide a truly unique and powerful set of tools for big data.
- 8. Big data isā¦. Hans Rosling ā uses big data to analyze world health trends Key Takeaway #1 transactions, interactions, observations Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 8
- 9. Big Data = Transactions + Interactions + Observations Sensors/RFID/Devices User Generated Content Big Data Mega, Giga, Tera, Peta bytes Sentiment Social Interactions & Feeds Mobile Web Spatial & GPS coordinates User Clicks External Demographics Web logs WEB Business Data Feeds Offer history A/B testing Video, Audio, Images Dynamic pricing SMS/MMS CRM Segmentation Affiliate Networks Search Marketing ERP Offer details Purchase detail Customer Touchpoints Behavioral Targeting Purchase record Support Contacts Dynamic Funnels Payment record Increasing Data Variety and Complexity Source: Hortonworks Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 9
- 10. What is Big Data integration?
- 11. Traditional Data Flows CRM ETL Normalized Traditional Data ERP Data Data Warehouse Quality Finance ā¢āÆ Scheduledādaily or weekly, sometimes more frequently. Business Business Analyst User ā¢āÆ Volumes rarely exceed terabytes Warehouse Administrator Executives Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 11
- 12. The new world of big data Social Networking CRM ERP Big Data Finance Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 12
- 13. The new world of big data Social Networking CRM Mobile Devices ERP Big Data Finance Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 13
- 14. The new world of big data Social Networking CRM Mobile Devices ERP Transactions Finance Big Data Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 14
- 15. The new world of big data Social Networking CRM Mobile Devices ERP Transactions Finance Network Devices Big Data Sensors Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 15
- 16. Key Takeaway #2 Forces us to think Ā© Talend 2011 differently Ā© Talend 2011 ā Stri2y Private & Confidential 16
- 17. But for Talendā¦. Big data isā¦ ā¦everything that is old, is new again! Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 17
- 18. Data driven business enables data governance supports information decisions drives Information provides value to the business If you can't rely on your information then Your the result can be missed opportunities, or business higher costs. Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE). Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 18
- 19. BIG data driven business enables BIG data governance supports BIG BIG information decisions drives Information provides value to the business If you can't rely on your information then the result can be missed opportunities, or BIG higher costs. business Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE). Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 19
- 20. āBig Data for the Massesā
- 21. Goal: Democratize Big Data Talend Open Studio for Big Data Ā¾ļ¾āÆ āBig Data for the Massesā Ā¾ļ¾āÆ Improves efficiency of big data job design with graphic interface Ā¾ļ¾āÆ Abstracts and generates code Ā¾ļ¾āÆ Run transforms inside Hadoop Pig Ā¾ļ¾āÆ Native support for HDFS, Pig, HBase, Sqoop and Hive Ā¾ļ¾āÆ Apache License 2.0 Ā¾ļ¾āÆ Embedded in Hortonworks Data ā¦an open source Platform ecosystem Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 21
- 22. Let us show youā¦ Ā© Talend 2012
- 23. Where to next? Ā© Talend 2012
- 24. How is big data integration being used? Use Cases ā¢āÆ Recommendation Engine ā¢āÆ Sentiment Analysis ā¢āÆ Risk Modeling ā¢āÆ Fraud Detection ā¢āÆ Marketing Campaign Analysis ā¢āÆ Customer Churn Analysis ā¢āÆ Social Graph Analysis ā¢āÆ Customer Experience Analytics ā¢āÆ Network Monitoring ā¢āÆ Research And Development BUT: to what level is DQ required for your use case? Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 24
- 25. Poor Data Quality + Big Data = Big Problems Poor Data Quality * Big Data = Big Problems^2 Key Takeaway #3 In big dataā¦ poor data quality can be magnified at huge scale Ā© Talend 2011 25
- 26. Two methods for inserting data quality into a big data job 1.āÆ Pipelining: as part of the load process 2.āÆ Load the cluster than implement and execute a data quality map reduce job Ā© Talend 2011 26
- 27. E-T-L - Load Extract ā Transform Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 27
- 28. E- DQ -L Extract ā Improve/Cleanse - Load Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 28
- 29. Pipelining: data quality with big data CRM DQ ERP DQ Finance Big Data Social Networking ā¢āÆ Use traditional data quality tools ā¢āÆ No new programming, no PHDs ā¢āÆ Once and done Mobile Devices Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 29
- 30. Big data alternative: Load and improve within the cluster CRM DQ ERP DQ Finance Big Data Social Networking ā¢āÆ Load first, improve later ā¢āÆ Really complex to build, limited tools ā¢āÆ Constant on, increments Mobile Devices ā¢āÆ Insane performance Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 30
- 31. big 2012 data now Q4 2013 Talend Open Studio for Big Data Ā¾ļ¾āÆPackaged within Hortonworks Data Platform ā¦Eclipse tools for HIVE, HDFS, PIG, SCOOP ā¦supports Oozie, Hcatalog, Kerberos Ā¾ļ¾āÆFree to download and use under the Apache license ā¦democratizing big data through intuitive tools Ā© Talend 2011 ā Stri2y Private & Confidential Ā© Talend 2011 31
- 33. Sessions will resume at 11:25am Page 33