ソーシャルアプリでの Amazon Elastic MapReduce 活用事例
- 1. Amazon Elastic MapReduce Takahiro Kamatani gumi, Inc. 2010/09/26 Sunday, September 26, 2010
- 2. • • • Amazon Elastic MapReduce • Sunday, September 26, 2010
- 3. • Twitter: @buhii • • gumi @ http://www.kansei.tsukuba.ac.jp/~uchiyamalab/beacon • • beacon • gumi @ynil Sunday, September 26, 2010
- 4. Sunday, September 26, 2010
- 5. gumi • mixi, , GREE • python Django • Amazon Web Services (EC2 + RDS) • • DB Sunday, September 26, 2010
- 6. • PV , UU • DAU Daily Active Users • • • ÷ DAU • ARPU Average Revenue Per User • Sunday, September 26, 2010
- 7. Amazon Web Service AWS Sunday, September 26, 2010
- 8. Sunday, September 26, 2010
- 9. Sunday, September 26, 2010
- 10. Sunday, September 26, 2010
- 11. Sunday, September 26, 2010
- 12. Sunday, September 26, 2010
- 13. Amazon Elastic MapReduce Sunday, September 26, 2010
- 14. MapReduce Mapper Key, Value Mapper key Sort / Shuffle Reducer Reducer key, value Mapper, Reducer Sunday, September 26, 2010
- 15. Amazon Elastic MapReduce • Hadoop • Hadoop Streaming Mapper Reducer Ruby, Perl, Python, PHP, R, Bash, C++ • EC2 job • Sunday, September 26, 2010
- 16. Example Task • • • Sunday, September 26, 2010
- 17. • Mapper • Apache Log • ID key value Reducer Sunday, September 26, 2010
- 18. • Reducer • sort/shuffle ID Reducer • ID Sunday, September 26, 2010
- 19. Reducer 31758623 2010-08-20 42346572 2010-09-05,2010-09-06 31977736 2010-08-11,2010-08-12,2010-08-13,2010-08-14 14007991 2010-08-16 35995849 2010-08-12,2010-08-13,2010-08-14 34246688 2010-08-21,2010-08-22,2010-08-23,2010-08-27 ... PC Sunday, September 26, 2010
- 20. Amazon Elastic Mapreduce • AWS • S3 Mapper, Reducer → s3cmd, S3Fox Organizer, Cyberduck • Job OK • Sunday, September 26, 2010
- 21. Sunday, September 26, 2010
- 22. Sunday, September 26, 2010
- 23. Streaming Sunday, September 26, 2010
- 24. {Input, Output} Location, Mapper, Reducer S3 gzip Hadoop Extra Args -jobconf stream.recordreader.compression=gzip input Location Extra Args -input s3n://(bucket )/( )/access_log.* Sunday, September 26, 2010
- 25. Sunday, September 26, 2010
- 26. Sunday, September 26, 2010
- 27. Sunday, September 26, 2010
- 28. Debug Sunday, September 26, 2010
- 29. Sunday, September 26, 2010
- 30. • Hadoop • MapReduce • • Sunday, September 26, 2010
- 31. Hadoop • • S3 gzip • • hadoop EC2 • ( 20 ...) Sunday, September 26, 2010
- 32. @ynil MapReduce http://nlpyutori.g.hatena.ne.jp/yaruki_nil/20100911/1284089305 Sunday, September 26, 2010
- 33. Sunday, September 26, 2010
- 34. Sunday, September 26, 2010
- 35. Sunday, September 26, 2010
- 36. MapReduce MapReduce Google Map Reduce Map Reduce MapReduce C++ Java Python Wikipedia “MapReduce” http://ja.wikipedia.org/wiki/MapReduce Sunday, September 26, 2010
- 37. cron • PV, UU NFS CSV • DB → DB • PV, UU Sunday, September 26, 2010