《Flume：构建高可用、可扩展的海量日志采集系统》

《Flume：构建高可用、可扩展的海量日志采集系统》 内容介绍：

《Flume：构建高可用、可扩展的海量日志采集系统》从Flume 的基本概念和设计原理开始讲解，分别介绍了不同种类的组件、如何配置组件、如何运行Flume Agent 等。同时，分别讨论Source、Channel 和Sink 三种核心组件，不仅仅阐述每个组件的基本概念，而且结合实际的编程案例，深入、全面地介绍每个组件的详细用法，并且这部分内容也是整个Flume 框架的重中之重。之后，讲解拦截器、Channel选择器、Sink 组和Sink 处理器等内容，它们为Flume 提供灵活的扩展支持。最后，介绍了Flume 的高级使用，如何使用Flume 软件开发工具集（SDK）和Embedded Agent API，如何设计、部署和监控Flume 生产集群。 总而言之，《Flume：构建高可用、可扩展的海量日志采集系统》是一本理论结合实战，深度、广度兼备的海量日志采集系统的著作。

作者【美】Hari Shreedharan（哈里•史瑞德哈伦）介绍：

Hari Shreedharan是Cloudera的一名软件工程师，他工作于Apache Spark、Apache Flume和Apache Sqoop。他也是Flume项目的一个提交者和PMC成员，帮助项目的方向做决定。

《Flume：构建高可用、可扩展的海量日志采集系统》 目录大纲：

译者序 ........................................................................... v
序 ................................................................................xiii
前言 ............................................................................... x
第1 章 认识Apache Hadoop 和Apache HBase ............ 1
分布式文件系统HDFS ..........................................................................................1
HDFS 的数据格式 ...........................................................................................3
处理HDFS 中的数据 ......................................................................................4
Apache HBase ........................................................................................................4
总结 .......................................................................................................................5
参考文献 ................................................................................................................6
第2 章 用Apache Flume 处理流数据 ............................ 7
我们需要Flume .....................................................................................................7
Flume 是否适合呢？ .............................................................................................9
Flume Agent 内部原理 .........................................................................................10
配置Flume Agent .................................................................................................13
Flume Agent 之间的相互通信 ..............................................................................17
复杂的流 ..............................................................................................................17
复制数据到不同目的地 ........................................................................................20
动态路由 ..............................................................................................................21
Flume 的无数据丢失保证，Channel 和事务 ........................................................22
Flume Channel 中的事务 ...............................................................................23
Agent 失败和数据丢失 ........................................................................................25
批量的重要性 ......................................................................................................26
重复怎么样？ ......................................................................................................27
运行Flume Agent .................................................................................................27
总结 .....................................................................................................................29
参考文献 ..............................................................................................................30
第3 章 源（Source） .................................................. 31
Source 的生命周期 ...............................................................................................31
Sink-to-Source 通信 .............................................................................................33
Avro Source ...................................................................................................34
Thrift Source .................................................................................................37
RPC Sources 的失败处理 ..............................................................................39
HTTP Source ........................................................................................................40
针对HTTP Source 写处理程序* ..................................................................42
Spooling Directory Source ....................................................................................47
使用Deserializers 读取自定义格式* ............................................................50
Spooling Directory Source 性能.....................................................................55
Syslog Source .......................................................................................................55
Exec Source ..........................................................................................................59
JMS Source ..........................................................................................................61
转换JMS 消息为Flume 事件* .....................................................................63
编写自定义Source* .............................................................................................65
Event-Driven Source 和Pollable Source ........................................................66
总结 .....................................................................................................................73
参考文献 ..............................................................................................................73
第4 章 Channel ......................................................... 75
事务工作流 ..........................................................................................................76
Flume 自带的Channel .........................................................................................78
Memory Channel ...........................................................................................78
File Channel ..................................................................................................80
总结 .....................................................................................................................86
参考文献 ..............................................................................................................86
第5 章 Sink ............................................................... 87
Sink 的生命周期 ..................................................................................................88
优化Sink 的性能 .................................................................................................89
写入到HDFS ：HDFS Sink ..................................................................................89
理解Bucket ...................................................................................................90
配置HDFS Sink ............................................................................................93
使用序列化器控制数据格式* ..................................................................... 100
HBase Sink ......................................................................................................... 106
用序列化器将Flume 事件转换成HBase Put 和Increment* ....................... 108
RPC Sink ............................................................................................................ 113
Avro Sink ..................................................................................................... 113
Thrift Sink ................................................................................................... 115
Morphline Solr Sink ........................................................................................... 116
Elastic Search Sink ............................................................................................. 119
自定义数据格式* ....................................................................................... 121
其他Sink ：Null Sink、Rolling File Sink 和Logger Sink .................................. 124
编写自定义Sink* .............................................................................................. 125
总结 ................................................................................................................... 129
参考文献 ............................................................................................................ 129
第6 章 拦截器、Channel 选择器、Sink 组和
Sink 处理器 ................................................... 131
拦截器 ................................................................................................................ 131
时间戳拦截器 .............................................................................................. 132
主机拦截器 ................................................................................................. 133
静态拦截器 ................................................................................................. 133
正则过滤拦截器 .......................................................................................... 134
Morphline 拦截器 ........................................................................................ 135
UUID 拦截器 ............................................................................................... 136
编写拦截器* ............................................................................................... 137
Channel 选择器 .................................................................................................. 140
复制Channel 选择器 ................................................................................... 140
多路复用Channel 选择器 ........................................................................... 141
自定义Channel 选择器* ............................................................................ 144
Sink 组和Sink 处理器 ....................................................................................... 146
Load-Balancing Sink 处理器 ....................................................................... 148
Failover Sink 处理器 ................................................................................... 151
总结 ................................................................................................................... 153
参考文献 ............................................................................................................ 154
第7 章 发送数据到Flume* ....................................... 155
构建Flume 事件 ................................................................................................ 155
Flume 客户端SDK ............................................................................................. 156
创建Flume RPC 客户端 .............................................................................. 157
RPC 客户端接口 ......................................................................................... 157
所有RPC 客户端的公共配置参数 .............................................................. 158
默认RPC 客户端......................................................................................... 165
Load-Balancing RPC 客户端 ....................................................................... 168
Failover RPC 客户端 ................................................................................... 171
Thrift RPC 客户端 ....................................................................................... 172
嵌入式Agent ..................................................................................................... 173
配置嵌入式Agent ....................................................................................... 175
log4j Appender ................................................................................................... 180
Load-Balancing log4j Appender ................................................................... 181
总结 ................................................................................................................... 182
参考文献 ............................................................................................................ 183
第8 章 规划、部署和监控Flume ............................... 185
规划一个Flume 部署 ......................................................................................... 185
修复时间 ..................................................................................................... 185
我的Flume Channel 需要多少容量？ ......................................................... 186
多少层？ ..................................................................................................... 186
通过跨数据中心链接发送数据 .................................................................... 188
层分片 ......................................................................................................... 190
部署Flume ......................................................................................................... 191
部署自定义代码 .......................................................................................... 191
监控Flume ......................................................................................................... 193
从自定义组件报告度量 ............................................................................... 196
总结 ................................................................................................................... 196
参考文献 ............................................................................................................ 196
索引 ........................................................................... 197