flume日志收集系统,怎么定义flow out的文件名?

漏洞扫描、网关、防火墙、补丁升级、数据备份和迁移、系统故障排除
回复
170119566
帖子: 18
注册时间: 2009-10-05 22:09
送出感谢: 0
接收感谢: 0

flume日志收集系统,怎么定义flow out的文件名?

#1

帖子 170119566 » 2013-02-22 20:55

flume:一个类似scribe的日志收集系统官方网站如下
http://flume.apache.org/index.html

现在有一个问题就是我把日志收集后在sink到文件中的时候,flume自动生成的日志格式问是一些随机数(像1361524178237-1),而且我把把sink到hdfs中的时候也是这样,同样是生成以随机数的文件名.
我看官方文档的时候有一节说的是使用hdfs生成的日志可以用时间还命名,但是我将配置文件改成按时间生成日志时,就报如下错误,不知道有没有人用过这玩意,到这来和大家交流交流经验-_-! :em20 :em20 :em20

使用hdfs按时间生成日志时,报如下错误:
22 Feb 2013 20:54:20,981 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed
java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:160)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:343)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:397)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:404)
at java.lang.Long.valueOf(Long.java:540)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:158)
... 5 more
22 Feb 2013 20:54:20,983 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:160)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:343)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:397)
... 3 more
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:404)
at java.lang.Long.valueOf(Long.java:540)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:158)
... 5 more
回复

回到 “服务器维护和硬件相关”