你的位置:首页 > 数据库

[数据库]flume坑之channel.transactionCapacity和HdfsSink.batchSize


  不说过程了,直接说结果!一对相连接的channel-HdfsSink,无意间配置如下:
...
agent.channels.common-channel.transactionCapacity=10
...
agent.sinks.hdfs-sink.hdfs.batchSize=20

  简单测试之后发现flume报如下异常,倒也正常……

[2015-12-17 11:42:09:694 ERROR][org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:467)]process failed
org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 10 full, consider committing more frequently, increasing capacity, or in creasing thread count
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
        at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:382)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
[2015-12-17 11:42:09:696 ERROR][org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)]Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 10 full, consider committing more frequently, increasing capacity, or increasing thread count
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:471)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 10 full, consider committing more frequently, increasing capacity, or increasing thread count
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
        at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:382)
        ... 3 more

  但是......但是老子发现停掉发送,channelSize一直不减,hdfs里的数据也一直在涨!!!而且永远停不下来,数据被永远重放!!!
  查看相关HdfsSink代码如下:

 public Status process() throws EventDeliveryException {  Channel channel = getChannel();  Transaction transaction = channel.getTransaction();  List<BucketWriter> writers = Lists.newArrayList();  transaction.begin();  try {   int txnEventCount = 0;   for (txnEventCount = 0; txnEventCount < batchSize; txnEventCount++) {    Event event = channel.take();    if (event == null) {     break;    }    ......         bucketWriter.append(event);    ......   transaction.commit();   if (txnEventCount < 1) {    return Status.BACKOFF;   } else {    sinkCounter.addToEventDrainSuccessCount(txnEventCount);    return Status.READY;   }  } catch (IOException eIO) {   transaction.rollback();   LOG.warn("HDFS IO error", eIO);   return Status.BACKOFF;  } catch (Throwable th) {   transaction.rollback();   LOG.error("process failed", th);   if (th instanceof Error) {    throw (Error) th;   } else {    throw new EventDeliveryException(th);   }  } finally {   transaction.close();  } }

 

  看到了嘛!异常在取第11个数据时channel.take();出现,然后此次事物被回滚,但是,但是尼玛之前取出来的10个Event都被bucketWriter.append(event);了,也就是被写到hdfs了;
然后就是最初的现象了,事物一直不断的被回滚,但部分取到的数据却也写到hdfs了,这尼玛算是什么事务……
  虽然是不合理的配置参数,但flume启动时有一大陀检测参数的代码也没检测到这些,至少给报个错或WARN嘛!