Storm OpenTSDB 集成

Storm OpenTSDB Bolt 和 TridentState

OpenTSDB 为时间序列数据提供了可扩展且高可用性的存储. 它由 Time Series Daemon (TSD) servers 以及命令行工具组成. 每个 TSD 连接到配置的 HBase 集群以 push/query（推送/查询） 数据.

时间序列数据点包括: - a metric name. - a UNIX timestamp (seconds or milliseconds since Epoch). - a value (64 bit integer or single-precision floating point value). - a set of tags (key-value pairs) that describe the time series the point belongs to.

Storm bolt 和 trident state 从一个基于给定的 TupleMetricPointMapper 的 tuple 中创建了上面的时间序列数据.

该模块提供了 core Storm 和 Trident bolt 实现, 用户将数据写入 OpenTSDB.

时间序列数据点被写入时有 at-least-once（至少一次）的语义保证, 并且重复的数据点应该像 OpenTSDB 的 这里 提及的一样来处理.

示例

Core Bolt

下面的示例描述了 org.apache.storm.opentsdb.bolt.OpenTsdbBolt cord bolt 的使用方法

 OpenTsdbClient.Builder builder =  OpenTsdbClient.newBuilder(openTsdbUrl).sync(30_000).returnDetails();
        final OpenTsdbBolt openTsdbBolt = new OpenTsdbBolt(builder, Collections.singletonList(TupleOpenTsdbDatapointMapper.DEFAULT_MAPPER));
        openTsdbBolt.withBatchSize(10).withFlushInterval(2000);
        topologyBuilder.setBolt("opentsdb", openTsdbBolt).shuffleGrouping("metric-gen");

Trident State

 final OpenTsdbStateFactory openTsdbStateFactory =
                new OpenTsdbStateFactory(OpenTsdbClient.newBuilder(tsdbUrl),
                        Collections.singletonList(TupleOpenTsdbDatapointMapper.DEFAULT_MAPPER));
        TridentTopology tridentTopology = new TridentTopology();
        final Stream stream = tridentTopology.newStream("metric-tsdb-stream", new MetricGenSpout());
        stream.peek(new Consumer() {
            @Override
            public void accept(TridentTuple input) {
                LOG.info("########### Received tuple: [{}]", input);
            }
        }).partitionPersist(openTsdbStateFactory, MetricGenSpout.DEFAULT_METRIC_FIELDS, new OpenTsdbStateUpdater());