I’d really like to get my data into an instance of InfluxDB. I’m currently doing so with the HomeAssistant service but obviously that’s a limited data set. Any chance of re-incarnating the previous InFlux connector to optionally send data to an external InfluxDB in addition to what goes to VictoriaMetrics for the core functionality?
This is not something we’d want to officially support, but the actual conversion of Brewblox history data → Influx Line Protocol is trivial, to the point where I may as well put it in a script. Depending on preference, this can be done in Python or Javascript (for Node-RED).
Just getting back to this after a busy summer. I haven’t really looked at how data ingress works with InfluxDB besides a general understanding of configuring Telegraf for various sources. This does indeed look trivial and I think I might take a shot at it myself in Javascript.
Looking into this over the weekend I think the best path forward for me is to configure Telegraf to use the history data from MQTT. I think I’ll probably do that by deploying a Telegraf docker container. Other than the the Telegraf configuration file, nothing custom will be required. That gives me a chance to learn some about consuming data from MQTT into InfluxDB which will likely come in handy later.
Skimming through the Telegraf documentation, this appears very doable. The topic for Spark history data includes the service name, and the field path within the JSON object is deterministic (no lists).
Please let us know how it works out. I suspect more people would be interested in possibly taking this path.
Got it working!
# Telegraf Configuration
[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
hostname = ""
omit_hostname = false
[[outputs.influxdb_v2]]
urls = ["http://<influxdb servername>:8086"]
token = "<secret>"
organization = "<org>"
bucket = "brewblox"
[[inputs.mqtt_consumer]]
servers = ["wss://brewblox:443/eventbus"]
topics = ["brewcast/history/+",]
insecure_skip_verify = true
qos = 0
connection_timeout = "30s"
persistent_session = true
client_id = "telegraftest"
data_format = "xpath_json"
xpath_native_types = true
[[inputs.mqtt_consumer.topic_parsing]]
topic = "brewcast/history/+"
measurement = "measurement/_/_"
tags = "_/_/service"
[[inputs.mqtt_consumer.xpath]]
metric_name = "'measurment1'"
metric_selection = "//data/*"
field_selection = "*"
field_name = "replace(name(.), ' ', '_')"
[inputs.mqtt_consumer.xpath.tags]
block = "replace(name(.), ' ', '_')"
1 Like
Doesn’t this clash with the metrics name parsed from topic?
