2023-04-26

Send data from single inputs.exec to multiple outputs.kafka in telegraf

I have data coming to telegraf and it has multiple fields. I want to be able to have one inputs.exec plugin and multiple outputs.kafka plugin to send each field to respective kafka topics.

My data is in this format:

{
    "field1": [
        {
            "abc" : 0,
            "efg" : 1,
            "hij" : 4,
            "jkl" : 5
        }
    ],
    "field2": [
        {
            host : "admin1",
            timestamp: 1682314679774
        },
        {
            host : "admin2",
            timestamp: 1682314679775
        },
        {
            host : "admin3",
            timestamp: 1682314679773
        }
    ]
}

I want to send field1 to "field1" kafka topic and field2 to "field2" kafka topic. My expected output is:

In field1 kafka topic: { "abc" : 0, "efg" : 1, "hij" : 4, "jkl" : 5 }

In field2 kafka topic: { host : "admin1", timestamp: 1682314679774 }, { host : "admin2", timestamp: 1682314679775 }, { host : "admin3", timestamp: 1682314679773 }

How do i implement this in telegraf configuration file?

I tried writing the telegraf conf file in this format. But it is not working.

[[outputs.kafka]]
    brokers = ["admin:9091"]
topic = "field1"
data_format = "json"
flush_interval = "1m"

[[outputs.kafka]]
    brokers = ["admin:9092"]
## Kafka topic for producer messages
topic = "field2"
data_format = "json"
flush_interval = "1m"

[[inputs.exec]]
interval = "1m"
commands = ["/opt/clustertest/bin/script.py"]
timeout = "10s" # I want the script to execute every ten seconds
data_format = "json"
flush_interval = "1m"
json_query = "{'field1': [], 'field2': []}"

Can anyone please help?



No comments:

Post a Comment