New decoding and framing options for sources

Often, when consuming data from a source, the first operation you have to do on it is decode the data from its source representation. To make this easier, we’ve added new decoding options to most sources .

For example, if you have a kafka source that has JSON-encoded messages, now you can simply add decoding.codec = "json" to your source configuration like:

[sources.kafka]
type = "kafka"
bootstrap_servers = "localhost:9200"
topics = ["my_topic"]
decoding.codec = "json"

This will decode your messages from JSON, thus saving you from an additional remap transform.

In addition, we’ve added a new framing option to allow configuration for sources that have non-standard framing (for example a custom-delimiter separating messages).

For example, if you have an http source where the messages are delimited by commas instead of newlines, you can configure this like:

[sources.http]
type = "http"
address = "0.0.0.0:8080"
framing.method = "character_delimited"
framing.character_delimited.delimiter = ","

To have Vector parse each comma-delimited element as a new message. This can be used with decoding option specified above. See the docs for other framing options.