Move Messages
Transform and Copy messages between Kafka topics
Provided with Professional license.
This is one of the most requested tasks - copying selected messages from one Kafka topic to another, with some transformation along the way.
Click on a source topic, select target topic, and click “Start Copying” button. That’s it. Kafka Magic will try copying up to ‘Max Results’ messages from source to target using default schema assignments for the topics.
But most likely, you need something more complicated than that. Here come the magical features.
It is highly recommended to click “Test Run” before copying messages for real. In the Test Run Kafka Magic will perform search and transformation, and will display the results without actually copying messages to the target topic. This will help you confirm that search and transformation work as intended before committing to topic publishing.
Select a specific Schema to use when serializing messages to a target topic.
You can select by Schema Subject or ID, choose to write messages as string without referencing any schema, or copy each message as is, byte by byte, maintaining whatever schema assignment the message has.
In complex scenarios, you can also assign schema to message dynamically using JavaScript after analyzing message’s content and metadata (see below).
If you choose to use schema, you have an option to select appropriate Subject Name Strategy.
This might sound confusing, but don’t worry, in most practical cases our magic default Ignore strategy works just fine.
Other options are standard values TopicNameStrategy, RecordNameStrategy, and TopicRecordNameStrategy - they define if the producer will try to register a new subject for your target schema, and what that subject will be. For this to work, you need to set “Auto-register schemas” flag to “true” in the cluster connection configuration, and enable that in your cluster.
Specify Message Compression options
This option defines if Kafka Magic producer will compress messages, which algorithm it will use, and the compression level specific for particular algorithm.
Provide filtering and transforming JavaScript code
In most cases you need to select a particular set of messages from the source topic based on message fields or metadata. Sometimes you also need to transform messages to conform to target schema, or to modify or add some fields.
Check the “Use JavaScript to filter and transform messages” checkbox, and enter the JavaScript code. This code will be applied to every message coming from the source topic. The code must return Context object for messages you want to send to the target topic, or null for messages to be discarded.
Here is an example of the code:
if (Context.Message.myField === 'foo' && Context.Message.anotherField >= 0) {
Context.Message.myField = 'bar';
Context.Message.newField = 1234;
return Context;
}
else {
return null;
}This code selects only those messages which have myField === 'foo' && anotherField >= 0, all other messages will not be selected. For selected messages the code modifies value for the field myField, creates new field newField, and returns the Context object with modified message which will be published to the target topic.
The Context object is the same object you saw when you queried topics (see here).
Limit search scope by Timestamp and Partition/Offset
This part works exactly the same way as in topic querying, limiting the source topic’s search scope (see here).