Query Topics

Search for messages in a topic using JavaScript

Whether it is Json or Avro serialized message you can search for it using JavaScript query referencing any combination of the message fields, headers, and metadata. You can even declare JavaScript functions to handle a complicated case. Full support for ECMAScript 5.1 standard.

Kafka Magic: Query Topics using JavaScript

The query references the Context object to get required properties and returns boolean value.

Context object

When writing a JavaScript query you are provided with the Context object holding message content and metadata.

Here is the definition for the Context:

{
    Timestamp: Date;
    Topic: string;
    Partition: number;
    Offset: number;
    SchemaId: number;
    SchemaType: string;
    Key: number[];
    Headers: any;
    Message: any;
    Error: string;
}

Context properties

  • Timestamp - the time either set by the producer on message creation time, or by the broker on message insertion time (depending on cluster configuration)
  • Topic - the topic name
  • Partition - partition id
  • Offset - message offset in the partition
  • SchemaId - Schema Id which was used by producer of the message. null if no schema
  • SchemaType - Type of the Schema which was used by producer of the message. null if no schema
  • Key - binary representation of the Key - sequence of numbers each representing a byte
  • Headers - an object which properties represent message headers
  • Message - JSON representation of the message, or string representation of the byte array if there was no schema or deserialization failed
  • Error - Exceptions thrown when deserializing the message, null if none

Feel free to use all these values in your query.

Kafka Magic Search Query

Your query can be pretty much any JavaScript statement, or a set of statements. The only requirements is that at some point it must return a boolean value specifying if this is the message you are looking for.

Query can be as simple as a single return true; statement, or a complex code inspecting the Context object, making calculations, declaring variables and functions. Just don’t forget to return true if you want this particular message to be included in the results, or false otherwise.

Example

In this example you want to find messages which satisfy three conditions:

  • the header ‘myHeader’ has value ‘myHeaderValue’
  • the message field ‘title’ has value ‘Title value’
  • the message field ‘price’ has value greater than 12.34

In this case the query should return true when all these conditions are true:

return Context.Headers.myHeader === 'myHeaderValue' && Context.Message.title === 'Title value' && Context.Message.price > 12.34;

When you enter a JavaScript query to find particular messages in a Kafka topic, the system applies this query to each and every message in the topic. For large topic this can take a lot of time.

Filtering by timestamp or partition/offset allows you to significantly reduce time system needs to find a particular message.

Filtering by Timestamp

Since version 0.10.0 Apache Kafka has an index based on message timestamps, which makes this operation very efficient. So, while using Kafka Magic to search for messages in a topic you can limit the scope of the search by specifying Start and Stop timestamps.

Kafka Magic: Filtering by Timestamp

Filtering by Partition/Offset

If you know approximate offset values for the Kafka message you are trying to find, filtering by partition/offset can be a very efficient way of reducing the search time.

Kafka Magic: Filtering by Partition/Offset

Transforming Messages

You can transform selected messages before displaying them in the results pane. You can change any field value, or add new fields.

You apply your changes to the Context object you receive. To make your changes visible in the results pane return the transformed Context instead of returning true from your script.

The changes in the Context object you make are reflected only in the displayed results. You can’t change any message residing in the topic.

Here is an example of the Search query adding new field before displaying message.

Context.Message.myNewField = 'bar';
return Context;