Blog

Updates from the Arroyo team

Announcing Arroyo 0.3.0

The Arroyo 0.3.0 release adds UDFs, DDL statements, custom event time and watermarks, web UI improvements, and more.

Micah Wylde

Micah Wylde

CEO of Arroyo

We're thrilled to announce the 0.3.0 release of Arroyo, our second minor release as an open-source project. Arroyo is a state-of-the-art stream processing engine designed to allow anyone to build complex, stateful real-time data pipelines with SQL.

Table of Contents

Overview

The Arroyo 0.3 release focused on improving the flexibility of the system and completeness of SQL support, with the MVP for UDF support, DDL statements, and custom event time and watermarks. There have also many substantial improvements to the Web UI, including error reporting, backpressure monitoring, and under-the-hood infrastructure improvements.

We've also greatly expanded our docs since the last release. Check them out at https://doc.arroyo.dev.

New contributors

We are excited to welcome three new contributors to the project with this release:

Thanks to all new and existing contributors!

What's next

Looking forward to the 0.4 release, we have a lot of exciting changes planned. We're adding the ability to create updating tables with native support for Debezium, allowing users to connect Arroyo to relational databases like MySQL and Postgres. Other planned features include external joins, session windows, and Delta Lake integration.

Excited to be part of the future of stream processing? Come chat with the team on our discord, check out a starter issue and submit a PR, and let us know what you'd like to see next in Arroyo!

Features

UDFs

With this release we are shipping initial support for writing user-defined functions (UDFs) in Rust, allowing users to extend SQL with custom business logic. See the udf docs for full details.

For example, we can register a Rust function:

// Returns the great-circle distance between two coordinates
fn gcd(lat1: f64, lon1: f64, lat2: f64, lon2: f64) -> f64 {
    let radius = 6371.0;
 
    let dlat = (lat2 - lat1).to_radians();
    let dlon = (lon2 - lon1).to_radians();
 
    let a = (dlat / 2.0).sin().powi(2) +
        lat1.to_radians().cos() *
            lat2.to_radians().cos() *
                (dlon / 2.0).sin().powi(2);
    let c = 2.0 * a.sqrt().atan2((1.0 - a).sqrt());
 
    radius * c
}

and call it from SQL:

SELECT gcd(src_lat, src_long, dst_lat, dst_long)
FROM orders;

SQL DDL statements

It's now possible to define sources and sinks directly in SQL via CREATE TABLE statements:

CREATE TABLE orders (
  customer_id INT,
  order_id INT,
  date_string TEXT
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json'
);

These tables can then be selected from and inserted into to read and write from those systems. For example, we can duplicate the orders topic by inserting from it into a new table:

CREATE TABLE orders_copy (
  customer_id INT,
  order_id INT,
  date_string TEXT
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json'
);
 
 
INSERT INTO orders_copy SELECT * FROM orders;

In addition to connection tables, this release also adds support for views and virtual tables, which are helpful for splitting up complex queries into smaller components.

Custom event time and watermarks

Arroyo now supports custom event time fields and watermarks, allowing users to define their own event time fields and watermarks based on the data in their streams.

When creating a connection table in SQL, it is now possible to define a virtual field generated from the data in the stream and then assign that to be the event time. We can then generate a watermark from that event time field as well.

A complete example looks like this:

CREATE TABLE orders (
  customer_id INT,
  order_id INT,
  date_string TEXT,
  event_time TIMESTAMP GENERATED ALWAYS AS (CAST(date_string as TIMESTAMP)),
  watermark TIMESTAMP GENERATED ALWAYS AS (event_time - INTERVAL '15' SECOND)
) WITH (
  connection = 'my_kafka',
  topic = 'order_topic',
  serialization_mode = 'json',
  event_time_field = 'event_time',
  watermark_field = 'watermark'
);

For more on the underlying concepts of event times and watermarks, see the concept docs.

Additional SQL features

Beyond UDFs and DDL statements, we have continued to expand the completeness of our SQL support with addition of case statements and regex functions:

Server-Sent Events source

We've added a new source which allows reading from Server-Sent Events APIs (also called EventSource). SSE is a simple protocol for streaming data from HTTP servers and is a common choice for web applications. See the SSE source documentation for more details, and take a look at the new Mastodon trends tutorial that makes uses of it

  • Add event source source operator by @mwylde in #106
  • Add HTTP connections and add support for event source tables in SQL by @mwylde in #119

Web UI

This release has seen a ton of improvements to the web UI.

Improvements

Fixes

The full change-log is available here.