© 2019 Swarm64, Inc.

Measuring the impact of FPGA acceleration on PostGIS concurrent query performance at Toyota In a benchmark run by Toyota, accelerating open source PostgreSQL/PostGIS with FPGA and Swarm64 yielded 3x faster, and more predictable performance under concurrent load.

© 2019 Swarm64, Inc. 1

Executive summary Advances in database performance enable companies to collect more data and analyze it in new ways to create new customer experiences, new services, and new cost-saving measures. Databases have steadily increased performance over the last 30 years through parallel processing innovations. Today, parallel processing distributes the work of loading and querying data across multiple threads, multiple CPU cores, or even multiple servers clustered together.

Hardware acceleration is the next innovation in database parallel processing

Hardware accelerators such as FPGAs enhance database servers with hundreds of extra processes that work in parallel to speed up and scale the database. In this paper, we describe a benchmark developed in July, 2019 by the Toyota InfoTech End to End Computing Group to test the potential effects of FPGA acceleration. The benchmark was created using FPGA extensions for PostgreSQL, developed by Intel-backed Swarm64. The tests showed that running PostGIS queries under increasing concurrent load on an FPGA- equipped server provides faster, and more stable performance than on a non-FPGA server:

In an internally developed benchmark at Toyota, PostGIS queries ran an average of 3x faster, and with more predictable performance on an FPGA-accelerated server.

With FPGA hardware becoming more mainstream (e.g., FPGA-equipped instances now available on most major cloud platforms), the purpose of this benchmark is to help database architects make better-informed infrastructure decisions by considering alternatives to CPU-only database servers.

© 2019 Swarm64, Inc. 2

About the benchmark organizers

Sylvain Lefebvre, Toyota Motor Corporation, Advanced Development InfoTech Division, End To End Computing Group

Toyota InfoTech End-To-End Computing group brings together researchers from networking and distributed database technologies in order to design Toyota’s future data platform.

Sylvain Lefebvre joined InfoTech and Toyota’s E2E Computing group in April 2019. His research interests are distributed database systems and he is now focusing on new hardware platforms for these systems.

Swarm64 Field Engineering team

The Toyota team was supported by members of the Swarm64 field engineering team in Berlin Germany. The Swarm64 field engineers are expert FPGA technologists and PostgreSQL developers. Their support helped validate and improve findings throughout the benchmark process.

Intel K.K.

Intel K.K. provided the FPGA hardware for the benchmark. The Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA (Intel® PAC with Intel® Arria® 10 GX FPGA) is a PCI Express-based FPGA accelerator card for data centers providing the performance and versatility of FPGA acceleration and supported by the Acceleration Stack for Intel® Xeon® CPU with FPGAs. Intel® PAC with Intel® Arria® 10 GX FPGA can be implemented in many market segments, such as big data analytics, artificial intelligence, genomics, video transcoding, cybersecurity, and financial trading.

Macnica, Inc.

Macnica Altima Company is a reseller of Swarm64 in the Japanese market and provides technical support to customers in Japan. Macnica is a global leader in supplying semiconductors, electronic components, network equipment, software products, and various services to customers worldwide in the electronics and enterprise IT markets. Macnica Altima Company is also the distributor of Intel FPGA products, and provides end-to-end support to customers from R&D to deployment.

© 2019 Swarm64, Inc. 3

Scenario: Real-time location analytics for connected cars The context for the benchmark was a use case analyzing data generated by Internet-connected cars. Toyota’s requirements included:

● Accelerate PostgreSQL PostGIS queries (PostGIS is an open source extension to PostgreSQL that adds support for geographic objects allowing location queries to be run in SQL.)

● Deliver consistently fast query performance as concurrent users increased ● Continue using free, open source PostgreSQL (avoid an expensive proprietary DBMS)

The benchmark team proposed running PostgreSQL on a commodity server equipped with an Intel PAC FPGA accelerator. Swarm64 DA, a PostgreSQL FPGA extension developed by Intel-backed Swarm64, Inc., runs on the FPGA to accelerate PostgreSQL performance.

About FPGA acceleration FPGAs (Field Programmable Gate Arrays) are reprogrammable hardware devices that can implement custom-developed logic that accelerate a specific workload such as database SQL execution.

FPGAs accelerate computing via parallel processing, and also by taking workload off of the CPUs in the server, freeing them to do other tasks.

In 2003, Netezza revolutionized analytics with its data warehouse appliance. It famously featured a cabinet full of FPGA boards working in parallel to ingest and query large volumes of data warehouse data on PostgreSQL.

Today, FPGA hardware is readily available and affordable to everyone. Swarm64 DA software enables companies to accelerate database performance running on commodity servers that are equipped with FPGAs in order to achieve game-changing breakthroughs in price-performance and scalability.

© 2019 Swarm64, Inc. 4

Benchmark test design In July 2019, the Toyota InfoTech End to End Computing Group developed a benchmark to measure the effects of FPGA hardware acceleration on PostgreSQL and PostGIS performance.

Technology stack

The test was conducted on a single server configured with: Hardware:

● Compute: 48-core, Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz ● Memory: 263GB RAM ● Storage: 500GB SSD Disk Space (224GB available for users) ● FPGA: Intel Programmable Acceleration Card with Intel Arria® 10 GX FPGA

DBMS: PostgreSQL 11.5 (Ubuntu 11.5-1.pgdg16.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609, 64-bit Database container:

● 24 cores ● 16GB RAM*

*The container limited the amount of RAM available to the database to simulate production at scale, where the amount of RAM would be smaller than the amount of storage space. Database Accelerator: Swarm64 DA is FPGA-based software that accelerates PostgreSQL database performance. It populates the FPGA with hundreds of processes that work in parallel to read and write data to PostgreSQL, thus improving PostgreSQL/PostGIS performance.

Test Scenarios

Toyota developed two test scenarios, with different levels of query complexity, to test the effect of FPGA acceleration on query performance and concurrency.

● Test Scenario 1 – Obstacle detection. Find the list of vehicles that have been within 2km of a random location within the last 6 minutes. The goal of this scenario is to assess the efficiency of FPGA acceleration against the Geospatial optimized functions of PostGIS.

● Test Scenario 2 – Analytics scenario on the NYC Taxi Dataset see [1]

© 2019 Swarm64, Inc. 5

Scenario 1: Obstacle detection and warning The dataset used in this scenario is a small dataset: The T-Drive trajectory data sample [1] [2] . This dataset consists of 1GB of trajectory data from taxis in the Beijing area.

Scenario 1 test data

PostGIS table definition: TABLE taxi_trips ( id INTEGER, ts TIMESTAMP, location GEOGRAPHY(POINT, 4326) ); Gist(location)

Swarm64 DA foreign data table definition: CREATE FOREIGN table taxi_trips ( id INTEGER, ts TIMESTAMP, location GEOGRAPHY(POINT, 4326) ) SERVER swarm64da_server OPTIONS(optimized_columns 'ts');

The Swarm64 DA accelerator is an extension to open source PostgreSQL, and it accelerates querying and data loading into Postgres foreign tables.

Scenario 1 test queries

The scenario simulates the detection of an obstacle on the road by selecting the list of vehicles that are within 2km and 6 minutes drive from a randomly selected point.

PostGIS query: SELECT tt.id, ST_AsText(tt.location), tt.ts FROM ( SELECT id, ts, location FROM taxi_trips where id=${pointid} ORDER BY ts LIMIT 1 ) AS obstacle, taxi_trips AS tt WHERE ST_DWithin(obstacle.location, tt.location, 2000.0) AND tt.ts BETWEEN obstacle.ts-interval '360 seconds' AND obstacle.ts ORDER BY id,ts;

Swarm64 DA query

© 2019 Swarm64, Inc. 6

DROP FUNCTION IF EXISTS foo(INT); CREATE FUNCTION foo(INT) RETURNS TABLE(id int, loc text, ts timestamp) AS $$ DECLARE ts timestamp; location geometry; BEGIN SELECT tt.ts, tt.location INTO ts, location FROM taxi_trips tt WHERE tt.id = $1 ORDER BY ts LIMIT 1; RETURN QUERY EXECUTE ' SELECT tt.id, ST_AsText(tt.location), tt.ts FROM taxi_trips AS tt WHERE ST_DWithin($2, tt.location::geometry, 2000.0) AND tt.ts BETWEEN $1 - INTERVAL ''360 seconds '' AND $1 ORDER BY id, ts' USING ts, location; RETURN; END; $$ LANGUAGE plpgsql;

The Swarm64 DA version of the query was originally implemented as a subquery, just as the PostGIS query. However, it was learned that Swarm64 DA executes the parent query on the FPGA, but not the nested subquery, which gets executed by PostgreSQL. In order to execute the entire query on the FPGA and maximize acceleration, the query was rewritten to leverage a SQL function (above), thus forcing all parts of the query to the FPGA and resulting in much faster performance. Complete support for subqueries will be offered in a future version of Swarm64 DA.

Scenario 1 results

3200 queries were executed on PostGIS, with and without FPGA acceleration. Query latency (time from query request sent to query results received) was measured by JMeter. A histogram of the query latencies at each level of concurrency, from 1 client thread, to 32 concurrent threads are shown below:

© 2019 Swarm64, Inc. 7

Chart 1: Histogram of Swarm64 DA-accelerated PostgreSQL/PostGIS query times vs. standard PostgreSQL/PostGIS

as concurrent users scales from 1 to 32. It’s clear from the histogram that FPGA delivers more consistent query performance--35x less variance--as the number of users scales. A graph of average query response time shows that, on average, FPGA acceleration delivers 3x faster performance as concurrent clients scales from 1 to 32:

Chart 2: Average query times for Swarm64 DA-accelerated PostgreSQL/PostGIS vs. standard PostgreSQL/PostGIS

as concurrent users scales from 1 to 32. This improvement has a profound effect on the cost of a clustered PostGIS deployment. With each FPGA-accelerated server capable of servicing 3x or more concurrent clients, a smaller cluster of FPGA-equipped servers can handle the load. For example, a 5-node cluster versus a 15-node cluster, which is less expensive, easier to administer, and less failure prone.

© 2019 Swarm64, Inc. 8

Scenario 2: Location analytics queries In the second scenario, the goal was to evaluate the performance of large analytics queries on a larger-than-RAM table.

The data set used in this test is the full NYC Taxi Dataset [3]. This data amounts to a 267GB table with 1.1 billion rows of data. Queries from this scenario were inspired by the work of M. Litwintschik [4].

Scenario 2 Test data

PostgreSQL Table CREATE TABLE trips (

id INTEGER, cab_type_id INTEGER, vendor_id text, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, store_and_fwd_flag text, rate_code_id INTEGER, pickup_longitude NUMERIC, pickup_latitude NUMERIC, dropoff_longitude NUMERIC, dropoff_latitude NUMERIC, passenger_count INTEGER, trip_distance NUMERIC, fare_amount NUMERIC, …..

PRIMARY KEY (id)); CREATE INDEX trips_pickup_datetime_idx brin (pickup_datetime) WITH (pages_per_range='32');

© 2019 Swarm64, Inc. 9

Swarm64 DA foreign data table CREATE FOREIGN TABLE trips_swarm ( id INTEGER, cab_type_id INTEGER, vendor_id text, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, store_and_fwd_flag text, rate_code_id INTEGER, pickup_longitude NUMERIC, pickup_latitude NUMERIC, dropoff_longitude NUMERIC, dropoff_latitude NUMERIC, passenger_count INTEGER, trip_distance NUMERIC, fare_amount NUMERIC, …..) SERVER swarm64da_server OPTIONS (optimized_columns 'passenger_count,pickup_datetime', primary_key 'PRIMARY KEY(id)');

Scenario 2 Test Queries

The query syntax used for PostgreSQL and FPGA-accelerated PostgreSQL were identical.

1. Average fee per number of passengers SELECT passenger_count, avg(total_amount) FROM trips GROUP BY 1;

2. Passenger count distribution per month SELECT passenger_count, EXTRACT(MONTH FROM pickup_datetime), COUNT(*) FROM trips GROUP BY 1,2;

3. Number of passengers per distance per month SELECT passenger_count, EXTRACT(MONTH FROM pickup_datetime), CAST(trip_distance AS INT), COUNT(*) FROM trips GROUP BY 1, 2, 3 ORDER BY 2,4 DESC;

© 2019 Swarm64, Inc. 10

Scenario 2 results

Each query was run 10 times, and the average response time (in milliseconds) was measured. Swarm64 DA accelerated PostGIS query performance by 2x to 3x:

Chart 3: Average query times (milliseconds) for 3 PostGIS queries of 1B+ rows of New York City taxi data.

Swarm64 DA acceleration

Query 1 2.4x

Query 2 2.9x

Query 3 1.9x

These results show that a meaningful performance increase can be achieved with Swarm64 and FPGA--without having to rewrite SQL queries and while continuing to use free, open source PostgreSQL.

Conclusions With FPGA technology costs becoming more affordable and accessible, Toyota designed a benchmark to test whether parallel processing introduced by the FPGA and Swarm64 DA would accelerate PostgreSQL/PostGIS. Their motivation was to find a cost-effective and easy way to scale free, open source PostgreSQL to handle many concurrent clients. Their tests showed that:

● FPGA does indeed accelerate PostgreSQL query performance under concurrent load, by a factor of more than 3x, and with much more predictable performance.

● Ad-hoc PostGIS queries on a 1.1 billion row data set execute 2x to 3x faster.

© 2019 Swarm64, Inc. 11

Next steps If you would like to experience FPGA acceleration of your PostgreSQL servers, please visit https://swarm64.com to learn more about it. Swarm64 invites you to contact us if you have any questions or would like to schedule a demo or a free trial. Swarm64 also invites you to contact our partner, Macnica (https://www.macnica.co.jp/business/semiconductor/manufacturers/swarm64/) for Swarm64 DA sales and support. Macnica connects cutting-edge technologies and intelligence to create unique solutions that move businesses into the future.

Resources and additional links [1] Jing Yuan, Yu Zheng, Chengyang Zhang, Wenlei Xie, Xing Xie, Guangzhong Sun and

Yan Huang, " T-drive: driving directions based on taxi trajectories.," in In Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, GIS ’10, New York, NY, USA, 2010.

[2] Jing Yuan, Yu Zheng, Xing Xie and Guangzhong Sun., "Driving with knowledge from the physical world," in The 17th ACM SIGKDD international conference on Knowledge Discovery and Data mining, KDD’11, New York, NY, USA, 2011.

[3] T. W. Schneider, "New York City Taxi and For-Hire Vehicle Data," 10 2019. [Online]. Available: https://github.com/toddwschneider/nyc-taxi-data. [Accessed 10 2019].

[4] M. Litwintschik, "Big Data Consultant," [Online]. Available: https://tech.marksblogg.com.