Benchmarking PyrrhoV7alpha

 It is natural to wonder whether Pyrrho v7's unusual data structures and approach to transactions are practical. Since 2018 PyrrhoDB has been tested using a modified version of the TPC-C benchmark

The TPC-C benchmark simulates an OLTP system for ordering items by telephone from an e-business. The business operates a number of warehouses, with 30000 customers at each divided into 10 districts. There are 100000 items available. There are normal speed tests, e.g. what is the top speed for a sequence of transactions? Pyrrho's performance here on a PC is modest at 10 new-order transactions per second.

If the specification is followed more closely, there is one clerk per warehouse to deal with a mix of tasks including new order, payment, stock enquiry, and delivery. The clerk's behaviour is simulated so that tasks take a realistic amount of real time (e.g. 23 seconds for a new order with between 5 and 15 different items). The specification sets these out in great detail. For one warehouse no DBMS following the rules can complete more than 17 new orders in 10 minutes.

To provide greater challenge for the DBMS's transaction mechanism, Pyrrho sets the number of warehouses to 1, but increases the number of clerks for that warehouse to 50 to 100. Pyrrho always requires serialisable transactions, so after 10 clerks, the chances of transaction conflict rise sharply. All commercial DBMS have trouble with this test, and performance generally collapses well before 30 clerks. 

With StrongDBMS the throughput continued to increase to 100 clerks, when CPU resources became a problem.  This link gives access not only to software for StrongDBMS's version of the test, but also versions for other DBMS we tried. The Pyrrhov7alpha folder in this repository contains the current alpha source code for PyrrhoV7, with the version of software used for the test described below.

We have analysed the reasons for Strong's unexpected performance, and it relates to the use of optimistic concurrency mechanisms. Pyrrho uses similar mechanisms but up to version 6.3 its results on this test were poor, leading to the development of PyrrhoV7, which seeks to learn lessons from StrongDBMS's implementation. A full account of this process was given in a tutorial at DBKDA 2021.

With PyrrhoV7 alpha, results with 50 clerks have always been impressive. At the current stage in development, however, Pyrrho's rowset review mechanism, still under development, does not yet carry out a good implementation of the TPCC StockLevel task, which is a relatively rare CPU-intensive task that conflicts with a great many ongoing transactions. Without this task, CPU and memory usage for server and clients remain low even at 100 clerks (1GB server memory, peak 6% CPU). Experimentally, peak throughput occurs with around 60 clerks.

The results below were obtained using the server version dated 30 May 2021.

Clerks	NewOrders Committed	Conflicts reported
1	17	0
2	29	15
5	62	74
10	91	254
20	108	678
50	135	2009
75	133	3088
100	105	4089

This page will be updated when the test includes the StockLevel task.