Interworking with MySQL databases

Previous posts in this blog have discussed the use of HTTP for direct operations with the DBMS. With Pyrrho's support for RESTViews it becomes very important to have REST interfaces to other DBMS, and there are several products such as dreamfactory and restifydb that offer this.

From today the Pyrrho distribution includes a very simple solution for MySQL, called RestIfD , and controllers to support other DBMS will probably be added over time. There is a GitHub repository for Restif at https://github.com/MalcolmCrowe/Restif .

RestIfD should be started up by an administrator on the same host as the MySQL instance, and left running. It requires no configuration, creates no files and uses very little memory. It provides a REST service on http://localhost:8803/REST/MySQL , and all requests should include an Authentication header giving the user and password for accessing MySQL.

To obtain a list of databases in Json format, use GET to http://localhost:8803/REST/MySQL . To send one or more commands to MySQL, place them in posted data to POST http://localhost:8803/REST/MySQL .

To obtain a list of tables on database db in Json format, use GET to http://localhost:8803/REST/MySQL/db .To send one or more commands for database db to MySQL, place them in posted data to POST http://localhost:8803/REST/MySQL/db .The group of commands is executed in a single transaction.

To obtain the rows of table tb on database db, in Json format, use GET to http://localhost:8803/REST/MySQL/db/tb .

To filter these results using a where condition or document w, use GET to http://localhost:8803/REST/MySQL/db/tb/w . You can delete rows from the table by using a DELETE requests to a similar url.

To update a single row in table tb on database db, send it in Json format as posted data to POST http://localhost:8803/REST/MySQL/db/tb . The row to be updated will be identified by the key fields in the supplied data.

To add a single row to table tb in database db, send it in Json format as posted data to POST http://localhost:8803/REST/MySQL/db/tb .

Comments welcome.

Pyrrho Version 6.0

The new version completes some of the changes announced in previous posts in this blog.

A: Open Source Version and Append Storage

Append storage has been a build option for some time in Pyrrho, and the new version uses it by default for the Open Source version. Unfortunately this means that databases built with the previous default version of OSPSvr (up to 5.7) cannot be used with the new default build (v.6) because they won't have the encrypted end-of-file marker.
Workarounds for users incovenienced in this way include continuing to use v5.7 for now, manually to strip the last 5 bytes from the binary file, or rebuild v6's OSPSvr without the APPEND flag. But see the next point.
v.6 OSPSvr by default doesn't support the Distributed and Partitioned features either but again these can be restored by removing the LOCAL build flag. The Pro version by default continues to use the end-of-file tamper-proof lock and distributed/partitioned features. These have been enhanced as described below.

B: Using the latest version of C#

The codebase uses the 2017 version of C# with its shortcut implicit declarations e.g. if (x is SomeClass y), and .NET's clever new Task model for some purposes, including for Push, described next.

C: Push and Pull for update notifications

Distributed and Partitioned operation up to v 5.5 was not very scalable as transaction masters and base partitions were contacted at the start of every transaction by every replica and partition to check that the local copies were up to date. Some versions since v.5.5 have contained bugs in this area. This is now remedied in v6.0 with the following interesting design, visible in the Open Source codebase and used in the default PyrrhoSvr build.

From v6.0 servers holding slave databases and partitions check their copy information is up-to-date only on start-up, and thereafter receive Push notifications about any changes to database length. They will retrieve the details of the changes when and if they need to. This mechanism works in a hierarchical way so that the Push load is shared by intermmediate servers if these have been configured. All this did complicate the implementation, since server-server communication channels need to be between threads, but it is worthwhile for theoretical reasons.

I will update the screenshots and documentation in the distributed and partitioned tutorials in the next days.

D: Enhancements to the HTTP/REST services

Previous posts in this blog have referred to ETags and RESTViews. A paper on these has now been published, thanks to Friz Laux, Mart Laiho and Carolyn Begg. The practical effect of these features could be adopted much more widely than Pyrrho, and I hope to return to this aspect soon.

Basically, it would be great if all DBMS provided a really simple REST service so that a simple GET would get a table (or other query results) in Json format together with an RFC7232 ETag. Even better if such a view was also potentially updatable with PUT, POST and DELETE. If this is done, database applications can always use WebRequests instead of the clunky APIs: JDBC,ODBC, ADO.NET, JPA.... The authorisation arrangements for this should also be simple and preferably role based, and use the If-Match HTTP header to check ETags.

This works already for Pyrrho (If-Match from v6), and there is more:

HTTP is stateless so the simple steps described above don't provide a way of making several changes in a single HTTP call. But at least for a single database, ACID transactions can be supported by a simple POST of an SQL statement to the database (+role) URL, since SQL statements can be compound and have exception handlers etc. And the POST can be guarded by If-Match to if the SQL statement contains data previously obtained from the database.

It is definitely a step too far in my opinion to try to use these mechanisms for transactions that change more than one database.

As usual, any comments welcome, preferably to malcolm.crowe at uws.ac.uk (make sure the subject line of the email is relevant).