FlureeDB is an open source blockchain based graph database with many great features that is suitable, among other things as technology for making distributed side-chains for blockchains like HIVE. As a database that is written in the clojure language, and as a relatively new piece of database technology, the ecosystem surrounding this database was relatively limited from the perspective of the Python/TypeScript/C++ environment me and my project team are working in.
To overcome this limited ecosystem, we created and open sourced two related projects (available through four repos). The aioflureedb asynchronous Python library, and the Fluree Schema Scenario Tool or fsst, a tool aimed at CICD pipeline usage mostly, but suitable for a wide range of testing and debugging activities when developing with FlureeDB.
On march 30th 2023 FlureePBC released the 2.0 version of their blockchain based graph database. I don't have a complete picture of everything that changed in the 2.0 version compared to the 1.0, but a few important ones:
- A considerable performance increase
- More powerful queries in FlureeQL
- Multiple bug fixed
- Removal of depricated APIs
When the 2.0 version of FlureeDB was released, the fsst tool stoped working on the stable build of the fluree+fsst docker images. In this repo, there are docker files and shell scripts for building the docker images and to push them to docker hub so fsst can use them. When a new version of FlureeDB comes out, we have a small collection with currently failing scenarios we want to test the new release with that are in this repo.
There was a tiny problem with the release of the 2.0 version in that the API documentation at that point in time had not yet been updated to include documentation for the new API endpoints that were supposed to replace the now removed deprecated API endpoints. For the most part it was just a name change from API endpoints and request parameters with db in the name to API endpoints and request parameters with ledger in the name.
aioflureedb API changes
Basically the API changes for aioflureedb reflect those for FlureeDB itself. They are mostly transparent.
async with aioflureedb.FlureeClient(masterkey=key,
host=host,
port=port) as flureeclient:
await flureeclient.health.ready()
await flureeclient.new_db(ledger_id=dbase)
fdb = await flureeclient[dbase]
async with fdb(key) as database:
await database.ready()
The above code works with the 1.0 version of FlureeDB, yet not with the version 2.0 version.
async with aioflureedb.FlureeClient(masterkey=key,
host=host,
port=port) as flureeclient:
await flureeclient.health.ready()
await flureeclient.new_ledger(ledger_id=dbase)
fdb = await flureeclient[dbase]
async with fdb(key) as database:
await database.ready()
By replacing new_db with new_ledger, the above code works not just with the new 2.0 version of fluree but also still works with the old 1.0 version. In the same way the method dbs that lists all the ledgers on the server is now called ledgers, and the method delete_db is now named delete_ledger.
There are two other changes to aioflureedb when running the above code that happen beneath the service.
The internals of the line await flureeclient.health.ready() have been changed. This line makes aioflureedb repeatedly query the /fdb/health API endpoint untill it considers the flureedb instance to be ready. But recent tests done with restarting FlureeDB in a new advanced fsst test with hooks has revealed the old implementation of ready() wasn't quite complete.
When FlureeDB starts up, and it isn't the first time flureeDB starts, ever, and the server is almost up, first the field ready is set to True. A little bit later the field status is updated to have a non-null value. The call to ready() now waits for the second condition because if it doesn't, signed transactions will fail during the window will the second condition is met.
A second under the hood change to aioflureedb involves cryptographical database ownership. In the 1.0 version of FlureeDB new_db used to make a database bound to initially generated server key. That behavior is now simmulated by aioflureedb, but this happens under the hood through the use of an extra API parameter that is added to the requests.
Getting the latest versions
If you used fsst before, update to the newest version first. Like before, you have the choice to do so with or without support for local domain-api functionality.
python3 -m pip install 'fsst[docker,domainapi]' --force
or
python3 -m pip install 'fsst[docker,domainapi]' --force
Because aioflureedb is a dependency of fsst, you will now have the latest version of both.
After this, it is suggested you remove any remaining local fsst docker docker images if you have them.
docker rmi --force $(docker images -q 'pibara/fsst'|uniq)
You can now pull the latest fsst/flureedb docker image for both version 1.0 and version 2.0 of FlureeDB
docker pull pibara/fsst:stable
docker pull pibara/fsst:v1
Note that this step isn't strictly needed. The fsst tool will do a docker pull itself of the image if its not available local, but this will make the first run slower because it is fetching the docker image first.
New fsst features
Making fsst run with the 2.0 version of fluree took some time and effort, and most of all debugging. To allow for this debugging, a few new features were added to the fsst command line.
debug
The fsst tool already had the verbosefluree flag to make it output the FlureeDB console log inside of docker to the console for the fsst command. Next to the verbosefluree flag for debug purposes, fsst now also has a debug flag. When using this flag, aioflureedb inside of the running docker container, is given an environment variable prompting it to log all API queries and responses to the console. You can use this flag on its own or combined with the verbosefluree flag.
If we run dockertest on this demo directory without the debug flag,
fsst dockertest
we get:
COMMAND: fsst guesttest --target default --network test91340 --runs 1 --stages ALL
IMAGE: pibara/fsst:stable
NOTICE: stable not found, trying to fetch from docker hubi, this may take a moment.
- Fetched
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
WARNING: No hooks file found: hooks/hooks.py
RUN: 0
- Database: test91340/api-trias-politica
- collecting transactions from build subdirs
Going through first 1 stages
- trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
### trias_politica test_index = 0 ###
### Test the trias politica scenarios ###
- dbase = test91340/api-trias-politica
- processing schema transaction sub-set
- ok, completed 4 transactions on test91340/api-trias-politica
- ok, added hard-coded _auth collection
- ok, added auth roles for TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG : ['root']
- ok, added auth roles for Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88 : ['root']
- ok, added test user test_trias_politica TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
- ok, added test user auth31 Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
+ Running run_test_give_role as TexyZ1iY2m8iHQoAGxvFd3e9ixWeUUjHdcG
['artist', 'judicial']
['farmer', 'executive']
['baker', 'legislator']
res= True EXPECTED
['judge', 'executive']
ERROR: res= False NOT EXPECTED Shouldn't be able to give a judge executive powers
Unexpected result from scenario for scenario1 at sub-test 0
Now fasten your seatbelt, because when we add debug,
fsst dockertest --debug
we get the very verbose output:
COMMAND: fsst guesttest --target default --network test91371 --runs 1 --stages ALL
IMAGE: pibara/fsst:stable
# waiting for default-private-key.txt to appear
# waiting for default-private-key.txt to appear
Started FlureeDB and got createkey from newly created keyfile
DEBUG: True []
WARNING: No hooks file found: hooks/hooks.py
RUN: 0
- Database: test91371/api-trias-politica
- collecting transactions from build subdirs
Going through first 1 stages
- trias_politica
NOTICE: Loaded Domain-API testing module trias_politica
domain API test, run = 0
Unsigned GET: url = http://localhost:8090/fdb/health , ssl_verify_disabled = False
Result:
{"ready":true,"status":"leader","utilization":0.5}
### trias_politica test_index = 0 ###
### Test the trias politica scenarios ###
- dbase = test91371/api-trias-politica
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
Signed POST: url = http://localhost:8090/fdb/new-ledger , headers = {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:53 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1c30440220604670952419f4bd205462d5d2842781269a6734d847bc99a0edad2b7cd2442c02200d1bba64f93f35aa56142d4fda651be6050b9d973552364cb1508caee93e24bd"', 'Digest': 'SHA-256=m78LYfJtTeF+tQAXPjZ2Z4bypT1i+jXWcHgsn+WAOXc='} ,ssl_verify_disabled = False
body = {"ledger/id":"test91371/api-trias-politica","owners":["Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS"]}
Result:
"3b02d5c8f21045068ffd806827718f84262a1db629f97f1ec3dc7326b5219448"
Signed POST: url = http://localhost:8090/fdb/ledgers , headers = {'Content-Type': 'application/json'} ,ssl_verify_disabled = False
body = {}
Result:
[["test91371","api-trias-politica"]]
Signed POST: url = http://localhost:8090/fdb/ledgers , headers = {'Content-Type': 'application/json'} ,ssl_verify_disabled = False
body = {}
Result:
[["test91371","api-trias-politica"]]
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:53 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3046022100e31737f92e3f2f8a3a52e57f293d1636c351b21ef0f69ff0424aae0044357ccc022100b5626383c8ad99840cd45f9d1375ef26f66a741aa99ccc364656f0a294596329"', 'Digest': 'SHA-256=LKglIDoqt+3BTB9EtXjCP+TIaxz43vdeb47iP95JX8U='}
body: {"select":["_collection/name"],"from":"_collection"}
400
rval: {"status":400,"message":"Ledger test91371/api-trias-politica is not currently available. Status is: :initialize.","error":"db/unavailable"}
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:56 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3045022052124a4203ab47e5826edf3160ef01023e887d2cbfba0a878125d8be1dfb4794022100cb707b31b0162232455e64c68c97d10d69b579b9b2b9a5bf755644c4df8f6fb3"', 'Digest': 'SHA-256=LKglIDoqt+3BTB9EtXjCP+TIaxz43vdeb47iP95JX8U='}
body: {"select":["_collection/name"],"from":"_collection"}
200
rval: [{"_collection/name":"_ctx","_id":17592186044426},{"_collection/name":"_setting","_id":17592186044425},{"_collection/name":"_rule","_id":17592186044424},{"_collection/name":"_role","_id":17592186044423},{"_collection/name":"_auth","_id":17592186044422},{"_collection/name":"_user","_id":17592186044421},{"_collection/name":"_fn","_id":17592186044420},{"_collection/name":"_tag","_id":17592186044419},{"_collection/name":"_shard","_id":17592186044418},{"_collection/name":"_collection","_id":17592186044417},{"_collection/name":"_predicate","_id":17592186044416}]
- processing schema transaction sub-set
Signing with: Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/command False
headers: {'content-type': 'application/json'}
body: {
"cmd": "{\"type\": \"tx\", \"tx\": [{\"_id\": \"_fn$lastblock\", \"name\": \"lastBlock\", \"doc\": \"Get the number of the last known block on the ledger\", \"code\": \"(query (str \\\"{\\\\\\\"select\\\\\\\": \\\\\\\"?maxBlock\\\\\\\", \\\\\\\"where\\\\\\\": [[\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?bNum\\\\\\\"], [\\\\\\\"?maxBlock\\\\\\\", \\\\\\\"#(max ?bNum)\\\\\\\"], [\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?maxBlock\\\\\\\"]]}\\\" ) )\"}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS\", \"fuel\": 1000, \"nonce\": 5166419320643064, \"expire\": 1683913856508}",
"sig": "1b304502200a432ba313cb55b9a4987c0c75bdd1f829551c1b19ccfb3f9d4956aab5cf3a58022100c5617df2a8a2791869b387599d4e651fdb0bde13f932debe13e353a240ed857c"
}
200
rval: {"tempids":{"_fn$lastblock":70368744178665},"block":2,"hash":"efa8b0bab51890a112bce50f922072f99608a34f654b436ac47a698c087a5038","instant":1683913736572,"type":"tx","duration":"83ms","fuel":1216,"auth":"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS","status":200,"id":"25999844945b4bd664616395684037865995216c1fdcd308a3fcd2edccc3f996","bytes":1206,"t":-3,"flakes":[[70368744178665,90,"lastBlock",-3,true,null],[70368744178665,92,"(query (str \"{\\\"select\\\": \\\"?maxBlock\\\", \\\"where\\\": [[\\\"?s\\\", \\\"_block/number\\\", \\\"?bNum\\\"], [\\\"?maxBlock\\\", \\\"#(max ?bNum)\\\"], [\\\"?s\\\", \\\"_block/number\\\", \\\"?maxBlock\\\"]]}\" ) )",-3,true,null],[70368744178665,93,"Get the number of the last known block on the ledger",-3,true,null],[-3,99,"73bb9c7a9423673dc36c37ec22d7a2244189d390f427e5f9b34d6c3a9cf87be7",-3,true,null],[-3,100,"25999844945b4bd664616395684037865995216c1fdcd308a3fcd2edccc3f996",-3,true,null],[-3,101,105553116266496,-3,true,null],[-3,103,5166419320643064,-3,true,null],[-3,106,"{\"type\": \"tx\", \"tx\": [{\"_id\": \"_fn$lastblock\", \"name\": \"lastBlock\", \"doc\": \"Get the number of the last known block on the ledger\", \"code\": \"(query (str \\\"{\\\\\\\"select\\\\\\\": \\\\\\\"?maxBlock\\\\\\\", \\\\\\\"where\\\\\\\": [[\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?bNum\\\\\\\"], [\\\\\\\"?maxBlock\\\\\\\", \\\\\\\"#(max ?bNum)\\\\\\\"], [\\\\\\\"?s\\\\\\\", \\\\\\\"_block/number\\\\\\\", \\\\\\\"?maxBlock\\\\\\\"]]}\\\" ) )\"}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf5AhKTwYphG9CNnVp6vNUPmjmyG1FEKitS\", \"fuel\": 1000, \"nonce\": 5166419320643064, \"expire\": 1683913856508}",-3,true,null],[-3,107,"1b304502200a432ba313cb55b9a4987c0c75bdd1f829551c1b19ccfb3f9d4956aab5cf3a58022100c5617df2a8a2791869b387599d4e651fdb0bde13f932debe13e353a240ed857c",-3,true,null],[-3,108,"{\"_fn$lastblock\":70368744178665}",-3,true,null]]}
I'm leaving out the middle part to keep this post digestible, so here is the end:
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:58 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1c3045022015954f419348d6225cc61db1a9671300b62ab78695dfd811ad89f0b8767ffa8902210096cdb6e5b230c3b46a079ad48c4ca1df995753cbfc5a6f5888252170f6253d8e"', 'Digest': 'SHA-256=bnIsZEIemFsAtLHZpH6lRKvEiXvUAC6+mBHYKKaI6mg='}
body: {"select":["*"],"from":["_tx/id","58bfe4610286d84e942ab409a29e13a4c63f27788eec541365803c01a47ce832"]}
200
rval: [{"_id":-25,"_tx/hash":"291e7cb361e0f2e368bbf6766193f9070828414751ca7c46c5bf963d1fc008e2","_tx/id":"58bfe4610286d84e942ab409a29e13a4c63f27788eec541365803c01a47ce832","_tx/auth":{"_id":105553116267528},"_tx/nonce":8140585842157148}]
res= True EXPECTED
['judge', 'executive']
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/command False
headers: {'content-type': 'application/json'}
body: {
"cmd": "{\"type\": \"tx\", \"tx\": [{\"_id\": [\"_user/username\", \"judge\"], \"roles\": [[\"_role/id\", \"executive\"]]}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88\", \"fuel\": 1000, \"nonce\": 6974785935112664, \"expire\": 1683913858449}",
"sig": "1b3046022100aa43718ea5962dfd41d94b582986347b916ea5b5729305d9a28e047f713eb331022100c098e5ebe5c01be77accc714a81ba141aed20e49ddfac93781b678b62c2a3dff"
}
200
rval: {"block":14,"hash":"c2049f510231cfb473779afd11055531af96df0d2b8f861f25f702e77879ac06","instant":1683913738474,"type":"tx","duration":"22ms","fuel":553,"auth":"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88","status":200,"id":"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d","bytes":546,"t":-27,"flakes":[[87960930223083,52,123145302311914,-27,true,null],[-27,99,"e2935a9eaf80a9e8673a53b0b2d2d33a377965b8fafe789cc6741ac7c9cc028f",-27,true,null],[-27,100,"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d",-27,true,null],[-27,101,105553116267528,-27,true,null],[-27,103,6974785935112664,-27,true,null],[-27,106,"{\"type\": \"tx\", \"tx\": [{\"_id\": [\"_user/username\", \"judge\"], \"roles\": [[\"_role/id\", \"executive\"]]}], \"ledger\": \"test91371/api-trias-politica\", \"auth\": \"Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88\", \"fuel\": 1000, \"nonce\": 6974785935112664, \"expire\": 1683913858449}",-27,true,null],[-27,107,"1b3046022100aa43718ea5962dfd41d94b582986347b916ea5b5729305d9a28e047f713eb331022100c098e5ebe5c01be77accc714a81ba141aed20e49ddfac93781b678b62c2a3dff",-27,true,null]]}
Signing with: Tf8JAdpa3MASMLdWtcTvayHKdWBngZcWW88
_post_body_with_headers http://localhost:8090/fdb/test91371/api-trias-politica/query False
headers: {'Content-Type': 'application/json', 'X-Fluree-Date': 'Fri, 12 May 2023 17:48:58 GMT', 'Signature': 'keyId="na",headers="(request-target) x-fluree-date digest",algorithm="ecdsa-sha256",signature="1b3046022100c22c03601eae72ed6f21147d956f6a0a0303050f5c41988e8ebfc5eb8d9247c6022100ea940d0bff54af5969d8b346c420c0f5c296dc2175220baa9b875c5d1ae275c2"', 'Digest': 'SHA-256=FTYMZO1hQmIzpJLoK8YujWNyzsDU3SGv1XvVwPQ45/4='}
body: {"select":["*"],"from":["_tx/id","082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d"]}
200
rval: [{"_id":-27,"_tx/hash":"e2935a9eaf80a9e8673a53b0b2d2d33a377965b8fafe789cc6741ac7c9cc028f","_tx/id":"082b34dbab44c9a4de27008cf640f2337fdf416692d4d17a68e9186181ea689d","_tx/auth":{"_id":105553116267528},"_tx/nonce":6974785935112664}]
ERROR: res= False NOT EXPECTED Shouldn't be able to give a judge executive powers
Unexpected result from scenario for scenario1 at sub-test 0
As you can see, we get the complete FlureeDB requests, queries and transactions and their responses, what can be useful when debugging.
runs and hooks
Another new feature that the latesr version of fsst provides are multi-run tests with hooks. Before we look into how these work, lets talk about why we created them for fsst.
We have had FlureeDB running on a k8s system that as it turns out has seen some major crashes. I'm no expert in the details of k8s storage, but the result of the crashes has been that for FlureeDB the raft file ended up corrupted, but corrupted in a special way. Basicly spurious zeros were added to the end of the raft file, and we've been able to restore from this situation bysimply removing the trai