178 lines
12 KiB
Markdown
178 lines
12 KiB
Markdown
-
|
||
- ---
|
||
- title: "Unlearning SQL to Learn LogSeq's Datalog"
|
||
- date: 2022-10-20
|
||
- lastmod: 2022-10-22
|
||
- slug: "unlearning-sql-to-learn-logseqs-datalog"
|
||
- categories:
|
||
- - "Personal Knowledge Management"
|
||
- ---
|
||
- ## Introduction
|
||
- Welcome to the post I wish I had when I was first tackling datalog a few months ago.
|
||
- The purpose of this post is NOT to teach you how to do Advanced Queries in general. There are a number of resources for that already. My favorites are...
|
||
- - [The LogSeq Advanced Queries Documentation](https://docs.logseq.com/#/page/advanced%20queries)
|
||
- - [Learn Datalog Today!](http://www.learndatalogtoday.org/)
|
||
- - [The Datomic docs]([https://docs.datomic.com/on-prem/query/query.html](https://docs.datomic.com/on-prem/query/query.html))
|
||
- - [This blog post with both simple and advanced query examples](https://bgrolleman.gitlab.io/logseq_publish_toolsontech/#/page/logseq%2Fadvanced%20queries)
|
||
- - [This Github gist with a bunch of example queries extracted from Discord](https://gist.github.com/houshuang/ca4c3e79a088f660dfd9e2c902da6abb)
|
||
- - [The LogSeq Schema](https://github.com/logseq/logseq/blob/master/deps/db/src/logseq/db/schema.cljs)
|
||
- Instead, the goal of this post is to get rid of some of the thinking patterns that served you well for SQL, but will cause you to bang your head against the wall in datalog. As part of this, we'll walk through a query I wrote that took me a long time to wrap my head around, because shedding my SQL brain without any guidance was hard.
|
||
- Hopefully my struggles will be helpful to you in your exploration as well.
|
||
- ## Big Idea 1: Do **NOT** presume you know what you need
|
||
- When I got started with my problem query, I had a map in my head of how the SQL should look. It was a fairly straightforward Left Outer Join. Maybe with a little subquery depending on the details of the database.
|
||
- So naturally, I just needed to figure out how to do a left join and/or subquery in datalog, right?
|
||
- {{< admonition type=danger title="NO!" open=true >}}
|
||
- **VERY VERY WRONG! DO NOT DO THIS!**
|
||
- {{< /admonition >}}
|
||
- Ahem...
|
||
- When jumping into this with your SQL brain, you *must* let go of what you think is the right answer. Otherwise, you'll chase dead ends and get frustrated trying to make datalog do something in a way it isn't meant to.
|
||
- Datalog does things differently. Let go of assuming you know the right way to solve your problem. Then, you can move forward.
|
||
- Feel free to Google for the answers if you want. You'll find a disturbingly low number of hits. Even the pages that are useful have answers like "run two queries, because datalog isn't good at that."
|
||
- ## Big Idea 2: There are NO tables
|
||
- So the crazy thing about datalog is that there are no tables. Or, if you prefer to think of it another way, there's exactly one table. All entries have three columns (technically five, but for the sake of anything LogSeq query-related we're sticking to three).
|
||
- ```clojure
|
||
[entity_id attribute value]
|
||
```
|
||
- That's it. The whole database is a giant stack of these. That includes database info. That includes schema info. Foreign keys. Values. Everything.
|
||
- This leads to some interesting implications. Namely...
|
||
- ## Big Idea 3: There are NO joins
|
||
- It's not that nothing can be connected, but rather that you don't spell out the connections in the same way as SQL. You write a series of statements that must be true all at once, which constrains your results to the ones you want. Datalog just figures out what to do from there.
|
||
- This works because everything is just a `[entity_id attribute value]` triplet. Since all entity_ids are unique, any foreign keys would just have a different entity id as its value. The query engine just sort of figures it out from the links you define in the query.
|
||
- I've seen this pattern in the DB of a program or two I dealt with years ago. No idea what the advantage of such a schema would be in a normal SQL database, but if you've run into an app DB that had a suspiciously small number of tables, you might have seen the pattern before.
|
||
- {{< admonition type=warning title="Caution" open=true >}}
|
||
- If you go googling, commands with the word `join` in them do exist in datalog/datomic/datascript, but they *don't* do what your SQL brain thinks they do. They actually do something much more akin to subqueries. We'll see an example later.
|
||
- {{< /admonition >}}
|
||
- ## Big Idea 4: Everything is NOT a filter, despite being in the `where:` clause
|
||
- In the simple LogSeq queries, everything feels a bit like a filter. You're always narrowing things down to the specific stuff that you want.
|
||
- In advanced queries, that's no longer the case. Even though we put everything in a `where:` clause, there's a lot more work to do there than in SQL.
|
||
- Complicated queries will have you binding things from different places without any filtering at all until later in the query. This is hugely important as you get into chaining different pieces of the graph together.
|
||
- ## Example Time!
|
||
- Let's walk through my own query that gave me a headache for more nights than I'm proud to admit.
|
||
- The query we'll build is one that drives my little CRM in LogSeq.
|
||
- To be more precise, I have multiple pages tagged `contact`, and they have a property `list` with an A,B,C,D value to give the contact a priority. For each alphbetical group, I want to see the contacts where I have NOT written any journal entries about them within a certain timeframe. A listers should be contacted frequently, while D listers can just get holiday cards.
|
||
- ### High level considerations
|
||
- Because I want different time values for each group, I'll probably need a different query for each letter. No biggie. I've heard of this copy and paste thing before. LogSeq is already good with running multiple queries on a page, so I don't anticipate a problem there.
|
||
- As for writing the where clause itself, we'll have to step through multiple triplets to establish what we want. Because the presence of the contact on a dated journal page is our condition, *not* a value on the contact page itself we'll need to go Advanced. If it was a property on the contact page, a simple query for a date would've been much easier. I actually thought about writing a plugin to just update a property when it was linked to from the journal instead... but I believed in the power of queries (and also felt too lazy to figure out plugins).
|
||
- Clearly I need the set of contacts for a letter, but also to exclude the contacts that meet a condition. The latter is where all the complicated stuff comes in, so let's do that part first.
|
||
- ### Getting the triplets in place
|
||
- To start!
|
||
- Well I know I'm going to need to scope the journal entries to a date range in the query, so let's start there, pulling from one of the [Advanced Query examples](https://docs.logseq.com/#/page/advanced%20queries):
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?b [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
(between ?b ?start ?today)
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- Note that I brought the ending bracket for the `where:` clause down to a new line because I like to paste things in without thinking about its placement all the time.
|
||
- I would read the query as "give me ALL the journal blocks in the past week." Thinking in those big terms helps, because each operation basically acts globally on a whole set. That all the triplets are implicitly AND'ed together is what's doing the restriction.
|
||
- So next, we'll need "bind ALL the links (entity id's to other pages) to a variable." You'll note that I'm moving `?b` down as we go to keep seeing the latest results when I run the query.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?b [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
(between ?journals ?start ?today)
|
||
[?journals :block/refs ?b]
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- Thus, a list of all the blocks that are linked to by journal entries within the last week. These results might be a little weird, especially if you linked to a lot of things in that time, but the contacts are present!
|
||
- Next, lets filter those blocks to the ones with a `list` property. Note this is a two step process, because we need to pull out the property from the full list of properties first. I'm pretty sure `get` is a Clojure function because it's enclosed in brackets. You'll note with `between`, that the parentheses (a *rule* in the datalog... or a simple query expression if you prefer) are *not* enclosed.
|
||
- There is a simple rule for this, but be careful using simple rules in Advanced Queries, because sometimes they filter results in ways that are hard to understand.
|
||
- We're also not moving the `?b` down the query, because we still want the same result set. So we keep returning the entity id's of the pages in those journal links... just with a few more constraints.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?b [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
(between ?journals ?start ?today)
|
||
[?journals :block/refs ?b]
|
||
[?b :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- One more filter, and that gets us to only the blocks that represent the pages that are in the A List.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?b [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
(between ?journals ?start ?today)
|
||
[?journals :block/refs ?b]
|
||
[?b :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
[(= ?list "A")]
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- We can replace the `?b` with `?p` to make the results table prettier, since we know the results will be pages. Remember, in Advanced Queries, the `?b` and `?p` are special, focused on returning blocks or pages respectively.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?p [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
(between ?journals ?start ?today)
|
||
[?journals :block/refs ?p]
|
||
[?p :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
[(= ?list "A")]
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- Great! The only problem is those are the pages we *don't* want. The point of this query is to follow up with people we don't have any notes for. So we'll start by commenting out the work we just did and pulling in all the A listers again above them. A little whitespace helps keep the sections apart.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?p [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
[?p :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
[(= ?list "A")]
|
||
;(between ?journals ?start ?today)
|
||
;[?journals :block/refs ?p]
|
||
;[?p :block/properties ?prop]
|
||
;[(get ?prop :list) ?list]
|
||
;[(= ?list "A")]
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- Now the tricky part. We've essentially built a subquery with results to exclude, so we can put that whole commented section in a `(not)` or `(not-join)` clause.
|
||
- The difference between them is that `(not)` forces all your variable bindings to match the ones outside the clause. `(not-join)` lets you pick which ones to match.
|
||
- Since we make so many chained connections, I'd rather use `(not-join)`. It makes copying and pasting easier.
|
||
- ```clojure
|
||
#+BEGIN_QUERY
|
||
{:title "The A Listers that need follow up"
|
||
:query [:find (pull ?p [*])
|
||
:in $ ?start ?today
|
||
:where
|
||
[?p :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
[(= ?list "A")]
|
||
(not-join [?p ?start ?today]
|
||
(between ?journals ?start ?today)
|
||
[?journals :block/refs ?p]
|
||
[?p :block/properties ?prop]
|
||
[(get ?prop :list) ?list]
|
||
[(= ?list "A")]
|
||
)
|
||
]
|
||
:inputs [:7d-before :today]}
|
||
#+END_QUERY
|
||
```
|
||
- And TA DA! A list of A list contacts, that only shows people we haven't taken notes on recently. It doesn't seem to tough in retrospect. |