Google proposes Logica data language for building more manageable SQL code
- Reference: 1618482309
- News link: https://www.theregister.co.uk/2021/04/15/google_logica_language/
- Source link:
"Good programming is about creating small, understandable, reusable pieces of logic that can be tested, given names, and organized into packages which can later be used to construct more useful pieces of logic," explain Google software engineers Konstantin Tretyakov and Evgeny Skvortsov in [1]a post to Google's open source blog. "SQL resists this workflow."
[2]
Tretyakov and Skvortsov propose using a new open source logic programming language called, aptly enough, [3]Logica , to craft database interactions using the syntax of mathematical propositional logic instead of the chains of English words used in SQL.
SQL now a dirty word for Oracle, at least in cloudy data warehouses [4]READ MORE
[5]
Logica, its creators say, "stands for Logic with aggregation." The project's description on GitHub makes its target audience a bit more clear: "Logica is for engineers, data scientists and other specialists who want to use logic programming syntax when writing queries and pipelines to run on BigQuery."
Logica code compiles to SQL which Google hopes will get run on BigQuery, the ad giant's data platform-as-a-service. But it can be run locally and SQL is portable; it also offers experimental support for targeting PostgreSQL and SQLite.
The language is "more concise and supports the clean and reusable abstraction mechanisms that SQL lacks," or so say Tretyakov and Skvortsov.
Here's how a basic query might look in Logica code: MagicComment(comment_text:) :-
`comments`(user_id:, comment_text:),
user_id == 5;
And here's the equivalent in SQL: SELECT comment_text FROM comments WHERE user_id = 5;
Logica is the successor of a [6]Datalog -like language Google developed internally and discussed in 2015 called Yedalog, which tried to provide a tool for querying large, semi-structured data sets.
Tretyakov and Skvortsov don't explain why Yedalog needed a successor but a 2016 paper
[7]PDF
by the Yedalog's creators suggests that Google's Yedalog implementation automated control decisions normally left to those working with more general purpose languages.Because that created uncertainty about who had operational responsibility for keeping things running, the Yedalog developers drafted documentation describing a suggested operational model. But they found that their users learned by experience, rather than reading, and that experience led them to make assumptions that the Yedalog team hadn't anticipated.
"The model they learned assumes some forms of automation beyond any we had planned to support," the paper explains.
Whatever the current state of Yedalog at Google these days, Logica aims to move beyond verbose COBOL-style caps lock English to the language of formal logic, in the hope database code can be broken up into more manageable pieces.
[8]
"This inherent resistance to decomposition of logic into bite-sized pieces is what leads into the contrived, lengthy queries, the copy-pasted chunks of code and, eventually, unmaintainable, unstructured (note the irony) SQL codebases," quip Tretyakov and Skvortsov. ®
Get our [9]Tech Resources
[1] https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YHhjHqETqlc5GdV6gt6HGAAAAJI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://github.com/EvgSkv/logica
[4] https://www.theregister.com/2021/03/18/oracle_cloud_data_warehouse/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YHhjHqETqlc5GdV6gt6HGAAAAJI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://en.wikipedia.org/w/index.php?title=Datalog&oldid=1015453010
[7] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45983.pdf
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YHhjHqETqlc5GdV6gt6HGAAAAJI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://whitepapers.theregister.com/
No.
It's not just you.
I could be interested in seeing how a query with several outer joins would be represented, but not interested enough in googling it.
I'd also like to be the first to XKCD this board: https://xkcd.com/927/
I would like to see a more complex example, also - and see how it really becomes more readable and "structured" than SQL.
Also, why the "=="? Niklaus Wirth was right, you need to change the assignment operator, not the equality one.
And the ":-" ?
I don't know why they decided to use the "penis" operator. Is not gender neutral.
Penis operator
My sentiment exactly, unbelievably insensitive towards e.g. penilewise challenged trans males and probably many other minorities yet to be discovered!!1!
:- is from Prolog. All of these languages descend from Prolog via Datalog I think (Datalog is pretty much Prolog without the Turing-completeness, or with less Turing-completeness, so queries are guaranteed to terminate).
I'm not sure why this is more interesting than Datalog though.
Oh it's better than that, it's much better than that.
They use '==' both for assignment and equality, depending on context.
So, you know, it's like people writing maths where you often lazily say 'f(x) = x^2' to mean 'I am defining the function f(x) to be x^2' and then 'sin(t) = t' to mean 'I am interested in the values of t for which sin(t) is equal to t' rather than 'I am defining sin(t) to be 0 for all t' (the answer to that one is 't=0' of course, which is again not defining t as 0'). Very occasionally people are more explicit but usually not.
Except for some fucked-up reason of their own they've chosen to use '==' for both things instead of being compatible with several hundred years of lazy mathematicians and using '=' .
This language is ... not going anywhere, is it?
Yep it's a poor trivial example. Having written/maintained several SQL based reporting systems over the years, often querying unverified data, SQL can get very messy & very quickly as you add complexity. Much of my work was to manipulate the data at the right point in the process to simplify the process, validate the data and end up with optimised queries that ran at acceptable speeds.
So Logica might be a good thing if it significantly reduces the work necessary to do the above but is impossible to tell from this article.
On the other hand, one of SQL's big advantages is that a non-coder can build their own simple (and not so simple sometimes) queries to help them do their job and something like Logica could maybe exclude those users if the SQL interface was walled off.
But I presume Google are looking at this from a perspective of multiple magnitudes of complexity higher than anything I've ever done and I can see that at that level it makes sense to use code that is much more powerful and maintainable (if that actually is the case with Logica).
Worse than that. What happens when you 'join' multiple bits together - you'll have no way of understanding the efficiency of the final SQL.
Re: no way of understanding the efficiency of the final SQL
Can you do that anyway without looking at the query plan?
Will post about this on Hangouts and Orkut
Do bookies take bets on when Google will abandon yet another project? Asking for a friend.
Re: Will post about this on Hangouts and Orkut
I can't picture Google doing that, maybe I'll look in Picassa
Re: Will post about this on Hangouts and Orkut
This thing is explicitly a research project. For which mean 'they've already abandoned it'.
Bring back QUEL
Bring back [1]QUEL , that's what I say!!
[1] https://en.wikipedia.org/wiki/QUEL_query_languages
Re: Bring back QUEL
Is Logica a typo? Caligo (latin) is an actual description ... it's a messy mess.
Re: Bring back QUEL
Yes, returning to the fundamentals of relational calcula would have great advantages, especially as a "wire" protocol betweent the client language and the server.
Is this a flash in the pan
My first thought was 'how long will this last. I'm not knocking it as I primarily write in SQL and have seen, written long bits of code**, I just want others to give their views on how realistic this is and could it take off.
Basically, is it worth learning?
**Projects often dictate how fast you have to go and this leads to very messy quick fixes just to get a result (thats my defense and I'm sticking to if(haha))
Re: Is this a flash in the pan
"Is this a flash in the pan"
No, more like a "floater"
Now I can't code my way out of a wet paper bag but I did learn SQL.
For some reason it just made sense. I think it was the "English" style of expression that helped me overcome my mental block on anything resembling maths or logic. The same with Cobol which I also had to learn. As others here have said the example looks to me like the SQL version is more comprehensible. The Logica one has more of a feel of a high level programming language so my mental block comes into play and it all looks very confusing. I would hate to have to work through a more complicated example.
Still having said all that I am not in the target audience for this so others may find it easier to use.
Re: Is this a flash in the pan
It looks awful in that example, but I'd like to see a more complex example.
It might be better.
Re: Is this a flash in the pan
very messy quick fixes
So you're the one!
To run Logica locally you need Python3
Pass.
All depends on the author coding
A complex SQL stored proc (SP) can be nice and readable, using lots of other SPs or functions.
This fits with the idea of "small, understandable, reusable pieces of logic that can be tested, given names"
And obviously the use of parameters / named variables.
However there's nothing to stop someone writing a 1000+ line monster SP
Unless logica in some magic way stops that, readability will still depend on the authors coding style
Re: All depends on the author coding
But we're not talking about stored procs here, we're talking about database queries. No stored proc in the world is going to address the readability of a query which is several hundred lines long and designed for speed.
Re: All depends on the author coding
If it's several hundred lines long it needs to be designed for speed.
Re: All depends on the author coding
A stored proc is a query....
Performance advantage & code reuse as main advantages (plus having code under source control f you go that route on your SPs)
Re: All depends on the author coding
I normally use them heavily, though I did nearly go insane when doing something really easy on Postgres because I didnt realise it allowed SP overloading so you could have lots of SPs with the same name but different arguments so as you were developing and modifying the DB and associated code weird things would happen even though I could swear the code was right. It wasn't until I found out the SQL to list all the SPs that I laughed long and loud and scared the whole office.
Re: All depends on the author coding
Although for me anything ORACLE should normally be avoided, the one thing I DID like about ORACLE SQL was the fact you could have a single package with multiple methods, functions, some public others private. Easier to keep things all together. SQL server the same thing could have 10's of seperate SQL stored procs and functions. Makes to code base a bit more messy.
And yes, once you get over a hundred lines it can become a paid to read. Not really designed to do lots of complicated stuff easilly.
having 16 joins or more in a SQL is a Pain.....
having people design tables without knowing what they are doinf and they having to deal with the consquences afterwards is also a real pain. Not everyone has the option of refactoring a database (OR part of it) to make it more meaningful.
Re: All depends on the author coding
Experience as a sometime DBA stongly suggests that a major source of poor RDBMS performance is that the system designers have no idea of which queries will be the most frequently used and that this information is seldom available to the DBA. This isn't really their fault since very often the system's sponsors and future users don't know this either. The result is that all too often a well-designed RDBMS schema generates a database that runs like a lobotomised snail.
Very often its performance can be hugely improved by merely adding indexes that actually support the queries that the application's logic needs to do its designed task and.or reorder the columns within indexes so the queries return rows in the required sequence without needing to sort then as they are extracted from the database.
Its quite remarkable how much a star schema can be sped up by merely reordering columns in the prime indexes and, of course, this sort of fix doesn't require any of the SQL queries or the code calling them to be rewritten.
Whats needed to improve the average RDBMS performance or maintainability is not a sexy new way of writing queries: instead all that's needed is an SQL interface with an EXPLAIN capability to analyse how any given query will interact with the RDBMS content, structure and query mix plus a DBA who is well enough trained to use the EXPLAIN output to better match the database's physical structure to the queries executed against it.
Re: All depends on the author coding
I've always used SPs for web access - you can normally secure a DB so the web user can only run SPs and not random queries - initially a a security step but it quickly became apparent you could actually write really complicated code by breaking it down into well chosen SPs and objects and end up with libraries of SPs and functions that made life really easy for yourself and isolate the DB structure from the user (or web user) and make quite major changes without fucking up the web interface or the three hundred random copies of excel about the office.
I do wonder whether Logica is another one of these inventions by someone who hasn't read the SQL manual to the end.
They seem to be missing the real reason SQL becomes a problem in large systems - which is that it's usually embedded in a general purpose programming language as strings that can't be analysed and refactored.
The solution is something like jOOQ, which provides an API that has type safety, compile time checking and the possibility of refactoring. I'm sure similar solutions exist for other general purpose programming languages.
Being in strings shouldn't prevent that, more a tooling problem. Not sure if I'm recalling correctly as I don't like the language, but didn't vb.net support inline xml that was treated pretty much as code rather than a string?
Thats what ODBC helps with and JDBC fixes almost entirely, because both generally return columns in the programming language's nearest match to a particular column's data type.
Of course, if the database was "designed" by the sort of numpty who thinks integer values should be stored as character strings, then all you can do is light the blue touch paper and run.
I'm not familiar with ODBC, but JDBC does not solve the "queries as opaque strings" problem, nor does it offer type safety. Something like jOOQ generates mappings by querying the schema, so you go from JDBC like:
ResultSet rs = statement.executeQuery("SELECT foo, bar FROM baz");
while (rs.next()) {
...
}
To jOOQ like:
List
Both the query and mapper work in a type safe, compile time checked, manner. If the underlying DB schema changes, it's caught at compile time.
SQL resists this workflow
It may be the case that:
Good programming is about creating small, understandable, reusable pieces of logic
But you don't (shouldn't) write entire programs in SQL. You can talk about operations on a relational database in terms of algebra (which has some implicit notion of procedure and order) or in terms of calculus (where you define the transformation rather than individual steps). Most people would find it easier to conceive of what is going on in procedural terms and SQL is an expression of that form (firstly JOIN this to that then secondly SELECT a subset WHERE come condition applies) but in practical databases the query optimiser may well reorder the operations to get the best performance so SQL is in reality only an expression of the desired outcome.
Again, in a practical system, you might need to chain some small number of operations together in a transaction to achieve some specific purpose and you might create a stored procedure for convenience, but if it isn't small, understandable and reusable you're doing it wrong.
In any case, given that SQL is just a specific syntax for expressing the basic maths, I'm not sure that simply inventing a new syntax to express the same concepts is going to help the cause of understanding especially if you have users determined to "learn by experience, rather than reading". I can see nothing about:
MagicComment(comment_text:) :-
`comments`(user_id:, comment_text:),
user_id == 5;
with its magic punctuation that is inherently clearer than (note that the CAPS-lock is typically optional):
select comment_text from comments where user_id = 5
[Edit: sorry, it looks like this has all been said already - phone call intervened before "submit"]
Re: SQL resists this workflow
Yes, but FWIW SQL is Turing complete.
4GL
This just seems to be a 4GL a la the Eighties.
Also, do CGI still own the Logica trademark?
Rapport anyone?
Little late for an April fool joke
the whole point of SQL is that it is easy (quick) to parser and optimise on the server side, relative to the time taken to execute.
For further optimisation, a hash indexed cache of execution-plans and cursor handles avoids most of the actual parsing.
where things can be improved is in the area of mapping (to classes), and client side syntax checking.. where C# LINQ has basically got it nailed
from c in database.Customers where c.Id == 5 select c;
=>
select c.* from customers where c.Id = 5;
Re: Little late for an April fool joke
For repetitive queries, where the parameters are passed separately, parsing time is not relevant.
SQL wasn't developed for the server but as the client interface for non-programmers. It was then more or less deliberately knobbled by vendors who wanted to be able to push their own proprietary solutions.
Re: Little late for an April fool joke
SQL is a nightmare to parse!
Large numbers of keywords to support.
Inconsistent grammar (sometimes "var = x " sometimes "(var1, var2) (x, y)"
Massively recursive you can place another "select" almost anywhere.
Really inconsistent as in you cannot place another "select" just anywhere.
It can also be tedious to write matching lists of variables to lists of parameters or values is just plain error prone.
Codd hated SQL! It took the beautiful mathematically provable relational algebra and turned it into a messy natural language with all its ambiguities and opportunities for misunderstanding.
Here are a few examples of SQL parsers:
https://github.com/antlr/grammars-v4/tree/master/sql
Most of these come in at 2000 plus line of ANTLR code compared with a language like smalltalk with comes in at 104 lines of code.
Simplifications
Maybe someday someone will invent views to encapsulate complicated query logic. Or am I dreaming?
Backticks for the fail
Any programming language that requires backticks is doomed because they can be a real problem for non-ASCII keyboards and users.
Re: Backticks for the fail
Backticks are used in .MD files (markdown documentation) and Javascript.
It should be called RSI
17 punctuation characters in the simplest of queries, 9 of which require the use of the shift key.
File alongside the hundred previous attempts to create a "better" query language that have all been worse than SQL.
Lawyer fodder?
I suspect that [1]CGI , which bought the venerable UK software outfit [2]Logica in 2012, may still have a vague desire to hang onto its trademark.
[1] https://en.wikipedia.org/wiki/CGI_Inc.
[2] https://en.wikipedia.org/wiki/Logica
Re: Lawyer fodder?
Yes, this was what I thought. Did they do any research before thinking of the name, like, I don't know, typing 'logica' into some search engine or other (perhaps they don't have access to a search engine, I don't know)?
'Designed to run on BigQuery'
Wake me when it runs on MSSQL, PostgreSQL, and even SQLLite
A research project with no manual, a few examples only some of which appear to be better expressed in Logica than SQL is not entirely enticing.
SQL can be a real pain to write at times, but I'm not immediately seeing this as the solution.
Maybe it's just me, or it's a bad example, but the sample shown looks far quicker and easier to understand as SQL than Logica?