Introduce function API #1022

dmos62 · 2022-01-25T11:53:06Z

This is a continuation of the Backend Filtering Refactor PR (#921).

This PR does not affect the current APIs. It only adds the endpoints and data structures related to the new function API, but does not replace the previous APIs. There will be another PR that will replace the current filtering API with a new filtering API that will use this function API.

Technical details

Should be considered a minor PR, since the functions API doesn't affect the rest of the code and its main "user" in the foreseeable future will be the to-be-implemented filtering API. In other words, this adds an internal API that will not yet be used on merge.

There are some minimal coverage tests for checking if DbFunction subclasses can be used for filtering. I expect to add tests as implementation breadth improves and as the upcoming filtering API is built.

There are two significant improvements planned for the functions API (though not for this PR):

named parameters (currently it's using indexed parameters);
generics (for hinting at the relationship between various inputs and outputs).

Adds the endpoints /api/v0/functions and /api/v0/db_types.

Here are their sample responses. Note that the hints currently applied are not accurate and are not meant to be at this stage. The uses of the hints system are somewhat unclear at this stage, since the frontend will not be using the functions API directly to build up the UI. The near-term future of the hints system depends on what the new filtering API will be like, which, as I said, is yet to be seen. I'd expect the principle of the hints system to be under review and not the details of their current use.

`/api/v0/functions` endpoint sample response


GET /api/v0/functions/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
  {
      "id": "and",
      "name": "And",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          }
      ]
  },
  {
      "id": "column_reference",
      "name": "Column Reference",
      "hints": [
          {
              "id": "parameter_count",
              "count": 1
          },
          {
              "id": "parameter",
              "index": 1,
              "hints": [
                  {
                      "id": "column"
                  }
              ]
          }
      ]
  },
  {
      "id": "empty",
      "name": "Empty",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 1
          }
      ]
  },
  {
      "id": "equal",
      "name": "Equal",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 2
          }
      ]
  },
  {
      "id": "extract_uri_authority",
      "name": "Extract URI Authority",
      "hints": [
          {
              "id": "parameter_count",
              "count": 1
          },
          {
              "id": "parameter",
              "index": 1,
              "hints": [
                  {
                      "id": "uri"
                  }
              ]
          }
      ]
  },
  {
      "id": "greater",
      "name": "Greater",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 2
          },
          {
              "id": "all_parameters",
              "hints": [
                  {
                      "id": "comparable"
                  }
              ]
          }
      ]
  },
  {
      "id": "in",
      "name": "In",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 2
          },
          {
              "id": "parameter",
              "index": 2,
              "hints": [
                  {
                      "id": "array"
                  }
              ]
          }
      ]
  },
  {
      "id": "lesser",
      "name": "Lesser",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 2
          },
          {
              "id": "all_parameters",
              "hints": [
                  {
                      "id": "comparable"
                  }
              ]
          }
      ]
  },
  {
      "id": "list",
      "name": "List",
      "hints": null
  },
  {
      "id": "literal",
      "name": "Literal",
      "hints": [
          {
              "id": "parameter_count",
              "count": 1
          },
          {
              "id": "parameter",
              "index": 1,
              "hints": [
                  {
                      "id": "literal"
                  }
              ]
          }
      ]
  },
  {
      "id": "not",
      "name": "Not",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 1
          }
      ]
  },
  {
      "id": "or",
      "name": "Or",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          }
      ]
  },
  {
      "id": "starts_with",
      "name": "Starts With",
      "hints": [
          {
              "id": "returns",
              "hints": [
                  {
                      "id": "boolean"
                  }
              ]
          },
          {
              "id": "parameter_count",
              "count": 2
          },
          {
              "id": "all_parameters",
              "hints": [
                  {
                      "id": "string_like"
                  }
              ]
          }
      ]
  },
  {
      "id": "to_lowercase",
      "name": "To Lowercase",
      "hints": [
          {
              "id": "parameter_count",
              "count": 1
          },
          {
              "id": "all_parameters",
              "hints": [
                  {
                      "id": "string_like"
                  }
              ]
          }
      ]
  }
]

`/api/v0/db_types` endpoint sample response

GET /api/v0/db_types/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "id": "_array",
        "hints": null
    },
    {
        "id": "bigint",
        "hints": null
    },
    {
        "id": "bit varying",
        "hints": null
    },
    {
        "id": "bit",
        "hints": null
    },
    {
        "id": "boolean",
        "hints": [
            {
                "id": "boolean"
            }
        ]
    },
    {
        "id": "bytea",
        "hints": null
    },
    {
        "id": "\"char\"",
        "hints": null
    },
    {
        "id": "character varying",
        "hints": [
            {
                "id": "string_like"
            }
        ]
    },
    {
        "id": "character",
        "hints": [
            {
                "id": "string_like"
            }
        ]
    },
    {
        "id": "cidr",
        "hints": null
    },
    {
        "id": "date",
        "hints": null
    },
    {
        "id": "daterange",
        "hints": null
    },
    {
        "id": "decimal",
        "hints": null
    },
    {
        "id": "double precision",
        "hints": null
    },
    {
        "id": "float",
        "hints": null
    },
    {
        "id": "hstore",
        "hints": null
    },
    {
        "id": "inet",
        "hints": null
    },
    {
        "id": "int4range",
        "hints": null
    },
    {
        "id": "int8range",
        "hints": null
    },
    {
        "id": "integer",
        "hints": null
    },
    {
        "id": "interval",
        "hints": null
    },
    {
        "id": "json",
        "hints": null
    },
    {
        "id": "jsonb",
        "hints": null
    },
    {
        "id": "macaddr",
        "hints": null
    },
    {
        "id": "money",
        "hints": null
    },
    {
        "id": "name",
        "hints": null
    },
    {
        "id": "numeric",
        "hints": [
            {
                "id": "comparable"
            }
        ]
    },
    {
        "id": "numrange",
        "hints": null
    },
    {
        "id": "oid",
        "hints": null
    },
    {
        "id": "real",
        "hints": null
    },
    {
        "id": "regclass",
        "hints": null
    },
    {
        "id": "smallint",
        "hints": null
    },
    {
        "id": "text",
        "hints": [
            {
                "id": "string_like"
            }
        ]
    },
    {
        "id": "time",
        "hints": null
    },
    {
        "id": "time with time zone",
        "hints": null
    },
    {
        "id": "time without time zone",
        "hints": null
    },
    {
        "id": "timestamp",
        "hints": null
    },
    {
        "id": "timestamp with time zone",
        "hints": null
    },
    {
        "id": "timestamp without time zone",
        "hints": null
    },
    {
        "id": "tsrange",
        "hints": null
    },
    {
        "id": "tstzrange",
        "hints": null
    },
    {
        "id": "tsvector",
        "hints": null
    },
    {
        "id": "uuid",
        "hints": null
    },
    {
        "id": "email",
        "hints": null
    },
    {
        "id": "uri",
        "hints": [
            {
                "id": "uri"
            }
        ]
    },
    {
        "id": "money",
        "hints": null
    }
]

Checklist

My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the master branch of the repository
My commit messages follow best practices.
My code follows the established code style of the repository.
I added tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Also includes implementation for converting to SA filter spec.

Apparently dataclass defaults don't carry over from mixins.

I was wrong. Apparently I just got the order of mixin application wrong.

There's nothing faux about it.

dmos62 · 2022-02-01T16:51:45Z

Requesting re-review @kgodey @mathemancer. Changes implemented. See this comment for a list.

@mathemancer I did not end up using your suggested setup. Current setup works ok and seems simple enough. I had fun exploring your idea though.

Most changes are minimal and to the point.

A noteworthy change is that I lowercased database type names in the table API. It seems that the SQLAlchemy method used to get those type names was uppercasing them. The rest of our code and at least some of the other SA methods return/expect database type names in lower case, so I forced it on the output of that method too.

dmos62 · 2022-02-01T16:59:35Z

I may have jumped the gun. There are failing tests due to switching to lowercase for database type names on the table API.

kgodey · 2022-02-01T17:13:22Z

@dmos62 Why update the existing API, why not use the same SQLAlchemy method in the new API?

Edited to add: Updating the exisitng API also makes this a breaking API change, see workflow here.

This reverts commit 06cea56.

dmos62 · 2022-02-01T17:32:31Z

@kgodey I'm reverting my changes to the casing of database types. After investigating further, it seems that there's a fair amount of logic about the casing of those types, which I didn't notice earlier and I don't fully understand yet, and I'm not comfortable making any changes to it at the moment.

I think that resolving this is outside the scope of this PR. I could uppercase the ids in the db_types endpoint as a partial workaround.

Why update the existing API, why not use the same SQLAlchemy method in the new API?

Because I'm working with database type Enums found under db.types. The code I referred to that's returning uppercased names is working with SQLAlchemy classes, so I can't pass it the data I have. Wrapping in SQLAlchemy classes and then unwrapping could be a workaround, but doesn't feel right: we should be able to compare database type ids without SA.

kgodey · 2022-02-01T18:55:51Z

I think that resolving this is outside the scope of this PR. I could uppercase the ids in the db_types endpoint as a partial workaround.

@dmos62 Can you open a new issue to track this? I agree it makes sense to not block this PR on it, but I don't think the db_types endpoint is usable by the frontend until the ID is consistent with other usage of types.

Ideally, we would have a single source of truth for type IDs and use those everywhere.

dmos62 · 2022-02-01T19:07:16Z

I don't think the db_types endpoint is usable by the frontend until the ID is consistent with other usage of types

Ideally, we would have a single source of truth for type IDs and use those everywhere.

@kgodey agreed. I'll open a ticket for this tomorrow.

dmos62 · 2022-02-02T10:06:23Z

Opened the ticket for inconsistent type name casing #1036

kgodey

@dmos62 Nice work, I had some more comments, mainly about the APIs. One of my previous comments is also unaddressed and I added a comment about that.

kgodey · 2022-02-02T20:51:39Z

db/tests/functions/operations/test_filter.py

+    ("date", "eq", "2000-01-01", 1),
+    ("array", "eq", "{0,0}", 1),
+    # ne
+    # ("varchar", "ne", "string42", 99),


@dmos62 This comment has not been addressed

kgodey · 2022-02-02T21:00:32Z

mathesar/api/viewsets/db_types.py

+class DBTypeViewSet(viewsets.ViewSet):
+    def list(self, request):
+        try:
+            db_name = request.query_params['db_name']


We should be using IDs to refer to API resources, not names. Also, this shouldn't be a required parameter, we should be able to loop through all installed DBs if this is not provided. Filtering by DB ID should be handled similarly to other APIs, see mathesar/api/filters.py.

If there's some reason to require a database, then it's better to put this under /api/v0/databases/<id>/types/. We should not have multiple patterns to provide a database identifier in the URL.

All of this feedback also applies to the functions API.

Fixed. Made new endpoints sub-resources of databases. So /api/v0/databases/<id>/db_types/ and /api/v0/databases/<id>/functions/. When creating the db and abstractions namespaces, I'll split the DatabasesViewSet into two view sets. Will have to think of a way to not retreat to calling one of them DbDatabasesViewSet. I'll probably create Python namespaces: something like mathesar.api.db.viewsets.DatabasesViewSet and mathesar.api.abstractions.viewsets.DatabasesViewSet.

kgodey · 2022-02-02T21:03:59Z

mathesar/api/serializers/functions.py

+
+
+class DbFunctionSerializer(serializers.Serializer):
+    id = serializers.CharField()


It looks like you've instead chosen to only make these available on a single database. In that case, they should not be a top level resource, please see my comment below. If we do choose to make them available on multiple databases, then you'll need to specify the database ID in the serializer.

dmos62 · 2022-02-03T11:40:57Z

@kgodey @mathemancer re-requesting review. Made requested changes. Namely, removed commented tests and made new endpoints' routing coherent with the rest.

kgodey

Looks good to me, I'll defer to @mathemancer to complete review and merge.

mathemancer

I noticed a problem with the function that checks whether a type is installed. To be honest, that logic is a bit all over the place both in the db/types/base.py file as well as the db/types/operations/cast.py file, and should probably be tidied up. I'm okay with deferring fixing the problem I noted for a future PR that also cleans up those functions generally. I'll approve, and leave it to you to decide whether to make the change now or later (or perhaps it's just not needed).

mathemancer · 2022-02-04T15:28:40Z

db/types/base.py

+def get_available_known_db_types(engine):
+    """
+    Returns database types that are both available on the database and known through our Enums
+    above.
+    """


This doesn't check the DB, but rather gets the types which are both available in the engine mapping, as well as the Enums. Does it need to ensure the type is installed on the DB in order to fulfill its purpose? If not, the doc string should change. Otherwise, we should add a way to check that.

I use available in this context to mean that it's installed on the DB. I made a mistake thinking that the engine mapping is a proxy for that.

I was going by the docstring on db/types/base.py::get_available_types:

def get_available_types(engine): """ Returns a dict where the keys are database type names defined on the database associated with provided Engine, and the values are their SQLAlchemy classes. """ return engine.dialect.ischema_names

The original docstring was also a mistake it seems.

dmos62 · 2022-02-07T13:33:49Z

@mathemancer I'll merge this PR in now, and I'll open a ticket to track the problem with checking whether a type is installed.

mathemancer · 2022-02-07T13:40:51Z

@mathemancer I'll merge this PR in now, and I'll open a ticket to track the problem with checking whether a type is installed.

If you don't mind, tag me (or ping me in matrix) when you do. We should make sure the ticket covers the whole problem, which has sprawled across multiple files at this point.

dmos62 · 2022-02-07T13:57:41Z

@mathemancer sure, here's the ticket: #1050.

dmos62 added 30 commits November 22, 2021 16:03

Intro dataclass hierarchy for modeling filtering

f714909

Also includes implementation for converting to SA filter spec.

Rename primitive to leaf and operator to branch

6bdb935

Rename subject to branch

90710c3

Introduce BranchType and ParameterType

b51cf6b

Refactor to not use mixins

1819c0b

Apparently dataclass defaults don't carry over from mixins.

Implement remaining predicates

639c14b

Revert to using mixins

5025f87

I was wrong. Apparently I just got the order of mixin application wrong.

Ammend fauxStatic comment

906cbd1

Rename fauxStatic to static

efd3dc2

There's nothing faux about it.

Rename parameterType to Count; other minor changes

aa9239e

Implement exposing filtering options through REST API

828814a

Dead code

58a9432

switch to snake_case

7705bae

Make name more specific

7379d8c

Impl. parsing our custom filter spec to Predicate

ee1f01a

Rename Leaf.field to Leaf.column

c18e0fe

Move serialization and deserialization routines

74c6626

Rename function to something more specific

78bf19b

Add basic serialization test

311456e

Implement Predicate parameter constraints

201837f

Change filter api nomenclature

54e143a

Fix SA spec serialization

086d1d3

Impl. testing deserialization

6424c1f

Make filters aware of MA type comparability

b1910b7

Reorder MA filter spec object keys

30ea5c4

Switch to using None as default filter

db73bda

Use snake_case for vars and funcs

8d16a0f

Remove duplicate_only (get_duplicates) filter

79172bc

Finish changing casing

28eb1b3

Remove get_duplicates tests

485184e

Linter fixes

9b552c8

dmos62 requested review from kgodey and mathemancer February 1, 2022 16:44

Fix import

fd474b6

kgodey self-assigned this Feb 1, 2022

Revert "Fix table API returning db types uppercased"

ea3f200

This reverts commit 06cea56.

kgodey requested changes Feb 2, 2022

View reviewed changes

dmos62 added 2 commits February 3, 2022 13:30

Move new endpoints to databases resource

2564c0b

Remove commented tests

0053051

dmos62 requested a review from kgodey February 3, 2022 11:39

dmos62 added 2 commits February 3, 2022 13:41

Fix whitespace

fc0f8f6

Fix new endpoint tests

7de8eee

kgodey approved these changes Feb 3, 2022

View reviewed changes

mathemancer approved these changes Feb 4, 2022

View reviewed changes

kgodey mentioned this pull request Feb 4, 2022

Introduce ui and db API namespaces #1041

Merged

7 tasks

Merge branch 'master' into function-api-denuked

70ebae2

dmos62 mentioned this pull request Feb 7, 2022

Fix checks for whether a database type is installed #1050

Closed

dmos62 merged commit 5a6db9f into master Feb 7, 2022

dmos62 deleted the function-api-denuked branch February 7, 2022 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce function API #1022

Introduce function API #1022

dmos62 commented Jan 25, 2022 •

edited

Loading

dmos62 commented Feb 1, 2022

dmos62 commented Feb 1, 2022 •

edited

Loading

kgodey commented Feb 1, 2022 •

edited

Loading

dmos62 commented Feb 1, 2022 •

edited

Loading

kgodey commented Feb 1, 2022 •

edited

Loading

dmos62 commented Feb 1, 2022

dmos62 commented Feb 2, 2022

kgodey left a comment

kgodey Feb 2, 2022

kgodey Feb 2, 2022

dmos62 Feb 3, 2022

kgodey Feb 2, 2022

dmos62 commented Feb 3, 2022

kgodey left a comment

mathemancer left a comment

mathemancer Feb 4, 2022

dmos62 Feb 7, 2022 •

edited

Loading

dmos62 Feb 7, 2022

mathemancer Feb 7, 2022

dmos62 commented Feb 7, 2022 •

edited

Loading

mathemancer commented Feb 7, 2022

dmos62 commented Feb 7, 2022



		class DbFunctionSerializer(serializers.Serializer):
		id = serializers.CharField()

Introduce function API #1022

Introduce function API #1022

Conversation

dmos62 commented Jan 25, 2022 • edited Loading

Checklist

Developer Certificate of Origin

dmos62 commented Feb 1, 2022

dmos62 commented Feb 1, 2022 • edited Loading

kgodey commented Feb 1, 2022 • edited Loading

dmos62 commented Feb 1, 2022 • edited Loading

kgodey commented Feb 1, 2022 • edited Loading

dmos62 commented Feb 1, 2022

dmos62 commented Feb 2, 2022

kgodey left a comment

Choose a reason for hiding this comment

kgodey Feb 2, 2022

Choose a reason for hiding this comment

kgodey Feb 2, 2022

Choose a reason for hiding this comment

dmos62 Feb 3, 2022

Choose a reason for hiding this comment

kgodey Feb 2, 2022

Choose a reason for hiding this comment

dmos62 commented Feb 3, 2022

kgodey left a comment

Choose a reason for hiding this comment

mathemancer left a comment

Choose a reason for hiding this comment

mathemancer Feb 4, 2022

Choose a reason for hiding this comment

dmos62 Feb 7, 2022 • edited Loading

Choose a reason for hiding this comment

dmos62 Feb 7, 2022

Choose a reason for hiding this comment

mathemancer Feb 7, 2022

Choose a reason for hiding this comment

dmos62 commented Feb 7, 2022 • edited Loading

mathemancer commented Feb 7, 2022

dmos62 commented Feb 7, 2022

dmos62 commented Jan 25, 2022 •

edited

Loading

dmos62 commented Feb 1, 2022 •

edited

Loading

kgodey commented Feb 1, 2022 •

edited

Loading

dmos62 commented Feb 1, 2022 •

edited

Loading

kgodey commented Feb 1, 2022 •

edited

Loading

dmos62 Feb 7, 2022 •

edited

Loading

dmos62 commented Feb 7, 2022 •

edited

Loading