{"action":"created","issue":{"url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/22","repository_url":"https://api.github.com/repos/davidagold/jplyr.jl","labels_url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/22/comments","events_url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/22/events","html_url":"https://github.com/davidagold/jplyr.jl/issues/22","id":175136343,"number":22,"title":"Parametrized queries/interpolation","user":{"login":"davidagold","id":11862683,"avatar_url":"https://avatars.githubusercontent.com/u/11862683?v=3","gravatar_id":"","url":"https://api.github.com/users/davidagold","html_url":"https://github.com/davidagold","followers_url":"https://api.github.com/users/davidagold/followers","following_url":"https://api.github.com/users/davidagold/following{/other_user}","gists_url":"https://api.github.com/users/davidagold/gists{/gist_id}","starred_url":"https://api.github.com/users/davidagold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidagold/subscriptions","organizations_url":"https://api.github.com/users/davidagold/orgs","repos_url":"https://api.github.com/users/davidagold/repos","events_url":"https://api.github.com/users/davidagold/events{/privacy}","received_events_url":"https://api.github.com/users/davidagold/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-09-05T23:28:36Z","updated_at":"2016-09-13T23:38:36Z","closed_at":null,"body":"This issue is a proper home for the discussion that ended up emerging in #20 re: syntax and semantics for parametrized queries (i.e. (?) prepared statements) and interpolation (the original issue concerned the functionality of extending existing `Query`s via `@query`).\r\n\r\nBy interpolation we mean the ability for users to designate a name within an `@query` invocation as referring to a value in the scope surrounding the `@query` context. For instance, letting `$` denote an interpolated name (value?), the following query\r\n\r\n```julia\r\nc = .5\r\n@query filter(tbl, A > $c)\r\n```\r\n\r\nwould be equivalent to\r\n\r\n```julia\r\n@query filter(tbl, A > .5)\r\n```\r\n\r\nThis sort of functionality is (I think) relatively straightforward. However, it is not the most efficient means of satisfying what I expect will be the dominant use patterns involving this functionality. These use patterns involve collecting a `Query` over an array of values for query parameters, e.g.\r\n\r\n```julia\r\nresults = Array{Float64}()\r\nfor c in consts\r\n    res = @collect tbl |>\r\n        groupby(A > $c) |>\r\n        summarize(avg_B = mean(B))\r\n    push!(results, res)\r\nend\r\n```\r\nThe above is suboptimal because it generates a new `Query` object, with a new `graph` and new `QueryHelper`s for each value of `c`, where the structure of each of the foregoing objects is actually invariant. What would be most efficient would be to generate a single `Query` with a parameter `c` that can be assigned values from `consts` without having to regenerate the `Query` each time. This requires syntaxes that (i) allow users to designate parameters within their queries and (ii) allow users to bind parameters to specific values before `collect`ing the `Query`s. I envision something like the following in place of the interpolation solution above:\r\n\r\n```julia\r\nresults = Array{Float64}()\r\nqry = @query tbl |>\r\n    groupby(A > $c)\r\n    summarize(avg_B = mean(B))\r\nfor c in consts\r\n    bind!(qry, c = c)\r\n    res = collect(qry)\r\n    push!(results, res)\r\nend\r\n```\r\nMy plan is that parameter information will be stored in some mutable aspect of the `Query` object, say in an object of type `Parameters`, and that users will use `bind!` will to bind each parameter to a specific value via keyword arguments, where each key specifies the name of the parameter to bind.\r\n\r\nIt will be important, at least for `collect` machinery for in-memory Julia tabular data structure, that pulling a query parameter from a `Query` object and passing it, say, to a filtering kernel (lambda) be type-inferable in order to avoid boxing the result of applying the lambda to the row and parameter arguments. \r\n\r\nIn the normal case of applying such a lambda just to the values of an iterator over `Table` rows, (i.e. without any query parameters), type information about such row values (which are tuples) is propagated through the type of the iterator, which is just the result of `zip`ing together the individual columns that are arguments to the filtering lambda. That is to say, the type parameters of the zipped iterator convey the element types of the relevant columns of the `Table` data source, since those element types are themselves represented as type parameters of the columns (which are `NullableArray`s). As long as this iterator is passed through a function barrier before being iterated over, type inference can identify the type of the argument (a tuple) passed to the filtering kernel. \r\n\r\nIn the case of a parametrized query, an obvious implementation of this functionality involves passing the parameters from the appropriate field of the `Query` object (after they've been bound) to, in this case, the filtering kernel as a second argument tuple. However, if the mutable structure in which the parameters are stored does not convey the type information of the parameters, then this information may not propagate to the point at which the kernel is actually applied to the row-tuple and the parameter-tuple, which may result in the compiler boxing the result of applying the kernel. Of course, we may assume that a filtering kernel always returns a `Nullable{Bool}`, but in the case of, say, a select kernel we can't make any such assumptions. I think (but could be wrong about this) that the only sure-fire way to avoid such boxing is to convey the type information of the parameters in the type of the `Parameters` object that wraps them. \r\n\r\nThus `Parameters` should be a parametric type, and in order for us to generate useful type parameters, we need to know the types of the parameters before they are ever bound in the `Query`. This suggests a syntax for declaring query parameters and their types. A natural candidate is type assertion:\r\n\r\n```julia\r\nqry = @query filter(tbl, A > c::Float64) # we declare c to be an unbound query parameter of type Float64\r\n```\r\n\r\nThis syntax choice would preclude use of the `::` type assertion syntax in query arguments. I would be okay paying this price.\r\n\r\nIt may turn out, however, that we can find a way to communicate the type information of parameters to the compiler without storing it in the `Parameters` wrapper, in which case `$` syntax for designating parameters would be sufficient. \r\n\r\nNote that query parametrization is a solely a matter of designating parameters within \"query arguments\", that is, non-data source arguments to manipulation verbs within an `@query` invocation. The present package achieves an analogue of parametrization for data sources by means of [dummy sources](https://github.com/davidagold/jplyr.jl#dummy-sources). [As mentioned in comments in #20](https://github.com/davidagold/jplyr.jl/issues/20#issuecomment-244812365), this syntactic distinction reflects a conceptual distinction: one collects the same query against multiple backends by means of dummy sources, and one collects the a query with constant structure but varying values against a fixed backend by using parametrization. \r\n\r\nThoughts? \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/comments/246860438","html_url":"https://github.com/davidagold/jplyr.jl/issues/22#issuecomment-246860438","issue_url":"https://api.github.com/repos/davidagold/jplyr.jl/issues/22","id":246860438,"user":{"login":"davidagold","id":11862683,"avatar_url":"https://avatars.githubusercontent.com/u/11862683?v=3","gravatar_id":"","url":"https://api.github.com/users/davidagold","html_url":"https://github.com/davidagold","followers_url":"https://api.github.com/users/davidagold/followers","following_url":"https://api.github.com/users/davidagold/following{/other_user}","gists_url":"https://api.github.com/users/davidagold/gists{/gist_id}","starred_url":"https://api.github.com/users/davidagold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidagold/subscriptions","organizations_url":"https://api.github.com/users/davidagold/orgs","repos_url":"https://api.github.com/users/davidagold/repos","events_url":"https://api.github.com/users/davidagold/events{/privacy}","received_events_url":"https://api.github.com/users/davidagold/received_events","type":"User","site_admin":false},"created_at":"2016-09-13T23:38:35Z","updated_at":"2016-09-13T23:38:35Z","body":"Those are good questions. It really comes down to defining a collection machinery that implements the semantics you're looking for. Part of this is understanding what is stored in a `QueryNode`. Let's take querying an `Array{Int}` and filtering out even elements as an example. \r\n\r\n```julia\r\njulia> using jplyr\r\n\r\njulia> A = collect(1:10);\r\n\r\njulia> qry = @query filter(A, isodd(a))\r\nQuery with Array{Int64,1} source\r\n```\r\nIf I look at the `qry.graph` I see the above query represented in terms of a `FilterNode`:\r\n\r\n```julia\r\njulia> dump(qry.graph)\r\njplyr.FilterNode\r\n  input: jplyr.DataNode\r\n    input: #undef\r\n  args: Array{Union{Expr,Symbol}}((1,))\r\n    1: Expr\r\n      head: Symbol call\r\n      args: Array{Any}((2,))\r\n        1: Symbol isodd\r\n        2: Symbol a\r\n      typ: Any\r\n  helpers: Array{jplyr.FilterHelper{F}}((1,))\r\n    1: jplyr.FilterHelper{##3#4}\r\n      f: #3 (function of type ##3#4)\r\n      arg_fields: Array{Symbol}((1,))\r\n        1: Symbol a\r\n```\r\nThe `FilterNode` contains information about the input data source (in this case, a `DataNode` signifies that a source, rather than the results of another manipulation, are being fed to the `filter`, the raw query argument `isodd(a)` as an `Expr` , and an array of `FilterHelper`s. Each `FilterHelper` represents a predicate argument to the `filter` verb. So in this case there's just one. It contains a function that represents the structure of the predicate `isodd(a)`, with some modifications. I can use this function to extend the default collection machinery and thereby define behavior for `collect`ing an `Array{Int}` against a `FilterNode`:\r\n```julia\r\njulia> function jplyr._collect(A::Array{Int}, q::jplyr.FilterNode)\r\n           res = Array{Int}(0)\r\n           f = q.helpers[1].f\r\n           for a in A\r\n               f((a,)) && push!(res, a)\r\n           end\r\n           return res\r\n       end\r\n\r\njulia> collect(qry)\r\n5-element Array{Int64,1}:\r\n 1\r\n 3\r\n 5\r\n 7\r\n 9\r\n```\r\nIn these semantics, the name `a` as it appears in the query is just a placeholder for an element of `A`. As the provider of these querying semantics, I'd make sure to document this behavior. You're right that in the context of querying against tabular data structures, the name `a` is taken to refer to an attribute of the table. But this is just a particular semantics provided by the collection machinery defined on tabular data structures. \r\n\r\nIt would probably be useful to add some sort of syntax akin to what you have in Query.jl, e.g.\r\n```\r\nqry = @query a in A, b in B begin\r\n    filter(a * b > .5)\r\nend\r\n```\r\nsince without as much there's no way to enforce coherence of placeholder names from verb to verb. But it's not strictly necessary, as evidenced above.\r\n\r\nDoes this help?\r\n"}}
{"actor":{"display_login":"davidagold"}}