Quantcast
Channel: Table.Buffer for cashing intermediate query results or how workaround Unnecessary Queries Issue
Viewing all articles
Browse latest Browse all 21

Table.Buffer for cashing intermediate query results or how workaround Unnecessary Queries Issue

$
0
0

Hi All!

I'd like to use Table.Buffer to workaround the issue of multiple unnecessary calls of
A. steps of a Query and
B. Queries in a query chain in Power Query.

The issue is that PQ make a lot unnecessary calls (at least I see them through tracing). If you change some step in a Query (in this text a Query is a field of #sections[Section1] of global PQ environment), then:
A. if we have N steps in a Query, then all the N steps will be called in this manner: first step will be called N times, second - N-1, and so on (the last step will be called once). Let's illustrate it as NQs1, N-1Qs2 ... 1QsN.
B. All the previous AND next queries in a Query chain will be called in this manner - N time every previous queries in a chain, and 1 time every of the next.

So if you have a chain of Q1 -> Q2 -> Q3 -> Q4 -> Q5 queries (Q2 uses Q1, and so on), and Q3 has 5 steps, and you change any step in Q3, you'll have 5 Q1, 5 Q2, 5 Q3s1 4 Q3s2.. 1 Q3s5, 1 Q4 and 1 Q5 calls. Seems like digital cash mining just to see a result of a small change, isn't it?

So if you make a small change in a Query step, PQ will do a lot of work, recalling the same queries, while me as a user perfectly know that there weren't ANY changes anywhere except this small change I just made in one particular step and would be happy to see the results ASAP. In case of simple queries it's okay , and you workarounded calls to sources issue through cashing. But in case of more or less complex ETL (which means a lot of work AFTER you downloaded data from source) it just makes such workload not possible to develop in UI.

SO - I'd like to buffer complex calculation whether inside a Query for it's steps or between Queries, but it seems that Table.Buffer doesn't provide any service you may intuitively expect from this function (lack of official documentation let us use our imagination at full force ;-)).

So I have 3 questions:
1. What exactly Table.Buffer does?
2. Is it possible to make some workaround of Unnecessary Queries Issue? It seems that I even cannot use laziness for that - you calculate all the expressions whether they are directly shown in UI of steps (e.g. in cells of a table) or I hide them in e.g. lists and never use them directly anywhere!
3. Why do you do SO MANY all these Queries?

Viewing all articles
Browse latest Browse all 21

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>