Ehren, thanks a lot for your very helpful information.
And I have question - why, if I do a refresh of a top-level Query in PQ UI (what you wrote about), PQ performs it through as many independent refresh processes (through several separate Mashup processes), as we have steps in the top-level Query? I think that these refresh processes are totally independent from each other (even if several of them are performed through one Mashup process), because Buffer doesn't work for these processes. So I'll have N queries of a data source and all the previous calculations for every of N steps in the Query, regardless of whether I use Buffer or not. I understand that I use Buffer properly because in this situationPQ makes 1 request to the datasource if I refresh #"Q2'":
Section1;
Q1 = Table.Buffer(datasource);
Q2 = let s1 = Q1, .. sN in sN;
Q2' = Q2;
SoI think that when you say about "refreshing of top-level query", let's say refresh Q2, you mean "independent refreshing of every step of the Query", not just one refresh for the Query as a whole - at least that's what I see via tracing. And that's happens even if "Allow data preview to download in the background" is set off. The latter has effect only in case of changing the Query, not refreshing it.
Taking into account, that refreshing of all these N steps is done through just up to 4 separate Mashup processes (from my experience), and inside of that processes refresh of every step is performed on one-by-one basis, I definitely may expect slow down of the things in case of big data sizes or complex ETL.
But situation is even more inconvenient. In case of a Query with a number of steps,if I click (right after refresh of the Query)on some steps in the UI, instead of showing me results of the step instantly, refreshing of that particular step is launched again, and I cannot see it's results until the refreshing is finished. Again - the full refresh of the Query is just finished and I haven't done anything else apart of clicking on it's steps in UI. And the large a Query the more proportion of steps behave in that manner.
And that's happens on a PC with 64 bit Excel and 70% free RAM of total size of 40Gb and Peak RAM usage by a Mashup process of 90 - 223 K (according to Task Manager). And for data sources of total size of just several MB. And that's for just ETL with 4 source-reading plus 5 top-level Queries in a chain and 6 functions called from tables on different steps, 4 of which where created from top-level Queries. That's doesn't seem to beoverwhelming.
That latter experience (with refreshes of particular steps on the fly despite a full Query refresh) is particularly frustrating. Of course I can hide all the steps in lists of tables' cells in order to use laziness. But, apart from my own inconvenience, PQ is created (as far as I understand) for self service purposes by clients. And, if I provide such a cryptic table to my clients, the maintaining of it of any kind will be very inconvenient for them. At the end, you did a lot of amazing job in order to let us perform _a very complex ETL_ VISUALLY - I'm not tired to be fascinated by that - that it is very pity to throw away much of your UI achievements and start just programming AND - provide just a program to clients, instead of the nice self-served solution that may be (at least partly) supported by clients themselves.
So what may be done in order to workaround the issue of unnecessary calls for Refresh of a Query and still use UI ETL at full scale? My be there are some configuration parameters for that, or some other techniques?
All my questions are just for the sake of my clients and in order to have a great product at hands, so I'm sorry for any inconvenience and thank you very much in advance for your answer!