Thank you Ehren, this is extremely useful information. I have a few follow-up questions:
1) Is the information that Curt gives in this thread about memory usage:
https://social.technet.microsoft.com/Forums/en-US/471b18e6-9086-489d-9445-12b3a6205120/pq-query-optimization?forum=powerquery
...still correct? For example, is there still a 256MB cap on the amount of memory used by a single top-level refresh?
2) If so, is there any way apart from using Task Manager to know whether this limit is being hit and which step is causing the problem (for example is there something in the diagnostic logs)?
3) Even if you don't use Table.Buffer(), am I right in thinking that some caching still takes place? Can you explain what the "cache shared nodes" optimisation that was in Power BI Desktop and has only recently moved into Excel does?
https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/16852819-power-query-cache-shared-nodes
I think there are a lot of advanced Power Query/Power BI users out there writing complicated queries that are slow, and better understanding of the internals of Power Query could help us a lot with performance tuning.
Thanks!
Chris
Check out my MS BI blog I also do SSAS, Power BI, MDX, DAX and M consultancy