Hi Ehren and Chris, and All!
Ehren, this is extremely helpful information, it's really appreciated! IMHO the earlier we have such info, the better, regardless the possible problems in details. It's because it anyway helps to see the whole picture. And gives a lot of info we need time to think about before you may do some changes. Thanks again for the information!
Ehren, this is extremely helpful information, it's really appreciated! IMHO the earlier we have such info, the better, regardless the possible problems in details. It's because it anyway helps to see the whole picture. And gives a lot of info we need time to think about before you may do some changes. Thanks again for the information!
And yes, I still have questions, and still about Buffer :-).
1. Is Buffering "flat" or "deep"? I mean - If a table consists from cells with not only with scalar values (lists, tables, records), and some of non-scalars already where evaluated - will non-scalar values be stored in Buffer? And will
not evaluated non-scalars not? It seems for me that Buffer is "flat", and I have to Buffer non-scalar cells in a table and table itself separately, other wise I have Buffered table and not Buffered non-scalar Cells :-).
1.2. By the way (regardless of Buffer) - how cells of a table with scalar expressions are calculated - immediately as they appear in a expression or upon access (obviously the question is not about the situation, when you see the table in PQ UI)? I read about streaming semantics and lazy evaluation of Records, but from my point of view there is no clear answer there.
If they a lazy evaluated, what happens when we use Buffer for a table with some not evaluated scalar expressions? They are calculated at the point we Buffer the table? Logs make me feel that yes, but it is interpreted, not direct, info.
2. Immutability of variables and their recalculation. May PQ, under some circumstances, calculate really _immutable_ variables again? Theoretically, in case of memory shortage, PQ may remove the variable from memory in order to free it, and recalculate
the variable upon next access. When I stress _immutable_ - that means exactly that there weren't used any mutable operands for variable calculation, whether by nature of operands or through Buffer. I understand that _I_ cannot change the value of variable,
but it doesn't mean that the value may not be destroyed (and recalculated again) by PQ, during evaluation.
Theoretically, PQ may flashing values of immutable variables like for Buffered values and then restoring from disk when needed, instead of just removing the variable. Or just throw an error. It seems for me (via logs), that PQ sometime recalculates, but
you never sure whether you interpret such info correctly.
3. Is it possible that PQ released Buffered memory (by removing buffered table from memory) BEFORE the end of evaluation? In general it may be possible, because, while evaluating, M analyzer may find, that Buffered variable is not used in the remain steps
of the evaluation plan. If we have low remain memory or big data tables, it may be very effective approach. We could break big data table on pieces, calculate them separately, using Buffer for every part as we want, and hope that PQ will free memory from previous
part for calculating the next part of the table.