Insufficient virtual stack with 64bits

Like the Gilbert and Sullivan policeman, the compiler developer's 'lot is not a happy one'. They must 'do what it says on the tin' but try at the same time to make up for naive mistakes. All within the time and resources available.

I remember a case where a user was having problems inverting a very large matrix via determinants. Why was the compiler so slow!

The current thread reminds me that FTN95 could do more to compensate for the unnecessary use of array sections. It also reveals a stack limitation that has become out of date.

The feedback is useful and hopefully will make FTN95 even better in the future.

Paul,

Could you provide some more information on 'Vstack'.

Is it a general replacement for the STACK, enabling much larger local or automatic arrays without the need to redefine the Stack size ? (perhaps 2 stacks, with Vstack used for large local or automatic arrays would work very well, while subroutine argument references and local variables on a small near stack)

As it is a memory address, it could be an address offset greater than the physical memory installed ( say 128 GBytes ) If it needs a long address, it can be anywhere. This has no affect on available memory as it only takes physical memory when required.

Ifort can place some very large memory strides for stack and heap addresses, without any severe performance hits.

Admittedly an array section that is larger than half the physical memory or configured virtual memory will always crash the program, so we may need a better test that the array section is not contiguous before resorting to a temporary copy. Unnecessary temporary copies of array sections is a major cause of FTN95's poor performance in the Polyhedron examples. Please do not resort to the Ifort approach of supporting non-contiguous memory arrays, as it breaks the F77_wrapper approach.

John

The current virtual stack might be replaced subject to the planned review of this issue. So it would probably be best to await the outcome before providing further information. I will make sure that this has a high priority.

Paul,

The concept of a larger stack for automatic or large local arrays, plus for temporary arrays is very good.

Also the use of large virtual address strides provides flexibility for a very large Vstack and heap. You should review Gfortran and Ifort load maps to identify the strides that they provide, with no appreciable performance problem.

This could leave the conventional stack as small for managing subroutine argument lists and smaller local variables and so use short addresses.

In my programs, most arrays are on the heap, which use a long address but still provide good performance.

I look forward to the review and hope that this can lead to fewer stack overflow errors !

John

Here is the outcome of the promised review of what has been called the 64 bit 'virtual stack'.

The next release of FTN95 and its associated DLLs will be amended and the following instructions will apply (i.e. these will be the new instructions).

The compiler generates temporary blocks of data, for example, when passing non-contiguous array sections and when functions return array valued results.

For 64 bit programs, by default this temporary data is allocated from a private heap that is different from the global heap which is used for ALLOCATE statements in the user's program.

The compiler uses the global heap rather than this private heap when /ALLOCATE is added to the FTN95 command line but code created using this option could run more slowly.

The default size of this private heap is 128GB. This is the reserved and not the committed size so reducing this value should have no impact on performance. It should not be necessary to increase this value. If it is too small then runtimes will probably be unacceptable because of the amount of data being copied. However, the default can be set by using /VSTACK <size> on the FTN95 command line (<size> is the required number of GBs). Alternatively the default can be changed by a call to HEAP_RESERVE...

SUBROUTINE HEAP_RESERVE(RESERVE) INTEGER RESERVE This routine must be called before calling other routines. It sets the reserve size of the private heap as the number of GBs required. There is no known advantage in setting this value below its default value.