Size of all arrays

If your program has years in development you will try many different things and sometimes forget to clean up after failed attempts. Sometimes you forget to reduce back huge static array you created and it lives there for years. Often you just do not realize that you've just created monstrous array. This happened to me many times. No mater what the reason both in 32bit and even with the 64bits it is always important to know the array sizes

Which FTN95 option tells about sizes of all static arrays in the program? I remember it existed but was so inconvenient that I failed to find what I wanted. If this does not exist in the convenient summary form showing largest arrays first like this i made following suggestion of John Campbell because could not handle by myself the bizarre format like this b'z,zzz,zzz,zz# 😃

    write (*,11) size (Arr1), size (Arr1) /1e6 * 4   
    write (*,11) size (Arr2), size (Arr2) /1e6 * 4
    write (*,11) size (Arr3), size (Arr3) /1e6 * 4
    write (*,11) size (Arr4), size (Arr4) /1e6 * 4
 11 format ('Total size, array elements = ',b'z,zzz,zzz,zz#', '   Total size, MB = ',b'z,zzz,zzz,zz#') 

then i'd consider this as the request for the new option.

The dynamic arrays is different story and need programmer's own careful bookkeeping and control. I do not know if compiler can tell what is the size of allocated arrays in the program?

Dan,

You can use task manager to monitor the memory allocation as the program is running. While not a clear indication, if the amount of memory being used looks much greater than expected, it does indicate that there could be unexpected arrays being allocated.

For large allocated arrays, I use a reporting approach like the following, which reports the 'stat' and size of the array. It does require that allocate is not in a big loop. ALLOCATE ( IEQN(6,NANODE), stat=stat ) ; call alloc_test (stat, 'IEQN(6,NANODE)', 6,NANODE)

(most of my arrays are 1d or 2d)

I always have kept an audit of memory usage when developing programs, both for 32-bit and now 64-bit, as although address space has exploded, exceeding installed memory is a practical stop for most calculations. I also try to allocate all large arrays, say larger than 1k, which means they are not taking space when not in scope. This would mean that the compiler would not be able to report the memory demand of most of my routines.

Some possibly helpful ideas ?

John

You can request a dump of the variables in the listing file (/DUMP). The format of the report is:

/VALIDATE_QUAD/ TYPE(QUAD_REC)       QUADRANGLE(32761)   @ 192 (at line 2860)
/VALIDATE_QUAD/ INTEGER(KIND=3)      NUM_QUADS           @ 0   (at line 2859)
/CLOGPLTCOM/    TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS_LOGPLOT(65536) @ 157024340 (at line 2855)
/CLOGPLTCOM/    TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS(65536) @ 80  (at line 2855)
/CLOGPLTCOM/    REAL(KIND=2)         SCALE               @ 72  (at line 2854)
/CLOGPLTCOM/    LOGICAL(KIND=3)      LFLAGS(17)          @ 4   (at line 2853)

As best I can figure, there are no embedded spaces in the first 'column'. I also use /SAVE for my code, so all the 'local' variables show in the first column as 'SAVE'. For those locals that use the stack, there is not an indicator for local; this area is left blank.

	Local Variables
                INTEGER(KIND=3)      (Alternate address of FIRST_CHAR) @ -24 (at line 6780)
                INTEGER(KIND=3)      (Alternate address of CAT) @ -20 (at line 6780)
                INTEGER(KIND=3)      (Alternate address of REASON) @ -16 (at line 6780)
/HDRFLT/        REAL(KIND=2)         ADOBE_MAX_WIDTH     @ 112 (at line 6587)
/HDRFLT/        REAL(KIND=2)         ADOBE_MAX_LENGTH    @ 104 (at line 6587)

The third 'column' can have embedded spaces (see the example). Unfortunately, with long variable names, the columns don't line up.

/REF_COMMON/    LOGICAL(KIND=3)      DIRECTION_REFERENCE(0:5, 10) @ 66158 (at line 3511)

You can use Excel to load in the file using the @ as a separator, and the '=' as a separator. This should give you access to the variables alone. Unfortunately, for an array with more than one dimension, the other sizes may have embedded spaces. So using spaces as separators for excel will give you more difficult output to view.

If you have TYPE's, these will also show here.

SAVE            TYPE(QSTSC_INPUT_STRUCT) STRUCT              @ 32784 (at line 3166)

John Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows.

Bill, thanks for the hint. I know why i did not use and always forget this option: the elitist inarticulate FTN95 /? tells that /DUMP will show you variable dump in the listing file but forgets to tell that besides /DUMP the /LIS is required. Plus one of subroutines crashed compilation with /64 /DUMP /LIS ...

Quoted from DanRRight Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows.

Dan,

What is this devilry bug? I don't find this and do find task manager very useful.

wrt memory usage, there is a usage difference between allocating an array and actually using it. This may show a difference between 'Working Set', 'Peak working set' and 'Memory (active private'

It does help to establish an audit of large arrays, rather than blame the devil !

John and Dan,

What I have found with named common is: Using /RELEASE: If you don't use it, it doesn't get allocated. This differs from /CHECKMATE. /CHECKMATE will pre-allocate all the named common blocks.

I discovered this when running a program that waits until all options are selected to begin processing the desired data. The memory went from a few 10's of megabytes to hundreds of megabytes by the end.

This is an example of the same code at the opening screen. The CHECKMATE version is at 850MB, and the RELEASE version is 30 mb.

https://www.dropbox.com/s/bw0r7p5qb7wdc9u/C-MasterCheckmate%20vs%20Release.png?dl=0

Thanks Bill. And even that is not all. No matter how much i use the computer i get Task Manager rarely showing more than 40-50% memory usage all the time even with multiple browsers with hundreds of tabs and my code for sure grabbing >90% of RAM via allocatable arrays. I can not load anything more since sometimes these browsers crash and for sure not single one of my programs are loading anymore.

Worst what happening why i wait for AMD 3950 which has 128GB limit is that when the memory demanded by my code increase i sometimes see how NVIDIA graphics driver crashes leaving screen black. Task Manager though before crash shows all is fine, 'don't worry, be happy', 'Everything is alright, my beautiful Marquise' (humorous song from 1930th where the whole estate was burnt, but the estate manager calls the owner marquise and slowly tells all is alright)

Dan,

Task Manager works for me, I think you are not seeing the difference between committed memory and used memory.

The following program may be a useful test. It is the first time I have tested allocatable arguments with FTN95 and appears to work with with Ver 8.51

If you run it, responding to the ? with a 1, you will see the memory usage increase in task manager. Note the difference working set and commit size. Physical memory % relates to working set and not commit size, although I expect you may run out of memory based on commit size.

Anyway, good test of allocatable array arguments for Ver 8.5+

!  program to first allocate big arrays, then initialise big arrays
!   requires FTN95 Ver 8.5 to work with allocatable arguments
!   
module sz
      real*4    :: sz_mb = 500.   ! array size in mb
      real*4    :: sz_ni          ! array elements
      integer*4 :: ni             ! dimension of array
end module sz
!
   program use_memory
     use sz
!      
      interface
        subroutine allocate_array ( aa )
         use sz
          integer*4, allocatable, dimension(:,:) :: aa
        end subroutine allocate_array
      end interface
!
      integer*4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g
      integer*4 :: i,j
!
      sz_gb = 500   ! gb
      sz_n  = sz_gb * 1024.**2 / 4
      ni    = sqrt (sz_n)
      write (*,*) '2D array dimension =',ni
!
      do i = 1,8
        select case (i)
          case (1)
            call allocate_array ( a )
          case (2)
            call allocate_array ( b )
          case (3)
            call allocate_array ( c )
          case (4)
            call allocate_array ( d )
          case (5)
            call allocate_array ( e )
          case (6)
            call allocate_array ( f )
          case (7)
            call allocate_array ( g )
          case default
        end select
        write (*,*) i, ' arrays allocated: check task manager'
        read  (*,*) j
      end do
!
      do i = 1,8
        select case (i)
          case (1)
            call initialise_array ( a )
          case (2)
            call initialise_array ( b )
          case (3)
            call initialise_array ( c )
          case (4)
            call initialise_array ( d )
          case (5)
            call initialise_array ( e )
          case (6)
            call initialise_array ( f )
          case (7)
            call initialise_array ( g )
          case default
        end select
        write (*,*) i, ' arrays initialised: check task manager'
        write (*,*) '?'
        read  (*,*) j
      end do
!
   end program use_memory

   subroutine allocate_array ( aa )
     use sz
      integer*4, allocatable, dimension(:,:) :: aa
      integer*4 :: stat
      allocate ( aa(ni,ni), stat=stat )
      write (*,*) 'array allocated : stat=',stat                  
   end subroutine allocate_array

   subroutine initialise_array ( aa )
     use sz
      integer*4 :: aa(ni,ni)
      integer*4 :: i,j, n
      n = 0
      do j = 1,ni
        do i = 1,ni
          aa(i,j) = i+j
          n = n+1
        end do
      end do
      write (*,*) 'array initialised',n
   end subroutine initialise_array

Hopefully no errors in this coding !!

I have now run task manager in Win7 and Win10.

With Win 10, on the performance tab, the memory graph is for usage, while 'In use' includes the Allocated arrays being used and 'Committed' does show the total memory, included Allocated arrays, but not yet in use.

??? hope that was clear !!, Anyway Used memory is for memory that has been taken and given a value, while Committed also includes memory (Fortran) allocated but not yet physically allocated and given a value.

I am not sure how you can crash if memory committed exceeds available installed memory ? as this should not happen until arrays are being used and physical memory is required. Once you exceed physical memory available, then it goes to virtual memory on pagefile.sys and the system usually slows down dramatically (looks like it has stopped) Your system could crash if pagefile.sys is not big enough to support the virtual memory requested. I have seen pc's with pagefile.sys configured to be too small (smaller than physical memory). This is hardly Task manager's fault ?

Never was able to allocate RAM+pagefile.sys, maximum was close to RAM size. Would be nice to allocate RAM+pagefile. But probably as I suspect there exist limit of allocation which is Intel processors RAM limit. It was 32 then 64 and now new models have 128 GB

My crash of video driver as I suspect happen when I reach limit of processor despite the RAM size + pagefile.sys are 2x larger

OK, I see what was always confusing with this effing default view of Task Manager on Win10. Before in previous versions there was no such confusion. When I load my program the Task Manager shows on graph memory usage hits its peak but few seconds later monitor shows around 50% which is what they call in-Use memory while hiding part they call Standby both are around 50:50 in size

There are also Available, Cached, Total, Installed and Commit'ted. Also for the all processes running on PC there exist additional such values. And Committed from the Resource Manager seems indeed looks like what my arrays occupy. Exclusing COMMON files lazy allocation Bill mentioned

Dan,

If you are having video problems, I assume you are sharing memory ?

'Cached' is interesting. I assume it refers to disk I/O cache (buffer) and not processor cache.

Although I do have pagefile.sys configured to larger than installed memory, I rarely use it for my programs, as the system basically stops. Back in 70's we used the equivalent of pagefile.sys all the time and thought it was great (although 1,000 times slower than today)

Not sure I understand your problem(s) as I now try to use allocatable arrays in modules, rather than COMMON.

Anyway, good test for using allocatable arrays as arguments.

To follow up on the test program; the test creates 8 arrays and checks them for being available with the correct values.

This test demonstrates FTN95 Ver8.5 supporting allocatable arrays as arguments (the call requires an interface definition) It works for both 32-bit and 64-bit /64 .

Interestingly, if I create 8 x 400mb arrays, this demonstrates for both 32-bit and 64-bit, using 3.2 GB of addressable memory.

The tidied up program is: ! program to first allocate big arrays, then initialise big arrays ! requires FTN95 Ver 8.5 to work with allocatable arguments !
module sz real4 :: sz_mb ! array size in mb real4 :: sz_ni ! array elements integer4 :: ni ! dimension of array end module sz ! program use_memory use sz !
interface subroutine allocate_array ( aa ) use sz integer4, allocatable, dimension(:,:) :: aa end subroutine allocate_array end interface ! integer4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g,h integer4 :: i,j ! sz_mb = 500 ! MBytes ! call prompt ('What array size to test (in MBytes, default 500) ?', j) if ( j > 1 .and. j < 2048 ) sz_mb = j sz_n = sz_mb * 1024.**2 / 4 ni = sqrt (sz_n) write (,) '2D I4 array dimension =',ni ! do i = 1,9 select case (i) case (1) call allocate_array ( a ) case (2) call allocate_array ( b ) case (3) call allocate_array ( c ) case (4) call allocate_array ( d ) case (5) call allocate_array ( e ) case (6) call allocate_array ( f ) case (7) call allocate_array ( g ) case (8) call allocate_array ( h ) case default write (,) 'all allocates done : now to initialise' exit end select write (,) i, ' arrays allocated: check task manager' call prompt ('Proceed ?',j) end do ! do i = 1,9 select case (i) case (1) call initialise_array ( a, i ) case (2) call initialise_array ( b, i ) case (3) call initialise_array ( c, i ) case (4) call initialise_array ( d, i ) case (5) call initialise_array ( e, i ) case (6) call initialise_array ( f, i ) case (7) call initialise_array ( g, i ) case (8) call initialise_array ( h, i ) case default write (,) 'all initialise done : now to test' exit end select write (,*) i, ' arrays initialised: check task manager' call prompt ('Proceed ?',j) end do !

ctd/.. ! do i = 1,9 select case (i) case (1) call test_array ( a, i ) case (2) call test_array ( b, i ) case (3) call test_array ( c, i ) case (4) call test_array ( d, i ) case (5) call test_array ( e, i ) case (6) call test_array ( f, i ) case (7) call test_array ( g, i ) case (8) call test_array ( h, i ) case default write (,) 'all tests now done : final ? before exiting' end select if ( i < 9) write (,) i, ' arrays tested: check task manager' call prompt ('Proceed ?',j) end do ! end program use_memory

   subroutine allocate_array ( aa )
     use sz
      integer*4, allocatable, dimension(:,:) :: aa
      integer*4 :: stat
      allocate ( aa(ni,ni), stat=stat )
      write (*,*) 'array allocated : stat=',stat                  
   end subroutine allocate_array

   subroutine initialise_array ( aa, ii )
     use sz
      integer*4 :: aa(ni,ni), ii
      integer*4 :: i,j, n
      n = 0
      do j = 1,ni
        do i = 1,ni
          aa(i,j) = i+j+ii
          n = n+1
        end do
      end do
      write (*,*) 'array initialised',n
   end subroutine initialise_array

   subroutine test_array ( aa, ii )
     use sz
      integer*4 :: aa(ni,ni), ii
      integer*4 :: i,j, n, e
      n = 0
      e = 0
      do j = 1,ni
        do i = 1,ni
          if ( aa(i,j) /= i+j+ii ) e = e+1
          n = n+1
        end do
      end do
      write (*,*) 'array tested',n, ' :',e,' errors'
   end subroutine test_array   
   
   subroutine prompt (string, j)
      character string*(*)
      integer j, iostat
      write (*,fmt='(a)',advance='NO') string
      read  (*,fmt='(bn,i6)', iostat=iostat) j
      if (iostat /= 0) j = -1
   end subroutine prompt

John, Allocatable arrays are shown in Commit part. COMMON arrays are not till they are used (lazy allocation). Run this code and look at memory in Task Manager and then uncomment two lines with '!!!' in subroutine sub and run again

common/a1/a, aa(7000,7000), aaa(7000,7000)
A=1
call sub
end

subroutine sub
common/a1/a, aa(7000,7000), aaa(7000,7000)
print*, a
!!!aa(:,:)=2
!!!aaa(:,:)=3
pause
end

If compile FTN95 aaa.f95 /link FTN95 aaa.f95 /link /64

there are also differences in In-Use memory. So damn things are confusing and you will forget what is what if you are not visit these things often.

Currently with 64 bits all these COMMON are restricted to 2GB but when (or IF ) the Silverfrost will remove the stack limits this could become a problem when you set huge array and forget about it or even do not know from the beginning that you have set a huge sucker of memory

Dan,

Try the following with a variety of compile options

32 bit (default) /debug /checkmate

64 bit /64 /64 /debug /64 /checkmate

module nc_def
integer*4, parameter :: nc = 14000
end module nc_def

use nc_def
common /a1/ a, aa(nc,nc), aaa(nc,nc)
real*4 a,aa,aaa

A=1
pause 1
call sub
end

subroutine sub
use nc_def
common /a1/ a, aa(nc,nc), aaa(nc,nc)
real*4 a,aa,aaa

print*, a
aa(:,:)=2
aa(:,:)=3
pause 2
end

This demonstrates that common is not allocated memory until variables are initialised (used) for release mode, but initially for checkmate mode.

My 32-bit release and /64 shows that the memory is not allocated to after pause 1, which is what could be a useful way to interrogate your program, certainly when comparing working set to commit

Also, FTN95 /64 supports common larger than 2gb. Use nc = 30000 and do the 64 bit compiles. (other 64 bit compilers do not do this, or support pause, which is a significant feature of FTN95)

the following approach works for large common and /64 module nc_def integer*4, parameter :: nc = 30000 ! 30000 end module nc_def

the following approach fails when adapted to the program above module nc_def integer4, parameter :: nc = 30000 ! 30000 common /a1/ a, aa(nc,nc), aaa(nc,nc) real4 a,aa,aaa end module nc_def

John

Unless i misunderstood you, John, you have just repeated what we said with Bill that arrays not used remain hidden to Task Manager till they are used or they are exposed if compiled with /undef.

And the larger than 2GB arrays in COMMON already available making things dangerous already today. The only what left to put the last nail is to allow stack to be more than 2GB like in this example which does not work yet but az said that is already fixed Program aaa a=1 call sub contains subroutine sub real aa(30000,30000) aa(:,:)=2 print*, a pause end subroutine end

That means (returning to our sheep from the start of this thread) that using Task Manager is very confusing to hunt for the large arrays existing in the program. They may expose themselves in one type of compilation (checkmate) and do not in another (release) if they are not in use.

Much better would be the option /dump /list which if a bit modified by Silverfrost to place arrays in descending size order (and also showing the total sum) will reveal the largest arrays in the program. Currently this option is also very confusing. And though we now understand where devilry was hiding a bit better the complete hunting him out is still work in progress. Agree?

Quoted from DanRRight Unless i misunderstood you, John, you have just repeated what we said with Bill that arrays not used remain hidden to Task Manager till they are used or they are exposed if compiled with /undef.

Dan,

They (common) are not hidden, as they are reported in 'Commit' at start. Allocate arrays (to heap) are hidden until they are allocated ( then shown in Commit ) then when used/initialised (shown in working set ). The 'heap' just extends. Automatic arrays (to stack) are part of the pre-allocated stack ? Stack would be in Commit initially and in Working set as it is used. I think FTN95 /64 does sent send large automatic arrays to the heap.

Using /undef to expose them could be an approach, but not for production version of the program. ( there is a problem for debugging with different ways of allocating arrays between /undef, /check /debug and release mode, which can change the appearance of bugs. )

Complaining about FTN95 allowing COMMON to be larger than 2GB doesn't look like a good idea to me !

Perhaps /statistics /list could be enhanced to report memory usage for each routine, say sum of local arrays, sum of common arrays for each routine. (I am not sure if this is an easy stat to report)

Paul, has there been any thought to report /list as a .csv file, as is done with /timing ? I know I could use this mod.

A previous compiler I used had options /xref and also /xrefs which excluded Common variables that were not referenced. (might have also identified arguments and declared variables that were not referenced) Was that FTN77 ?

John

John

As far as I recall the work on /TIMING was not done by Silverfrost/Salford Software so to my knowledge no thought has been given to producing .csv lists for /LIST.

Paul,

Whoever did the /timing implementation should be invited back, as I am a fan of that analysis. .csv file output makes it easier to import into excel or read, using the comma as a unique field delimiter,

thanks,

John