forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Size of all arrays
Goto page 1, 2  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2124
Location: South Pole, Antarctica

PostPosted: Wed Sep 18, 2019 2:01 am    Post subject: Size of all arrays Reply with quote

If your program has years in development you will try many different things and sometimes forget to clean up after failed attempts. Sometimes you forget to reduce back huge static array you created and it lives there for years. Often you just do not realize that you've just created monstrous array. This happened to me many times. No mater what the reason both in 32bit and even with the 64bits it is always important to know the array sizes

Which FTN95 option tells about sizes of all static arrays in the program? I remember it existed but was so inconvenient that I failed to find what I wanted. If this does not exist in the convenient summary form showing largest arrays first like this i made following suggestion of John Campbell because could not handle by myself the bizarre format like this b'z,zzz,zzz,zz# Smile

Code:
    write (*,11) size (Arr1), size (Arr1) /1e6 * 4   
    write (*,11) size (Arr2), size (Arr2) /1e6 * 4
    write (*,11) size (Arr3), size (Arr3) /1e6 * 4
    write (*,11) size (Arr4), size (Arr4) /1e6 * 4
 11 format ('Total size, array elements = ',b'z,zzz,zzz,zz#', '   Total size, MB = ',b'z,zzz,zzz,zz#')


then i'd consider this as the request for the new option.

The dynamic arrays is different story and need programmer's own careful bookkeeping and control. I do not know if compiler can tell what is the size of allocated arrays in the program?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Wed Sep 18, 2019 3:19 am    Post subject: Reply with quote

Dan,

You can use task manager to monitor the memory allocation as the program is running. While not a clear indication, if the amount of memory being used looks much greater than expected, it does indicate that there could be unexpected arrays being allocated.

For large allocated arrays, I use a reporting approach like the following, which reports the "stat" and size of the array. It does require that allocate is not in a big loop.
ALLOCATE ( IEQN(6,NANODE), stat=stat ) ; call alloc_test (stat, 'IEQN(6,NANODE)', 6,NANODE)

(most of my arrays are 1d or 2d)

I always have kept an audit of memory usage when developing programs, both for 32-bit and now 64-bit, as although address space has exploded, exceeding installed memory is a practical stop for most calculations.
I also try to allocate all large arrays, say larger than 1k, which means they are not taking space when not in scope. This would mean that the compiler would not be able to report the memory demand of most of my routines.

Some possibly helpful ideas ?

John
Back to top
View user's profile Send private message
wahorger



Joined: 13 Oct 2014
Posts: 666
Location: Morrison, CO, USA

PostPosted: Wed Sep 18, 2019 4:01 pm    Post subject: Reply with quote

You can request a dump of the variables in the listing file (/DUMP). The format of the report is:
Code:

/VALIDATE_QUAD/ TYPE(QUAD_REC)       QUADRANGLE(32761)   @ 192 (at line 2860)
/VALIDATE_QUAD/ INTEGER(KIND=3)      NUM_QUADS           @ 0   (at line 2859)
/CLOGPLTCOM/    TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS_LOGPLOT(65536) @ 157024340 (at line 2855)
/CLOGPLTCOM/    TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS(65536) @ 80  (at line 2855)
/CLOGPLTCOM/    REAL(KIND=2)         SCALE               @ 72  (at line 2854)
/CLOGPLTCOM/    LOGICAL(KIND=3)      LFLAGS(17)          @ 4   (at line 2853)


As best I can figure, there are no embedded spaces in the first "column". I also use /SAVE for my code, so all the "local" variables show in the first column as "SAVE". For those locals that use the stack, there is not an indicator for local; this area is left blank.
Code:

   Local Variables
                INTEGER(KIND=3)      (Alternate address of FIRST_CHAR) @ -24 (at line 6780)
                INTEGER(KIND=3)      (Alternate address of CAT) @ -20 (at line 6780)
                INTEGER(KIND=3)      (Alternate address of REASON) @ -16 (at line 6780)
/HDRFLT/        REAL(KIND=2)         ADOBE_MAX_WIDTH     @ 112 (at line 6587)
/HDRFLT/        REAL(KIND=2)         ADOBE_MAX_LENGTH    @ 104 (at line 6587)


The third "column" can have embedded spaces (see the example). Unfortunately, with long variable names, the columns don't line up.
Code:

/REF_COMMON/    LOGICAL(KIND=3)      DIRECTION_REFERENCE(0:5, 10) @ 66158 (at line 3511)


You can use Excel to load in the file using the @ as a separator, and the "=" as a separator. This should give you access to the variables alone. Unfortunately, for an array with more than one dimension, the other sizes may have embedded spaces. So using spaces as separators for excel will give you more difficult output to view.

If you have TYPE's, these will also show here.
Code:

SAVE            TYPE(QSTSC_INPUT_STRUCT) STRUCT              @ 32784 (at line 3166)
Back to top
View user's profile Send private message Visit poster's website
DanRRight



Joined: 10 Mar 2008
Posts: 2124
Location: South Pole, Antarctica

PostPosted: Sat Sep 21, 2019 9:07 am    Post subject: Reply with quote

John
Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows.

Bill,
thanks for the hint. I know why i did not use and always forget this option: the elitist inarticulate FTN95 /? tells that /DUMP will show you variable dump in the listing file but forgets to tell that besides /DUMP the /LIS is required. Plus one of subroutines crashed compilation with /64 /DUMP /LIS ...
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sat Sep 21, 2019 1:13 pm    Post subject: Re: Reply with quote

DanRRight wrote:
Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows.


Dan,

What is this devilry bug? I don't find this and do find task manager very useful.

wrt memory usage, there is a usage difference between allocating an array and actually using it. This may show a difference between "Working Set", "Peak working set" and "Memory (active private"

It does help to establish an audit of large arrays, rather than blame the devil !
Back to top
View user's profile Send private message
wahorger



Joined: 13 Oct 2014
Posts: 666
Location: Morrison, CO, USA

PostPosted: Sat Sep 21, 2019 4:26 pm    Post subject: Reply with quote

John and Dan,

What I have found with named common is: Using /RELEASE: If you don't use it, it doesn't get allocated. This differs from /CHECKMATE. /CHECKMATE will pre-allocate all the named common blocks.

I discovered this when running a program that waits until all options are selected to begin processing the desired data. The memory went from a few 10's of megabytes to hundreds of megabytes by the end.
Back to top
View user's profile Send private message Visit poster's website
wahorger



Joined: 13 Oct 2014
Posts: 666
Location: Morrison, CO, USA

PostPosted: Sat Sep 21, 2019 4:50 pm    Post subject: Reply with quote

This is an example of the same code at the opening screen. The CHECKMATE version is at 850MB, and the RELEASE version is 30 mb.

https://www.dropbox.com/s/bw0r7p5qb7wdc9u/C-MasterCheckmate%20vs%20Release.png?dl=0
Back to top
View user's profile Send private message Visit poster's website
DanRRight



Joined: 10 Mar 2008
Posts: 2124
Location: South Pole, Antarctica

PostPosted: Sun Sep 22, 2019 2:18 am    Post subject: Reply with quote

Thanks Bill. And even that is not all. No matter how much i use the computer i get Task Manager rarely showing more than 40-50% memory usage all the time even with multiple browsers with hundreds of tabs and my code for sure grabbing >90% of RAM via allocatable arrays. I can not load anything more since sometimes these browsers crash and for sure not single one of my programs are loading anymore.

Worst what happening why i wait for AMD 3950 which has 128GB limit is that when the memory demanded by my code increase i sometimes see how NVIDIA graphics driver crashes leaving screen black. Task Manager though before crash shows all is fine, "don't worry, be happy", "Everything is alright, my beautiful Marquise" (humorous song from 1930th where the whole estate was burnt, but the estate manager calls the owner marquise and slowly tells all is alright)
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sun Sep 22, 2019 5:33 am    Post subject: Reply with quote

Dan,

Task Manager works for me, I think you are not seeing the difference between committed memory and used memory.

The following program may be a useful test.
It is the first time I have tested allocatable arguments with FTN95 and appears to work with with Ver 8.51

If you run it, responding to the ? with a 1, you will see the memory usage increase in task manager.
Note the difference working set and commit size.
Physical memory % relates to working set and not commit size, although I expect you may run out of memory based on commit size.

Anyway, good test of allocatable array arguments for Ver 8.5+

Code:
!  program to first allocate big arrays, then initialise big arrays
!   requires FTN95 Ver 8.5 to work with allocatable arguments
!   
module sz
      real*4    :: sz_mb = 500.   ! array size in mb
      real*4    :: sz_ni          ! array elements
      integer*4 :: ni             ! dimension of array
end module sz
!
   program use_memory
     use sz
!     
      interface
        subroutine allocate_array ( aa )
         use sz
          integer*4, allocatable, dimension(:,:) :: aa
        end subroutine allocate_array
      end interface
!
      integer*4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g
      integer*4 :: i,j
!
      sz_gb = 500   ! gb
      sz_n  = sz_gb * 1024.**2 / 4
      ni    = sqrt (sz_n)
      write (*,*) '2D array dimension =',ni
!
      do i = 1,8
        select case (i)
          case (1)
            call allocate_array ( a )
          case (2)
            call allocate_array ( b )
          case (3)
            call allocate_array ( c )
          case (4)
            call allocate_array ( d )
          case (5)
            call allocate_array ( e )
          case (6)
            call allocate_array ( f )
          case (7)
            call allocate_array ( g )
          case default
        end select
        write (*,*) i, ' arrays allocated: check task manager'
        read  (*,*) j
      end do
!
      do i = 1,8
        select case (i)
          case (1)
            call initialise_array ( a )
          case (2)
            call initialise_array ( b )
          case (3)
            call initialise_array ( c )
          case (4)
            call initialise_array ( d )
          case (5)
            call initialise_array ( e )
          case (6)
            call initialise_array ( f )
          case (7)
            call initialise_array ( g )
          case default
        end select
        write (*,*) i, ' arrays initialised: check task manager'
        write (*,*) '?'
        read  (*,*) j
      end do
!
   end program use_memory

   subroutine allocate_array ( aa )
     use sz
      integer*4, allocatable, dimension(:,:) :: aa
      integer*4 :: stat
      allocate ( aa(ni,ni), stat=stat )
      write (*,*) 'array allocated : stat=',stat                 
   end subroutine allocate_array

   subroutine initialise_array ( aa )
     use sz
      integer*4 :: aa(ni,ni)
      integer*4 :: i,j, n
      n = 0
      do j = 1,ni
        do i = 1,ni
          aa(i,j) = i+j
          n = n+1
        end do
      end do
      write (*,*) 'array initialised',n
   end subroutine initialise_array


Hopefully no errors in this coding !!
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sun Sep 22, 2019 6:58 am    Post subject: Reply with quote

I have now run task manager in Win7 and Win10.

With Win 10, on the performance tab, the memory graph is for usage, while "In use" includes the Allocated arrays being used and "Committed" does show the total memory, included Allocated arrays, but not yet in use.

??? hope that was clear !!, Anyway Used memory is for memory that has been taken and given a value, while Committed also includes memory (Fortran) allocated but not yet physically allocated and given a value.

I am not sure how you can crash if memory committed exceeds available installed memory ? as this should not happen until arrays are being used and physical memory is required.
Once you exceed physical memory available, then it goes to virtual memory on pagefile.sys and the system usually slows down dramatically (looks like it has stopped)
Your system could crash if pagefile.sys is not big enough to support the virtual memory requested. I have seen pc's with pagefile.sys configured to be too small (smaller than physical memory). This is hardly Task manager's fault ?
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2124
Location: South Pole, Antarctica

PostPosted: Sun Sep 22, 2019 8:28 am    Post subject: Reply with quote

Never was able to allocate RAM+pagefile.sys, maximum was close to RAM size. Would be nice to allocate RAM+pagefile. But probably as I suspect there exist limit of allocation which is Intel processors RAM limit. It was 32 then 64 and now new models have 128 GB

My crash of video driver as I suspect happen when I reach limit of processor despite the RAM size + pagefile.sys are 2x larger

OK, I see what was always confusing with this effing default view of Task Manager on Win10. Before in previous versions there was no such confusion. When I load my program the Task Manager shows on graph memory usage hits its peak but few seconds later monitor shows around 50% which is what they call in-Use memory while hiding part they call Standby both are around 50:50 in size

There are also Available, Cached, Total, Installed and Commit'ted. Also for the all processes running on PC there exist additional such values. And Committed from the Resource Manager seems indeed looks like what my arrays occupy. Exclusing COMMON files lazy allocation Bill mentioned
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sun Sep 22, 2019 8:49 am    Post subject: Reply with quote

Dan,

If you are having video problems, I assume you are sharing memory ?

"Cached" is interesting. I assume it refers to disk I/O cache (buffer) and not processor cache.

Although I do have pagefile.sys configured to larger than installed memory, I rarely use it for my programs, as the system basically stops. Back in 70's we used the equivalent of pagefile.sys all the time and thought it was great (although 1,000 times slower than today)

Not sure I understand your problem(s) as I now try to use allocatable arrays in modules, rather than COMMON.

Anyway, good test for using allocatable arrays as arguments.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sun Sep 22, 2019 9:03 am    Post subject: Reply with quote

To follow up on the test program; the test creates 8 arrays and checks them for being available with the correct values.

This test demonstrates FTN95 Ver8.5 supporting allocatable arrays as arguments (the call requires an interface definition)
It works for both 32-bit and 64-bit /64 .

Interestingly, if I create 8 x 400mb arrays, this demonstrates for both 32-bit and 64-bit, using 3.2 GB of addressable memory.

The tidied up program is:
Code:
!  program to first allocate big arrays, then initialise big arrays
!   requires FTN95 Ver 8.5 to work with allocatable arguments
!   
module sz
      real*4    :: sz_mb          ! array size in mb
      real*4    :: sz_ni          ! array elements
      integer*4 :: ni             ! dimension of array
end module sz
!
   program use_memory
     use sz
!     
      interface
        subroutine allocate_array ( aa )
         use sz
          integer*4, allocatable, dimension(:,:) :: aa
        end subroutine allocate_array
      end interface
!
      integer*4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g,h
      integer*4 :: i,j
!
      sz_mb = 500                       ! MBytes
!
      call prompt ('What array size to test (in MBytes, default 500) ?', j)
      if ( j > 1 .and. j < 2048 ) sz_mb = j
      sz_n  = sz_mb * 1024.**2 / 4
      ni    = sqrt (sz_n)
      write (*,*) '2D I*4 array dimension =',ni
!
      do i = 1,9
        select case (i)
          case (1)
            call allocate_array ( a )
          case (2)
            call allocate_array ( b )
          case (3)
            call allocate_array ( c )
          case (4)
            call allocate_array ( d )
          case (5)
            call allocate_array ( e )
          case (6)
            call allocate_array ( f )
          case (7)
            call allocate_array ( g )
          case (8)
            call allocate_array ( h )
          case default
            write (*,*) 'all allocates done : now to initialise'
            exit
        end select
        write (*,*) i, ' arrays allocated: check task manager'
        call prompt ('Proceed ?',j)
      end do
!
      do i = 1,9
        select case (i)
          case (1)
            call initialise_array ( a, i )
          case (2)
            call initialise_array ( b, i )
          case (3)
            call initialise_array ( c, i )
          case (4)
            call initialise_array ( d, i )
          case (5)
            call initialise_array ( e, i )
          case (6)
            call initialise_array ( f, i )
          case (7)
            call initialise_array ( g, i )
          case (8)
            call initialise_array ( h, i )
          case default
            write (*,*) 'all initialise done : now to test'
            exit
        end select
        write (*,*) i, ' arrays initialised: check task manager'
        call prompt ('Proceed ?',j)
      end do
!
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2147
Location: Sydney

PostPosted: Sun Sep 22, 2019 9:05 am    Post subject: Reply with quote

ctd/..
Code:
!
      do i = 1,9
        select case (i)
          case (1)
            call test_array ( a, i )
          case (2)
            call test_array ( b, i )
          case (3)
            call test_array ( c, i )
          case (4)
            call test_array ( d, i )
          case (5)
            call test_array ( e, i )
          case (6)
            call test_array ( f, i )
          case (7)
            call test_array ( g, i )
          case (8)
            call test_array ( h, i )
          case default
            write (*,*) 'all tests now done : final ? before exiting'
        end select
        if ( i < 9) write (*,*) i, ' arrays tested: check task manager'
        call prompt ('Proceed ?',j)
      end do
!
   end program use_memory

   subroutine allocate_array ( aa )
     use sz
      integer*4, allocatable, dimension(:,:) :: aa
      integer*4 :: stat
      allocate ( aa(ni,ni), stat=stat )
      write (*,*) 'array allocated : stat=',stat                 
   end subroutine allocate_array

   subroutine initialise_array ( aa, ii )
     use sz
      integer*4 :: aa(ni,ni), ii
      integer*4 :: i,j, n
      n = 0
      do j = 1,ni
        do i = 1,ni
          aa(i,j) = i+j+ii
          n = n+1
        end do
      end do
      write (*,*) 'array initialised',n
   end subroutine initialise_array

   subroutine test_array ( aa, ii )
     use sz
      integer*4 :: aa(ni,ni), ii
      integer*4 :: i,j, n, e
      n = 0
      e = 0
      do j = 1,ni
        do i = 1,ni
          if ( aa(i,j) /= i+j+ii ) e = e+1
          n = n+1
        end do
      end do
      write (*,*) 'array tested',n, ' :',e,' errors'
   end subroutine test_array   
   
   subroutine prompt (string, j)
      character string*(*)
      integer j, iostat
      write (*,fmt='(a)',advance='NO') string
      read  (*,fmt='(bn,i6)', iostat=iostat) j
      if (iostat /= 0) j = -1
   end subroutine prompt
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2124
Location: South Pole, Antarctica

PostPosted: Mon Sep 23, 2019 1:48 am    Post subject: Reply with quote

John,
Allocatable arrays are shown in Commit part. COMMON arrays are not till they are used (lazy allocation). Run this code and look at memory in Task Manager and then uncomment two lines with "!!!" in subroutine sub and run again

Code:
common/a1/a, aa(7000,7000), aaa(7000,7000)
A=1
call sub
end

subroutine sub
common/a1/a, aa(7000,7000), aaa(7000,7000)
print*, a
!!!aa(:,:)=2
!!!aaa(:,:)=3
pause
end


If compile
FTN95 aaa.f95 /link
FTN95 aaa.f95 /link /64

there are also differences in In-Use memory. So damn things are confusing and you will forget what is what if you are not visit these things often.

Currently with 64 bits all these COMMON are restricted to 2GB but when (or IF ) the Silverfrost will remove the stack limits this could become a problem when you set huge array and forget about it or even do not know from the beginning that you have set a huge sucker of memory


Last edited by DanRRight on Mon Sep 23, 2019 3:49 am; edited 1 time in total
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group