|
forums.silverfrost.com Welcome to the Silverfrost forums
|
View previous topic :: View next topic |
Author |
Message |
DanRRight
Joined: 10 Mar 2008 Posts: 2816 Location: South Pole, Antarctica
|
Posted: Wed Sep 18, 2019 2:01 am Post subject: Size of all arrays |
|
|
If your program has years in development you will try many different things and sometimes forget to clean up after failed attempts. Sometimes you forget to reduce back huge static array you created and it lives there for years. Often you just do not realize that you've just created monstrous array. This happened to me many times. No mater what the reason both in 32bit and even with the 64bits it is always important to know the array sizes
Which FTN95 option tells about sizes of all static arrays in the program? I remember it existed but was so inconvenient that I failed to find what I wanted. If this does not exist in the convenient summary form showing largest arrays first like this i made following suggestion of John Campbell because could not handle by myself the bizarre format like this b'z,zzz,zzz,zz#
Code: | write (*,11) size (Arr1), size (Arr1) /1e6 * 4
write (*,11) size (Arr2), size (Arr2) /1e6 * 4
write (*,11) size (Arr3), size (Arr3) /1e6 * 4
write (*,11) size (Arr4), size (Arr4) /1e6 * 4
11 format ('Total size, array elements = ',b'z,zzz,zzz,zz#', ' Total size, MB = ',b'z,zzz,zzz,zz#') |
then i'd consider this as the request for the new option.
The dynamic arrays is different story and need programmer's own careful bookkeeping and control. I do not know if compiler can tell what is the size of allocated arrays in the program? |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Wed Sep 18, 2019 3:19 am Post subject: |
|
|
Dan,
You can use task manager to monitor the memory allocation as the program is running. While not a clear indication, if the amount of memory being used looks much greater than expected, it does indicate that there could be unexpected arrays being allocated.
For large allocated arrays, I use a reporting approach like the following, which reports the "stat" and size of the array. It does require that allocate is not in a big loop.
ALLOCATE ( IEQN(6,NANODE), stat=stat ) ; call alloc_test (stat, 'IEQN(6,NANODE)', 6,NANODE)
(most of my arrays are 1d or 2d)
I always have kept an audit of memory usage when developing programs, both for 32-bit and now 64-bit, as although address space has exploded, exceeding installed memory is a practical stop for most calculations.
I also try to allocate all large arrays, say larger than 1k, which means they are not taking space when not in scope. This would mean that the compiler would not be able to report the memory demand of most of my routines.
Some possibly helpful ideas ?
John |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1217 Location: Morrison, CO, USA
|
Posted: Wed Sep 18, 2019 4:01 pm Post subject: |
|
|
You can request a dump of the variables in the listing file (/DUMP). The format of the report is:
Code: |
/VALIDATE_QUAD/ TYPE(QUAD_REC) QUADRANGLE(32761) @ 192 (at line 2860)
/VALIDATE_QUAD/ INTEGER(KIND=3) NUM_QUADS @ 0 (at line 2859)
/CLOGPLTCOM/ TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS_LOGPLOT(65536) @ 157024340 (at line 2855)
/CLOGPLTCOM/ TYPE(MASTER_CATALOG_REC_EXTENDED) EXTRACT_LOGS(65536) @ 80 (at line 2855)
/CLOGPLTCOM/ REAL(KIND=2) SCALE @ 72 (at line 2854)
/CLOGPLTCOM/ LOGICAL(KIND=3) LFLAGS(17) @ 4 (at line 2853)
|
As best I can figure, there are no embedded spaces in the first "column". I also use /SAVE for my code, so all the "local" variables show in the first column as "SAVE". For those locals that use the stack, there is not an indicator for local; this area is left blank.
Code: |
Local Variables
INTEGER(KIND=3) (Alternate address of FIRST_CHAR) @ -24 (at line 6780)
INTEGER(KIND=3) (Alternate address of CAT) @ -20 (at line 6780)
INTEGER(KIND=3) (Alternate address of REASON) @ -16 (at line 6780)
/HDRFLT/ REAL(KIND=2) ADOBE_MAX_WIDTH @ 112 (at line 6587)
/HDRFLT/ REAL(KIND=2) ADOBE_MAX_LENGTH @ 104 (at line 6587)
|
The third "column" can have embedded spaces (see the example). Unfortunately, with long variable names, the columns don't line up.
Code: |
/REF_COMMON/ LOGICAL(KIND=3) DIRECTION_REFERENCE(0:5, 10) @ 66158 (at line 3511)
|
You can use Excel to load in the file using the @ as a separator, and the "=" as a separator. This should give you access to the variables alone. Unfortunately, for an array with more than one dimension, the other sizes may have embedded spaces. So using spaces as separators for excel will give you more difficult output to view.
If you have TYPE's, these will also show here.
Code: |
SAVE TYPE(QSTSC_INPUT_STRUCT) STRUCT @ 32784 (at line 3166)
|
|
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2816 Location: South Pole, Antarctica
|
Posted: Sat Sep 21, 2019 9:07 am Post subject: |
|
|
John
Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows.
Bill,
thanks for the hint. I know why i did not use and always forget this option: the elitist inarticulate FTN95 /? tells that /DUMP will show you variable dump in the listing file but forgets to tell that besides /DUMP the /LIS is required. Plus one of subroutines crashed compilation with /64 /DUMP /LIS ... |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sat Sep 21, 2019 1:13 pm Post subject: Re: |
|
|
DanRRight wrote: | Task Manager of Windows 10 also has devilry bug inside. Hell knows what it shows. |
Dan,
What is this devilry bug? I don't find this and do find task manager very useful.
wrt memory usage, there is a usage difference between allocating an array and actually using it. This may show a difference between "Working Set", "Peak working set" and "Memory (active private"
It does help to establish an audit of large arrays, rather than blame the devil ! |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1217 Location: Morrison, CO, USA
|
Posted: Sat Sep 21, 2019 4:26 pm Post subject: |
|
|
John and Dan,
What I have found with named common is: Using /RELEASE: If you don't use it, it doesn't get allocated. This differs from /CHECKMATE. /CHECKMATE will pre-allocate all the named common blocks.
I discovered this when running a program that waits until all options are selected to begin processing the desired data. The memory went from a few 10's of megabytes to hundreds of megabytes by the end. |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1217 Location: Morrison, CO, USA
|
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2816 Location: South Pole, Antarctica
|
Posted: Sun Sep 22, 2019 2:18 am Post subject: |
|
|
Thanks Bill. And even that is not all. No matter how much i use the computer i get Task Manager rarely showing more than 40-50% memory usage all the time even with multiple browsers with hundreds of tabs and my code for sure grabbing >90% of RAM via allocatable arrays. I can not load anything more since sometimes these browsers crash and for sure not single one of my programs are loading anymore.
Worst what happening why i wait for AMD 3950 which has 128GB limit is that when the memory demanded by my code increase i sometimes see how NVIDIA graphics driver crashes leaving screen black. Task Manager though before crash shows all is fine, "don't worry, be happy", "Everything is alright, my beautiful Marquise" (humorous song from 1930th where the whole estate was burnt, but the estate manager calls the owner marquise and slowly tells all is alright) |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sun Sep 22, 2019 5:33 am Post subject: |
|
|
Dan,
Task Manager works for me, I think you are not seeing the difference between committed memory and used memory.
The following program may be a useful test.
It is the first time I have tested allocatable arguments with FTN95 and appears to work with with Ver 8.51
If you run it, responding to the ? with a 1, you will see the memory usage increase in task manager.
Note the difference working set and commit size.
Physical memory % relates to working set and not commit size, although I expect you may run out of memory based on commit size.
Anyway, good test of allocatable array arguments for Ver 8.5+
Code: | ! program to first allocate big arrays, then initialise big arrays
! requires FTN95 Ver 8.5 to work with allocatable arguments
!
module sz
real*4 :: sz_mb = 500. ! array size in mb
real*4 :: sz_ni ! array elements
integer*4 :: ni ! dimension of array
end module sz
!
program use_memory
use sz
!
interface
subroutine allocate_array ( aa )
use sz
integer*4, allocatable, dimension(:,:) :: aa
end subroutine allocate_array
end interface
!
integer*4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g
integer*4 :: i,j
!
sz_gb = 500 ! gb
sz_n = sz_gb * 1024.**2 / 4
ni = sqrt (sz_n)
write (*,*) '2D array dimension =',ni
!
do i = 1,8
select case (i)
case (1)
call allocate_array ( a )
case (2)
call allocate_array ( b )
case (3)
call allocate_array ( c )
case (4)
call allocate_array ( d )
case (5)
call allocate_array ( e )
case (6)
call allocate_array ( f )
case (7)
call allocate_array ( g )
case default
end select
write (*,*) i, ' arrays allocated: check task manager'
read (*,*) j
end do
!
do i = 1,8
select case (i)
case (1)
call initialise_array ( a )
case (2)
call initialise_array ( b )
case (3)
call initialise_array ( c )
case (4)
call initialise_array ( d )
case (5)
call initialise_array ( e )
case (6)
call initialise_array ( f )
case (7)
call initialise_array ( g )
case default
end select
write (*,*) i, ' arrays initialised: check task manager'
write (*,*) '?'
read (*,*) j
end do
!
end program use_memory
subroutine allocate_array ( aa )
use sz
integer*4, allocatable, dimension(:,:) :: aa
integer*4 :: stat
allocate ( aa(ni,ni), stat=stat )
write (*,*) 'array allocated : stat=',stat
end subroutine allocate_array
subroutine initialise_array ( aa )
use sz
integer*4 :: aa(ni,ni)
integer*4 :: i,j, n
n = 0
do j = 1,ni
do i = 1,ni
aa(i,j) = i+j
n = n+1
end do
end do
write (*,*) 'array initialised',n
end subroutine initialise_array |
Hopefully no errors in this coding !! |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sun Sep 22, 2019 6:58 am Post subject: |
|
|
I have now run task manager in Win7 and Win10.
With Win 10, on the performance tab, the memory graph is for usage, while "In use" includes the Allocated arrays being used and "Committed" does show the total memory, included Allocated arrays, but not yet in use.
??? hope that was clear !!, Anyway Used memory is for memory that has been taken and given a value, while Committed also includes memory (Fortran) allocated but not yet physically allocated and given a value.
I am not sure how you can crash if memory committed exceeds available installed memory ? as this should not happen until arrays are being used and physical memory is required.
Once you exceed physical memory available, then it goes to virtual memory on pagefile.sys and the system usually slows down dramatically (looks like it has stopped)
Your system could crash if pagefile.sys is not big enough to support the virtual memory requested. I have seen pc's with pagefile.sys configured to be too small (smaller than physical memory). This is hardly Task manager's fault ? |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2816 Location: South Pole, Antarctica
|
Posted: Sun Sep 22, 2019 8:28 am Post subject: |
|
|
Never was able to allocate RAM+pagefile.sys, maximum was close to RAM size. Would be nice to allocate RAM+pagefile. But probably as I suspect there exist limit of allocation which is Intel processors RAM limit. It was 32 then 64 and now new models have 128 GB
My crash of video driver as I suspect happen when I reach limit of processor despite the RAM size + pagefile.sys are 2x larger
OK, I see what was always confusing with this effing default view of Task Manager on Win10. Before in previous versions there was no such confusion. When I load my program the Task Manager shows on graph memory usage hits its peak but few seconds later monitor shows around 50% which is what they call in-Use memory while hiding part they call Standby both are around 50:50 in size
There are also Available, Cached, Total, Installed and Commit'ted. Also for the all processes running on PC there exist additional such values. And Committed from the Resource Manager seems indeed looks like what my arrays occupy. Exclusing COMMON files lazy allocation Bill mentioned |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sun Sep 22, 2019 8:49 am Post subject: |
|
|
Dan,
If you are having video problems, I assume you are sharing memory ?
"Cached" is interesting. I assume it refers to disk I/O cache (buffer) and not processor cache.
Although I do have pagefile.sys configured to larger than installed memory, I rarely use it for my programs, as the system basically stops. Back in 70's we used the equivalent of pagefile.sys all the time and thought it was great (although 1,000 times slower than today)
Not sure I understand your problem(s) as I now try to use allocatable arrays in modules, rather than COMMON.
Anyway, good test for using allocatable arrays as arguments. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sun Sep 22, 2019 9:03 am Post subject: |
|
|
To follow up on the test program; the test creates 8 arrays and checks them for being available with the correct values.
This test demonstrates FTN95 Ver8.5 supporting allocatable arrays as arguments (the call requires an interface definition)
It works for both 32-bit and 64-bit /64 .
Interestingly, if I create 8 x 400mb arrays, this demonstrates for both 32-bit and 64-bit, using 3.2 GB of addressable memory.
The tidied up program is: Code: | ! program to first allocate big arrays, then initialise big arrays
! requires FTN95 Ver 8.5 to work with allocatable arguments
!
module sz
real*4 :: sz_mb ! array size in mb
real*4 :: sz_ni ! array elements
integer*4 :: ni ! dimension of array
end module sz
!
program use_memory
use sz
!
interface
subroutine allocate_array ( aa )
use sz
integer*4, allocatable, dimension(:,:) :: aa
end subroutine allocate_array
end interface
!
integer*4, allocatable, dimension(:,:) :: a,b,c,d,e,f,g,h
integer*4 :: i,j
!
sz_mb = 500 ! MBytes
!
call prompt ('What array size to test (in MBytes, default 500) ?', j)
if ( j > 1 .and. j < 2048 ) sz_mb = j
sz_n = sz_mb * 1024.**2 / 4
ni = sqrt (sz_n)
write (*,*) '2D I*4 array dimension =',ni
!
do i = 1,9
select case (i)
case (1)
call allocate_array ( a )
case (2)
call allocate_array ( b )
case (3)
call allocate_array ( c )
case (4)
call allocate_array ( d )
case (5)
call allocate_array ( e )
case (6)
call allocate_array ( f )
case (7)
call allocate_array ( g )
case (8)
call allocate_array ( h )
case default
write (*,*) 'all allocates done : now to initialise'
exit
end select
write (*,*) i, ' arrays allocated: check task manager'
call prompt ('Proceed ?',j)
end do
!
do i = 1,9
select case (i)
case (1)
call initialise_array ( a, i )
case (2)
call initialise_array ( b, i )
case (3)
call initialise_array ( c, i )
case (4)
call initialise_array ( d, i )
case (5)
call initialise_array ( e, i )
case (6)
call initialise_array ( f, i )
case (7)
call initialise_array ( g, i )
case (8)
call initialise_array ( h, i )
case default
write (*,*) 'all initialise done : now to test'
exit
end select
write (*,*) i, ' arrays initialised: check task manager'
call prompt ('Proceed ?',j)
end do
! |
|
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sun Sep 22, 2019 9:05 am Post subject: |
|
|
ctd/.. Code: | !
do i = 1,9
select case (i)
case (1)
call test_array ( a, i )
case (2)
call test_array ( b, i )
case (3)
call test_array ( c, i )
case (4)
call test_array ( d, i )
case (5)
call test_array ( e, i )
case (6)
call test_array ( f, i )
case (7)
call test_array ( g, i )
case (8)
call test_array ( h, i )
case default
write (*,*) 'all tests now done : final ? before exiting'
end select
if ( i < 9) write (*,*) i, ' arrays tested: check task manager'
call prompt ('Proceed ?',j)
end do
!
end program use_memory
subroutine allocate_array ( aa )
use sz
integer*4, allocatable, dimension(:,:) :: aa
integer*4 :: stat
allocate ( aa(ni,ni), stat=stat )
write (*,*) 'array allocated : stat=',stat
end subroutine allocate_array
subroutine initialise_array ( aa, ii )
use sz
integer*4 :: aa(ni,ni), ii
integer*4 :: i,j, n
n = 0
do j = 1,ni
do i = 1,ni
aa(i,j) = i+j+ii
n = n+1
end do
end do
write (*,*) 'array initialised',n
end subroutine initialise_array
subroutine test_array ( aa, ii )
use sz
integer*4 :: aa(ni,ni), ii
integer*4 :: i,j, n, e
n = 0
e = 0
do j = 1,ni
do i = 1,ni
if ( aa(i,j) /= i+j+ii ) e = e+1
n = n+1
end do
end do
write (*,*) 'array tested',n, ' :',e,' errors'
end subroutine test_array
subroutine prompt (string, j)
character string*(*)
integer j, iostat
write (*,fmt='(a)',advance='NO') string
read (*,fmt='(bn,i6)', iostat=iostat) j
if (iostat /= 0) j = -1
end subroutine prompt |
|
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2816 Location: South Pole, Antarctica
|
Posted: Mon Sep 23, 2019 1:48 am Post subject: |
|
|
John,
Allocatable arrays are shown in Commit part. COMMON arrays are not till they are used (lazy allocation). Run this code and look at memory in Task Manager and then uncomment two lines with "!!!" in subroutine sub and run again
Code: | common/a1/a, aa(7000,7000), aaa(7000,7000)
A=1
call sub
end
subroutine sub
common/a1/a, aa(7000,7000), aaa(7000,7000)
print*, a
!!!aa(:,:)=2
!!!aaa(:,:)=3
pause
end
|
If compile
FTN95 aaa.f95 /link
FTN95 aaa.f95 /link /64
there are also differences in In-Use memory. So damn things are confusing and you will forget what is what if you are not visit these things often.
Currently with 64 bits all these COMMON are restricted to 2GB but when (or IF ) the Silverfrost will remove the stack limits this could become a problem when you set huge array and forget about it or even do not know from the beginning that you have set a huge sucker of memory
Last edited by DanRRight on Mon Sep 23, 2019 3:49 am; edited 1 time in total |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|