 |
forums.silverfrost.com
Welcome to the Silverfrost forums
|
| View previous topic :: View next topic |
| Author |
Message |
KennyT
Joined: 02 Aug 2005
Posts: 318
|
 Posted: Tue Sep 28, 2010 5:40 am Post subject: Use of ALLOCATE and the HEAP for large arrays |
 |
|
Hi,
I have to allocate memory for a large bitmap (>150MB) and my ALLOCATE statement is failing. From what I can gather, ALLOCATE uses the heap which is set at 100MB. However, I can't see any explicit examples of use of the HEAP directive in SLINK so I have some questions:
1 Will increasing the HEAP help?
2 What is the HEAP value I need to be able to allocate 200MB at a time?
3 Should the HEAP directive be applied to my EXE, the relevant DLL, all DLLs, or what?
TIA
K |
|
| Back to top |
|
 |
PaulLaidler
Site Admin
Joined: 21 Feb 2005
Posts: 8210
Location: Salford, UK
|
| Posted: Tue Sep 28, 2010 10:15 am Post subject: |
|
|
If you are using a recent version of salflibc.dll and you are not using /check (or anything that implies it) then ALLOCATE will simply take memory from the global heap (i.e. a GlobalAlloc type of statement).
Typically with a 32bit operarting system you can ALLOCATE a total of about 1.7MB for the whole program data. I would need to check this but I don't think that changing the SLINK parameters makes any difference.
In theory you can use the /3MB switch on the SLINK command line but I have not heard of anyone who has found this helpful. |
|
| Back to top |
|
|
KennyT
Joined: 02 Aug 2005
Posts: 318
|
| Posted: Tue Sep 28, 2010 10:26 am Post subject: |
|
|
Thanks for pipcking this up Paul. I'm fairly sure there's a something like a 100Mb limit on an ALLOCATE though, since with testing, 45Mb works, 96Mb works but 120Mb fails...
I've got 3.2Gb of disc cacheing set and I've recently "defraggled" the drive so there should be plenty of contiguous memory space.
K |
|
| Back to top |
|
|
PaulLaidler
Site Admin
Joined: 21 Feb 2005
Posts: 8210
Location: Salford, UK
|
| Posted: Tue Sep 28, 2010 12:39 pm Post subject: |
|
|
Qusetion is how much memory is being allocated in the rest of the program. A two or three line program with one ALLOCATE statement should allow an integer array of aprroximate size x where 4x = 1.7GB etc.
i.e a default integer occupies 4 bytes. |
|
| Back to top |
|
|
KennyT
Joined: 02 Aug 2005
Posts: 318
|
| Posted: Tue Sep 28, 2010 12:58 pm Post subject: |
|
|
OK, some test results:
A "Noddy" program works and allocates 216Mb successfully.
My main app fails. Before starting it, using Task Manager, shows memory usage on the machine as 2.32Gb. When the ALLOCATE fails, this has risen to 2.67Gb, so as far as I can tell, only 350Mb is in use by the app, unless there's a more certain way of establishing usage...
I'll try making a more complex "Noddy" (perhaps with DLLs?) and report back.
K |
|
| Back to top |
|
|
KennyT
Joined: 02 Aug 2005
Posts: 318
|
| Posted: Tue Sep 28, 2010 1:31 pm Post subject: |
|
|
OK, sussed it!
I had some "VC" and "BASE" statements in my EXE and DLL SLINK scripts (don't ask why, the reasons are lost in the mists of time!).
So I took them out and it started to work!
Thanks for the help.
K  |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Tue Sep 28, 2010 1:38 pm Post subject: |
|
|
Kenny,
The problem with allocating large chunks of memory is that even if your program initially has only a small memory footprint, SLINK may locate the stack, heap and other bits throughout the 2gb of addressable memory. For example, while your program may initially use say 0.3gb of the 1.7gb of practically available addressable memory, it is unlikely that SLINK will leave you a 1.4gb chunk of contiguous free memory. If you want to ensure you have space for large arrays, it is sometimes better to pre-define them in common. Trying to allocate 4 arrays of 400mb would probably fail as there is not a 1.6gb vacant block to start with. If the initial memory footprint included two contiguous spaces of 0.7gb, then allocating 2 x 0.4gb arrays would use up all this space and a 3rd allocation of 0.4gb would fail, even though there is 0.6gb of memory free. You can test this by trying to allocate a very large array, say 1.8gb, then keep reducing it in size until it does not fail, then repeating it for a second, third and 4th array. You will fine the size quickly reduces below the 0.4gb size you were wanting.
Paul provided me with a program to test the memory for the /3GB switch, by re-allocating the same pointer array (which doesn't release the past allocations). By testing the start address of all these arrays (say 20mb in size), I was able to map what parts of memory were not available for contiguous allocation of larger arrays. There were a number of pre-allocated spots throughout the addressable space. This changed as the initial size of teh program changed. The only way to get a 1.6gb single array is to define it in common.
I also tried to use virtual common, but that appeared to start at 1gb, providing space for an array of about 1gb in size.
However if you want to allocate lots of arrays, say 100mb each, then you should be able to allocate about 5 to 10 of these, as ALLOCATE looks for a free space that will fit the array. Make sure you test the status of the ALLOCATE.
I do not know a way to get slink to leave a large amount of contiguous free memory (say 1.0gb plus in size) for later use by ALLOCATE.
John
| Code: | ! program to find the largest available ALLOCATABLE memory space
!
real*8, allocatable, dimension(:) :: A1, a2, a3, a4
integer*4 i1,i2,i3,i4,iostat
!
1001 format ('Array A',i0,' alloacted as ',i0,' mb, at address ',b'z,zzz,zzz,zz#')
! A1
do mb = 1900,10,-10
i1 = mb * 2**17
allocate (A1(i1),stat=iostat)
if (iostat == 0) exit
end do
write (*,1001) 1,mb,loc(A1)
!
! A2
do mb = 1900,10,-10
i2 = mb * 2**17
allocate (A2(i2),stat=iostat)
if (iostat == 0) exit
end do
write (*,1001) 2,mb,loc(A2)
!
! A3
do mb = 1900,10,-10
i3 = mb * 2**17
allocate (A3(i3),stat=iostat)
if (iostat == 0) exit
end do
write (*,1001) 3,mb,loc(A3)
!
! A4
do mb = 1900,10,-10
i4 = mb * 2**17
allocate (A4(i4),stat=iostat)
if (iostat == 0) exit
end do
write (*,1001) 4,mb,loc(A4)
!
end |
|
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Tue Sep 28, 2010 2:24 pm Post subject: |
|
|
The following example will show the effect of using a larger defined array if you comment out the code associated with the array A0. Note 1500mb is bigger than the 1290mb that is allocatable.
I'm not sure how this would run with other compilers ?
| Code: | ! program to find the largest available ALLOCATABLE memory space
!
integer*4, parameter :: one_mb = 2**18
integer*4, parameter :: i0 = 1500 * one_mb
integer*4 a0(i0)
common /aa/ a0
!
integer*4, pointer, dimension(:) :: ii
integer*4 i,iostat, m, l
integer*4 get_free_memory_size
external get_free_memory_size
!
write (*,1002) 0,size(a0)*4, loc(A0)
!
1002 format ('Array A',i0,' alloacted as ',i11,' bytes, at address ',i11, i5)
!
do i = 1,20
m = get_free_memory_size (i)
!
allocate (ii(m),stat=iostat)
l = loc(ii)
!
write (*,1002) i,m*4,l, iostat
if (iostat /= 0) exit
if (m < 1) exit
end do
!
end
integer*4 function get_free_memory_size (i)
!
integer*4 i, ml, mh, m, iostat
integer*4, allocatable, dimension(:) :: ii
!
ml = 0
mh = 2**29
!
do
m = ml + (mh-ml)/2
if (m == ml) exit
allocate (ii(m), stat=iostat)
if (iostat /= 0) then
mh = m
else
ml = m
deallocate (ii)
end if
end do
!
get_free_memory_size = m
!
end function get_free_memory_size |
|
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 1:53 am Post subject: |
|
|
The situation gets more confusing !!
I have further modified the program above and run it on my XP 32-bit and XP 64-bit OS and I appear to have 4gb of available memory on the 64-bit version; thats if my routine "get_free_memory_size" is working.
My latest program is | Code: | ! program to find the largest available ALLOCATABLE memory space
!
integer*4, parameter :: one_mb = 2**18
integer*4, parameter :: i0 = 1500 * one_mb
integer*4 a0(i0)
common /aa/ a0
!
integer*4, pointer, dimension(:) :: ii
integer*4 i,iostat, m, k
integer*4 get_free_memory_size
external get_free_memory_size
integer*8 l, nblock, block_start(0:30), block_size(0:30)
!
nblock = 0
block_start = 0
block_size = 0
!
l = loc(a0)
write (*,1002) 0,size(a0)*4, l
!
block_start(1) = l
block_size(1) = size(a0)*4
nblock = 2
block_start(2) = nblock**32
!
i = 0
do k = 0,nblock
write (*,*)i,k,block_start(k),block_size(k)
end do
!
1002 format ('Array A',i0,' allocated as ',b'zz,zzz,zzz,zz#',' bytes, at address ',b'zz,zzz,zzz,zz#', i5)
!
do i = 1,20
m = get_free_memory_size (i)
!
allocate (ii(m),stat=iostat)
l = loc(ii)
!
write (*,1002) i,m*4,l, iostat
if (iostat /= 0) exit
if (m < 1) exit
!
do k = nblock,0,-1
if (block_start(k) < l) then
block_start(k+1) = l
block_size(k+1) = m*4
nblock = nblock+1
exit
else
block_start(k+1) = block_start(k)
block_size(k+1) = block_size(k)
end if
end do
! do k = 0,nblock
! write (*,*)i,k,block_start(k),block_size(k)
! end do
end do
!
! report all blocks and leading gaps
write (*,*) ' Blk Gap Start Size'
do k = 1,nblock
write (*,2002) k, block_start(k) - (block_start(k-1)+block_size(k-1)), block_start(k), block_size(k)
end do
2002 format (i5,3(b'zz,zzz,zzz,zz#'))
end
integer*4 function get_free_memory_size (i)
!
integer*4 i, ml, mh, m, iostat
integer*4, allocatable, dimension(:) :: ii
!
ml = 0
mh = 2**29
!
do
m = ml + (mh-ml)/2
if (m == ml) exit
allocate (ii(m), stat=iostat)
if (iostat /= 0) then
mh = m
else
ml = m
deallocate (ii)
end if
end do
!
get_free_memory_size = m
!
end function get_free_memory_size
|
The output on my 64-bit is below. |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 1:56 am Post subject: |
|
|
64-bit run | Code: | Array A0 allocated as 1,572,864,000 bytes, at address 4,206,668
0 0 0 0
0 1 4206668 1572864000
0 2 4294967296 0
Array A1 allocated as 2,147,221,468 bytes, at address 2,147,418,144 0
Array A2 allocated as 184,418,268 bytes, at address 1,798,111,264 0
Array A3 allocated as 64,225,244 bytes, at address 2,025,390,112 0
Array A4 allocated as 25,559,004 bytes, at address 1,769,603,104 0
Array A5 allocated as 17,432,540 bytes, at address 2,113,142,816 0
Array A6 allocated as 12,058,588 bytes, at address 2,013,265,952 0
Array A7 allocated as 10,878,940 bytes, at address 1,991,049,248 0
Array A8 allocated as 7,995,356 bytes, at address 1,982,857,248 0
Array A9 allocated as 6,946,780 bytes, at address 1,710,161,952 0
Array A10 allocated as 4,128,732 bytes, at address 2,004,549,664 0
Array A11 allocated as 2,359,260 bytes, at address 2,099,773,472 0
Array A12 allocated as 2,293,724 bytes, at address 1,795,489,824 0
Array A13 allocated as 2,097,116 bytes, at address 2,110,652,448 0
Array A14 allocated as 1,114,076 bytes, at address 2,104,426,528 0
Array A15 allocated as 851,932 bytes, at address 2,108,686,368 0
Array A16 allocated as 720,860 bytes, at address 2,098,004,000 0
Array A17 allocated as 655,324 bytes, at address 2,002,583,584 0
Array A18 allocated as 520,180 bytes, at address 1,036,296 0
Array A19 allocated as 344,052 bytes, at address 1,556,488 0
Array A20 allocated as 258,036 bytes, at address 2,106,130,504 0
Blk Gap Start Size
1 1,036,296 1,036,296 520,180
2 12 1,556,488 344,052
3 2,306,128 4,206,668 1,572,864,000
4 133,091,284 1,710,161,952 6,946,780
5 52,494,372 1,769,603,104 25,559,004
6 327,716 1,795,489,824 2,293,724
7 327,716 1,798,111,264 184,418,268
8 327,716 1,982,857,248 7,995,356
9 196,644 1,991,049,248 10,878,940
10 655,396 2,002,583,584 655,324
11 1,310,756 2,004,549,664 4,128,732
12 4,587,556 2,013,265,952 12,058,588
13 65,572 2,025,390,112 64,225,244
14 8,388,644 2,098,004,000 720,860
15 1,048,612 2,099,773,472 2,359,260
16 2,293,796 2,104,426,528 1,114,076
17 589,900 2,106,130,504 258,036
18 2,297,828 2,108,686,368 851,932
19 1,114,148 2,110,652,448 2,097,116
20 393,252 2,113,142,816 17,432,540
21 16,842,788 2,147,418,144 2,147,221,468
22 327,684 4,294,967,296 0
|
|
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 1:58 am Post subject: |
|
|
32 bit OS run | Code: | Array A0 allocated as 1,572,864,000 bytes, at address 4,206,668
0 0 0 0
0 1 4206668 1572864000
0 2 4294967296 0
Array A1 allocated as 201,064,412 bytes, at address 1,759,313,952 0
Array A2 allocated as 75,497,436 bytes, at address 2,013,265,952 0
Array A3 allocated as 22,609,884 bytes, at address 1,960,837,152 0
Array A4 allocated as 19,070,940 bytes, at address 2,099,118,112 0
Array A5 allocated as 18,546,652 bytes, at address 2,119,434,272 0
Array A6 allocated as 16,842,716 bytes, at address 1,637,548,064 0
Array A7 allocated as 8,126,428 bytes, at address 2,139,029,536 0
Array A8 allocated as 7,602,140 bytes, at address 1,983,905,824 0
Array A9 allocated as 6,225,884 bytes, at address 2,002,911,264 0
Array A10 allocated as 5,963,740 bytes, at address 1,991,704,608 0
Array A11 allocated as 2,228,188 bytes, at address 1,998,258,208 0
Array A12 allocated as 1,441,756 bytes, at address 2,009,530,400 0
Array A13 allocated as 655,324 bytes, at address 1,636,827,168 0
Array A14 allocated as 520,180 bytes, at address 299,016 0
Array A15 allocated as 491,508 bytes, at address 819,208 0
Array A16 allocated as 564 bytes, at address 298,440 0
Array A17 allocated as 548 bytes, at address 271,192 0
Array A18 allocated as 516 bytes, at address 269,968 0
Array A19 allocated as 76 bytes, at address 284,232 0
Array A20 allocated as 68 bytes, at address 289,464 0
Blk Gap Start Size
1 269,968 269,968 516
2 708 271,192 548
3 12,492 284,232 76
4 5,156 289,464 68
5 8,908 298,440 564
6 12 299,016 520,180
7 12 819,208 491,508
8 2,895,952 4,206,668 1,572,864,000
9 59,756,500 1,636,827,168 655,324
10 65,572 1,637,548,064 16,842,716
11 104,923,172 1,759,313,952 201,064,412
12 458,788 1,960,837,152 22,609,884
13 458,788 1,983,905,824 7,602,140
14 196,644 1,991,704,608 5,963,740
15 589,860 1,998,258,208 2,228,188
16 2,424,868 2,002,911,264 6,225,884
17 393,252 2,009,530,400 1,441,756
18 2,293,796 2,013,265,952 75,497,436
19 10,354,724 2,099,118,112 19,070,940
20 1,245,220 2,119,434,272 18,546,652
21 1,048,612 2,139,029,536 8,126,428
22 2,147,811,332 4,294,967,296 0
|
Are others able to confirm this result ?
What is left is to confirm if the extra 2gb of memory provided by WOW64 is useable. Which would increase the available memory from 1.7gb to about 4gb.
My interpretation of the results presented above is the leading GAP reported is memory taken by SLINK at run time. There are certainly a lot of little blocks being used. I think the stack is about 60 mb. It would be good if SLINK could clean up all it's droppings !!
I should also do a run with the A0:common array much smaller.
John
Last edited by JohnCampbell on Wed Sep 29, 2010 4:20 am; edited 1 time in total |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 2:16 am Post subject: |
|
|
The following is a run on XP-64 with array A0 only 1mb. This "map" shows a lot of bits taken throughout the second gb of memory.
size limit - This can be deleted
Last edited by JohnCampbell on Wed Sep 29, 2010 2:18 am; edited 1 time in total |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 2:17 am Post subject: |
|
|
The following is a run on XP-64 with array A0 only 1mb. This "map" shows a lot of bits taken throughout the second gb of memory.
| Code: | Array A0 allocated as 1,048,576 bytes, at address 4,206,668
0 0 0 0
0 1 4206668 1048576
0 2 4294967296 0
Array A1 allocated as 2,147,221,468 bytes, at address 2,147,418,144 0
Array A2 allocated as 995,098,588 bytes, at address 270,860,320 0
Array A3 allocated as 528,613,340 bytes, at address 1,266,548,768 0
Array A4 allocated as 184,418,268 bytes, at address 1,798,111,264 0
Array A5 allocated as 80,543,708 bytes, at address 187,891,744 0
... (dam size limit!!)
Array A26 allocated as 53,236 bytes, at address 2,010,316,872 0
Array A27 allocated as 45,044 bytes, at address 4,128,840 0
Array A28 allocated as 4,020 bytes, at address 4,173,896 0
Blk Gap Start Size
1 1,036,296 1,036,296 520,180
2 12 1,556,488 344,052
3 2,228,300 4,128,840 45,044
4 12 4,173,896 4,020
5 28,752 4,206,668 1,048,576
6 182,636,500 187,891,744 80,543,708
7 2,424,868 270,860,320 995,098,588
8 589,860 1,266,548,768 528,613,340
9 327,716 1,795,489,824 2,293,724
10 327,716 1,798,111,264 184,418,268
11 327,716 1,982,857,248 7,995,356
12 196,644 1,991,049,248 10,878,940
13 655,396 2,002,583,584 655,324
14 1,310,756 2,004,549,664 4,128,732
15 393,292 2,009,071,688 77,812
16 1,167,372 2,010,316,872 53,236
17 2,895,844 2,013,265,952 12,058,588
18 65,572 2,025,390,112 64,225,244
19 8,388,644 2,098,004,000 720,860
20 1,048,612 2,099,773,472 2,359,260
21 1,245,260 2,103,377,992 61,428
22 987,108 2,104,426,528 1,114,076
23 589,900 2,106,130,504 258,036
24 331,788 2,106,720,328 61,428
25 856,076 2,107,637,832 102,388
26 946,148 2,108,686,368 851,932
27 1,114,148 2,110,652,448 2,097,116
28 393,252 2,113,142,816 17,432,540
29 16,842,788 2,147,418,144 2,147,221,468
30 327,684 4,294,967,296 0
|
Wow 64 !! This shows 3 big arrays may be possible of 2gb, 990mb and 520mb. ( Unfortunalely, my original problem that prompted my testing of /3gb needed a single array of 2.3gb )
Again, can anyone else confirm similar results.
It would be great if WOW64 provided 4gb of addressible memory, even if the biggest chunk was about 2gb !
John |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
| Posted: Wed Sep 29, 2010 3:57 am Post subject: |
|
|
I have written a simple big array program and run it, checking the vitrual memory allocation in Windows Task Manager as 3,534,880k and it appeared to run successfully on XP win64. It will not run on XP-32. The array sizes were based on my last post.
Can anyone else confirm this on a 64 bit os ?
| Code: | ! PROGRAM to test allocating 3 big arrays
integer*4, allocatable, dimension(:) :: a1, a2, a3
integer*4 l1, l2, l3, iostat
!
l1 = 2100 * 1000 * 1000 / 4
l2 = 990 * 1000 * 1000 / 4
l3 = 520 * 1000 * 1000 / 4
!
allocate (a1(l1), stat=iostat)
write (*,*) 'A1 allocated',iostat
!
allocate (a2(l2), stat=iostat)
write (*,*) 'A2 allocated',iostat
!
allocate (a3(l3), stat=iostat)
write (*,*) 'A3 allocated',iostat
!
call define_array (a1,l1)
call define_array (a2,l2)
call define_array (a3,l3)
!
call check_array (a1,l1)
call check_array (a2,l2)
call check_array (a3,l3)
END PROGRAM
subroutine define_array (a,l)
integer*4 l, a(l), i, n
real*8 mb
n = l/2
do i = 1,l
a(i) = n
n = n-1
end do
mb = l
mb = mb * 4. / 1024. /1024.
write (*,*) 'Array defined for size =',l, mb
end subroutine
subroutine check_array (a,l)
integer*4 l, a(l), i, n, er
n = l/2
er = 0
do i = 1,l
if (a(i) /= n) er = er+1
n = n-1
end do
write (*,*) 'Array checked for size =',l,' error =',er
end subroutine
|
|
|
| Back to top |
|
|
JohnHorspool
Joined: 26 Sep 2005
Posts: 270
Location: Gloucestershire UK
|
| Posted: Wed Sep 29, 2010 8:06 am Post subject: |
|
|
John
Does this help?
cheers,
John
program output
| Code: | A1 allocated 0
A2 allocated 0
A3 allocated 0
Array defined for size = 525000000 2002.71606445
Array defined for size = 247500000 944.137573242
Array defined for size = 130000000 495.910644531
Array checked for size = 525000000 error = 0
Array checked for size = 247500000 error = 0
Array checked for size = 130000000 error = 0 |
machine Xeon quadcore 3.46 GHz 24 GB ram
OS is XP64 Pro SP2 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
