forums.silverfrost.com

[PaulLaidler]

Hi John

We have considered the questions that you raise and suggest that the best way forward would be for you to send us a small program/project that illustrates the problems that you are encountering.

In the mean time here is a sample program that allocates about 2.8 GB on my machine

[JohnCampbell]

Paul,

I emailed you the file big_new_memory.zip on 16 Sep to your hotmail account, which was the account you had previously advised for emailing sample programs. It is a small program with little code, but big arrays. It's a bit of a mess as there is a bit of commented out code, as I tried different ideas.
My example used explicit array sizeing with common or modules. After 1.85 gb of arrays, the executable generated gives "is not a valid Win32 application" error ?

I will try your example, using allocate. My big memory PC is at work.
What pool of memory does allocate get the big chunks of memory from ?

John

[PaulLaidler]

ALLOCATE ends up using the Windows API GlobalAlloc or HeapAlloc.

We will look at your sample code as soon as we can.

Paul

[JohnCampbell]

Paul,

I tried the allocate approach with real*8 arrays and it failed after 1.15gb.
the following is the commands I used to run it.

ftn95 big_alloc /lis /xref
slink big_alloc.obj /3gb /map:big_alloc.map /file:big_alloc.exe

[JohnCampbell]

the post worked in preview, but clipped when sent. this is the rest of the code.

[JohnCampbell]

Paul,

I also tried various block sizes in your program with unusual results.
While 80mg blocks gives 2,768mb, the allocated memory reduces with bigger block sizes.
(> 597mg) 600mg blocks gives only 2 blocks and fails at only 1,887mb while 1,000mg blocks gives up to 2,097mb
1,200mg and above gives nothing !!

Unfortunately, I'm from old Fortran and am not familiar with pointers !

All this with FTN95_NEW_MEMORY

[JohnCampbell]

Paul,

I again modified your example to allocate multiple real*8 arrays and test the gaps between them using LOC.

Using 312,000,000 byte chunks, the allocation started at about 258 mb,
it allocated 4 blocks from 258mb to 1448 mb
it then left a 41mb gap at 1449mb and
another 261mb gap at 1787 mb
finally 3 blocks from 2048mb to 2940 mb

Do you know why it started at 258mb and what are the 2 other gaps ?
I'd expect if I used 80mb blocks, there may be other gaps.
This also shows a maximum contiguous area of 1200 mb without any influence.
Presumably when we get arrays of 1.8gb, some of these gaps are moved around.

The following is the code I used, which gave these interesting results.

[JohnCampbell]

Again there was too much code:

[JohnCampbell]

Paul,

I again took your program and tried to extend the "cheating" on multiple allocates for pointer arrays, to test the memory availability. I put the program in as a subroutine, so that when it exits, I thought all the allocates would be automatically deallocated (fortran standard ?). This does not appear to work.
However with a block size of 20mb, it shows in more detail what areas of memory are not available. The program also appears to confirm that the /3gb switch is providing the 3gb of accessible memory.
I'm assuming the contiguous areas can be addressed as merged blocks. I will take the previous real*8 array program in an earlier post and confirm if this works.

It remains to be able to shift the "unavailable parts" to a better area to maximise the size of contiguous available memory. Isn't this what Slink does ?

[JohnCampbell]

Paul,

I took the memory test subroutine and placed it in the large program I am wanting to use for extended memory.
I ran it at the start for a number of different working array sizes, with and without /3gb switch in slink.

1) without the 3gb in slink, only 2gb of memory is accessible.

2) with 300mb working array, i get a memory usage of :
first 371 mg is taken by program ( 300mb + 71mb)
at 1429mb, there is a 24.6mb gap
at 1486mb, a 3.5mb gap
at 1874mb, a 46.1mb gap
at 1984mb, a 63.9mb gap

3) with 1000mb working, a similar set of gaps

4) with 1600mb working array, i get a memory usage of :
first 1664 mg is taken by program ( 1600mb + 64mb)
gaps at 1429mb and 1486mb have been overwritten, ie omitted
at 1889mb, 46.1mb gap has been reduced to 30.8mb
at 1984mb, a 63.9mb gap

5) with 1750mb, similar (.map version)

ie, the gaps have been removed and do not appear to have been shifted

I did not run the program past the memory test.
There appears to be 3 parts of memory reserved for a small program, but only 2 for a larger program.
If the program encroaches on the 2 beyond 1889mb, then it becomes an illegal .exe
What are these 3 parts of memory being used for ?
Can they be relocated, perhaps to 2900mb or possibly to 64mb, and place the .bss section after them ?

I am attaching to an email the memory availability dump I generated from the memory scanning subroutine for these 5 tests and the last .map file for the program.

I did not run the program.
I have not yet tested the earlier real*8 array test to see if the memory allocated is useable.

Does this make any sense ?

Regards

John

[PaulLaidler]

Ian

I have not managed to find your sample code.
Can you send it again please to my hotmail account.

Paul

[JohnCampbell]

Paul,

It appears that the attachments may have been deleted, as they contained .exe files. I will send it again, without the .exe.
I have included all the recent test programs.
Please let me know if the attachments are not there this time. Otherwise I will have to use my home email account !

John

[JohnCampbell]

Paul,

I have done a review of the large program I am wanting to run in 3gb of memory. The following is an approximate breakdown of the memory usage in the .map file:-

0.64mb .text - which is my code
0.12mb .data - which is salflibc/kernel32
0.69mb .salfdbg - presumably due to link map and /debug compilation
1.93mb .bss - my common excluding the big array

The total memory usage is 3.44 mb, excluding the big array

In summary, the memory usage, excluding the big array would be 0.1% of memory.

To use anything significantly above 2gb, the big array must be 99.9% of this. Either I must be able to have a single array bigger than 2gb or I have a major rewrite to use 2 big arrays or "chunks" in my algorithm.

The example problem I have been targeting to use for this /3gb approach requires an array of about 2.4gb in size. Will this be a possibility ?

Are any other forum users interested in this /3gb facility ?

Paul, thanks again for your assistance with this.

John

[PaulLaidler]

John

My understanding is that you cannot have a single array of more than 2GB.
Apart from rewriting your code, the only possible way forward would be to try running your program under WOW on a 64 bit machine and operating system. I don't know if anyone has tried this but you may be able to get 4GB this way. Alternatively you would have to switch to a 64 bit compiler, machine and operating system.

Paul

[DanRRight]