Silverfrost Forums

Welcome to our forums

Limits on GET_STORAGE@?

3 Apr 2012 2:33 #9938

I'm getting a failure using GET_STORAGE@ when I try to grab just over 100MB in one go. I'm on windows7-64 bit and there seems to be plenty of space left at the time the error occurs (system memory is about 50% so there should be about 4Gb available of my assigned swap space).

I'm wondering if the 100Mb is a coincidence or whether there's a system (or SLINK?) limit I can change?

K

3 Apr 2012 5:15 #9940

Some more info.

The same app running against the same datafile on an XP machine doesn't fail!

K

3 Apr 2012 5:18 #9941

It would be interesting to know which of the FTN77 storage allocation routines still work (in FTN95), and what their limitations are.

E

3 Apr 2012 6:17 #9942

Interestingly, I've got access to 'malloc' through a 3rd party library and that fails at exactly the same point!

K

3 Apr 2012 9:20 #9943

I've just tried it on a w7-32bit machine without error.

I'm going to defrag and reboot my 64 bit machine overnight just to be certain.

K

3 Apr 2012 10:02 #9944

Is it possible to use ALLOCATE. If you compile everything without /DEBUG or more checking, then you should get access to 4gb with ALLOCATE. I have not used GET_STORAGE@ to know if it gets access to above 2gb.

John

4 Apr 2012 6:31 #9946

GET_STORAGE@ and ALLOCATE (not /CHECKed) use the same storage allocation. Basically a call to GlobalAlloc.

4 Apr 2012 7:22 #9947

I have now defragged and rebooted and the problem remains under w7-64. It's OK under w7-32 and XP32. From browsing the web, it seems I'm not alone so I suspect this is a MS problem, rather than one that can be solved here!

I've seen a mention of a /LARGEADDRESSAWARE switch for C++ compilation and linking. Is there an equivalent in FTN?

K

5 Apr 2012 12:43 #9955

K,

I've been using ALLOCATE on Win7-64 and finding it's performance very robust. If you do not use /check or /debug, then you should get access to nearly 4gb of memory, which I have successfully used. I have experimented with mapping the available free memory to identify what should be possible. There appears to be a lot of memory defragmentation with SLINK. A good rule is to allocate the big arrays first, then the smaller ones can go in the remaining spaces. There is always a problem with stack conflict, which can be managed by minimising the use of large local arrays. I have not used get_storage@ as I am not familiar with a clean way of using the address. To use the address, do you transfer to a 'call interface_routine (core4(addr))' then use this array in the normal form or is there a simpler way to use the address ? Repeated use of CORE4 is very non-standard fortran.

John

5 Apr 2012 7:42 (Edited: 5 Apr 2012 8:12) #9956

Hi John,

This is 'old code' and the programmer responsible is semi-retired so changing it is not a trivial change but yes the returned address is passed around using core4/ccore as you said. We don't, in this particular case, have control over the order of the allocation as it's determined by the industry standard datafile structure. That datafile essentially says 'I've got xxxxMb of data coming up, I'll tell you how it's laid out later!'.

All the above doesn't alter the fact that it works perfectly well on 32-bit versions of Windows, but fails under w7-64 on a higher spec machine.

I don't know whether the failure occurs because of the particular size of the block being allocated or the total memory that has been requested to date.

K

edit: I tried, as a test, adding an ALLOCATABLE character buffer but when it tries to grab the 112972564 bytes, I get an 'Error 408: ALLOCATE was unable to obtain sufficient storage'. So I tried adding a 'first time only' ALLOCATE of the same size and that succeeded. So it's not the size of the buffer at the time that's the problem, it must be the total memory requested at that time. I'll do some more tests...

5 Apr 2012 7:54 #9957

/LARGEADDRESSAWARE will probably the same as what you get with the SLINK switch /3gb. For details search for 3gb in the current FTN95.chm.

5 Apr 2012 8:31 #9958

Thanks Paul! That seems to have fixed it!!!! 😃

K

edit: Interestingly, the app had only requested a total of 350Mb at the time it used to fail!

5 Apr 2012 11:37 #9959

K

Is allocation of storage in a loop? There may be missing RETURN_STORAGE@ calls. Failing for 350 mb does not appear to be the explaination, although it would depend on other storage demands.

John

6 Apr 2012 7:16 #9961

There might be some small amounts of 'leakage', (one can never be certain about these things!!!) but all the major amounts of memory are released once they're finished with.

K

8 Apr 2012 2:15 #9966

K,

Look at the load map (.map) and see how much memory is used by static allocation. In a recent post, I showed that different OS versions have different memory maps of free memory. This free memory is not contguous, so the mix of available array sizes differs between XP and Win7.

However, I would have expected Win7 to be better (??) With win7-64, you can get access to above 2gb, if you don't use any checking compilation options, this should help.

(Paul, can we get beta access to the /DEBUG fix?)

John

8 Apr 2012 6:04 #9967

At the moment we do not have a system for general beta access. This is mainly because work can be taking place on different parts of the compiler etc. which means that extra work is required on our part in order to make beta versions reasonably safe. From time to time we do make beta versions available to individuals who are willing to accept the risks involved and this does help us to reduce the number of regressions that can occur with our full releases.

9 Apr 2012 1:30 #9971

Quoted from JohnCampbell K,

Look at the load map (.map) and see how much memory is used by static allocation.... John

Sorry, where is this shown? In my .EXE, I get:

Executable section map:

Section   Base      Length    Flags   

.text     00401000  00001a56  60000020
.bss      00403000  00000054  c0000080
.comment  00404000  000000fa  00000a00
.data     00405000  00000398  c0000040
.rdata    00406000  00000038  40000040
.idata    00407000  0000090f  c0000040
.rsrc     00408000  000748f8  40000040
.CRT      0047d000  0000000c  c0000040
.salfmap  0047e000  00000800  40000040
.salfdbg  0047f000  00000408  42000040
.salfsys  00480000  00000018  c0000040
.salfvc   00481000  0000002c  c0000040
.reloc    00482000  00000000  42000040

And in my main DLL:

Section   Base      Length    Flags   

.text     10001000  00fdaa40  60000020
.bss      10fdc000  05e2e3cc  c0000080
.comment  16e0b000  0000313a  00000a00
.data     16e0f000  00609d2b  c0000040
.rdata    17419000  00000038  40000040
.idata    1741a000  0000847b  c0000040
.CRT      17423000  0000000c  c0000040
.edata    17424000  0002c763  40000040
.salfmap  17451000  00032258  40000040
.salfdbg  17484000  0003de98  42000040
.salfsys  174c2000  00000018  c0000040
.salfvc   174c3000  0000002c  c0000040
.reloc    174c4000  00000000  42000040

K

10 Apr 2012 7:38 #9974

K,

I am not familiar with using a DLL. However, I do have a subroutine which I use to map all available memory. You can call it from a location in your code to identify what memory is now available for ALLOCATE. Once you have called it, all memory is now locked so the program will not be able to proceed. It will give you some idea of what free memory is available if the routine had not been called. Again, I have not tested this when using DLL's.

I hope this may provide an indication of where your memory is being lost.

John

!     ******************************************************************
!     report_free_mem.f95
!     Copyright(c) JDC 2000
!
!     Created: 29/09/2010 9:47:18 AM
!     Author : JOHN CAMPBELL
!     Last change: JC 10/04/2012 5:29:02 PM
!     ******************************************************************
!
      call report_free_memory_chunks (0)
      end

      subroutine report_free_memory_chunks (ilu)
!
! routine to find the largest available ALLOCATABLE memory space now available
!
      integer*4  ilu  ! unit number to report results
!
      integer*4, parameter :: one_mb_i4 = 2**18 
      real*8,    parameter :: one_mb_r4 = 1024. * 1024. / 4.
      integer*8, parameter :: two       = 2
      integer*8, parameter :: four      = 4
!
!   Include a defined common for location
!z    integer*4, parameter :: i0 = 1500 * one_mb_i4     ! option to test large common
      integer*4, parameter :: i0 = 1 * one_mb_i4        ! small common option
      integer*4 a0(i0)
      common /aa/ a0
! 
      integer*4 i, lu, iostat, m_i4, k, mi, n
      integer*8 l, nblock, block_start(0:30), block_size(0:30)
      integer*8 m8, bytes
      real*8    mb, sec, sec_0
      real*4    block_sec(0:30)
!
      integer*4, allocatable, dimension(:) :: jj    ! allocatable array for test 1
      integer*4, pointer,     dimension(:) :: ii    ! array for test 2; repeat allocation
!
      integer*4 get_free_memory_size
      external  get_free_memory_size
!
      lu = ilu
      if (lu < 11) lu = 1
      if (lu > 1) open (lu, file='free_mem.log')
!
!  Test 1 : Allocate increasing size arrays to report the maximum array and
!           identify an error response
!
      write (lu,2000) 'TEST 1 : Testing for maximum free block size'
      mi = 100 * one_mb_i4
      do i = 1,32
         m_i4 = mi * i
         allocate (JJ(m_i4), stat=iostat)
         if (iostat == 0) then 
            call test_JJ (jj, m_i4, lu)
            deallocate (JJ) 
         else
            mb = m_i4 ; mb = mb / one_mb_r4
            write (lu,9001) ' Testing size =',m_i4, mb, iostat
            exit
         end if 
      end do 
 9001 format (a,b'zz,zzz,zzz,zz#',3x,b'zz,zz#.##',' mb : error code ',i0)
!
!  Test 2 : Now keep allocating the largest available memory block
!           until no more appears available (max of 28)
!
      call cpu_sec (sec_0)
      write (lu,2000) 'TEST 2 : Mapping free memory'
!
      n              = 0
      nblock         = 0
      block_start    = 0
      block_size     = 0
      block_sec      = 0
!
!   stats on common AA
!lf95 l              = pointer (a0)
      l              = loc (a0)
      write (lu,1002) 0, size(a0)*4, l
      n              = n+1
      block_start(n) = l                   ! common address
      block_size(n)  = size(a0)*4          ! size (bytes)
      call cpu_sec (sec) ; sec = sec - sec_0
      block_sec(n)   = sec
!
      n              = n+1
      block_start(n) = two**32             ! end of possible memory
      nblock         = n
10 Apr 2012 7:40 #9975

code continued ! do i = 1,28

         m_i4  = get_free_memory_size (i) 
! 
         do k = 0,m_i4                              ! this loop included for LF95, reducing
            allocate (ii(m_i4), stat=iostat) 
            if (iostat == 0) exit
            m_i4 = m_i4-1
            if (m_i4 < 4) exit
         end do
         if (k > 0) write (lu,*) 'Target array size reduced from', m_i4+k,' to',m_i4, iostat
!lf95           l = pointer (ii) 
         l     = loc (ii) 
!
         m8    = m_i4
         bytes = m8 * four 
         call cpu_sec (sec) ; sec = sec - sec_0
         write (lu,1002) i, bytes, l, iostat, sec
         if (iostat /= 0) exit 
         if (m_i4 < 4) exit
!
!      Insert block info into table
         do k = nblock,0,-1
            if (block_start(k) < l) then
               block_start(k+1) = l          ! start address
               block_size(k+1)  = bytes      ! memory size
               block_sec(k+1)   = sec
               nblock = nblock+1
               exit
            else
               block_start(k+1) = block_start(k)
               block_size(k+1)  = block_size(k)
               block_sec(k+1)   = block_sec(k)
            end if
         end do
      end do 
! 
 1002  format (' Array A',i2.2,' allocated as ',b'zz,zzz,zzz,zz#',' bytes, at address ',b'zz,zzz,zzz,zz#', i5, f10.4)
!
!  Now provide a map of free and reserved memory
!
      write (lu,2000) 'Free Memory Mapping'
      write (lu,2000) ' Blk       Lead Gap          Start     Size_bytes   Size_mb'
      do k = 1,nblock
         mb = block_size(k) / 1024. / 1024.
         write (lu,2002) k, block_start(k) - (block_start(k-1)+block_size(k-1)), block_start(k), block_size(k), mb, block_sec(k)
      end do
 2000 format (/1x,a)
 2002 format (i5,3(b'zzz,zzz,zzz,zz#'), f10.2, f10.4)
end 

      integer*4 function get_free_memory_size (i) 
!
!  This routinme searches for the largest available memory block still available
!  LF95 does not give non-zero STAT if array is > 2gb
! 
      integer*4 i, m_low, m_high, m, iostat 
      integer*4, allocatable, dimension(:) :: jj
!      real*4    mb
      integer*4, parameter :: m_high_start = 2**30 + 2**28    ! works as 2**29
      data  m_high / -1 /
! 
      if (m_high < 0) m_high = m_high_start
      m_low = 0 
! 
      do 
         m = m_low + (m_high-m_low)/2 
         if (m == m_low) exit 
         allocate (jj(m), stat=iostat)
!         mb = m ; mb = mb / one_mb_r4
!         write (*,2001)' testing', m, mb, iostat
!2001     format (a,b'zz,zzz,zzz,zz#',3x,b'zzz,zz#.##',' mb : error code ',i0)
         if (iostat /= 0) then 
            m_high = m 
         else 
            m_low  = m 
            deallocate (jj) 
         end if 
      end do 
! 
      get_free_memory_size = m 
! 
      end function get_free_memory_size

      subroutine test_JJ (jj, m, lu)
!
!  Tests array jj(m) exists by setting and checking values
!
      integer*4 m, jj(m), lu, i, k, er
      real*8    mb, sec, sec_0
      real*8,    parameter :: one_mb_r4 = 1024. * 1024. / 4.
      integer*8 byte_address
      data sec_0 / -1 /
!
      if (sec_0 < 0) call cpu_sec (sec_0)
      er = 0
      k  = m/2
      do i = 1,m
         k = k-1
         jj(i) = k
      end do
!  
      k  = m/2
      do i = 1,m
         k = k-1
         if (jj(i) /= k) er = er+1
      end do
!
10 Apr 2012 7:43 #9976

and again as submit limit exceeded !! ! mb = m ; mb = mb / one_mb_r4 !f95 byte_address = pointer(jj) byte_address = loc(jj) call cpu_sec (sec) write (lu,2001) ' Testing size =',m, mb, ' at address ', byte_address, ' error count = ',er, sec-sec_0 2001 format (a,b'zz,zzz,zzz,zz#',3x,b'zz,zz#.##',' mb',a,b'zz,zzz,zzz,zz#',a,i0,f8.2) sec_0 = sec end

      subroutine cpu_sec (sec)
      real*8 sec
      call use_QueryPerform (sec)
      end

      SUBROUTINE use_QueryPerform (time)
!
      real*8, intent (out) :: time
!
      STDCALL QUERYPERFORMANCEFREQUENCY 'QueryPerformanceFrequency' (REF):LOGICAL*4
      STDCALL QUERYPERFORMANCECOUNTER 'QueryPerformanceCounter' (REF):LOGICAL*4
!
      real*8 :: freq = -1.
      integer*8 num
      logical*4 ll
!
      if (freq < 0) then
         num  = 1
         ll   = QueryPerformanceFrequency (num)     ! cycle rate
         freq = 1.0d0 / dble (num)
      end if
!
      num  = 1
      ll   = QueryPerformanceCounter (num)          ! cycle count; assume ll is ok
!
      time = dble (num) * freq
!
      END SUBROUTINE use_QueryPerform

The free memory mapping reports the start and size of each free block of memory identified. The 'Lead Gap' is chunks of memory which are in use. 'Start' indicates when this used memory finished, which might give you some idea of what they are. I find that 30 blocks are enough, but this might change with your program.

John

Please login to reply.