A few bugs

Paul,

While testing ALLOCATE I have come up with a few small bugs.

1. When I converted the program I posted on 29 Sep to a subroutine, I found that the arrays I allocated via the pointer array ' integer*4, pointer, dimension(: ) :: ii ' ( which works by de-linking the allocated arrays) are not automatically de-allocated when exiting the subroutine. Is this a bug or is it caused by the non-standard use of the pointer re-allocation ? It would be good if all allocated arrays were released when exiting the subroutine.

2. The following program exhibits 2 problems with integer8 overflow; one with calculation of 2**32 and the other using I8 variable 'i8' as an array subscript. ! Program to test I8 problems in FTN95 ! integer8 block(0:3), i8 ! block = 0 i8 = 2 block(1) = 232 ! 232 is evaluated as 0 ; 2_432 may work block(i8) = i832 ! address i8 is evaluated as 0 write (,*) block ! should be 0 2^{32 2}32 0 end ! but is 2^32 0 0 0

block(i8) being interpreted as block(0), is a bit of a problem, which may be due to the array subscript being truncated to I*4.

3. I am using FILES8@ on XP-64 and it does not correctly size files bigger than 4gb. It also doesn't work on Win-32. It would appear that FILES8@ is an extension for correctly giving the size of files between 2gb and 4gb.

None of these are critical, but they are clear in identifying the problem.

John

John

232 will be evaluated as a default integer so you will need to use 2_432 for the correct result.

I can see other bugs that I need to work on.

Can you post a short program that illustrates the problem about automatic de-allocation. Anything that saves me time means more bug fixes for you guys.

Paul

Paul,

I have tidied up the memory scan program as a subroutine, which I put in a bigger program, but unfortunately after doing the scan, I could not do much more. The routine relies on the allocated arrays in the scan loop not being released, but the routine is only useful if all allocated arrays are released when the scan_free_memory subroutine finishes. I've posted the program in 2 parts. The Loop : 'Scan if free size changes' was required for other compilers, as available memory can be dynamic. It would be interesting to put an OPEN statement in the middle of the scan and see the change.

Where does ALLOCATE obtain the available memory list from ?

John

!     Last change:  JDC  14 Oct 2010   10:26 am
! program to find the largest available ALLOCATABLE memory space
!
      integer*4, parameter :: one_mb = 2**18    ! mb bytes for I*4 array
      integer*4, parameter :: i0 = 100 * one_mb
      integer*4 a0(i0) 
      common /aa/ a0 
!
      open (unit=98, file='scan_memory.log')
!
      call scan_free_memory (a0,i0)
      call scan_free_memory (a0,i0)
!
      end

      integer*4 function get_free_memory_size (n)
!
!   Routine to find largest available allocatable array; n is number of tests
!
      integer*4  n, ml, m, iostat
      integer*4, save :: mh = 2**30 + 2**28    ! works as 2**29 - now try a bigger number
      integer*4, allocatable, dimension(:) :: ii
!      real*4    mb
!
      ml = 0
      if (n < 0) then
         mh = 2**30
         write (*,fmt='(/a,i0)') ' Resetting scan size at ',mh
      end if
!
      do n = 1,huge(n)
         m = ml + (mh-ml)/2
         if (m == ml) exit
         if (m < 1) exit
         allocate (ii(m), stat=iostat)
!         mb = m ; mb = mb *4./1024./1024.
!         write (*,*)'testing', m, mb, iostat
         if (iostat /= 0) then
            mh = m
         else
            ml = m
            deallocate (ii)
         end if
      end do
!
      get_free_memory_size = m
!
      end function get_free_memory_size

      subroutine report_VC (lu)
!
      INTEGER*4 lu, BASE,SIZE,COMMIT,AMT_COMMIT
      real*8, parameter :: mb = 1024.*1024.
!
      write (lu,2000) ' ', 'Virtual Common Usage'
2000  format (a)
      CALL GET_VIRTUAL_COMMON_INFO@ ('free_mema.exe', BASE, SIZE, COMMIT, AMT_COMMIT)
      write (lu,2001) ' BASE       =', BASE,   dble (base) / mb
      write (lu,2001) ' SIZE       =', SIZE,   dble (size) / mb
      write (lu,2001) ' COMMIT     =', COMMIT, dble (commit) / mb
      write (lu,2001) ' AMT_COMMIT =', AMT_COMMIT
2001  format (a,b'zz,zzz,zzz,zz#',f9.3,' mb')
      END

The scan_free_memory subroutine

      subroutine scan_free_memory (a0,i0)
!
!  Routine to find all unused blocks in memory, available to ALLOCATE
!
      integer*4 i0, a0(i0)
!
      integer*4, pointer, dimension(:) :: ii
      integer*4 i,iostat, m, k, n, lu
      integer*8 l, nblock, block_start(0:40), block_size(0:40)
      character block_name(0:40)*10
      real*8    mb
      integer*4 get_free_memory_size
      external  get_free_memory_size
!
      nblock         = 2
      block_start    = 0
      block_size     = 0
      block_name     = ' '
!
      block_start(1) = loc (A0)
      block_size(1)  = size (a0)*4
      block_name(1)  = 'Common_A0'
!
      block_start(2) = 2_4**32
      block_name(2)  = '.end.'
!
      do i = 1,2
         if (i==1) lu = 1
         if (i==2) lu = 98
         write (lu,2000) ' Initial settings for Memory Scan'
         do k = 0,nblock
            write (lu,2001) k, block_start(k), block_size(k),block_name(k)
         end do
      end do
!
      n = -1
      do i = 1,38
!
!      Get largest free space
         m = get_free_memory_size (n)
          if (m < 1) exit
!
!      Scan if free size changes
         do k = 0,m
            allocate (ii(m),stat=iostat)
            if (iostat == 0) exit
            m = m-1
            if (m < 1) exit
         end do
!
         l = loc(ii)
         write (  *,1002) i, m*4, l, iostat, n, k
         write ( 98,1002) i, m*4, l, iostat, n, k
         if (iostat /= 0) exit
         if (m < 1) exit
!
         do k = nblock,0,-1
            if (block_start(k) < l) then
               block_start(k+1) = l
               block_size(k+1)  = m*4
               write (block_name(k+1),'(a,i0)') 'Free_',i
               nblock = nblock+1
               exit
            else
               block_start(k+1) = block_start(k)
               block_size(k+1)  = block_size(k)
               block_name(k+1)  = block_name(k)
            end if
         end do
      end do
!
      do i = 1,2
        if (i==1) lu = 1
        if (i==2) lu = 98
        write (lu,2000) ' Blk       Lead Gap          Start           Size      mb'
        do k = 1,nblock
           mb = block_size(k) / 1024. / 1024.
           write (lu,2002) k, block_start(k) - (block_start(k-1)+block_size(k-1)),   &
                              block_start(k),        &
                              block_size(k),         &
                              mb,                    &
                              block_name(k)
        end do
        call report_VC (lu)
      end do
!
1002  format ('Array A',i0,' allocated as ',b'zz,zzz,zzz,zz#',' bytes, at address ',b'zz,zzz,zzz,zz#', 3i5)
2000  format (/a)
2001  format (i4,2i11,2x,a)
2002  format (i5,3(b'zzz,zzz,zzz,zz#'), f8.2, 2x,a)
!
end

John

I have answered the question about ALLOCATE elsewhere on this forum quite recently. From memory I think I said that, unless you are using /CHECK, ALLOCATE leads to a call to something like GlobalAlloc. So the memory alignment and allocation is handled by Microsoft.

Your program is way too big for me to debug. I was looking for a few lines of code that illustrates the ALLOCATE problem that you mentioned.

Paul,

Thanks, for the info on GlobalAlloc. I will look it up on MSDN.

As for the program I sent, don't be worried about the size, as it is very straight foward. The function get_free_memory_size uses ALLOCATE and the STAT= return to find the largest available memory for a new array. The routine Scan_Free_Memory does up to 38 calls to this function then allocates the located memory size to the same pointer array 'II' Each time that II is allocated to a new memory area, the previous allocation is not de-allocated but left as unavailable to ALLOCATE. This action is required for the program to work. This is the basis of a program you sent me some time ago (in 2008?). The routine Scan_Free_Memory also sorts these allocated areas and reports them in memory order, noteing the size of the gap between allocated arrays, assumed to be memory not available to ALLOCATE. It is an interesting map of available memory, showing the size of arrays that could be allocated and the location of what I assume is 'reserved' memory.

The problem I reported yesterday is, when I exit Scan_Free_Memory, the (up to) 38 arrays that were created by ALLOCATE are not released. The Fortran 95 standard says this should happen automatically. I do not use a DEALLOCATE statement, as their identification is lost when I re-use the pointer array to point to the next ALLOCATE. I must admit I have limited experience of pointers. Fixing this problem would depend on how FTN95 keeps track of all arrays that have been allocated in the routine, to be automatically de-allocated.

By the way, all the work I have done on trying to use more memory in my program and reduce the amount of disk I/O, has been with the aim of using a 64-bit address space. This has included improving the reporting of run time performance and disk I/O usage. However performance testing I have done on the 64-bit OS, with only a 32-bit .exe is showing far better run time improvement that any of the gains I have made with changes to the program algorithms. Statistics now show disk I/O elapsed times are minimal. The bottom line is that the 64-bit OS (with 6gb of physical memory) has much better performance for my problems that use about 2gb to 7gb of 'effective' memory, using my 32-bit program. I think it is the improved disk I/O buffering available with 6gb of physical memory. More physical memory would probably be better for larger problems.

On the basis of this, if your program uses disk files of say 2 to 10 gb in size, I would certainly recommend switching to 64-bit OS, before contemplating moving to a 64-bit compiler, as there are significant performance gains available in the short term.

John

Paul,

A smaller example from the previous program

! program to count available memory blocks for ALLOCATABLE
!
      call test_blocks (100, ' 100mb Blocks are available')
      call test_blocks (100, ' shows no blocks released')
      call test_blocks ( 20, ' but smaller blocks available')
      call test_blocks ( 20, ' but again not released')
      end

      subroutine test_blocks (mb, desc)
!
      character  desc*(*)
      integer*4  mb, m, n, mstat
      integer*4, pointer, dimension(:) :: ii
      integer*4, parameter :: one_mb_I4 = 2**18
!
      m = mb * one_mb_I4
      do n = 0,m
         allocate (ii(m),stat=mstat)
         if (mstat /= 0) exit
      end do
      write (*,1001) n, mb, desc
1001  format (i5,' blocks of ',i0,' mb allocated : ',a)
      END

Quoted from JohnCampbell Paul, 3) I am using FILES8@ on XP-64 and it does not correctly size files bigger than 4gb. It also doesn't work on Win-32. It would appear that FILES8@ is an extension for correctly giving the size of files between 2gb and 4gb.

This has been fixed and will now return the correct size for files greater than 4GB

I have now fixed the INTEGER*8 bug for the next release.

Thank you for your program illustrating the behaviour of ALLOCATE for local arrays. As I recall the Fortran 95 standard provided for automatic deallocation of local arrays whilst this was not a feature of Fortran 90.

It seems that this has not been implemented in FTN95, at least not in this context. So for the time being it is necessary to include an explicit DEALLOCATE statement in your code.

In your code, you over write the address of the allocated memory with each cycle of the do loop so (even when FTN95 is fixed) the compiler will only be able to deallocate the memory for the last cycle. In other words the programmer will still need to use the standard intrinsic ALLOCATED if there is any doubt.

I will think about this but presumably the compiler is not expected to generate a runtime fault when memory is ALLOCATEd more than once to the same array without deallocation.

On further testing I now find that you do get automatic deallocation if you use the ALLOCATABLE rather than the POINTER attribute on the array.

This complies with the standard which does not require automatic deallocation in the case of the POINTER attribute (probably with good reason).