Use of Virtual Common

Paul and John,

I am trying to understand the benefits of using Virtual Common. The documentation states that if VC is used in SLINK, then the '.bss' section is removed and global data is allocated to virtual memory at runtime. There are two main sorts of memory, addressable and physical. Does the virtual memory discussed have an address at start/run time or allocated as accessed? Is it that only physical memory allocation is delayed to as accessed? My understanding is that VC starts from address 1gb. Potentially this limits the size of available addressed common to only 1gb, as I do not appear to be able to start a program where common extends beyond 2gb. Is this the case ? If so this would imply that VC is suited to programs that use only about 0.8gb of common. (beyond 2gb appears to be only accessible by ALLOCATE arrays with /3gb or win-64) Is it a requirement that all declared common must be less than 1gb (from 1gb to 2gb), ie the (virtual) memory address of all common is set at run time, or is memory address of common only allocated when/as it is used ? Without the use of VC I can allocate COMMON arrays of a total size of about 1.6gb. I have not been able to do that with VC. Does the comment 'The result is that pages of memory (4Kb each) are allocated from the system on demand.' refer to allocation of physical memory, or is it a more flexible interpretation for mapping of virtual memory? ie, if I have multiple named common, are they allocated a memory address when they are accessed or at run/start time? Potentially if when accessed, then I may have named COMMON defined only in program modules not accessed, not taking addressable memory. If not, then potentially the main benefit of VC is only to reduce the size of the .exe file and the memory demand at start run time. I'm not sure if it provides any way of overlaying common usage that was available in Win-16, which would provide for increased available virtual memory.

John

As a quick response to your question, have you tried experimenting whilst calling get_virual_common_info@. This may help you to understand where the addresses are coming from and how much is being allocated. LOC may also be useful in this context.

There is no overlaying of vitrual memory and this is paged into physical memory as and when it is accessed.

Virtual Common's main benefit is in being able to set aside a large area of memory and only have it allocated as it is used. So, for example, a 500MB array may be placed in virtual common and if only the first 1MB of it was used the other 499MB would never be committed leading to much lower memory use. However, in the present day where machines regularly have 4GB of RAM or more VC is probably obsolete and possibly even gets in the way.

The main resource now is not physical RAM but virtual memory space. VC if anything breaks up the virtual memory space because it is pre-allocated to a specific location.

I have further investigated the available memory size and done some testing of VC I have used the program below: free_mem.f95

On the basis of my tests:- VC provides for a maximum common size of about 900mb in Win-32 This reduces to about 600mb in Win-64 Without VC I can use common of about 1500mb The program then locates all available space for ALLOCATE arrays. On Win-64, the maximum space found is about 2gb, which appears to be a good improvement. The program is small and makes little use of the stack, so I don't know how this will translate to a large code.

On the basis of this, I am not identifying any VC advantages others have reported. I'd be pleased to be shown where I am going wrong.

To test VC I used the commands ftn95 free_mem slink free_mem.obj /VC

Without VC, I used the commands ftn95 free_mem /lgo

Robert, thanks for your comments. I agree that VC being fixed at address 500mb does appear to break up memory.

Listing of free_mem.f95
! program to find the largest available ALLOCATABLE memory space ! integer4, parameter :: one_mb = 2**18 integer4, parameter :: i0 = 1500 * one_mb ! this size works without VC ! integer4, parameter :: i0 = 900 * one_mb ! this size appears to be the limit with VC:win32 ! integer4, parameter :: i0 = 600 * one_mb ! this size appears to be the limit with VC:win64 integer4 a0(i0) common /aa/ a0 ! integer4, pointer, dimension(:) :: ii integer4 i,iostat, m, k, n integer4 get_free_memory_size external get_free_memory_size integer8 l, nblock, block_start(0:40), block_size(0:40) real8 mb ! nblock = 0 block_start = 0 block_size = 0 ! nblock = 2 block_start(1) = loc(a0) ! COMMON array block_size(1) = size(a0)4 block_start(2) = nblock32 ! block_start(nblock) = nblock32 ! this is an I8 addressing error ! write (,) ' Initial settings' l = loc(a0) write (,1002) 0,size(a0)4,l do k = 0,nblock write (,) k,block_size(k),block_start(k) end do ! 1002 format ('Array A',i0,' allocated as ',b'zz,zzz,zzz,zz#',' bytes, at address ',b'zz,zzz,zzz,zz#', 2i5) ! do i = 1,38

         m = get_free_memory_size (n)
!
         do
            allocate (ii(m),stat=iostat)
            if (iostat==0) exit
            m = m-1
            if (m < 1) exit
         end do
!
         l = loc(ii)
         write (*,1002) i, m*4, l, iostat, n
         if (iostat /= 0) exit
         if (m < 1) exit
!
         do k = nblock,0,-1
            if (block_start(k) < l) then
               block_start(k+1) = l
               block_size(k+1)  = m*4
               nblock = nblock+1
               exit
            else
               block_start(k+1) = block_start(k)
               block_size(k+1)  = block_size(k)
            end if
         end do
!         do k = 0,nblock
!            write (*,*)i,k,block_start(k),block_size(k)
!         end do
      end do
!
      write (*,*) ' Blk      Lead Gap         Start          Size      mb'
      do k = 1,nblock
         mb = block_size(k) / 1024. / 1024.
         write (*,2002) k, block_start(k) - (block_start(k-1)+block_size(k-1)), block_start(k), block_size(k), mb
      end do
2002  format (i5,3(b'zz,zzz,zzz,zz#'), f8.2)
!
      call report_VC
!
end

shame about the size limit !! the rest of program

      integer*4 function get_free_memory_size (n)
!
!   Routine to find largest available allocatable array; n is number of tests
!
      integer*4  n, ml, m, iostat
      integer*4, save :: mh = 2**30 + 2**28    ! works as 2**29 - now try a bigger number
      integer*4, allocatable, dimension(:) :: ii
!      real*4    mb
!
      ml = 0
!
      do n = 1,huge(n)
         m = ml + (mh-ml)/2
         if (m == ml) exit
         if (m ==  0) exit
         allocate (ii(m), stat=iostat)
!         mb = m ; mb = mb *4./1024./1024.
!         write (*,*)'testing', m, mb, iostat
         if (iostat /= 0) then
            mh = m
         else
            ml = m
            deallocate (ii)
         end if
      end do
!
      get_free_memory_size = m
!
      end function get_free_memory_size

      subroutine report_VC
!
!      INTEGER (KIND=3)::BASE,SIZE,COMMIT,AMT_COMMIT
      INTEGER*4 BASE,SIZE,COMMIT,AMT_COMMIT
!
      write (*,2000) ' ', 'Virtual Common Usage'
2000  format (a)
      CALL GET_VIRTUAL_COMMON_INFO@ ('free_mem.exe', BASE, SIZE, COMMIT, AMT_COMMIT)
      write (*,2001) ' BASE       =', BASE,   dble(base)/1024./1024.
      write (*,2001) ' SIZE       =', SIZE,   dble(size)/1024./1024.
      write (*,2001) ' COMMIT     =', COMMIT, dble(commit)/1024./1024.
      write (*,2001) ' AMT_COMMIT =', AMT_COMMIT
2001  format (a,b'zz,zzz,zzz,zz#',f9.3,' mb')
      END

John Appleyard wrote very clearly about VC some years ago:

'What this means is that you declare an array of any size you like (or indeed you don't specify a size - you just say it's of indeterminate size) and the system allocates real (or virtual) memory to it in pages, but only at the last possible moment, when you actually do something with an element. If you never access elements 1-1000, then nothing is allocated for them. If the only element you ever access is number -123456, then a single page will be allocated for that element and its neighbours.

I remember demonstrating a striking example of this - a program with a sparsely accessed 1 Gigabyte array running on a 386SX with 4 MBytes RAM and a 40 MBytes hard disk.

As computers approached the limits of 32 bit addressing, the usefulness of this feature declined somewhat, as it was easy enough to fill a 32 bit address space with real or virtual memory. However now that 64 bit is becoming mainstream, I look forward to the return of lazy allocation

• and one of the benefits is arrays that can grow dynamically without copying'

That's exactly what i've expected this compiler will be having first in 64bits. But alas yet...

Dan,

Where have you taken this quote from ?

Certainly the concept of VC is attractive, and will be more so with 64-bit, for as the virtual (addressable size) of the program increases, the last thing you would want is for all the memory image to be allocated to physical memory and then paged out to the paging file. My recent work at trying to improve the performance of my programs has been to reduce the disk I/O that is taking place, both explicit in my program and also associated with pagefile.sys. I'm achieving this by placing medium sized allocated arrays in a Key_Variables module, and limiting the few large ones that I dump to disk. Previously, when there was much less physical memory, most information was stored on disk and now as memory size and processing speed has increased, I've lost the tolerance for larger problems that sit there while the disk is transferring gb's of data. Those problems that ran overnight are now expected to run like a screen update on the internet !!

John

My demand is just simple, even primitive as that: I NEED 64bit, i need MORE ADDREESSABLE MEMORY to the extent of switching compiler. Soon not just desktops, the laptops and even cell phones will be out of 2-4 GB roof as they will be powerful gaming devices...Progress is damn quick. 32 years ago Cray1 had the same Linpack score like my current cellphone ~7 MFlops (same phone on latest Froyo OS gets 20+). Why that was easy to developers do that 25 years ago when they switched from 16bit is beyond me. Hello devs, hello Dave Bailey where you've been last decade?

Dan,

Although I have a 64-bit OS, I don't yet have a 64-bit compiler (will soon). I will be interested to see a large memory program run. One of the problems I anticipate is when running a large memory progam, is if it has to be paged out to pagefile.sys then there could be a significant performance penalty writing the virtual image of the run memory to disk. I think we will need to ensure there is sufficient physical memory for the program and the OS so there is minimal pageing.

John