Fails to save arrays > 4GB

There exist two major methods to save arrays. Suppose we have 2D REAL*4 array Arr4(nA,nB). First method is when we explicitly save each element or each dimension individually

    open (11, file='LargeFile.dat', FORM='UNFORMATTED')
    do i=1,nB
      write(11) Arr4(:,i)
    enddo
    close(11)

and the second method is when we save the entire array as a whole

    open (11, file='LargeFile.dat', FORM='UNFORMATTED')
    write(11) Arr4
    close(11)

.

Second method can be an order of magnitude faster reaching speeds 3GB per second on current hardware (with m2 NVMe PCIe 4.0) drives. But it fails to save, flashing the Silverfrost exception, if array is larger than ~4GB while first slow method works with no problems. Here is demo which allocates OK any size and saves files using second method when file size is 4GB (array size nB =1e8) but fails to save with nB=2e8 and file size twice larger. It fails also to save even REAL*8 arrays. Compilation with /64 switch.

! compilation: ftn95 aaa.f90 /link /64 >z
! 
    real*4, dimension(:,:), allocatable :: Arr4

    nA = 11
    nB = 2.e8

!...Allocating array

    Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
    allocate ( Arr4 (nA, nB), stat = ierr) 

    if (ierr.eq.0) then
       Print*,'Allocation success'
    else
       Pause 'Fail to allocate'
       goto 1000
    endif  

!...Filling the array with some data
    do i=1,nB
      Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
    enddo

!   __  __  ____  ____  _   _  _____  ____      __ 
!  (  \/  )( ___)(_  _)( )_( )(  _  )(  _ \    /  )
!   )    (  )__)   )(   ) _ (  )(_)(  )(_) )    )( 
!  (_/\/\_)(____) (__) (_) (_)(_____)(____/    (__)
!....................................................

    Print*,'Trying to save the data Method 1 '
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
    do i=1,nB
      write(11) Arr4(:,i)
    enddo
    close(11)	
    call cpu_time(t1)  

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,' Speed of write Method 1 =', SpeedGBps   !   typically  ~0.5 GB/s 

!   __  __  ____  ____  _   _  _____  ____     ___  
!  (  \/  )( ___)(_  _)( )_( )(  _  )(  _ \   (__ \ 
!   )    (  )__)   )(   ) _ (  )(_)(  )(_) )   / _/ 
!  (_/\/\_)(____) (__) (_) (_)(_____)(____/   (____)
!....................................................
    Print*,'Trying to save the data Method 2'
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
    write(11) Arr4
    close(11)	
    call cpu_time(t1)  

!...Speeed of writing Method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,' Speed of write  Method 2=', SpeedGBps   !   typically  ~2.6 GB/s 

      pause 'File LargeFile.dat created OK'
      goto 1000

!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
    goto 1000
910 Print*,'Can not save file LargeFile.dat'


1000 Continue
    End

Dan

Thanks for the feedback. I have made a note that this needs investigating.

my understanding of unformatted binary records is that FTN95 does not support records longer than about 2^31 bytes ( ie the header is byte 1 = -1 and bytes 2:5 must be a positive 4-byte integer )

Both ifort and gfortran support a complex record header/footer structure of sub records, which I recall are 2^31-9 bytes in size. the header/footer is 4-bytes long. The sub records are identified by -ve sub record sizes, enclosed by +ve sub record size for first and last sub records.

Does FTN95 Stream I/O support integer*8 addressing, as that could be the simplest to implement ?

Alternatively, write the array as a group of records which are less than 2^{31 bytes long, choosing 'Arr4(:,i)' so that it is < 2}31 bytes. This will be the most practical and 'portable' solution. Stream I/O is portable between FTN95 and ifort/gfortran, as is fixed length direct access files ( again with records smaller than 2^31 bytes ) ( FTN95 unformatted records are not portable )

John

Can you explain what you mean by 'Stream I/O integer*8 addressing'?

For 64 bit FTN95 the I/O library is in clearwin64.dll which is 64 bits.

Quoted from PaulLaidler Can you explain what you mean by 'Stream I/O integer*8 addressing'?

To write a record longer than 2^{31 for stream I/O, I would expect that stream I/O must support 64 bit integer addressing. Perhaps for sequential access, writing a 'record' larger than 2}31 bytes may be ok ?

As there is no record structure, the following would not be any different on the file to write(11) Arr4. do i=1,nB write(11) Arr4(:,i) enddo

John

Stream I/O uses a nominal record length with value one. This could be compiler dependent but probably isn't.

I am assuming that the size of a file will not be a problem (for 64 bit FTN95).

Yes, ideally files created with any of these two methods have to be readable by each of these methods. But if this is not the case among the whole line of 12-13 Fortran compilers then it's OK not to be method-independent.

Currently if you create the file with Method1 it only can be read back with the same Method1, and Method2 - with Method2. Not super bad inconvenience, it is possible to make a note in the file which method was used and then read the file.

Dan,

Part of the problem is not clearly understanding the limitations of each binary file type/format ( unformatted sequential, direct access fixed length records or now, stream access)

Paul,

I have been successfully using Fotran direct access binary files (fixed length records) for many years as a portable interface between different Fortran compilers.

Stream I/O now provides a similar portable access by,

1. allowing I/O lists of intrinsic types ( I/O lists are much more flexible than a single vector )
2. able to create header and footer labels to records, to be compatible with other compiler's unformatted sequential binary file formats.
3. provides random/direct addressing to files ( although I have had some problems with integer*8 addressing up to FTN95 Ver 8.8, which may now be fixed. I should find my past examples and retest with Ver 8.9 )

It would be good if the following could be investigated and possibly supported for unformatted sequential files:

1. an alternative could be provided to support other compilers unformatted binary file formats, via a compile option or in the OPEN statement.
2. confirm if records or files larger than 2^31 bytes or 'words' could be supported.
3. investigate support for derived type records to be used in I/O lists, rather than specifying as a list of intrinsic type components.

However, stream I/O now provides a very flexible and portable alternative to earlier unformatted file approaches.

John

Thanks for the feedback. Regarding your request for further investigations:

1. I don't see how we could offer to do this even for just one other compiler.

2. Do you have any experience that this does not work?

3. This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.

Paul,

1. I don't see how we could offer to do this even for just one other compiler.

Answer: addressing only records smaller than 2gb; both ifort and gfortran use a 4-byte header/footer for unformatted sequential files. They are portable for this file type. If FTN95 could have a (say) /4_byte_header option, (unlike the present 1-byte or 5-byte default format), this would improve portability for gfortran and ifort interface.

2. Do you have any experience that this does not work?

Answer: I provided a stream I/O example in: https://forums.silverfrost.com/Forum/Topic/4081&highlight=stream

I have now updated this program to better document the test.

https://www.dropbox.com/s/r6cy1xak6hp2msa/stream_read2.f90?dl=0 https://www.dropbox.com/s/wnj6pu7cq2vxapw/stream_read.log?dl=0

It works with gfortran (in Plato) but FTN95 does not support 'write ( unit=stream_unit, pos=address ) values' or 'read ( unit=stream_unit, pos=address ) one_byte' the value 'address' does not work in FTN95 and 'INQUIRE ( unit, pos=address )' does not appear to give the same value as gfortran.

The file stream_read.log now provides a report of the tests for gfortran Ver 11.1 and FTN95 Ver 8.80

3. This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.

integer*8 address will then need to be checked after b) is addressed.

I will test the program with integer*8 address and check it still works.

( I think I have provided examples of this before, but we first need 'pos=' to be supported in FTN95 )

I will download Ver 8.90 and test.

A further update to stream_read3.f90

this reports the address after writing the record. It shows that for:

         write ( unit=stream_unit, pos=address ) values
         inquire ( unit=stream_unit, pos = end_pos ) 

pos=address is positioning to the byte before 'address' (based on gfortran response)

https://www.dropbox.com/s/1wdfnbn9cmy3bix/stream_read3.f90?dl=0

https://www.dropbox.com/s/nh3gczdfup106pn/stream_read3.log?dl=0

further change for integer*8 :: address (etc) This shows that pos=address does not work with FTN95 Ver 8.80 I will download Ver 8.90 tomorrow !!

https://www.dropbox.com/s/g09v7rd81a7aflf/stream_read8.f90?dl=0

John

I will make a note of this and see what can be done.

The original failure on this thread (reported by DanRRight) has now been fixed for the next release of clearwin64.dll.

The change has an impact on FTN95 so it is possible that users may also need the next release of FTN95 (the one after v8.91). In other words, it is unlikely that you will also need a new FTN95.

Paul,

Can you please provide some clarification for what error was fixed ?

Did it relate to the following code where an array larger than 2GBytes is written as a sequential unformatted file { OPEN(access='sequential', FORM='UNFORMATTED',...) } open (11, file='LargeFile.dat', FORM='UNFORMATTED') write(11) Arr4 close(11)

If this is the case, could you please clarify what header/footer structure has been proposed.

Also, there may be some confusion regarding which of my 'c)' requests are to be addressed. My preference would be to consider the 'c)' related to integer*8 addressing for 'pos=address' with stream I/O.

Stream I/O can offer much potential for a simpler solution to unformatted I/O, especially for larger records, although designing a record structure using data packets smaller than 2 GBytes might be a more robust solution with minimal performance penalty.

( the other 'c)' issue of derived type I/O support, as part of F08(?) is a much more complex beast, that I find difficult to understand. I think many of us would be satisfied with writing out derived type components, which is supported in FTN95 ? )

John

As I understand it, there is no header or footer for stream I/O.

The aim is to look at all the issues that you have raised to see what is possible within our resources. But it is on a list of things to do.

John

At the moment 'pos = end_pos' only works for 32 bit end_pos.

I will aim to fix this and then upload a new set of DLLs. We can then consider other issues after you have tested this fix.

John

I have sent you a pm with links to a trial fix.

Thanks, Paul.

The following is my latest stream IO test program that demonstrates problems with stream I/O in FTN95 when overwriting files. Others may confirm my tests or identify problems with my approach ?

This is a program that can be run in PLATO, using either FTN95 /64 or gfortran

https://www.dropbox.com/s/cxe6wr8vdd3vl6n/stream_readc.f90?dl=0

I am using gfortran as a reference, as it appears to perform correctly. The comments in the program describe 3 tests in more detail, that produce 3 files which should all be the same.

FTN95 fails in tests 2 and 3 for WRITE ( pos=address ) when rewriting and for INQUIRE

Please build using ftn95 /64 stream_readc.f90 /link then run.

You can compare the 3 files from the 3 tests. stream_filec.bin stream_filez.bin stream_file$.bin ( you can delete these files before the test, but is not necessary to confirm errors.)

They should all be the same, but are not with FTN95

file stream_readc.log is appended with results of each test run to confirm errors.

John, Do your tests need new dll with latest fix of 4GB issue? And what are speeds of READ / WRITE ( pos=address ) in comparison with Method2 above ? I suspect it might not be worth to use it anymore