soccer jersey forums.silverfrost.com :: View topic - Fails to save arrays > 4GB
forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Fails to save arrays > 4GB
Goto page 1, 2, 3, 4, 5, 6, 7  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2866
Location: South Pole, Antarctica

PostPosted: Mon Sep 26, 2022 2:13 am    Post subject: Fails to save arrays > 4GB Reply with quote

There exist two major methods to save arrays. Suppose we have 2D REAL*4 array Arr4(nA,nB). First method is when we explicitly save each element or each dimension individually
Code:
    open (11, file='LargeFile.dat', FORM='UNFORMATTED')
    do i=1,nB
      write(11) Arr4(:,i)
    enddo
    close(11)

and the second method is when we save the entire array as a whole
Code:
    open (11, file='LargeFile.dat', FORM='UNFORMATTED')
    write(11) Arr4
    close(11)
.

Second method can be an order of magnitude faster reaching speeds 3GB per second on current hardware (with m2 NVMe PCIe 4.0) drives. But it fails to save, flashing the Silverfrost exception, if array is larger than ~4GB while first slow method works with no problems.
Here is demo which allocates OK any size and saves files using second method when file size is 4GB (array size nB =1e8) but fails to save with nB=2e8 and file size twice larger. It fails also to save even REAL*8 arrays. Compilation with /64 switch.
Code:
! compilation: ftn95 aaa.f90 /link /64 >z
!
    real*4, dimension(:,:), allocatable :: Arr4

    nA = 11
    nB = 2.e8

!...Allocating array

    Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
    allocate ( Arr4 (nA, nB), stat = ierr)

    if (ierr.eq.0) then
       Print*,'Allocation success'
    else
       Pause 'Fail to allocate'
       goto 1000
    endif 

!...Filling the array with some data
    do i=1,nB
      Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
    enddo

!   __  __  ____  ____  _   _  _____  ____      __
!  (  \/  )( ___)(_  _)( )_( )(  _  )(  _ \    /  )
!   )    (  )__)   )(   ) _ (  )(_)(  )(_) )    )(
!  (_/\/\_)(____) (__) (_) (_)(_____)(____/    (__)
!....................................................

    Print*,'Trying to save the data Method 1 '
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    do i=1,nB
      write(11) Arr4(:,i)
    enddo
    close(11)   
    call cpu_time(t1) 

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,' Speed of write Method 1 =', SpeedGBps   !   typically  ~0.5 GB/s

!   __  __  ____  ____  _   _  _____  ____     ___ 
!  (  \/  )( ___)(_  _)( )_( )(  _  )(  _ \   (__ \
!   )    (  )__)   )(   ) _ (  )(_)(  )(_) )   / _/
!  (_/\/\_)(____) (__) (_) (_)(_____)(____/   (____)
!....................................................
    Print*,'Trying to save the data Method 2'
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    write(11) Arr4
    close(11)   
    call cpu_time(t1) 

!...Speeed of writing Method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,' Speed of write  Method 2=', SpeedGBps   !   typically  ~2.6 GB/s

      pause 'File LargeFile.dat created OK'
      goto 1000

!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
    goto 1000
910 Print*,'Can not save file LargeFile.dat'


1000 Continue
    End


Last edited by DanRRight on Mon Sep 26, 2022 11:11 am; edited 5 times in total
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Mon Sep 26, 2022 7:44 am    Post subject: Reply with quote

Dan

Thanks for the feedback. I have made a note that this needs investigating.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Tue Sep 27, 2022 7:27 am    Post subject: Reply with quote

my understanding of unformatted binary records is that FTN95 does not support records longer than about 2^31 bytes ( ie the header is byte 1 = -1 and bytes 2:5 must be a positive 4-byte integer )

Both ifort and gfortran support a complex record header/footer structure of sub records, which I recall are 2^31-9 bytes in size. the header/footer is 4-bytes long. The sub records are identified by -ve sub record sizes, enclosed by +ve sub record size for first and last sub records.

Does FTN95 Stream I/O support integer*8 addressing, as that could be the simplest to implement ?

Alternatively, write the array as a group of records which are less than 2^31 bytes long, choosing "Arr4(:,i)" so that it is < 2^31 bytes.
This will be the most practical and "portable" solution.
Stream I/O is portable between FTN95 and ifort/gfortran, as is fixed length direct access files ( again with records smaller than 2^31 bytes )
( FTN95 unformatted records are not portable )
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Tue Sep 27, 2022 9:24 am    Post subject: Reply with quote

John

Can you explain what you mean by "Stream I/O integer*8 addressing"?

For 64 bit FTN95 the I/O library is in clearwin64.dll which is 64 bits.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Tue Sep 27, 2022 2:10 pm    Post subject: Re: Reply with quote

PaulLaidler wrote:
Can you explain what you mean by "Stream I/O integer*8 addressing"?


To write a record longer than 2^31 for stream I/O, I would expect that stream I/O must support 64 bit integer addressing.
Perhaps for sequential access, writing a "record" larger than 2^31 bytes may be ok ?

As there is no record structure, the following would not be any different on the file to write(11) Arr4.
Code:
    do i=1,nB
      write(11) Arr4(:,i)
    enddo
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Wed Sep 28, 2022 7:07 am    Post subject: Reply with quote

John

Stream I/O uses a nominal record length with value one. This could be compiler dependent but probably isn't.

I am assuming that the size of a file will not be a problem (for 64 bit FTN95).
Back to top
View user's profile Send private message AIM Address
DanRRight



Joined: 10 Mar 2008
Posts: 2866
Location: South Pole, Antarctica

PostPosted: Fri Sep 30, 2022 9:39 pm    Post subject: Reply with quote

Yes, ideally files created with any of these two methods have to be readable by each of these methods. But if this is not the case among the whole line of 12-13 Fortran compilers then it's OK not to be method-independent.

Currently if you create the file with Method1 it only can be read back with the same Method1, and Method2 - with Method2. Not super bad inconvenience, it is possible to make a note in the file which method was used and then read the file.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Sat Oct 01, 2022 8:28 am    Post subject: Reply with quote

Dan,

Part of the problem is not clearly understanding the limitations of each binary file type/format ( unformatted sequential, direct access fixed length records or now, stream access)

Paul,

I have been successfully using Fotran direct access binary files (fixed length records) for many years as a portable interface between different Fortran compilers.

Stream I/O now provides a similar portable access by,
a) allowing I/O lists of intrinsic types ( I/O lists are much more flexible than a single vector )
b) able to create header and footer labels to records, to be compatible with other compiler's unformatted sequential binary file formats.
c) provides random/direct addressing to files ( although I have had some problems with integer*8 addressing up to FTN95 Ver 8.8, which may now be fixed. I should find my past examples and retest with Ver 8.9 )

It would be good if the following could be investigated and possibly supported for unformatted sequential files:
a) an alternative could be provided to support other compilers unformatted binary file formats, via a compile option or in the OPEN statement.
b) confirm if records or files larger than 2^31 bytes or "words" could be supported.
c) investigate support for derived type records to be used in I/O lists, rather than specifying as a list of intrinsic type components.

However, stream I/O now provides a very flexible and portable alternative to earlier unformatted file approaches.
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Sat Oct 01, 2022 10:41 am    Post subject: Reply with quote

John

Thanks for the feedback. Regarding your request for further investigations:

a) I don't see how we could offer to do this even for just one other compiler.

b) Do you have any experience that this does not work?

c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Mon Oct 03, 2022 1:48 pm    Post subject: Reply with quote

Paul,

a) I don't see how we could offer to do this even for just one other compiler.

Answer: addressing only records smaller than 2gb; both ifort and gfortran use a 4-byte header/footer for unformatted sequential files. They are portable for this file type.
If FTN95 could have a (say) /4_byte_header option, (unlike the present 1-byte or 5-byte default format), this would improve portability for gfortran and ifort interface.

b) Do you have any experience that this does not work?

Answer: I provided a stream I/O example in:
http://forums.silverfrost.com/viewtopic.php?t=4565&highlight=stream

I have now updated this program to better document the test.

https://www.dropbox.com/s/r6cy1xak6hp2msa/stream_read2.f90?dl=0
https://www.dropbox.com/s/wnj6pu7cq2vxapw/stream_read.log?dl=0

It works with gfortran (in Plato) but FTN95 does not support
"write ( unit=stream_unit, pos=address ) values" or
"read ( unit=stream_unit, pos=address ) one_byte"
the value "address" does not work in FTN95 and "INQUIRE ( unit, pos=address )" does not appear to give the same value as gfortran.

The file stream_read.log now provides a report of the tests for gfortran Ver 11.1 and FTN95 Ver 8.80

c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.

integer*8 address will then need to be checked after b) is addressed.

I will test the program with integer*8 address and check it still works.

( I think I have provided examples of this before, but we first need "pos=" to be supported in FTN95 )

I will download Ver 8.90 and test.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Mon Oct 03, 2022 2:45 pm    Post subject: Reply with quote

A further update to stream_read3.f90

this reports the address after writing the record.
It shows that for:
Code:

         write ( unit=stream_unit, pos=address ) values
         inquire ( unit=stream_unit, pos = end_pos )

pos=address is positioning to the byte before "address" (based on gfortran response)

https://www.dropbox.com/s/1wdfnbn9cmy3bix/stream_read3.f90?dl=0

https://www.dropbox.com/s/nh3gczdfup106pn/stream_read3.log?dl=0


further change for integer*8 :: address (etc)
This shows that pos=address does not work with FTN95 Ver 8.80
I will download Ver 8.90 tomorrow !!

https://www.dropbox.com/s/g09v7rd81a7aflf/stream_read8.f90?dl=0
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Tue Oct 04, 2022 7:39 am    Post subject: Reply with quote

John

I will make a note of this and see what can be done.
Back to top
View user's profile Send private message AIM Address
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Thu Oct 06, 2022 10:38 am    Post subject: Reply with quote

The original failure on this thread (reported by DanRRight) has now been fixed for the next release of clearwin64.dll.

The change has an impact on FTN95 so it is possible that users may also need the next release of FTN95 (the one after v8.91). In other words, it is unlikely that you will also need a new FTN95.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2585
Location: Sydney

PostPosted: Fri Oct 07, 2022 4:07 am    Post subject: Reply with quote

Paul,

Can you please provide some clarification for what error was fixed ?

Did it relate to the following code where an array larger than 2GBytes is written as a sequential unformatted file { OPEN(access='sequential', FORM='UNFORMATTED',...) }
Code:
    open (11, file='LargeFile.dat', FORM='UNFORMATTED')
    write(11) Arr4
    close(11)


If this is the case, could you please clarify what header/footer structure has been proposed.

Also, there may be some confusion regarding which of my "c)" requests are to be addressed. My preference would be to consider the "c)" related to integer*8 addressing for "pos=address" with stream I/O.

Stream I/O can offer much potential for a simpler solution to unformatted I/O, especially for larger records, although designing a record structure using data packets smaller than 2 GBytes might be a more robust solution with minimal performance penalty.

( the other "c)" issue of derived type I/O support, as part of F08(?) is a much more complex beast, that I find difficult to understand. I think many of us would be satisfied with writing out derived type components, which is supported in FTN95 ? )
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 8018
Location: Salford, UK

PostPosted: Fri Oct 07, 2022 7:22 am    Post subject: Reply with quote

John

As I understand it, there is no header or footer for stream I/O.

The aim is to look at all the issues that you have raised to see what is possible within our resources. But it is on a list of things to do.
Back to top
View user's profile Send private message AIM Address
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support All times are GMT + 1 Hour
Goto page 1, 2, 3, 4, 5, 6, 7  Next
Page 1 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group