|
forums.silverfrost.com Welcome to the Silverfrost forums
|
View previous topic :: View next topic |
Author |
Message |
DanRRight
Joined: 10 Mar 2008 Posts: 2866 Location: South Pole, Antarctica
|
Posted: Mon Sep 26, 2022 2:13 am Post subject: Fails to save arrays > 4GB |
|
|
There exist two major methods to save arrays. Suppose we have 2D REAL*4 array Arr4(nA,nB). First method is when we explicitly save each element or each dimension individually
Code: | open (11, file='LargeFile.dat', FORM='UNFORMATTED')
do i=1,nB
write(11) Arr4(:,i)
enddo
close(11) |
and the second method is when we save the entire array as a whole
Code: | open (11, file='LargeFile.dat', FORM='UNFORMATTED')
write(11) Arr4
close(11) | .
Second method can be an order of magnitude faster reaching speeds 3GB per second on current hardware (with m2 NVMe PCIe 4.0) drives. But it fails to save, flashing the Silverfrost exception, if array is larger than ~4GB while first slow method works with no problems.
Here is demo which allocates OK any size and saves files using second method when file size is 4GB (array size nB =1e8) but fails to save with nB=2e8 and file size twice larger. It fails also to save even REAL*8 arrays. Compilation with /64 switch.
Code: | ! compilation: ftn95 aaa.f90 /link /64 >z
!
real*4, dimension(:,:), allocatable :: Arr4
nA = 11
nB = 2.e8
!...Allocating array
Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
allocate ( Arr4 (nA, nB), stat = ierr)
if (ierr.eq.0) then
Print*,'Allocation success'
else
Pause 'Fail to allocate'
goto 1000
endif
!...Filling the array with some data
do i=1,nB
Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
enddo
! __ __ ____ ____ _ _ _____ ____ __
! ( \/ )( ___)(_ _)( )_( )( _ )( _ \ / )
! ) ( )__) )( ) _ ( )(_)( )(_) ) )(
! (_/\/\_)(____) (__) (_) (_)(_____)(____/ (__)
!....................................................
Print*,'Trying to save the data Method 1 '
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
do i=1,nB
write(11) Arr4(:,i)
enddo
close(11)
call cpu_time(t1)
!...Speeed of writing method 1
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,' Speed of write Method 1 =', SpeedGBps ! typically ~0.5 GB/s
! __ __ ____ ____ _ _ _____ ____ ___
! ( \/ )( ___)(_ _)( )_( )( _ )( _ \ (__ \
! ) ( )__) )( ) _ ( )(_)( )(_) ) / _/
! (_/\/\_)(____) (__) (_) (_)(_____)(____/ (____)
!....................................................
Print*,'Trying to save the data Method 2'
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
write(11) Arr4
close(11)
call cpu_time(t1)
!...Speeed of writing Method 2
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,' Speed of write Method 2=', SpeedGBps ! typically ~2.6 GB/s
pause 'File LargeFile.dat created OK'
goto 1000
!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
goto 1000
910 Print*,'Can not save file LargeFile.dat'
1000 Continue
End
|
Last edited by DanRRight on Mon Sep 26, 2022 11:11 am; edited 5 times in total |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Mon Sep 26, 2022 7:44 am Post subject: |
|
|
Dan
Thanks for the feedback. I have made a note that this needs investigating. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
Posted: Tue Sep 27, 2022 7:27 am Post subject: |
|
|
my understanding of unformatted binary records is that FTN95 does not support records longer than about 2^31 bytes ( ie the header is byte 1 = -1 and bytes 2:5 must be a positive 4-byte integer )
Both ifort and gfortran support a complex record header/footer structure of sub records, which I recall are 2^31-9 bytes in size. the header/footer is 4-bytes long. The sub records are identified by -ve sub record sizes, enclosed by +ve sub record size for first and last sub records.
Does FTN95 Stream I/O support integer*8 addressing, as that could be the simplest to implement ?
Alternatively, write the array as a group of records which are less than 2^31 bytes long, choosing "Arr4(:,i)" so that it is < 2^31 bytes.
This will be the most practical and "portable" solution.
Stream I/O is portable between FTN95 and ifort/gfortran, as is fixed length direct access files ( again with records smaller than 2^31 bytes )
( FTN95 unformatted records are not portable ) |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Tue Sep 27, 2022 9:24 am Post subject: |
|
|
John
Can you explain what you mean by "Stream I/O integer*8 addressing"?
For 64 bit FTN95 the I/O library is in clearwin64.dll which is 64 bits. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
Posted: Tue Sep 27, 2022 2:10 pm Post subject: Re: |
|
|
PaulLaidler wrote: | Can you explain what you mean by "Stream I/O integer*8 addressing"? |
To write a record longer than 2^31 for stream I/O, I would expect that stream I/O must support 64 bit integer addressing.
Perhaps for sequential access, writing a "record" larger than 2^31 bytes may be ok ?
As there is no record structure, the following would not be any different on the file to write(11) Arr4. Code: | do i=1,nB
write(11) Arr4(:,i)
enddo |
|
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Wed Sep 28, 2022 7:07 am Post subject: |
|
|
John
Stream I/O uses a nominal record length with value one. This could be compiler dependent but probably isn't.
I am assuming that the size of a file will not be a problem (for 64 bit FTN95). |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2866 Location: South Pole, Antarctica
|
Posted: Fri Sep 30, 2022 9:39 pm Post subject: |
|
|
Yes, ideally files created with any of these two methods have to be readable by each of these methods. But if this is not the case among the whole line of 12-13 Fortran compilers then it's OK not to be method-independent.
Currently if you create the file with Method1 it only can be read back with the same Method1, and Method2 - with Method2. Not super bad inconvenience, it is possible to make a note in the file which method was used and then read the file. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
Posted: Sat Oct 01, 2022 8:28 am Post subject: |
|
|
Dan,
Part of the problem is not clearly understanding the limitations of each binary file type/format ( unformatted sequential, direct access fixed length records or now, stream access)
Paul,
I have been successfully using Fotran direct access binary files (fixed length records) for many years as a portable interface between different Fortran compilers.
Stream I/O now provides a similar portable access by,
a) allowing I/O lists of intrinsic types ( I/O lists are much more flexible than a single vector )
b) able to create header and footer labels to records, to be compatible with other compiler's unformatted sequential binary file formats.
c) provides random/direct addressing to files ( although I have had some problems with integer*8 addressing up to FTN95 Ver 8.8, which may now be fixed. I should find my past examples and retest with Ver 8.9 )
It would be good if the following could be investigated and possibly supported for unformatted sequential files:
a) an alternative could be provided to support other compilers unformatted binary file formats, via a compile option or in the OPEN statement.
b) confirm if records or files larger than 2^31 bytes or "words" could be supported.
c) investigate support for derived type records to be used in I/O lists, rather than specifying as a list of intrinsic type components.
However, stream I/O now provides a very flexible and portable alternative to earlier unformatted file approaches. |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Sat Oct 01, 2022 10:41 am Post subject: |
|
|
John
Thanks for the feedback. Regarding your request for further investigations:
a) I don't see how we could offer to do this even for just one other compiler.
b) Do you have any experience that this does not work?
c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
Posted: Mon Oct 03, 2022 1:48 pm Post subject: |
|
|
Paul,
a) I don't see how we could offer to do this even for just one other compiler.
Answer: addressing only records smaller than 2gb; both ifort and gfortran use a 4-byte header/footer for unformatted sequential files. They are portable for this file type.
If FTN95 could have a (say) /4_byte_header option, (unlike the present 1-byte or 5-byte default format), this would improve portability for gfortran and ifort interface.
b) Do you have any experience that this does not work?
Answer: I provided a stream I/O example in:
http://forums.silverfrost.com/viewtopic.php?t=4565&highlight=stream
I have now updated this program to better document the test.
https://www.dropbox.com/s/r6cy1xak6hp2msa/stream_read2.f90?dl=0
https://www.dropbox.com/s/wnj6pu7cq2vxapw/stream_read.log?dl=0
It works with gfortran (in Plato) but FTN95 does not support
"write ( unit=stream_unit, pos=address ) values" or
"read ( unit=stream_unit, pos=address ) one_byte"
the value "address" does not work in FTN95 and "INQUIRE ( unit, pos=address )" does not appear to give the same value as gfortran.
The file stream_read.log now provides a report of the tests for gfortran Ver 11.1 and FTN95 Ver 8.80
c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.
integer*8 address will then need to be checked after b) is addressed.
I will test the program with integer*8 address and check it still works.
( I think I have provided examples of this before, but we first need "pos=" to be supported in FTN95 )
I will download Ver 8.90 and test. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Tue Oct 04, 2022 7:39 am Post subject: |
|
|
John
I will make a note of this and see what can be done. |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Thu Oct 06, 2022 10:38 am Post subject: |
|
|
The original failure on this thread (reported by DanRRight) has now been fixed for the next release of clearwin64.dll.
The change has an impact on FTN95 so it is possible that users may also need the next release of FTN95 (the one after v8.91). In other words, it is unlikely that you will also need a new FTN95. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2585 Location: Sydney
|
Posted: Fri Oct 07, 2022 4:07 am Post subject: |
|
|
Paul,
Can you please provide some clarification for what error was fixed ?
Did it relate to the following code where an array larger than 2GBytes is written as a sequential unformatted file { OPEN(access='sequential', FORM='UNFORMATTED',...) } Code: | open (11, file='LargeFile.dat', FORM='UNFORMATTED')
write(11) Arr4
close(11) |
If this is the case, could you please clarify what header/footer structure has been proposed.
Also, there may be some confusion regarding which of my "c)" requests are to be addressed. My preference would be to consider the "c)" related to integer*8 addressing for "pos=address" with stream I/O.
Stream I/O can offer much potential for a simpler solution to unformatted I/O, especially for larger records, although designing a record structure using data packets smaller than 2 GBytes might be a more robust solution with minimal performance penalty.
( the other "c)" issue of derived type I/O support, as part of F08(?) is a much more complex beast, that I find difficult to understand. I think many of us would be satisfied with writing out derived type components, which is supported in FTN95 ? ) |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8018 Location: Salford, UK
|
Posted: Fri Oct 07, 2022 7:22 am Post subject: |
|
|
John
As I understand it, there is no header or footer for stream I/O.
The aim is to look at all the issues that you have raised to see what is possible within our resources. But it is on a list of things to do. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|