Welcome to our forums
John
Can you send me a short demo program that illustrates what you would like to happen?
Dan
I have looked at your program aaa.f90.
The result of 'read(11,err=930,end=932) Arr4' at line 63 is a call to the Microsoft API ReadFile where the number of bytes to read is an unsigned 32 bit integer.
So at the moment you can't read very large chunks in one go.
This can be fixed internally by buffering the input but at the moment you will need to do the buffering yourself as in your Method 1.
This read issue has now been fixed to the degree that you will get an explicit runtime error when trying to read very large chunks of data in one go.
In the longer term we could provide a fix by adding internal buffering but this would not help with regard to the stated object of reducing the runtime by using large chunks.
Paul, Thanks. Did i understand this correctly that this is Microsoft issue?
The Microsoft ReadFile function is limited and as far as I know there is no direct alternative.
Work on internal buffering is almost complete so, when finished, externally there will be no limit.
Just ran this test on ifort and gfortran. They work with files >4GB and speed of ifort is even 2.5x larger than anything we reported so far with any other compiler - 6GB/s with Method2. Of course FTN95 is still record king using proprietary READF@ and EQUIVALENCE to encode and decode data from READF@/WRITEF@ stream of data (i was getting 7-8 GB/s there) but that's different approach, and though it is simple but not as ultimately simple, standard, portable and straightforward as the one we discuss today. Intel though clearly made an error somewhere with Method1 because with it they got speed ... 200x less. That is like the speed of formatted I/O! Looks like they forgot napkins and scissors in the body during their surgery. Please use RAM drives or m2 Gen4 NVMe drives when testing to squeeze last bit of speed from software and for not missing something like Intel !
Quoted from PaulLaidler John
Can you send me a short demo program that illustrates what you would like to happen?
Paul, the following link has the 'alpha' code I am testing for my new random access file library.
https://www.dropbox.com/s/asj6g4dgucl7u7t/mslib.f90?dl=0
It is structured as my I/O library, so has routines for many basic tasks. 'pos=address' uses integer*8 address There is a lot of reporting of read/write in this alpha version, but hopefully the output is understandable.
For the test : line 426 adjusts the number of records in test line 309 adjusts the record sizes line 16 selects the record size header/trailer as 4 or 8 bytes, although only 8 bytes has been tested so far.
The test has 4 stages:
This version only supports 1 file but does demonstrate 2 problems. 1 : re-write may truncate file 2 : larger records are not recognised by INQUIRE
If I run the attached version using the latest FTN95 Ver 8.92 in PLATO x64 release, I get the following problems
Problem 1 : truncate file With the attached code: At stage 3, the last record re-written is record 8 At stage 4, the records 9 and 10 are no longer available, so it looks like a re-write might be truncating the file ? the OPEN statement is at line 20
Problem 2 : possible delay for updating file position in INQUIRE ? If I increase the record sizes ( use line 309 rather than line 310 ) I then have problems that (line 33) INQUIRE does not return the correct file position at the end of the write_record_data (line 260) ( for the first record written), but reports a lesser position value ( is the write buffered ? )
If I select ALT X in PLATO and use gfortran, these problems do not occur.
I hope this demonstrates the problem.
Question : would converting lines 184 and 260 to array syntax help ? ie, write ( unit=stream_lunit, pos=address, iostat=iostat ) rec_data(1:n)
When these 2 problems are solved, I hope to clean out unnecessary reporting and increase the number of records to test larger files.
John
Thanks. I will take a look at this.
Dan
The FTN95 I/O has now been fixed so that in future it will run your program aaa.f90.
Paul,
I have been investigating further, especially the problem of the file being truncated after a re-write.
I have also tidied up the error tests and messages, although still not complete. (see error_info)
I have also included a fifth test : sequential read; that checks the size and position of all records. ( This might not always work in general for my IO library, as records can be re-written as smaller records, meaning records are then only direct accessible by their file address.)
https://www.dropbox.com/s/1q2pyge5zgb9ezz/mslib_t2.f90?dl=0 https://www.dropbox.com/s/m8v905qtdu7fc1s/mslib_t2.log?dl=0 https://www.dropbox.com/s/cap59y3pc2jy67l/do_test.bat?dl=0
I am confused by the file truncation after a re-write, as it does not always occur. I am not sure of the rules for truncation or what attributes in the OPEN may control this. I understand sequential formatted text files are truncated, but not sure what conditions apply for unformatted stream I/O files ? I would expect stream access files should not be truncated, even if 'pos=' is not used, so used as a sequential access only file.
This attached test has also changed the record sizes (line 323) and now shows the 'INQUIRE ( pos=' error, (problem 2 above). As implemented, it appears to report the current end of file position, rather than the present file position. (This also helps identify when the file is truncated ? ) In test 1: sequential write, the returned INQUIRE file position is confusing, being less that the expected value. It looks like a delayed file update (asynchronous ?) I would expect pos= should return the 'current position', not the end of file position, including after a READ statement. I already calculate the 'current position', independent of INQUIRE, so this problem is not important for me.
With the attached program, once a file iostat = -1 occurs (tries to read past eof), all following file access gives iostat=104, even if pos= positions to before the end of file. This can be more clearly demonstrated by using line 477, rather than 478, so records past the truncated fiile are referenced. When this occurs, all records in the following sequential read fail. (I don't agree with this for stream access).
I need APPEND to the file defined as a STREAM. Both APPEND and STREAM use keyword ACCESS, i.e. ACCESS='APPEND' and ACCESS='STREAM'. The FTN95 uses for APPEND keyword STATUS='APPEND' and the problem is solved.
But the gFortran does not like any of that, it does not like two ACCESS keywords in the same place, does not like STATUS='APPEND' and also does not like POSITION='APPEND'. What's to do with this damn picky compiler? Is it strictly following Standard or just missing feature?
POSITION = 'APPEND' is in the Fortran 90 standard.
STATUS = 'APPEND' is not in the Fortran 90 standard but is allowed as an FTN95 extension to the Standard.
I don't know if 'APPEND' works for FTN95 ACCESS = 'STREAM' (or if it is meaningful in that context).
Meaningful implies STREAM files could only being rewritten, you can not append new data to them?
Problem solved. gFortran gave the false positive in wrong place. POSITION='APPEND' works, it was an extra ' at the end like this POSITION='APPEND''. In this situation FTN95 was spot on pointing (by words only unfortunately) on exact error place, while gFortran pointed out by finger on the letter P in POSITION saying this is inappropriate in this content.
This always rises for me rhetoric question - how all these sh#tty in the past compilers overjumped the leader of 1990th FTN95 in modern features so we now have to move there because FTN95 lags currently by not years but decades at parallelization and F2003/2008?
Quoted from PaulLaidler I don't know if 'APPEND' works for FTN95 ACCESS = 'STREAM' (or if it is meaningful in that context).
The documentation I have (Modern Fortran Explained) says for 'POSITION=' that the access method must be sequential, which might not be the same as saying ACCESS='SEQUENTIAL' ?
STREAM I/O can be either sequential or direct, as the use of POS= is optional. This is the reason for some ambiguity with the use of POSITION= in OPEN and if the file should be truncated.
Should truncating the file be valid if the access method is sequential, or only if ACCESS='SEQUENTIAL' ? Truncating the ACCESS='STREAM' file should probably be left to STATUS='REPLACE'
Paul, this is another smaller example, modified from https://fortranwiki.org/fortran/show/Stream+Input+Output
Running in Plato with (ALT+X) FTN95 or gfortran shows some problems. write (pos= is ignored in write, if past EOF read (pos= does not recover from IOSTAT /= 0, until file re-opened
use iso_fortran_env
write (*,*) 'Compiler Version :',compiler_version ()
call writeUstream
call readUstream
end
subroutine writeUstream
IMPLICIT NONE
INTEGER :: myvalue = 12345, mypos, out = 11, iostat
CHARACTER :: fmt*12
write (*,*) ' '
OPEN (UNIT=out, FILE='ustream.demo', STATUS='REPLACE', ACCESS='STREAM', iostat=iostat)
write (*,*) 'writing to unit=',out,' : iostat=',iostat
INQUIRE(UNIT=out, POS=mypos, FORM=fmt, iostat=iostat)
PRINT *, 'Myvalue will be written at position ', mypos,' : form= ',fmt,' : iostat=',iostat
WRITE(out, POS= 4, iostat=iostat) 'first' ; write (*,*) ' iostat=',iostat
WRITE(out, POS=14, iostat=iostat) 'second' ; write (*,*) ' iostat=',iostat
INQUIRE(UNIT=out, POS=mypos, iostat=iostat)
PRINT *, 'Myvalue will be written at position ', mypos,' : iostat=',iostat
Print *, ' should be position = 14+6 = 20'
WRITE(out, iostat=iostat) myvalue ; write (*,*) ' iostat=',iostat
CLOSE(UNIT=out)
END subroutine writeUstream
subroutine readUstream
IMPLICIT NONE
CHARACTER :: string*3, c
INTEGER :: n, in = 12, iostat, k
write (*,*) ' '
OPEN(UNIT=in, FILE='ustream.demo', STATUS='OLD', ACCESS='STREAM', iostat=iostat)
write (*,*) 'reading from unit=',in,' : iostat=',iostat
READ(in, POS=7, iostat=iostat) string
write (*,*) 'string :',string,' : iostat=',iostat
READ(in, POS=20, iostat=iostat) n
write (*,*) 'number :',n,' : iostat=',iostat
do k = 1,2
write (*,*) ' '
write (*,*) 'Now read all bytes in file'
write (*,*) ' Note FTN95 has ignored pos= for write'
do n = 1,25
read ( in, pos=n, iostat=iostat ) c
write (*,*) n, ichar(c), iostat
if ( iostat == -1 ) exit
end do
CLOSE(UNIT=in)
OPEN(UNIT=in, FILE='ustream.demo', STATUS='OLD', ACCESS='STREAM', iostat=iostat)
write (*,*) 'Reopen file to fix FTN95 iostat error'
end do
END subroutine readUstream
Thanks John.
Paul, While previous FTN95 version indeed was fixed and writes larger real4 arrays > 2GB (see the write demo above which allocates 16GB with nB=4e8), the latest version of FTN95 fails with anything larger than 2GB. Reading larger REAL4 arrays >2GB did not work with any FTN95 versions.
Here is new demo which designed to work for read and write as you can see with the arrays below 2GB (nB=3.e7) but it fails with anything larger than that. It creates and then reads the array using two methods: as Arr4(:,i) or as Arr4
! compilation: ftn95 aaa.f90 /link /64 >z
!
real*4, dimension(:,:), allocatable :: Arr4
nA = 11
nB = 2.e8
!...Allocating array
Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
allocate ( Arr4 (nA, nB), stat = ierr)
if (ierr.eq.0) then
Print*,'Allocation success'
else
Pause 'Fail to allocate'
goto 1000
endif
!...Filling the array with some data
do i=1,nB
Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
enddo
Print*,'Trying to save the data Method 1 '
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
do i=1,nB
write(11,err=910) Arr4(:,i)
enddo
close(11)
call cpu_time(t1)
!...Speeed of writing method 1
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,'Write OK. Speed of write Method 1 =', SpeedGBps ! typically ~0.5 GB/s
print*,'====================='
print*,'================ N O W R E A D ===================='
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
do i=1,nB
read(11,err=912,end=914) Arr4(:,i)
enddo
close(11)
call cpu_time(t1)
!...Speeed of writing method 1
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,'READ OK. Speed of read Method 1 =', SpeedGBps
print*,'============================='
Print*,'Trying to save the data Method 2'
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
write(11,err=920) Arr4
close(11)
call cpu_time(t1)
!...Speeed of writing Method 2
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,'Write OK. Speed of write Method 2=', SpeedGBps ! typically ~2.6 GB/s
print*,'====================='
print*,'================ N O W R E A D =================='
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access='STREAM', err=900)
read(11,err=930,end=932) Arr4
close(11)
call cpu_time(t1)
!...Speeed of writing method 2
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
print*,'READ OK. Speed of read Method 2 =', SpeedGBps
print*,'======================'
pause 'File LargeFile.dat created OK'
goto 1000
!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
goto 1000
910 Print*,'Error. Can not save file LargeFile.dat Method 1'
goto 1000
912 Print*,'Error. Can not load file LargeFile.dat Method 1'
goto 1000
914 Print*,'Abruptly tnd of file. Can not load file LargeFile.dat Method 1'
goto 1000
920 Print*,'Error. Can not write file LargeFile.dat Method 2'
goto 1000
930 Print*,'Error. Can not read file LargeFile.dat Method 2'
goto 1000
932 Print*,'Abruptly end of file. Can not read file LargeFile.dat Method 2'
pause
1000 Continue
End
Thank you for the feedback. I have made a note of this issue.
I think the status of FTN95 /64 in Nov 2022 was that you could write records longer than 2 GBytes, but they could not be read. ( The I/O software supported writing the buffer, but the header/footer syntax did not work) The problem is that the 4-byte header and footer (actually 5-byte) is not able to identify the record length.
The only way to fix this problem using the FTN95 approach, is to introduce a 13-byte header and footer, which would consist of: 1-byte = -1 : 4-bytes = -1 : 8-bytes = record length.
At present FTN95 supports the 1-byte = -1 to support records longer than 240 bytes. It now needs to support the 4-bytes = -1 to indicate a further 8-byte header is provided for records longer than 2^31-(say 16).
An alternative approach is to adopt the 4-byte sub-record approach of Intel and Gfortran, which adjusts the larger record as multiple 2^31-9 long sub records, where the sub record lengths are given as -ve to indicate there are repeated sub records. This approach is documented on the Intel Fortran web site. ( there is no support for small records < 240 bytes )
FTN95 can not support larger records without a change to the existing 5-byte syntax.
A problem with the Intel/Gfortran approach is that the sub records are not a multiple of 4 bytes (or 8 or 16) but the combined length of 4 bytes + 231-9 + 4 = 231-1, which is the largest +ve 4-byte integer. This odd length creates problems addressing memory using 4-byte or 8-byte integer arrays. Hence my recomendation of 2^31-16 a a sub-record length makes it easier to write the library in Fortran.