forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Fails to save arrays > 4GB
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Fri Oct 28, 2022 9:51 pm    Post subject: Reply with quote

Just ran this test on ifort and gfortran. They work with files >4GB and speed of ifort is even 2.5x larger than anything we reported so far with any other compiler - 6GB/s with Method2. Of course FTN95 is still record king using proprietary READF@ and EQUIVALENCE to encode and decode data from READF@/WRITEF@ stream of data (i was getting 7-8 GB/s there) but that's different approach, and though it is simple but not as ultimately simple, standard, portable and straightforward as the one we discuss today. Intel though clearly made an error somewhere with Method1 because with it they got speed ... 200x less. That is like the speed of formatted I/O! Looks like they forgot napkins and scissors in the body during their surgery. Please use RAM drives or m2 Gen4 NVMe drives when testing to squeeze last bit of speed from software and for not missing something like Intel !

Last edited by DanRRight on Sun Oct 30, 2022 7:25 am; edited 1 time in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sat Oct 29, 2022 4:48 am    Post subject: Re: Reply with quote

PaulLaidler wrote:
John

Can you send me a short demo program that illustrates what you would like to happen?


Paul, the following link has the "alpha" code I am testing for my new random access file library.

https://www.dropbox.com/s/asj6g4dgucl7u7t/mslib.f90?dl=0

It is structured as my I/O library, so has routines for many basic tasks.
"pos=address" uses integer*8 address
There is a lot of reporting of read/write in this alpha version, but hopefully the output is understandable.

For the test :
line 426 adjusts the number of records in test
line 309 adjusts the record sizes
line 16 selects the record size header/trailer as 4 or 8 bytes, although only 8 bytes has been tested so far.

The test has 4 stages:
1) define and write 10 records sequenitally
2) read and check 10 records randomly
3) re-write 10 records randomly ( last record is #8 in this FTN95 test )
4) read and check 10 records randomly (fails when testing #9)

This version only supports 1 file but does demonstrate 2 problems.
1 : re-write may truncate file
2 : larger records are not recognised by INQUIRE

If I run the attached version using the latest FTN95 Ver 8.92 in PLATO x64 release, I get the following problems

Problem 1 : truncate file
With the attached code:
At stage 3, the last record re-written is record 8
At stage 4, the records 9 and 10 are no longer available, so it looks like a re-write might be truncating the file ?
the OPEN statement is at line 20

Problem 2 : possible delay for updating file position in INQUIRE ?
If I increase the record sizes ( use line 309 rather than line 310 ) I then have problems that (line 33) INQUIRE does not return the correct file position at the end of the write_record_data (line 260) ( for the first record written), but reports a lesser position value ( is the write buffered ? )

If I select ALT X in PLATO and use gfortran, these problems do not occur.

I hope this demonstrates the problem.

Question : would converting lines 184 and 260 to array syntax help ? ie,
write ( unit=stream_lunit, pos=address, iostat=iostat ) rec_data(1:n)

When these 2 problems are solved, I hope to clean out unnecessary reporting and increase the number of records to test larger files.
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Sat Oct 29, 2022 7:43 am    Post subject: Reply with quote

John

Thanks. I will take a look at this.
Back to top
View user's profile Send private message AIM Address
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Sat Oct 29, 2022 3:29 pm    Post subject: Reply with quote

Dan

The FTN95 I/O has now been fixed so that in future it will run your program aaa.f90.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sun Oct 30, 2022 1:45 am    Post subject: Reply with quote

Paul,

I have been investigating further, especially the problem of the file being truncated after a re-write.

I have also tidied up the error tests and messages, although still not complete. (see error_info)

I have also included a fifth test : sequential read; that checks the size and position of all records. ( This might not always work in general for my IO library, as records can be re-written as smaller records, meaning records are then only direct accessible by their file address.)

https://www.dropbox.com/s/1q2pyge5zgb9ezz/mslib_t2.f90?dl=0
https://www.dropbox.com/s/m8v905qtdu7fc1s/mslib_t2.log?dl=0
https://www.dropbox.com/s/cap59y3pc2jy67l/do_test.bat?dl=0

I am confused by the file truncation after a re-write, as it does not always occur.
I am not sure of the rules for truncation or what attributes in the OPEN may control this.
I understand sequential formatted text files are truncated, but not sure what conditions apply for unformatted stream I/O files ?
I would expect stream access files should not be truncated, even if "pos=" is not used, so used as a sequential access only file.

This attached test has also changed the record sizes (line 323) and now shows the "INQUIRE ( pos=" error, (problem 2 above). As implemented, it appears to report the current end of file position, rather than the present file position. (This also helps identify when the file is truncated ? )
In test 1: sequential write, the returned INQUIRE file position is confusing, being less that the expected value. It looks like a delayed file update (asynchronous ?)
I would expect pos= should return the "current position", not the end of file position, including after a READ statement. I already calculate the "current position", independent of INQUIRE, so this problem is not important for me.

With the attached program, once a file iostat = -1 occurs (tries to read past eof), all following file access gives iostat=104, even if pos= positions to before the end of file. This can be more clearly demonstrated by using line 477, rather than 478, so records past the truncated fiile are referenced. When this occurs, all records in the following sequential read fail. (I don't agree with this for stream access).
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Mon Oct 31, 2022 1:17 pm    Post subject: Reply with quote

I need APPEND to the file defined as a STREAM. Both APPEND and STREAM use keyword ACCESS, i.e. ACCESS='APPEND' and ACCESS='STREAM'. The FTN95 uses for APPEND keyword STATUS='APPEND' and the problem is solved.

But the gFortran does not like any of that, it does not like two ACCESS keywords in the same place, does not like STATUS='APPEND' and also does not like POSITION='APPEND'. What's to do with this damn picky compiler? Is it strictly following Standard or just missing feature?
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Mon Oct 31, 2022 4:27 pm    Post subject: Reply with quote

POSITION = 'APPEND' is in the Fortran 90 standard.

STATUS = 'APPEND' is not in the Fortran 90 standard but is allowed as an FTN95 extension to the Standard.

I don't know if 'APPEND' works for FTN95 ACCESS = 'STREAM' (or if it is meaningful in that context).
Back to top
View user's profile Send private message AIM Address
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Mon Oct 31, 2022 8:38 pm    Post subject: Reply with quote

Meaningful implies STREAM files could only being rewritten, you can not append new data to them?
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Tue Nov 01, 2022 12:16 am    Post subject: Reply with quote

Problem solved. gFortran gave the false positive in wrong place. POSITION='APPEND' works, it was an extra ' at the end like this POSITION='APPEND''. In this situation FTN95 was spot on pointing (by words only unfortunately) on exact error place, while gFortran pointed out by finger on the letter P in POSITION saying this is inappropriate in this content.

This always rises for me rhetoric question - how all these sh#tty in the past compilers overjumped the leader of 1990th FTN95 in modern features so we now have to move there because FTN95 lags currently by not years but decades at parallelization and F2003/2008?


Last edited by DanRRight on Tue Nov 01, 2022 7:30 am; edited 2 times in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Tue Nov 01, 2022 3:06 am    Post subject: Re: Reply with quote

PaulLaidler wrote:
I don't know if 'APPEND' works for FTN95 ACCESS = 'STREAM' (or if it is meaningful in that context).


The documentation I have (Modern Fortran Explained) says for "POSITION=" that the access method must be sequential, which might not be the same as saying ACCESS='SEQUENTIAL' ?

STREAM I/O can be either sequential or direct, as the use of POS= is optional.
This is the reason for some ambiguity with the use of POSITION= in OPEN and if the file should be truncated.

Should truncating the file be valid if the access method is sequential, or only if ACCESS='SEQUENTIAL' ?
Truncating the ACCESS='STREAM' file should probably be left to STATUS='REPLACE'
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Wed Nov 02, 2022 5:50 am    Post subject: Reply with quote

Paul, this is another smaller example, modified from
https://fortranwiki.org/fortran/show/Stream+Input+Output

Running in Plato with (ALT+X) FTN95 or gfortran shows some problems.
write (pos= is ignored in write, if past EOF
read (pos= does not recover from IOSTAT /= 0, until file re-opened

Code:
  use iso_fortran_env
  write (*,*) 'Compiler Version :',compiler_version ()

  call writeUstream
  call readUstream

  end
 
  subroutine writeUstream
  IMPLICIT NONE
  INTEGER :: myvalue = 12345, mypos, out = 11, iostat
  CHARACTER :: fmt*12
 
  write (*,*) ' '
  OPEN (UNIT=out, FILE="ustream.demo", STATUS="REPLACE", ACCESS="STREAM", iostat=iostat)
  write (*,*) 'writing to unit=',out,' : iostat=',iostat
  INQUIRE(UNIT=out, POS=mypos, FORM=fmt, iostat=iostat)
  PRINT *, "Myvalue will be written at position ", mypos,' : form= ',fmt,' : iostat=',iostat

  WRITE(out, POS= 4, iostat=iostat) "first"  ; write (*,*) '  iostat=',iostat
  WRITE(out, POS=14, iostat=iostat) "second" ; write (*,*) '  iostat=',iostat
  INQUIRE(UNIT=out, POS=mypos, iostat=iostat)
  PRINT *, "Myvalue will be written at position ", mypos,' : iostat=',iostat
  Print *, " should be position = 14+6 = 20"
  WRITE(out, iostat=iostat) myvalue          ; write (*,*) '  iostat=',iostat
  CLOSE(UNIT=out)
 
  END subroutine writeUstream

  subroutine readUstream
  IMPLICIT NONE
  CHARACTER :: string*3, c
  INTEGER :: n, in = 12, iostat, k
 
  write (*,*) ' '
  OPEN(UNIT=in, FILE="ustream.demo", STATUS="OLD", ACCESS="STREAM", iostat=iostat)
  write (*,*) 'reading from unit=',in,' : iostat=',iostat

  READ(in, POS=7, iostat=iostat) string
  write (*,*) 'string :',string,' : iostat=',iostat
  READ(in, POS=20, iostat=iostat) n
  write (*,*) 'number :',n,' : iostat=',iostat

  do k = 1,2
    write (*,*) ' '
    write (*,*) 'Now read all bytes in file'
    write (*,*) ' Note FTN95 has ignored pos= for write'
    do n = 1,25
      read ( in, pos=n, iostat=iostat ) c
      write (*,*) n, ichar(c), iostat
     
      if ( iostat == -1 ) exit
    end do
   
    CLOSE(UNIT=in)
    OPEN(UNIT=in, FILE="ustream.demo", STATUS="OLD", ACCESS="STREAM", iostat=iostat)
    write (*,*) 'Reopen file to fix FTN95 iostat error'
  end do
 
  END subroutine readUstream
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Wed Nov 02, 2022 8:34 am    Post subject: Reply with quote

Thanks John.
Back to top
View user's profile Send private message AIM Address
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Wed Jul 12, 2023 2:44 am    Post subject: Reply with quote

Paul,
While previous FTN95 version indeed was fixed and writes larger real*4 arrays > 2GB (see the write demo above which allocates 16GB with nB=4e8), the latest version of FTN95 fails with anything larger than 2GB. Reading larger REAL*4 arrays >2GB did not work with any FTN95 versions.

Here is new demo which designed to work for read and write as you can see with the arrays below 2GB (nB=3.e7) but it fails with anything larger than that. It creates and then reads the array using two methods: as Arr4(:,i) or as Arr4
Code:
! compilation: ftn95 aaa.f90 /link /64 >z
!
    real*4, dimension(:,:), allocatable :: Arr4

    nA = 11
    nB = 2.e8

!...Allocating array

    Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
    allocate ( Arr4 (nA, nB), stat = ierr)

    if (ierr.eq.0) then
       Print*,'Allocation success'
    else
       Pause 'Fail to allocate'
       goto 1000
    endif

!...Filling the array with some data
    do i=1,nB
      Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
    enddo

    Print*,'Trying to save the data Method 1 '
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    do i=1,nB
      write(11,err=910) Arr4(:,i)
    enddo
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'Write OK. Speed of write Method 1 =', SpeedGBps   !   typically  ~0.5 GB/s
      print*,'====================='
      print*,'================ N O W    R E A D ===================='
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    do i=1,nB
      read(11,err=912,end=914) Arr4(:,i)
    enddo
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'READ OK. Speed of read Method 1 =', SpeedGBps   
      print*,'============================='

    Print*,'Trying to save the data Method 2'
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
      write(11,err=920) Arr4
    close(11)   
    call cpu_time(t1)

!...Speeed of writing Method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'Write OK.  Speed of write  Method 2=', SpeedGBps   !   typically  ~2.6 GB/s
      print*,'====================='
      print*,'================ N O W    R E A D =================='
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
      read(11,err=930,end=932) Arr4
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'READ OK. Speed of read   Method 2 =', SpeedGBps   
      print*,'======================'

      pause 'File LargeFile.dat created OK'
      goto 1000

!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
    goto 1000

910 Print*,'Error. Can not save file LargeFile.dat Method 1'
    goto 1000
912 Print*,'Error. Can not load file LargeFile.dat Method 1'
    goto 1000
914 Print*,'Abruptly tnd of file. Can not load file LargeFile.dat Method 1'
    goto 1000

920 Print*,'Error. Can not write file LargeFile.dat Method 2'
    goto 1000
930 Print*,'Error. Can not read file LargeFile.dat Method 2'
    goto 1000
932 Print*,'Abruptly end of file. Can not read file LargeFile.dat Method 2'
    pause
1000 Continue
    End


Last edited by DanRRight on Wed Jul 12, 2023 10:33 pm; edited 1 time in total
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Wed Jul 12, 2023 6:41 am    Post subject: Reply with quote

Thank you for the feedback. I have made a note of this issue.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Thu Jul 13, 2023 5:12 am    Post subject: Reply with quote

I think the status of FTN95 /64 in Nov 2022 was that you could write records longer than 2 GBytes, but they could not be read. ( The I/O software supported writing the buffer, but the header/footer syntax did not work)
The problem is that the 4-byte header and footer (actually 5-byte) is not able to identify the record length.

The only way to fix this problem using the FTN95 approach, is to introduce a 13-byte header and footer, which would consist of:
1-byte = -1 : 4-bytes = -1 : 8-bytes = record length.

At present FTN95 supports the 1-byte = -1 to support records longer than 240 bytes. It now needs to support the 4-bytes = -1 to indicate a further 8-byte header is provided for records longer than 2^31-(say 16).

An alternative approach is to adopt the 4-byte sub-record approach of Intel and Gfortran, which adjusts the larger record as multiple 2^31-9 long sub records, where the sub record lengths are given as -ve to indicate there are repeated sub records. This approach is documented on the Intel Fortran web site. ( there is no support for small records < 240 bytes )

FTN95 can not support larger records without a change to the existing 5-byte syntax.

A problem with the Intel/Gfortran approach is that the sub records are not a multiple of 4 bytes (or 8 or 16) but the combined length of 4 bytes + 2^31-9 + 4 = 2^31-1, which is the largest +ve 4-byte integer. This odd length creates problems addressing memory using 4-byte or 8-byte integer arrays. Hence my recomendation of 2^31-16 a a sub-record length makes it easier to write the library in Fortran.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
Page 4 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group