forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Fails to save arrays > 4GB
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support
View previous topic :: View next topic  
Author Message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Tue Oct 25, 2022 7:13 am    Post subject: Reply with quote

Paul,

I have downloaded FTN95 Ver 8.92 and tested the latest stream_readc.f95

This version now appears to work correctly for write ( lu, pos=address) ... when overwriting the file.

All 3 files that are generated from the 3 tests now appear to be the same.

Thanks very much for this update.

I will now try to perform a test for overwrite for a file larger than 4 gbytes ( using integer*8 :: address )
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Thu Oct 27, 2022 5:41 am    Post subject: Reply with quote

Write was fixed. Now same problem with READ. Does not work if array > 4 GB...
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Thu Oct 27, 2022 5:45 am    Post subject: Reply with quote

Code:
! compilation: ftn95 aaa.f90 /link /64 >z
!
    real*4, dimension(:,:), allocatable :: Arr4

    nA = 11
    nB = 1.e8

!...Allocating array
    Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
    allocate ( Arr4 (nA, nB), stat = ierr)

    if (ierr.eq.0) then
       Print*,'Allocation success'
    else
       Pause 'Fail to allocate'
       goto 1000
    endif

!...Filling the array with some data
    do i=1,nB
      Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
    enddo

    Print*,'Trying to save the data Method 1 '
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    do i=1,nB
      write(11,err=910) Arr4(:,i)
    enddo
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'Write OK. Speed of write Method 1 =', SpeedGBps   !   typically  ~0.5 GB/s

            print*,'================ N O W    R E A D ===================='
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
    do i=1,nB
      read(11,err=912,end=914) Arr4(:,i)
    enddo
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 1
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'READ OK. Speed of read Method 1 =', SpeedGBps   

          Print*,'Trying to save the data Method 2'
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
      write(11,err=920) Arr4
    close(11)   
    call cpu_time(t1)

!...Speeed of writing Method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'Write OK.  Speed of write  Method 2=', SpeedGBps   !   typically  ~2.6 GB/s
            print*,'================ N O W    R E A D =================='
    call cpu_time(t0)
    open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
      read(11,err=930,end=932) Arr4
    close(11)   
    call cpu_time(t1)

!...Speeed of writing method 2
      SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
      print*,'READ OK. Speed of read   Method 2 =', SpeedGBps   
     
      pause 'File LargeFile.dat created OK'
      goto 1000

!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
    goto 1000

910 Print*,'Error. Can not save file LargeFile.dat Method 1'
    goto 1000
912 Print*,'Error. Can not load file LargeFile.dat Method 1'
    goto 1000
914 Print*,'Abruptly tnd of file  LargeFile.dat Method 1'
    goto 1000


920 Print*,'Error. Can not write file LargeFile.dat Method 2'
    goto 1000
930 Print*,'Error. Can not read file LargeFile.dat Method 2'
    goto 1000
932 Print*,'Abruptly end of file  LargeFile.dat Method 2'
    pause

1000 Continue
    End
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Thu Oct 27, 2022 7:08 am    Post subject: Reply with quote

I have made a note of this issue.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Thu Oct 27, 2022 1:28 pm    Post subject: Reply with quote

Paul,

I am trying to go to the next step with using stream I/O to generate a random access binary file library for variable length records.

I am finding a problem with the returned file position after a random access read, ie
read ( unit=lu, pos=address, iostat=iostat ) (rec_data(i),i=1,n)
inquire ( unit=lu, pos=new_address, iostat=iostat )

Inquire appears to be returning the new_address after the last write, rather than the address after this last read. For this read test, the returned address is always the end of the file (after have previously written all records sequentially)

I could send the test program, if that helps, although it is very much an alpha version.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Thu Oct 27, 2022 11:30 pm    Post subject: Reply with quote

John,
Are these tricks you are developing are just to make an interface between gFortran you now use for simulations and Clearwin to plot the results and control the runs?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Fri Oct 28, 2022 1:18 am    Post subject: Reply with quote

Dan,

At present I use Fortran fixed length record random access binary file format for my binary file data. Fortunately, it has been compiler independent, as I have used many Fortran compilers.

(I wrote this library in 1970's to emulate the CDC random access file structure, using "standard" fortran. I say "F77 standard" as the approach is to provide the "record" address in memory and the length in 4-byte words. I can get a lot of mixed type errors! )

This file is based 4-byte addressing and is limited to 8 GByte file size.
This includes a record buffering approach as the "program" variable size records are transferred to the 64kbyte fixed length file record.

I have always wanted to update the approach by:
1) extend the file size limit by converting to an 8-byte addressing and,
2) remove the duplicate buffering, as Windows also provides very effective file buffering using free memory.
3) testing different logical record header/footer formats to overcome the long standing portability problems that FTN95 (1 or 5 bytes), gfortran and ifort have ( the mess with records larger than 2 gbytes )
4) expanding the header/footer format to include data type and kind ?

The other feature I am also looking to address is to include the flexibility of a Fortran I/O list for the record, rather than a record memory address. This could possibly include a derived type record. However, I am very use to the F77 approach of building a record using EQUIVALENCE or TRANSFER, so a modern Fortran approach to this can wait.

I am not sure about the need for header/footer labeling for records, as they were initially provided for BACKSPACE. Does anyone still use this ?

All this has become a bit academic with 64-bit, as I have transferred most of my binary file data to an in-memory derived type data structure using allocatable components. This data can now be transferred between file at the start/end of a run, as a sequential dump, simplifying the indexing structure.

Defining my own binary file data structure using stream I/O removes many of the portability, performance and capacity issues.
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Fri Oct 28, 2022 1:54 am    Post subject: Reply with quote

John,
But i am curious what for you need all that? It does not resonate with any my needs no matter how hard i try to imagine any future trends and progress of computing Smile
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Fri Oct 28, 2022 2:42 am    Post subject: Reply with quote

Dan,

Very simply: My MSLIB ( random access file library ) from 1970's uses 4-byte addressing, which is limited to 8 GByte file sizes.
So I need to a new library with 8-byte addressing, for larger files (and 8-byte address tables in the program)
I must use new routine names so I don't miss updating some old routine usage.

Why not change to Stream I/O at the same time, to see if it offers any other functional advantages ?

Hence my interest in confirming stream I/O works in FTN95 for large files and confirm in a way I expect.

( Happy to learn if it is different from what I expect as I don't always RTFS. Has anyone actually tried to read the latest Fortran standard? Approaching 1,000 pages !! )
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Fri Oct 28, 2022 6:41 am    Post subject: Reply with quote

I opened it for few seconds recently. Decently, i did not RTFS even one for Fortran95. Like probably we all here.

Please read it, you are our the only hope. You were the only who correctly felt the missing features years ago and were bringing this info here for our attention. For example 5 years ago on Sep 12 you wrote

Using gFortran provides a number of advantages, which FTN95 is not expected to address in the near future.
# Inclusion of F03 and F08 standard features.
# !$OMP support with -fopenmp
# Extensive vector instruction support, eg -O3 -mavx -march=native -ffast-math


At that time i was deaf and was not listening but you were right at saying that. Today i suffer from missing features hard. Tomorrow others will.
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Fri Oct 28, 2022 7:09 am    Post subject: Reply with quote

John

Can you send me a short demo program that illustrates what you would like to happen?
Back to top
View user's profile Send private message AIM Address
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Fri Oct 28, 2022 1:52 pm    Post subject: Reply with quote

Dan

I have looked at your program aaa.f90.

The result of "read(11,err=930,end=932) Arr4" at line 63 is a call to the Microsoft API ReadFile where the number of bytes to read is an unsigned 32 bit integer.

So at the moment you can't read very large chunks in one go.

This can be fixed internally by buffering the input but at the moment you will need to do the buffering yourself as in your Method 1.
Back to top
View user's profile Send private message AIM Address
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Fri Oct 28, 2022 3:47 pm    Post subject: Reply with quote

This read issue has now been fixed to the degree that you will get an explicit runtime error when trying to read very large chunks of data in one go.

In the longer term we could provide a fix by adding internal buffering but this would not help with regard to the stated object of reducing the runtime by using large chunks.
Back to top
View user's profile Send private message AIM Address
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Fri Oct 28, 2022 4:20 pm    Post subject: Reply with quote

Paul, Thanks. Did i understand this correctly that this is Microsoft issue?
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Fri Oct 28, 2022 8:22 pm    Post subject: Reply with quote

The Microsoft ReadFile function is limited and as far as I know there is no direct alternative.

Work on internal buffering is almost complete so, when finished, externally there will be no limit.
Back to top
View user's profile Send private message AIM Address
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
Page 3 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group