forums.silverfrost.com

DanRRight · Posted: Fri Oct 28, 2022 9:51 pm Post subject:

Just ran this test on ifort and gfortran. They work with files >4GB and speed of ifort is even 2.5x larger than anything we reported so far with any other compiler - 6GB/s with Method2. Of course FTN95 is still record king using proprietary READF@ and EQUIVALENCE to encode and decode data from READF@/WRITEF@ stream of data (i was getting 7-8 GB/s there) but that's different approach, and though it is simple but not as ultimately simple, standard, portable and straightforward as the one we discuss today. Intel though clearly made an error somewhere with Method1 because with it they got speed ... 200x less. That is like the speed of formatted I/O! Looks like they forgot napkins and scissors in the body during their surgery. Please use RAM drives or m2 Gen4 NVMe drives when testing to squeeze last bit of speed from software and for not missing something like Intel !

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

PaulLaidler · Posted: Sat Oct 29, 2022 7:43 am Post subject:

John

Thanks. I will take a look at this.

PaulLaidler · Posted: Sat Oct 29, 2022 3:29 pm Post subject:

Dan

The FTN95 I/O has now been fixed so that in future it will run your program aaa.f90.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

Paul,

I have been investigating further, especially the problem of the file being truncated after a re-write.

I have also tidied up the error tests and messages, although still not complete. (see error_info)

I have also included a fifth test : sequential read; that checks the size and position of all records. ( This might not always work in general for my IO library, as records can be re-written as smaller records, meaning records are then only direct accessible by their file address.)

https://www.dropbox.com/s/1q2pyge5zgb9ezz/mslib_t2.f90?dl=0
https://www.dropbox.com/s/m8v905qtdu7fc1s/mslib_t2.log?dl=0
https://www.dropbox.com/s/cap59y3pc2jy67l/do_test.bat?dl=0

I am confused by the file truncation after a re-write, as it does not always occur.
I am not sure of the rules for truncation or what attributes in the OPEN may control this.
I understand sequential formatted text files are truncated, but not sure what conditions apply for unformatted stream I/O files ?
I would expect stream access files should not be truncated, even if "pos=" is not used, so used as a sequential access only file.

This attached test has also changed the record sizes (line 323) and now shows the "INQUIRE ( pos=" error, (problem 2 above). As implemented, it appears to report the current end of file position, rather than the present file position. (This also helps identify when the file is truncated ? )
In test 1: sequential write, the returned INQUIRE file position is confusing, being less that the expected value. It looks like a delayed file update (asynchronous ?)
I would expect pos= should return the "current position", not the end of file position, including after a READ statement. I already calculate the "current position", independent of INQUIRE, so this problem is not important for me.

With the attached program, once a file iostat = -1 occurs (tries to read past eof), all following file access gives iostat=104, even if pos= positions to before the end of file. This can be more clearly demonstrated by using line 477, rather than 478, so records past the truncated fiile are referenced. When this occurs, all records in the following sequential read fail. (I don't agree with this for stream access).

DanRRight · Posted: Mon Oct 31, 2022 1:17 pm Post subject:

I need APPEND to the file defined as a STREAM. Both APPEND and STREAM use keyword ACCESS, i.e. ACCESS='APPEND' and ACCESS='STREAM'. The FTN95 uses for APPEND keyword STATUS='APPEND' and the problem is solved.

But the gFortran does not like any of that, it does not like two ACCESS keywords in the same place, does not like STATUS='APPEND' and also does not like POSITION='APPEND'. What's to do with this damn picky compiler? Is it strictly following Standard or just missing feature?

PaulLaidler · Posted: Mon Oct 31, 2022 4:27 pm Post subject:

POSITION = 'APPEND' is in the Fortran 90 standard.

STATUS = 'APPEND' is not in the Fortran 90 standard but is allowed as an FTN95 extension to the Standard.

I don't know if 'APPEND' works for FTN95 ACCESS = 'STREAM' (or if it is meaningful in that context).

DanRRight · Posted: Mon Oct 31, 2022 8:38 pm Post subject:

Meaningful implies STREAM files could only being rewritten, you can not append new data to them?

DanRRight · Posted: Tue Nov 01, 2022 12:16 am Post subject:

Problem solved. gFortran gave the false positive in wrong place. POSITION='APPEND' works, it was an extra ' at the end like this POSITION='APPEND''. In this situation FTN95 was spot on pointing (by words only unfortunately) on exact error place, while gFortran pointed out by finger on the letter P in POSITION saying this is inappropriate in this content.

This always rises for me rhetoric question - how all these sh#tty in the past compilers overjumped the leader of 1990th FTN95 in modern features so we now have to move there because FTN95 lags currently by not years but decades at parallelization and F2003/2008?

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

Paul, this is another smaller example, modified from
https://fortranwiki.org/fortran/show/Stream+Input+Output

Running in Plato with (ALT+X) FTN95 or gfortran shows some problems.
write (pos= is ignored in write, if past EOF
read (pos= does not recover from IOSTAT /= 0, until file re-opened

PaulLaidler · Posted: Wed Nov 02, 2022 8:34 am Post subject:

Thanks John.

DanRRight · Posted: Wed Jul 12, 2023 2:44 am Post subject:

Paul,
While previous FTN95 version indeed was fixed and writes larger real*4 arrays > 2GB (see the write demo above which allocates 16GB with nB=4e8), the latest version of FTN95 fails with anything larger than 2GB. Reading larger REAL*4 arrays >2GB did not work with any FTN95 versions.

Here is new demo which designed to work for read and write as you can see with the arrays below 2GB (nB=3.e7) but it fails with anything larger than that. It creates and then reads the array using two methods: as Arr4(:,i) or as Arr4

PaulLaidler · Posted: Wed Jul 12, 2023 6:41 am Post subject:

Thank you for the feedback. I have made a note of this issue.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

I think the status of FTN95 /64 in Nov 2022 was that you could write records longer than 2 GBytes, but they could not be read. ( The I/O software supported writing the buffer, but the header/footer syntax did not work)
The problem is that the 4-byte header and footer (actually 5-byte) is not able to identify the record length.

The only way to fix this problem using the FTN95 approach, is to introduce a 13-byte header and footer, which would consist of:
1-byte = -1 : 4-bytes = -1 : 8-bytes = record length.

At present FTN95 supports the 1-byte = -1 to support records longer than 240 bytes. It now needs to support the 4-bytes = -1 to indicate a further 8-byte header is provided for records longer than 2^31-(say 16).

An alternative approach is to adopt the 4-byte sub-record approach of Intel and Gfortran, which adjusts the larger record as multiple 2^31-9 long sub records, where the sub record lengths are given as -ve to indicate there are repeated sub records. This approach is documented on the Intel Fortran web site. ( there is no support for small records < 240 bytes )

FTN95 can not support larger records without a change to the existing 5-byte syntax.

A problem with the Intel/Gfortran approach is that the sub records are not a multiple of 4 bytes (or 8 or 16) but the combined length of 4 bytes + 2^31-9 + 4 = 2^31-1, which is the largest +ve 4-byte integer. This odd length creates problems addressing memory using 4-byte or 8-byte integer arrays. Hence my recomendation of 2^31-16 a a sub-record length makes it easier to write the library in Fortran.