forums.silverfrost.com

DanRRight · Posted: Mon Sep 26, 2022 2:13 am Post subject: Fails to save arrays > 4GB

There exist two major methods to save arrays. Suppose we have 2D REAL*4 array Arr4(nA,nB). First method is when we explicitly save each element or each dimension individually

PaulLaidler · Posted: Mon Sep 26, 2022 7:44 am Post subject:

Dan

Thanks for the feedback. I have made a note that this needs investigating.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

my understanding of unformatted binary records is that FTN95 does not support records longer than about 2^31 bytes ( ie the header is byte 1 = -1 and bytes 2:5 must be a positive 4-byte integer )

Both ifort and gfortran support a complex record header/footer structure of sub records, which I recall are 2^31-9 bytes in size. the header/footer is 4-bytes long. The sub records are identified by -ve sub record sizes, enclosed by +ve sub record size for first and last sub records.

Does FTN95 Stream I/O support integer*8 addressing, as that could be the simplest to implement ?

Alternatively, write the array as a group of records which are less than 2^31 bytes long, choosing "Arr4(:,i)" so that it is < 2^31 bytes.
This will be the most practical and "portable" solution.
Stream I/O is portable between FTN95 and ifort/gfortran, as is fixed length direct access files ( again with records smaller than 2^31 bytes )
( FTN95 unformatted records are not portable )

PaulLaidler · Posted: Tue Sep 27, 2022 9:24 am Post subject:

John

Can you explain what you mean by "Stream I/O integer*8 addressing"?

For 64 bit FTN95 the I/O library is in clearwin64.dll which is 64 bits.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

PaulLaidler · Posted: Wed Sep 28, 2022 7:07 am Post subject:

John

Stream I/O uses a nominal record length with value one. This could be compiler dependent but probably isn't.

I am assuming that the size of a file will not be a problem (for 64 bit FTN95).

DanRRight · Posted: Fri Sep 30, 2022 9:39 pm Post subject:

Yes, ideally files created with any of these two methods have to be readable by each of these methods. But if this is not the case among the whole line of 12-13 Fortran compilers then it's OK not to be method-independent.

Currently if you create the file with Method1 it only can be read back with the same Method1, and Method2 - with Method2. Not super bad inconvenience, it is possible to make a note in the file which method was used and then read the file.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

Dan,

Part of the problem is not clearly understanding the limitations of each binary file type/format ( unformatted sequential, direct access fixed length records or now, stream access)

Paul,

I have been successfully using Fotran direct access binary files (fixed length records) for many years as a portable interface between different Fortran compilers.

Stream I/O now provides a similar portable access by,
a) allowing I/O lists of intrinsic types ( I/O lists are much more flexible than a single vector )
b) able to create header and footer labels to records, to be compatible with other compiler's unformatted sequential binary file formats.
c) provides random/direct addressing to files ( although I have had some problems with integer*8 addressing up to FTN95 Ver 8.8, which may now be fixed. I should find my past examples and retest with Ver 8.9 )

It would be good if the following could be investigated and possibly supported for unformatted sequential files:
a) an alternative could be provided to support other compilers unformatted binary file formats, via a compile option or in the OPEN statement.
b) confirm if records or files larger than 2^31 bytes or "words" could be supported.
c) investigate support for derived type records to be used in I/O lists, rather than specifying as a list of intrinsic type components.

However, stream I/O now provides a very flexible and portable alternative to earlier unformatted file approaches.

PaulLaidler · Posted: Sat Oct 01, 2022 10:41 am Post subject:

John

Thanks for the feedback. Regarding your request for further investigations:

a) I don't see how we could offer to do this even for just one other compiler.

b) Do you have any experience that this does not work?

c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

Paul,

a) I don't see how we could offer to do this even for just one other compiler.

Answer: addressing only records smaller than 2gb; both ifort and gfortran use a 4-byte header/footer for unformatted sequential files. They are portable for this file type.
If FTN95 could have a (say) /4_byte_header option, (unlike the present 1-byte or 5-byte default format), this would improve portability for gfortran and ifort interface.

b) Do you have any experience that this does not work?

Answer: I provided a stream I/O example in:
http://forums.silverfrost.com/viewtopic.php?t=4565&highlight=stream

I have now updated this program to better document the test.

https://www.dropbox.com/s/r6cy1xak6hp2msa/stream_read2.f90?dl=0
https://www.dropbox.com/s/wnj6pu7cq2vxapw/stream_read.log?dl=0

It works with gfortran (in Plato) but FTN95 does not support
"write ( unit=stream_unit, pos=address ) values" or
"read ( unit=stream_unit, pos=address ) one_byte"
the value "address" does not work in FTN95 and "INQUIRE ( unit, pos=address )" does not appear to give the same value as gfortran.

The file stream_read.log now provides a report of the tests for gfortran Ver 11.1 and FTN95 Ver 8.80

c) This may be possible but we would need more details, a sample program and what you would expect it to do; perhaps a program that already works in the way you want using gFortran or iFort.

integer*8 address will then need to be checked after b) is addressed.

I will test the program with integer*8 address and check it still works.

( I think I have provided examples of this before, but we first need "pos=" to be supported in FTN95 )

I will download Ver 8.90 and test.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

A further update to stream_read3.f90

this reports the address after writing the record.
It shows that for:

PaulLaidler · Posted: Tue Oct 04, 2022 7:39 am Post subject:

John

I will make a note of this and see what can be done.

PaulLaidler · Posted: Thu Oct 06, 2022 10:38 am Post subject:

The original failure on this thread (reported by DanRRight) has now been fixed for the next release of clearwin64.dll.

The change has an impact on FTN95 so it is possible that users may also need the next release of FTN95 (the one after v8.91). In other words, it is unlikely that you will also need a new FTN95.

JohnCampbell · Joined: 16 Feb 2006 Posts: 2593 Location: Sydney

Paul,

Can you please provide some clarification for what error was fixed ?

Did it relate to the following code where an array larger than 2GBytes is written as a sequential unformatted file { OPEN(access='sequential', FORM='UNFORMATTED',...) }

PaulLaidler · Posted: Fri Oct 07, 2022 7:22 am Post subject:

John

As I understand it, there is no header or footer for stream I/O.

The aim is to look at all the issues that you have raised to see what is possible within our resources. But it is on a list of things to do.