forums.silverfrost.com

[JohnCampbell]

You are wrong, as access=direct requires recl=n

You have some options for record structure of a binary file, defined by ACCESS=
ACCESS='DIRECT', RECL=n provides a fixed length record structure with record lengths of n bytes. You can write multiple records with a single WRITE statement, so selecting RECL=1 (or 4 or 8) can achieve this purpose.
ACCESS='SEQUENTIAL' will write "records" which are enclosed in a size indicator. This size indicator can vary between compilers and for FTN95 depending on the record length. (others know this byte pattern) If you are writing and reading the file sequentially, then this can be a good option.
ACCESS='TRANSPARENT' is FTN95's version of F03 stream I/O. You have the ability to write variable length records and use the rec= to randomly position in the file.
ACCESS=transparent is like access=direct, recl=1. I think the use of rec= in READ and WRITE is optional.

Be warned that for recl=n; "n" is not always in bytes but can be words with some other annoying compilers. The F90 standard let us down here.

You should experiment with these options first. With access=transparent, I am not sure how to get the position. It may be with INQUIRE (unit=lu,nextrec=pos. Otherwise you can calculate this position!
You can open any file with access=transparent and read each byte. This will give you the structure of formatted or access=sequential files.

I have always preferred access=direct as it is an F95 standard approach.
If you are writing variable length records, I'd recommend ACCESS=DIRECT, RECL=4 or 8, then experiment with INQUIRE, nextrec= to get the record position. This should work very well.

I think there is a file size limit of 2^31 records > 16gb for recl=8. I have certainly worked with files up to 16gb. (I have not used integer*8 position for rec=position, but it may work)
I am not sure of the file size limit for ACCESS='SEQUENTIAL' I don't know what the limit is, but certainly more than 8gb.

John

[mecej4]

You should not use ACCESS='DIRECT' for unformatted files when one or more of the following conditions apply:

(i) the record size is not known in advance and you do not wish to calculate it on the fly.

(ii) you wish to write more than one record to the file and the records will not all be the same length.

(iii) you plan to access the file sequentially.

You can specify a record size of 1 byte when opening a direct-access unformatted file, but that is a tactic to use when you do not know the structure of an existing file and you wish to poke around in the file.

If your intention is to write one or more arrays in a single WRITE statement to the file, that file will have only one logical record, so there is no logical difference between direct access and sequential access. However, with ACCESS='DIRECT' you do not have the protection that the record-length bytes in the unformatted sequential file give you to avoid errors.

[JohnCampbell]

This is an example of access=transparent for reading a text file.

[JohnCampbell]

Mecej4,

The big advantage of access=direct is that there are no record headers. This provides a more portable binary file format for moving between programs compiled with different compilers.
I have again forgotten the tricks for access=sequential to overcome the different record header/trailer formats.
I only use access=transparent when I want to interrogate a file sequentially and manage varying record header/trailer or endian problems, eg reading unix text files.

It depends on the file history

[JohnCampbell]

Unfortunately, you can not write multiple records with a single write statement using FTN95 or gFortran, so you would have to place the write in a do loop. ( Sorry, I thought that was possible)
Anyway, you can have a record of 8_million bytes or 8 bytes and read/write the array

[JohnCampbell]

Also, it appears that you can not use transparent with direct access, ie using rec= on read or write. The following updated code shows a mix of direct access write, then sequential transparent read. ( my apologies as this is probably more than you needed as an answer)

[mecej4]

Fortran is a higher level language (vis-a-vis, say, C) and, in such a language, the concept of a file and the contents of the file are abstract. In such a framework, everything works fine if one uses the terminology of the language ('record','end-of-record','end-of-file', 'direct-access','record-length', etc.) and does not attempt to interpret these entities in terms of the (low level) attributes of the physical device.

Much of the terminology and Fortran I/O conventions come to us from old technology (tape drives, DASD-s), whereas the most common file-systems today (Unix/Linux/Android, Windows) regard each file as a collection of bytes. Whether a file contains meta-data only, data only, or a combination of data and meta-data is something that has to be resolved in the application program, and is not inherent to the OS view of the device. There is one exception to this, and that is the convention regarding text-files, which have CR, LF or CR+LF bytes that are given special significance.

With such a disconnect between the logical view of the storage device and its physical characteristics, lots of confusion and errors can arise. For example, if the keyboard is the I/O device, can I open it for direct access? If I can, can I read with REC=10 followed by REC=9? What if I have I/O buffered?

Returning the topic of this thread: My view is as follows. A direct access unformatted Fortran file is a collection of bytes with no metadata stored in the file. When one writes a record to a direct access unformatted file opened with RECL=m and specifies REC=n in the WRITE statement, byte no. m X (n-1) is the first byte written, and the number of bytes read is the size in bytes of the I/O list in the WRITE statement. (Some compilers, such as Intel Fortran, have a default "file-storage-unit" of 4 bytes, but most compilers use a file-storage-unit of 1 byte.) It is the product m(n-1) that matters -- whether one writes record 5 with a record length of 3, or writes record 4 with a record length of 4, the file contents (bytes 12 and up) will be identical. Once the file has been written, it is impossible to determine the record length or the number of records it contains from the file contents alone -- in fact, there are no physical records in it at all. Given such a file, it is then reasonable to open it for direct access with RECL=1, read the contents and attempt to discern some recognizable pattern from the first few bytes of the file, and then reopen the file with an appropriate RECL. For an example, see the binary .MAT files that Matlab uses for data interchange with other software.

[mecej4]

Here is a small test program to illustrate the points that I brought up in the preceding post (in particular, the point about not having physical records, record marks, etc., in a direct access unformatted file). Note that the file is written with RECL=24 and then read with RECL=16.

[LitusSaxonicum]

Mecej4,

I thought your comments were most insightful. Nowhere is this 'nodding in the direction of old technology' more apparent than in the assumption of 80 character punched cards that pervaded Fortran books until quite recently. It was hard enough to map this to punched tape input in 1970, especially as the cards appeared to have a 6-bit character set, and tapes 7 (with an 8th for parity checks), but now they have disappeared from everywhere - except perhaps the US voting system?

'Old technology' lurks in the DNA of every evolved system, and while what you speak of relates to pre-PC computers, don't forget all the DOS relics, like devices called LPT: COM: and AUX: which haunt us today, and also 8, 16, 32 and now 64-bittedness.

One has only to look at one's own vestigal organs and how they create trouble: wisdom teeth, appendixes and so on, to see the trouble that the analogous 'vestigial organs' cause in Fortran.

[JohnCampbell]

mecej4,

Although a direct access file is just a sequence of bytes for the data written, with no record structure description, I think you may have overstated the abstract nature of direct access files. For an existing direct access file the record length must be a factor of the file size. Your example of a 48 byte file, the possible record lengths must be a factor of 48, so 48, 24, 16, 12, 8, 6, 4, 3, 2 or 1 are the only record lengths possible. If the file length is an odd number the possible lengths can be fewer. This is important when opening the file, if the file size is not a multiple of the record length, then OPEN will return an error.
The other point is that you need to know something of the content of the file. Your example above reads bytes 33:48 into a real. However if you used "read(12,rec=2)x" ; then the values in real x(1) and x(2) would contain integer values of f(5) and f(6).

The use of unformatted sequential access files has problems as each record has a (compiler dependent) size header and trailer. FTN95 used a 1 byte length header/trailer for records of length 1-255 bytes and a 5-byte header/trailer for records of length more than 255 bytes. Most other compilers use a 4-byte header/trailer ( I am not sure what happens when the record length exceeds 2gb, even for FTN95_64 )

All these problems can be managed with transparent access, which has been called access=stream in the F2003 standard.

The problem of record formats is less of an issue if the file is being used by the same program or programs compiled by the same compiler and same compiler settings. When moving data between computer systems, I prefer a text based format (say .csv) so the data is more easily reviewed. Consideration of precision of real variables is an issue.

[mecej4]

John,

Thanks for your contribution to the discussion of direct access unformatted files.

[JohnCampbell]

mecej4,

Some interesting but puzzling points you have selected. You had to go to F2008 to get them.

7 On output... surprises me as I am sure I have seen examples of writing multiple records with a single write statement. Was it Lahey or ifort prior to F08 ?

Why would the standard change the requirement that the record length must be a factor of the file size ? It looked like a fairly sensible rule for practical access to the data. It is disappointing the standards committee have put effort into changing something like this but ignored other extensions such as RESIZE to ALLOCATE that I have used as non-standard approaches since 70's. Or they could have provided a standard way of interpreting a comma in text input, like FTN95 has always supported.

I love the "file storage units" terminology. It smells of the abstract nature you describe. By F2008, how many Fortran compilers are there that can not support the concept of a byte ? Portability is lost with this approach ! It is now one of the first errors I find when transferring code.

[mecej4]

John,

I was mildly surprised when I read your statement that the record-length of a direct access unformatted file should be an exact divisor of the file size. I have now looked up the Fortran 95 standard (Final Draft), and I see only the following text in 9.3.4.5 of J3/97-007R2:

For an existing file, the value of the RECL=specifier shall be included in the set of allowed record lengths for the file. For a new file, the processor creates the file with a set of allowed record lengths that includes the specified value.

Thus, the standard does not require but does allow the highly restrictive rule used by FTN95. Nor can I find an explicit statement of this rule in the FTN95 documentation.

Adding an IOSTAT= clause to line-14 of the code of Fri Jan 29, 2016 6:37 pm shows that IOSTAT acquires the value 108.

Peculiarly, if the size of a file (in bytes) is a prime number, FTN95 allows the file to be opened only with RECL=1 or RECL=filesize. I am not aware of any other Fortran compiler that takes the same restrictive stance.

[JohnCampbell]

I still agree with the FTN95 approach. If the file was successfully created, then the size should be a multiple of the record size. If it is not, then either you have the record length wrong or the file was not completed correctly. Using ifort with the wrong record size selected can do this the first time you run a program when transferred to the ifort compiler.

In this case, you could always use stream access or recl=1 to interrogate the file. I think the error reports that you are going to have problems with the content of the file.

In the past I have had this error, which was because I changed the record length of a direct access database file, which would have not worked when opened with the new default length.

Referring to your point 7; if you select to write a record shorter that recl, most compilers I have used will pad the record. Although these contents are undefined, they are provided in the file, giving a correct file size.

Again if you randomly write a direct access file, all compilers I have used will not create virtual records, but will fill out the file with undefined (not necessarily zero) information. (Do you know of any compilers that generate a virtual file in this case ?) As this action is typically buffered, it may not cause a big problem. If the random record order exceeds the file buffer, I would suggest it is either a poor approach or an overhead that must be tolerated.

[mecej4]

I seldom use direct access files; when I need to process unstructured binary files I use C code.

The Fortran standard leaves unspecified the "set of allowed record lengths for the file" for an existing file. FTN95 does not provide (or I have not discovered the hidden information) a means of inquiring the I/O system to find these allowable RECLs or to ascertain whether a particular integer value is one of these allowable values.

Many of your comments convey an expectation that a direct access file opened for reading was written by a Fortran program as a direct access file. In real life, the file could have been written on a different system and by using a language other than Fortran. Consider HDF and NetCDF as file formats that do not have uniform-size records, and need to be read in a Fortran program.

I don't understand what you mean by "randomly write a direct-access file". The very purpose of the REC= clause of a WRITE statement is to be able to write records in any desired order, i.e, "random". Similarly, I don't know what you mean by "virtual record". There is one instance of "virtual" in the F2008 standard, and that one is "virtual memory". References/definitions, please?