forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Are Intel compiler unformatted files readable by FTN95?

 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Jul 10, 2022 1:24 pm    Post subject: Are Intel compiler unformatted files readable by FTN95? Reply with quote

The files created by gcc on Linux are readable by FTN95 on Windows no problem without any changes or conversions. The same files made by the Intel Fortran on Linux are not readable by FTN95 on Windows. Anyone had this problem?
Back to top
View user's profile Send private message
simon



Joined: 05 Jul 2006
Posts: 268

PostPosted: Mon Jul 11, 2022 12:59 am    Post subject: Reply with quote

Ian Lambley has posted something about this in at least a couple of places on the forum. Here's one:

http://forums.silverfrost.com/viewtopic.php?t=2722&highlight=endian
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Mon Jul 11, 2022 2:32 am    Post subject: Reply with quote

Thanks, Simon. If nothing else works, will try GFortran and if it will not work or will be slower than Intel, will return to READF@/WRITEF@ as it was suggested.

I used them before, but lately i just switched to the plain standard way of reading. Thanks to the numerous helpful discussions here of our great community i found that with the trick of reading/writing the entire array without mentioning any of its indexes as utterly simple as this
Code:

real, allocatable :: Array(:,:,:)

OPEN (100, form='unformatted'...
WRITE(100) Array
CLOSE...
.............................

OPEN (100 ...
READ (100) Array
CLOSE...


it is only 2x slower than READF@/WRITEF@ but with its warp 3GB per second speeds (and soon it will be 6GB/s with new PCIe 5 standard) it does not often matter. How not to buy such simplicity and speed? All that Matlabs, C/C++, Pythons, SDF, HDF5 and XDR formats nervously smocking outside.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Wed Jul 13, 2022 9:35 am    Post subject: Reply with quote

ifort and gfortran use a 4-byte record header/footer.
FTN95 use a 1-byte or 5-byte record header/footer. 1-byte formats are for records smaller than 256 bytes.
ifort use 4-byte header/footer for records up to 2^31 - 9 bytes. Larger records become much more complicated.

So, the binary unformatted files are NOT compatible.

There are two alternatives:
1) write a file tester then converter, using transparent/stream format. The advantage of this approach is that the I/O list structure is not required. This is a fairly simple approach, or
2) Ask FTN95 to provide a "4-byte-header" option in the OPEN statement, as you might need to open both ifort and FTN95 format record files.

I have probably written option 1) before, certainly using access=transparent to read one or 4-bytes to identify the record header and so determine the record_data. It is not a difficult conversion.
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sat Jul 16, 2022 8:08 pm    Post subject: Reply with quote

John,
Thanks, missed your post. I saw similar advises to use access=STREAM. I included it with Intel and gfortran compilers on Linux, and then after transferring files to Windows with FTN95 (both with the STREAM or without) still got that unformatted read does not work.

What is "transparent"?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Sun Jul 17, 2022 5:31 am    Post subject: Reply with quote

What is "transparent" ? Good question !

Access='transparent' is FTN95's earlier support for stream like access before Access='stream' now being supported by FTN95.

I have looked for a converter between ifort/gfortran and FTN95 unformatted sequential binary files, but could not find one. So I have now produced another "test" program convert_file_to_FTN95.f90 that achieves the result.
It can be compiled using FTN95 or gfortran, but the option "is_FTN95" is based on the file name ( see routine open_files) ( The program can run sucessfully in PLATO without command line options being provided )
It has the main functionality required but might not be robust for all file options.

https://www.dropbox.com/s/wom8lsaq1lbkbl8/convert_file_to_FTN95.f90?dl=0

https://www.dropbox.com/s/kxz6df4aqm15kh3/make_file.f90?dl=0
https://www.dropbox.com/s/pi636fynx2c7etr/count_char.f90?dl=0

When accessing FTN95 unformatted binary files, they can have either a 1-byte or 5-byte header and footer : 1-byte for records shorter than 255 bytes or byte-1=255(-1) + 4-byte record size for the header then 4-bytes + 1-byte for the trailer.
gfortran/ifort use a 4-byte for header and trailer for all record sizes up to 2^31 - 9 bytes ( need to confirm this, might be 2^32-9 ?)
They do have record formats for larger records, involving linked 1 or 2gb sub-records, but I think FTN95 does not support larger size records.

The program "make_file.f90" can be used to generate an unformatted binary test file, using either gfortran or FTN95 compiler.
I use " is_FTN95 = kind(x) /= 4" to test which compiler I am using when providing a file name.
Program convert_file_to_FTN95.f90 recognises the name of the file to identify the implied input file format (lazy approach which could be enhanced to possibly identify the compiler used from the first record, but should be a command line option! )

The program "count_char.f90" shows a use of stream access to identify what characters are in a text file and the DOS/UNIX text record format.

All programs use Access='stream' sequentially, so don't demonstrate the power of random access provided by stream I/O.

Quote:
The files created by gcc on Linux are readable by FTN95 on Windows no problem without any changes or conversions.

As far as I know, this outcome could only apply to fixed length record; direct access binary files, as these do not have header/footer labels. Unformatted binary sequential files created on Linux are unlikely to use FTN95 like header/footer. They would probably be compatible with files created by gcc on Windows. I am suspicious of this claim.
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Jul 17, 2022 9:34 pm    Post subject: Reply with quote

John,
Thanks. Will look at that. Preliminary, the first code - the converter - by some reason did not work with my files complaining about something. As a result the file size of converted file increased 3x, first few numbers of converted file were different but the rest were the same resembling zeroes. But my smaller size file seems was converted OK by your converter

Do i correctly understand that if we with FTN95 succeeded to read GFortran or Intel unformatted file which size is less than a quarter kilobytes, we could fail reading 1GB file?

And succeeding with FTN95 to read 1GB GFortran file we will again fail to read >2GB file ? All because headers/footers depend of file size

-----UPDATE -------------------
I'm currently totally lost....
This is not related to your code. When working with my own small test programs which are saving only 4 unformatted numbers, the Linux gFortran files with access='stream' are reading OK by FTN95 with access='stream'. And regular gFortran files without access='stream' also are freaking loading OK by FTN95 without access='stream'.

Here are my programs, the first one writes the file, second one reads it either on Linux or Windows and both work damn fine ! As i wrote, all works fine if we remove access=stream. The problem is why my larger code fails ?
Code:
!  Writing unformat file

    INTEGER, PARAMETER :: r4 = KIND(1.e0)
    REAL (r4) :: data(4)

    data = (/1.,2.,3.,4./)

    open ( unit=11, file='data_Stream.dat', form='unformatted', status='unknown', access='stream', err=900 )
    write(11) data
    close(11)
    goto 1000

!... Errors
900  Print*,'Can not open file'

1000 CONTINUE
    END


Code:
!  Reading unformat file

    INTEGER, PARAMETER :: r4 = KIND(1.e0)
    REAL (r4) :: data(4)

    open ( unit=11, file='data_Stream.dat', form='unformatted', status='unknown', access='stream', err=900 )
    read(11) data
    close(11)
    print*, data
!    pause

    goto 1000

!... Errors
900  Print*,'Can not open file'

1000 CONTINUE
    END


For anyone here who also wants to play with this - here is how it was compiled on Linux just in usual terminal mode:
Code:
mpif90 aaaW_Stream.f90  -o aaaW_Stream   ! compiling Writing code with  GNU Fortran
mpif90 aaaR_Stream.f90  -o aaaR_Stream   ! compiling Reading code with GNU Fortran

mpiifort aaaW_Stream.f90  -o aaaW_Stream   ! compiling Writing code with  Intel Fortran
mpiifort aaaR_Stream.f90  -o aaaR_Stream   ! compiling Reading code with Intel Fortran


./aaaW_Stream   ! running and producing the file data_Stream.dat
./aaaR_Stream   ! running reading this file. Result is:

1.00000000       2.00000000       3.00000000       4.00000000

The file data_Stream.dat then was archived, transferred to Windows and unzipped, and then similar Reading code compiled with FTN95
Code:
FTN95 aaaR_Stream.f90 /link /64
was reading it. By the way there was no any problems with CR/LF, the codes are compiled on Windows and Linux without any changes (contrary to C/C++ where you need to convert Windows source files to Linux ones)

Will try the same with larger sizes and Intel compiler


Last edited by DanRRight on Mon Jul 18, 2022 7:25 pm; edited 1 time in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Mon Jul 18, 2022 7:23 am    Post subject: Reply with quote

Dan,

We need to review which types of Fortran binary file formats are available.

These are selected via ACCESS= in the OPEN statement and the options are:

DIRECT these are fixed length records and provide random access of "records" via READ or WRITE "[I/O list].

SEQUENTIAL these are the clasic sequential access, variable length records that have been available for 50 years, based on magnetic tape technology. Each record has a header and footer to indicate the size and the position can be manipulated by READ or BACKSPACE.
These files are NOT interchangeable between gfortran/ifort and FTN95, as the record size header is different. An important feature of these is they can be read/written as a "record" based on their position using a READ or WRITE I/O list. Partial records can be read, then positioned to the start of the next record.

STREAM ( also TRANSPARENT (FTN95 and Lahey) ) These ARE interchangeable between gfortran, ifort and FTN95, as they do not have a header or footer to describe the record. They were officially introduced with F2003. Technically they do not have "records" and you must make sure you read the record completely if you want to use them as records. You can position at any byte in the file using READ (unit, pos=byte_address) [I/O list].
When reading the file, if you want to position to the next record, you must completely read the previous record or position to the byte address of the next record. (the byte address needs to be obtained when creating the file (via a record address table, see INQUIRE (POS=) or be calculated.)

My past posts assumed you have been using the clasic Fortran sequential access, variable length records, where the record size is defined by the WRITE (unit) "I/O list" and all READs to this file structure are positioned at the start of a record, based on header/footer sizes. Both gfortran and ifort use a 4-byte record size packet, while FTN95 uses 1-byte or 5-bytes, so the reason for the incompatibility.

One of the strengths of Fortran Binary files, is you can create a "record" of mixed variable types, using an I/O list. It is best to limit the I/O list to real, integer or complex intrinsic types, as derived type structures are not easily supported by Fortran 95+ (FTN95) in I/O lists.

Now, your latest post suggests you are using ACCESS = 'STREAM' !!!
This changes the relevance of my previous post which does not apply to STREAM I/O, although it does indicate the flexibility of this approach.

You must check what "ACCESS=" you are selecting.

The other issue with binary files is "endianess", although I hope this is not an issue for the file sources we have been discussing.

I hope this enables you to no longer be TOTALLY LOST

RTFM (Manual or Standard or FTN95.chm) for further clarification.


Last edited by JohnCampbell on Mon Jul 18, 2022 7:57 am; edited 1 time in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Mon Jul 18, 2022 7:48 am    Post subject: Re: Reply with quote

DanRRight wrote:
As i wrote, all works fine if we remove access=stream
Redo this test, as this should NOT be the case !!

The default "access=" for Form="unformatted" should be "sequential".
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Mon Jul 18, 2022 3:31 pm    Post subject: Reply with quote

John,
I think i was not clear as usually. And i could be wrong as usually since always makin tons of typos and errors. The tests i have shown tell that FTN95 is 100% compatible with GFortran on regular unformatted read. If use STREAM on GFortran and FTN95 - they are 100% compatible. If both do not use STREAM they are also compatible. I totally have lost any clues. Devilry as usual - demo works contrary to any previous knowledge, the real production code - fails. I am even more confused.

I/O in FTN95 definitely has to be synchronized with all major Fortran compilers. No excuse to have its own standard for each compiler here which gives zero advantages to anyone, only problems. Every compiler developer dancing by its own.

Want more confusion? Here is the link https://gcc.gnu.org/onlinedocs/gfortran/File-format-of-unformatted-sequential-files.html and there the code
Code:
 program main
  use iso_fortran_env, only: int32
  implicit none
  integer(int32) :: i
  real, dimension(10) :: a, b
  call random_number(a)
  open (10,file='test.dat',form='unformatted',access='stream')
  inquire (iolength=i) a
  write (10) i, a, i
  close (10)
  open (10,file='test.dat',form='unformatted')
  read (10) b
  if (all (a == b)) print *,'success!'
end program main

This code works on Linux GFortran i checked and does not on FTN95 and Linux Intel. On FTN95 error message : Unformatted record is corrupt. And on Intel the error is: forrtl: severe (67): input statement requires too much data, unit 10, file /Unformat/Gfortran_Sequential/test.dat)

And even more confusion: my own simple codes above compiled with Intel are accepted by FTN95 if both are with STREAM and not accepted if both are without STREAM. Same is true for larger files i checked.
Same question though : why STREAM seems fixing compatibility between GFortran and Intel and FTN95 on these small tests but not fixing with larger FTN95 code where there are still problems?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Tue Jul 19, 2022 9:23 am    Post subject: Reply with quote

Dan,

Here is a version of the program that will run on FTN95 for a short (<255 bytes) and a long record.

Paul,
It appears that "inquire (iolength=l4a) a" does not give the value I would expect for a "real :: a(10)" being 40 bytes, but reports 400 bits or what ??
I think this is a bug in FTN95 Ver 8.80.0 that I am using.

Code:
 program main
!  program to test FTN95 unformatted sequential file format

  use iso_fortran_env, only: int32
  implicit none

  integer*1 :: la, lb, byte
  integer*4 :: l4a, l4b, iostat
  equivalence ( la, l4a) , (lb, l4b)
  real, allocatable :: a(:), b(:), na(:), nb(:)

!  allocate array a smaller than 256 bytes and b larger than 256 bytes
  allocate ( a(10), b(100) )

  call random_number(a)
  call random_number(b)

  open (10, file='test.dat', form='unformatted', access='stream')

  inquire (iolength=l4a) a
  inquire (iolength=l4b) b
  write (*,*) ' array A is',l4a, ' bytes so 1-byte header'
  write (*,*) ' array B is',l4b, ' bytes so 5-byte header'

! FTN95 header/footer
  write (*,*) '  lengths of records corrected for error in FTN95 inquire ??'
  byte = -1
  l4a = 40       ! bytes
  l4b = 400      ! bytes
  write (10) la, a, la
  write (10) byte,l4b, b, l4b, byte

! gfortran header/footer
!  write (10) l4a, a, l4a
!  write (10) l4b, b, l4b
  close (10)

!
  allocate ( na(10), nb(100) )
  open (11,file='test.dat',form='unformatted',access='sequential')
  read (11, iostat=iostat) na  ; write (*,*) 'Reading a, iostat=',iostat
  read (11, iostat=iostat) nb  ; write (*,*) 'Reading b, iostat=',iostat

  if (all (a == na)) print *,'success for short record a !'
  if (all (b == nb)) print *,'success for long record b !'

end program main


The program runs sucessfully, but reports invalid record lengths using INQUIRE

Anyway, this program demonstrates how easy it is to use " access='stream' " to generate access='sequential' record formats in FTN95.

access='stream' is much easier to use and is portable, although you must get the record I/O list correct for both computers.

Stream is impressive, but I still use my old random access file library, based on Fortran access='direct'
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Wed Jul 20, 2022 1:10 am    Post subject: Reply with quote

Well, i did not find the error in my production code, i just completely rewrote the piece responsible for reading. Pity i lost 5 days searching. Seems the error was somewhere on my side and not in FTN95. The FTN95 communicates with other compilers using access=STREAM without any other changes so far seamlessly and my hot air on incompatibility seems was unjustified, for which i apologize. Thanks all for the help. John can claim his 12 y.o. Macallan, just give me the phone or address of your local shop Smile
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2551
Location: Sydney

PostPosted: Wed Jul 20, 2022 6:20 am    Post subject: Reply with quote

Paul,

It appears that "inquire (iolength=l4a) a" does not give the value I would expect for a "real :: a(10)", which should be 40 bytes, but FTN95 reports 400 (bits?) or what ??

I think this is a bug in the FTN95 Ver 8.80.0 (and 8.83.0) that I am using.

I shall look at this and an earlier stream problem I reported (regarding use of pos=) and see if the latest download available adresses the problem and update this post.

access='STREAM' looks to be a good improvement over 'DIRECT' or 'SEQUENTIAL' and is much more portable (until endian becomes an issue !!)

John
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7912
Location: Salford, UK

PostPosted: Wed Jul 20, 2022 7:37 am    Post subject: Reply with quote

Thanks John.

I have logged this as a bug.
Back to top
View user's profile Send private message AIM Address
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7912
Location: Salford, UK

PostPosted: Wed Jul 20, 2022 11:40 am    Post subject: Reply with quote

John

This bug has been fixed for the next release of FTN95.
Back to top
View user's profile Send private message AIM Address
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group