forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Unformatted record is corrupt

 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support
View previous topic :: View next topic  
Author Message
Konstantin8



Joined: 07 Jun 2013
Posts: 9

PostPosted: Wed Jan 22, 2014 4:18 am    Post subject: Unformatted record is corrupt Reply with quote

I have a program distributed by NASA. During debugging I see the mentioned error on reading a binary data file.
I'm able to attach the ftn95 project with source files and data file.
Or source files can be downloaded from
http://nssdcftp.gsfc.nasa.gov/models/atmospheric/hwm07/

I use Win32 build in Plato.

Thanks in advance.
Back to top
View user's profile Send private message
IanLambley



Joined: 17 Dec 2006
Posts: 490
Location: Sunderland

PostPosted: Wed Jan 22, 2014 1:15 pm    Post subject: Reply with quote

If those files were written by another compiler, then there is no real standard for unformatted binary files and the form of the record length part of each line changes for lines of length less or greater than 256 characters.
Try using this program on the .dat files:
Code:
winapp
! convert an unformatted binary file to the FTN95 format
integer*1 input(65536),work(65536*2),ibyte_long(4)
INTEGER*2 handle_in,handle_out, error_code,itest_ffh
integer*4 i,iwork_pointer,iwork_end,irec_len, ilong, nline,imove, ilen_extra, itransfer_4bytes, nbytes_read
equivalence (ibyte_long,ilong)
iwork_pointer  = 1
iwork_end      = 0
itest_ffh      = -1 !-1 as a single byte is ffh the code for a long record
irec_len       = 0
call openr@('otherftnformattedlong.dat',handle_in,error_code)
call openw@('ftn95formattedlong_new.dat',handle_out,error_code)
nline = 0
do
  call READF@(input, handle_in, 65536L, nbytes_read, error_code)
  if(nbytes_read .eq. 0)goto 9999
  nline = nline + 1
  print *,nline,nbytes_read,error_code
!move previously handled work to the start of the work buffer   
  imove = 0
  do i=iwork_pointer,iwork_end
    imove = imove + 1
    work(imove) = work(i)
  enddo
  iwork_pointer = 1
  iwork_end     = imove
!move the input line to the work 
  do i=1,nbytes_read
    imove = imove + 1
    work(imove) = input(i)
  enddo
  iwork_end     = imove
!
! now process the next chunk from iwork_pointer to iwork_end
  do while (iwork_pointer .lt. iwork_end)
!get the 4 byte record length
    irec_len = itransfer_4bytes(work(iwork_pointer))
    iwork_pointer = iwork_pointer + 4
    ilen_extra = 4
    print *,'irec_len',irec_len,ilen_extra
!process record to output data here
!if the record length is less than 255 long, then output a single byte
! record length
    if(irec_len .lt. 255)then
      ilong = irec_len 
      call WRITEF@(ibyte_long,handle_out, 1L, error_code)
      call WRITEF@(work(iwork_pointer),handle_out, irec_len, error_code)
      call WRITEF@(ibyte_long,handle_out, 1L, error_code)
    else
! prefix with record length as ffh + a 4 byte integer
      call WRITEF@(itest_ffh,handle_out, 1L, error_code)
      call WRITEF@(irec_len,handle_out, 4L, error_code)
!output the data
      call WRITEF@(work(iwork_pointer),handle_out, irec_len, error_code)
! postfix with record length as a 4 byte integer + ffH
      call WRITEF@(irec_len,handle_out, 4L, error_code)
      call WRITEF@(itest_ffh,handle_out, 1L, error_code)
    endif
  ! now advance to the next record start
    iwork_pointer = iwork_pointer + irec_len + ilen_extra
 
    print*,'next iwork_pointer',iwork_pointer
  enddo
enddo
9999 continue
call CLOSEF@(handle_in, error_code)
call CLOSEF@(handle_out, error_code)
print *,nline
end
integer*4 function itransfer_4bytes(idata1)
integer*1 idata1(4),jdata1(4)
integer*4 jdata4,i
equivalence (jdata1,jdata4)
do i=1,4
  jdata1(i) = idata1(i)
  print '(z2.2)',jdata1(i)
enddo
itransfer_4bytes = jdata4
print '(z8.8)',jdata4
end
Back to top
View user's profile Send private message Send e-mail
mecej4



Joined: 31 Oct 2006
Posts: 1882

PostPosted: Wed Jan 22, 2014 4:42 pm    Post subject: Re: Reply with quote

I was able to convert the unformatted data files after I enlarged the buffers in IanLambley's converter program to cover the longest of the input data files.

The NASA source code has some bugs that are easy to find and fix, but without those fixes you will not get correct results. Subroutine HWMupdate needs an explicit interface but one is not provided. There are some subroutine arguments that have declared INTENT(IN) but are modified. There are some variables that are not initialized. Here is a brief list of fixes:

File dwm07b.f90:

Change INTENT of GLON to "IN OUT" in subroutines DWM07B_HWM_INTERFACE and GD2QD

Initialize HR in subroutine GD2QD.

File hwm07e.f90:

Change INTENT of GLON to "IN OUT" in subroutines HWM07.

Initialize LAST() in subroutine HWMQT.

Change subroutine HWMUPDATE to a contained subroutine or provide an explicit interface to it in its caller(s).

Initialize GZWGHT in subroutine LOADMODEL.

File apexcord.f90: Change INTENT of GLON to "IN OUT" in subroutine APEX.

If you (original poster) are interested, I can see about posting the modified sources and the FTN95 compatibles *.dat files somewhere from where you can download them. (This forum does not seem to allow attaching files to messages).


Last edited by mecej4 on Fri Jan 24, 2014 3:51 pm; edited 1 time in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2536
Location: Sydney

PostPosted: Thu Jan 23, 2014 1:59 am    Post subject: Reply with quote

It always annoys me to see programs released with binary data files.
It would be much better if the .dat files were text files, where the data was written as text numbers.
The penalty to be paid for file size or run time is minimal, while the choice of number formats to limit precision errors is ususlly minimal, given the accuracy of the initial data estimates.
Using text files as an interface between programs also makes validation of the data much easier.

John

Ian, thanks for your program. It is very useful.
It fails where the record size is larger than the input array or the logical record is not fully contained in the work array. I think you need to test if the next logical record is fully contained in WORK, rather than "do while (iwork_pointer < iwork_end)". If you change 65536 to be larger than the file ( can change up to about 1gb ) then ther is no problem.
If you open the file as access='transparent', you can read the file 1 byte at a time and do away with the input buffer, reading the 4_byte header, then the info, then the 4_byte trailer.
You might like to consider the TRANSFER intrinsic to generate the integer*4 value.
The other problem is if the source file is from a unix or linux source, where the problem of ??_endian makes it more complex. You then need to know the structure of each record.
Back to top
View user's profile Send private message
Konstantin8



Joined: 07 Jun 2013
Posts: 9

PostPosted: Thu Jan 23, 2014 6:12 am    Post subject: Reply with quote

Hi all,
Thanks for help!

I try to use the posted program but have an error with undefined elements in the input array.

If I correctly understand I need to enlarge dimensions of input and work arrays and also enlarge NBYTES value in the READF@ call.

Is it correct?
Back to top
View user's profile Send private message
Konstantin8



Joined: 07 Jun 2013
Posts: 9

PostPosted: Thu Jan 23, 2014 6:26 am    Post subject: Re: Reply with quote

mecej4 wrote:
I was able to convert the unformatted data files after I enlarged the buffers in IanLambley's converter program to cover the longest of the input data files.

The NASA source code has some bugs that are easy to find and fix, but without those fixes you will not get correct results. Subroutine HWMupdate needs an explicit interface but one is not provided. There are some subroutine arguments that have declared INTENT(IN) but are modified. There are some variables that are not initialized.

If you (original poster) are interested, I can see about posting the modified sources and the FTN95 compatibles *.dat files somewhere from where you can download them. (This forum does not seem to allow attaching files to messages).



mecej4,
It seems it will be really simlier if you are able to update files. If it is possible, of course.

Thanks in advance!
Back to top
View user's profile Send private message
IanLambley



Joined: 17 Dec 2006
Posts: 490
Location: Sunderland

PostPosted: Thu Jan 23, 2014 1:24 pm    Post subject: Reply with quote

John,
Thanks for the info. I could have opened it as a direct access file with a recl=1, but I was trying to avoid too many reads as I thought that could slow down the processing.
Yes, the TRANSFER intrinsic would be much better but I'm caught in a Fortran 66 time warp, I must do better, but, heavens, I've just sold my 1961 Riley 1.5 and I'm still in shock! LOL
Regards
Ian
Back to top
View user's profile Send private message Send e-mail
mecej4



Joined: 31 Oct 2006
Posts: 1882

PostPosted: Thu Jan 23, 2014 6:13 pm    Post subject: Re: Reply with quote

Konstantin8:
Quote:
mecej4,
It seems it will be really simlier if you are able to update files. If it is possible, of course.

Thanks in advance!

I have combined (i) the modified F90 sources, (ii) the source code of a utility (in C) to convert Intel/GFortran/etc. Fortran unformatted files to FTN95 unformatted files, and (iii) modified FTN95 unformatted data files for the HWM07 package, all into a single Zip file that you may download from https://www.amazon.com/clouddrive/share?s=eA42kBlCS_EvdLiuDMUaUk. [This download link should be valid for at least a few days.]

Please note that there is a bug in the 7.00 FTN95 compiler (and, possibly, earlier versions) that prevents you from compiling with the /check or /undef options. If you use these options, the program will abort with the message "Attempt to call a subroutine with argument number ten containing too few array elements". I have prepared a pared-down reproducer and I have posted a bug report for this issue separately.

[UPDATE: Paul Laidler suggests in http://forums.silverfrost.com/viewtopic.php?t=2726 that you use the option /inhibit_check 5 to work around the bug if you want to be able to use /undef and similar checks.]


Last edited by mecej4 on Fri Jan 24, 2014 1:40 pm; edited 1 time in total
Back to top
View user's profile Send private message
Konstantin8



Joined: 07 Jun 2013
Posts: 9

PostPosted: Fri Jan 24, 2014 5:41 am    Post subject: Re: Reply with quote

mecej4 wrote:
Konstantin8:
Quote:
mecej4,
It seems it will be really simlier if you are able to update files. If it is possible, of course.

Thanks in advance!

I have combined (i) the modified F90 sources, (ii) the source code of a utility (in C) to convert Intel/GFortran/etc. Fortran unformatted files to FTN95 unformatted files, and (iii) modified FTN95 unformatted data files for the HWM07 package, all into a single Zip file that you may download from https://www.amazon.com/clouddrive/share?s=eA42kBlCS_EvdLiuDMUaUk. [This download link should be valid for at least a few days.]

Please note that there is a bug in the 7.00 FTN95 compiler (and, possibly, earlier versions) that prevents you from compiling with the /check or /undef options. If you use these options, the program will abort with the message "Attempt to call a subroutine with argument number ten containing too few array elements". I have prepared a pared-down reproducer and I have posted a bug report for this issue separately.



Thank you a lot!
Everything work fine!
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group