|
forums.silverfrost.com Welcome to the Silverfrost forums
|
View previous topic :: View next topic |
Author |
Message |
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Sat Apr 24, 2021 5:16 pm Post subject: Reading ESRI bil format binary file |
|
|
Hello,
I am interested to know how a binary file can be read to store x, y, z values.
The ESRI BIL format has a separate header with the following:
BYTEORDER I
LAYOUT BIL
NROWS 120
NCOLS 120
NBANDS 1
NBITS 32
BANDROWBYTES 480
TOTALROWBYTES 480
PIXELTYPE FLOAT
ULXMAP -4.99166666666667
ULYMAP -3.00833333333333
XDIM 0.0166666666666667
YDIM 0.0166666666666667
Can this information be utilised to read and write the contained data. I am presently comparing the difference between reading from an ESRI Ascii raster grid and a binary file (far smaller).
Thanks
Lester |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1227 Location: Morrison, CO, USA
|
|
Back to top |
|
|
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Thu Apr 29, 2021 3:05 pm Post subject: |
|
|
Hi Bill,
Thanks for the link, that helped to understand the structure of the ESRI BIL format. I tried the following just to attempt to read the data from the file 'faa.bil' into the allocatable variable (zdata), however it seems to fall over at the attempt to read from unit=26.
program binary_ESRI_bil
implicit none
integer :: nrows, ncols, nbands, nbits, bandrowbytes, totalrowbytes
real(kind=2) :: upperleft_x, upperleft_y, xdim, ydim
character(len=10) :: layout, pixeltype, byteorder
real, allocatable, dimension( :,: ) :: zdata
open (25, file='faa.hdr', status='old') ! get information from header file
read(25, '(15x,A)') byteorder
read(25, '(15x, A)') layout
read(25, '(15x,I3)') nrows ! 120
read(25, '(15x,I3)') ncols ! 120
read(25, '(15x,I1)') nbands
read(25, '(15x,I2)') nbits
read(25, '(15x,I3)') bandrowbytes
read(25, '(15x,I3)') totalrowbytes
read(25, '(15x,A)') pixeltype
read(25, '(15x,f15.6)') upperleft_x ! -4.99166666666667
read(25, '(15x,f15.6)') upperleft_y ! -3.00833333333333
read(25, '(15x,f15.6)') xdim ! 0.0166666666666667
read(25, '(15x,f15.6)') ydim ! 0.0166666666666667
open(26, file='faa.bil', form='formatted', access='sequential', status='old')
read(26) zdata ! dimensions (ncols*nrows)
close(25)
close(26)
end program binary_ESRI_bil
I tried various options with the open form = formatted/unformatted, access= sequential/transparent etc., but with no help. I am guessing it is the read access point, but not sure why? The data is square (120 * 120)
Thanks
Lester |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1227 Location: Morrison, CO, USA
|
Posted: Thu Apr 29, 2021 4:23 pm Post subject: |
|
|
If the file is binary, then a formatted read will not work. Formatted assumes the data is in ASCII.
To do an unformatted read using the parameters from the header is not trivial, and may require the use of lower-level I/O routines. I use these in a couple of my subroutines on specialized data.
Also, you have declared ZDATA but not actually allocated any space based on what the dimensions are. Use something like:
Code: |
allocate(ZDATA(nrows,ncols))
| to get the variable allocated.
BTW: When you get errors, there should be a description of the run-time error. Providing that should help guide assistance from others on the forum. Was the error you got from the I/O statement or was it a NULL pointer assignment error? Each is very different in where to attack!
You are on the right track, though.
Be aware that if the ESRI data is written from "C" versus Fortran, your accessing of the data in the matrix will be different because of how the data are stored/accessed. Just sayin'. |
|
Back to top |
|
|
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Thu Apr 29, 2021 5:36 pm Post subject: |
|
|
Hi
Added the correction to allocate the dimensions.
The run-time error is:
Silverfrost 64-bit exception report on D:\Geoscience\Fortran\Test\Binary2.exe Thu Apr 29 17:32:57 2021
Unformatted record is corrupt at address 1c0089a1
Within file Binary2.exe
in BINARY_ESRI_BIL at address 69e
RAX = 000000000000006e RBX = 000000000000006e RCX = 000000000000006e RDX = 0000000000000000
RBP = 000000001c000000 RSI = 000000000000006e RDI = 0000000000000000 RSP = 000000000240f580
R8 = fffff805fa571ff6 R9 = 00000000000000a2 R10 = 00000000000000a1 R11 = 000000000240f340
R12 = 0000000000000000 R13 = 000000000000e100 R14 = 0000000000000078 R15 = 0000000000000077
1c0089a1) int 9
I also set the format as unformatted - it is a fiddly one! I can verify that the file is fully readable and can be imaged in GIS.
Lester |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1227 Location: Morrison, CO, USA
|
Posted: Fri Apr 30, 2021 3:19 am Post subject: |
|
|
One thing about unformatted I/O is: it can be finicky.
You may need to specify a record length. From the HELP file:
Quote: | RECL=<value>
<value> is an integer expression, which gives record length in bytes. For direct access files, this specifier is obligatory.
As an extension to the standard, RECL may also be specified for a file opened for sequential access. This causes fixed length records to be read from or written to file and allows a BACKSPACE to be followed by a WRITE.
|
You could calculate the record length as 120x120x8 to read the data you wish to read.
Your open should include : ACCESS='SEQUENTIAL', RECL=115200
This might just do it! If not, I have some alternatives that you can use. Non-portable, but usable. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2593 Location: Sydney
|
Posted: Fri Apr 30, 2021 6:48 am Post subject: |
|
|
Your post is unclear, but if the file is a binary unformatted file, then you need to be careful how it was generated.
If so, then you need to identify the record formats.
Typically sequential binary (unformatted) files have a header and a trailer to identify the record size. (for reading or backspace)
FTN95 header/footer formats are different to other compilers, so you need to know how the records were generated. (FTN95 provides a 1-byte h/f for small record storage efficiency)
This makes the file usage slightly more difficult, but you can overcome this by using FTN95 transparent access and read and process the header/footer in a way that is compatible with their generation.
Typically FTN95 uses a 1-byte or 5-byte format (for records longer than 255 bytes), while most other Fortran compilers use 4-byte format for records shorter than 2GBytes. (Unsure of format for records longer than 2GBytes)
You can experiment with FTN95 OPEN (..., ACCESS='TRANSPARENT',...) and start writing a routine Get_Next_Record_Size (num_bytes ) and then go from there.
An unformatted is WRITE (unit) [I/O list] so increased difficulty comes if the I/O list for generating the record mixes real and integer kinds.
It is much easier to decode if the form is :
WRITE (unit) n
WRITE (unit) array(1:n).
It would be good if FTN95 supported a 4-byte header, via an OPEN option, which I don't think it does. Perhaps FORM='4-BYTE'
We should find out what any compilers do for unformatted records larger than 2GBytes.
In 2021, with large disks and fast processing, text files are much easier and more flexible for transfering information.
"READ (unit,*) array(1:n)" will read until n values are recovered, from an unknown number of lines of data.
You must get "n" correct and don't make each line too long. Errors in the data are difficult to manage.
Note:
FTN95 OPEN (..., ACCESS='TRANSPARENT',...) is very similar to F03+ OPEN (..., ACCESS='STREAM',...), which provides both sequential and direct access.
Fortran fixed length record, random access binary files do not have a record length header/footer so are the same format for all Fortran compilers I have tested.
These are very easy for transferring data, but must have the same fixed record length.
These are OPEN (..., ACCESS='DIRECT',...) "recl=" specifies the length of each record in processor dependent units (bytes or words!!)
I am not sure if all this is in FTN95.chm, but it should be. |
|
Back to top |
|
|
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Fri Apr 30, 2021 7:32 am Post subject: |
|
|
Hello all,
Finally sorted out the binary read; it is a fiddly job!
The binary file does not have any obvious header within it, but must have some reference to record length. The solution that worked was to have the following:
Code: | open(26, file='faa.bil', form='unformatted', access='transparent', recl=115200, status='old')
allocate(zdata(ncols,nrows))
read(26) zdata ! dimensions (ncols*nrows)
open(27, file='faa_bil.dat', status='new')
write (27,'(f15.10)') zdata |
The output in 'faa_bil.dat' is exactly what was expected, i.e. a total of 14400 values (120 x 120)
Thanks for all the guidance, got there in the end.
Lester |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 8036 Location: Salford, UK
|
Posted: Fri Apr 30, 2021 9:40 am Post subject: |
|
|
Regarding TRANSPARENT access, the standard conforming STREAM access is available in FTN95 v8.71 which can be downloaded via the Sticky post in the Support section of this forum. |
|
Back to top |
|
|
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Fri Apr 30, 2021 11:25 am Post subject: |
|
|
Thanks for the update Paul. I was using ftn95 v8.70. |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1227 Location: Morrison, CO, USA
|
Posted: Fri Apr 30, 2021 9:21 pm Post subject: |
|
|
John, most contemporary applications do not use a record format in the traditional sense. As arctica demonstrated, you just read the whole darn thing in one gulp. This is typical of "C" where you can point to an array of double precision values and say "read (or write) the next 115200 bytes". But "C" and FORTRAN store data differently, so one must be careful and understand how the data are structured in sequential memory.
AutoCAD binary file (DWG, etc.) are another example. I much prefer to use DXF (ASCII) because of the record format nature of a DXF file. You always know what the next record will be, and its next level down is determined by what kind of record it is, ad infinitum.
Records are surely easier to accommodate logically, but they are also limiting in applications where there is a truly variable amount of data, and "waste" is not allowed. IBM VBS (variable length block with spanning) was an attempt to get at that problem (logically), but had overhead in each record, and was actually a fixed record structure in every other way imaginable. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2593 Location: Sydney
|
Posted: Sat May 01, 2021 9:03 am Post subject: |
|
|
Paul,
Thanks very much for implementing stream I/O.
It certainly provides more flexibility for accessing information from other compilers and operating systems. "Unix" text files are another good example for stream I/O.
We need an example that demonstrates random access (write then read) to records of different lengths and kinds, as a clear demonstration of positioning in a binary stream file would eliminate confusion on what syntax the standard provides for OPEN, READ, WRITE & INQUIRE. Re-reading a block of data is a typical positioning example.
Bill,
Given the speed and storage capacity of modern computers, I doubt if binary files can be justified for transferring data between "groups".
I certainly use binary files when creating databases in a large calculations (eg FE analysis with variable length random access binary records that I started using on CDC Fortran), but a clearly defined text structure is much easier for transferring information to others.
I wish that I used text files for all the spreadsheets I have generated over the years, especially for all those Quatro-pro files and .xls files that are lists of data for old projects. Old Access databases would be much more accessible as a text file, especially for tables. Most of these are no longer readable !!
Text files for .xlsx files would certainly make auditing much easier and would not cost a lot of time to recreate. Dealing with charts and pivot tables may be more challenging?
Ever tried to identify what has changed in 2 versions of the same spreadsheet with different modified dates ? ( often very little although you can never be sure )
Lester,
You are lucky you could read the file in the way you did.
Check if the matrix is transposed ? |
|
Back to top |
|
|
arctica
Joined: 10 Sep 2006 Posts: 105 Location: United Kingdom
|
Posted: Sat May 01, 2021 11:48 am Post subject: |
|
|
Hi John,
The data were flipped, so I had to apply a fix, think it was largely because the data started at top left, not sure if this is the best method:
Code: | sz=size(zdata,1)
zdata = zdata(sz:1:-1, :)
zdata=transpose(zdata)
zdata = zdata(:, sz:1:-1) |
I could use separate software to know which way round the grid should be.
Added a call to GMT to grid up the data and check it:
Code: | call cissue@('xyz2grd faa_bil_final.dat -GBIL_test.grd -R-5/-3/-5/-3 -I0.016666666 -F -V',fail) |
The record length could be defined from the separate header file as:
bandrowbytes * nrows * 2
Lester |
|
Back to top |
|
|
wahorger
Joined: 13 Oct 2014 Posts: 1227 Location: Morrison, CO, USA
|
Posted: Sat May 01, 2021 3:13 pm Post subject: |
|
|
Lester, glad you were able to read the data and get it in the right form!
Good luck on the next steps of your project!
Bill |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|