Reading ESRI bil format binary file

Hello,

I am interested to know how a binary file can be read to store x, y, z values.

The ESRI BIL format has a separate header with the following:

BYTEORDER I LAYOUT BIL NROWS 120 NCOLS 120 NBANDS 1 NBITS 32 BANDROWBYTES 480 TOTALROWBYTES 480 PIXELTYPE FLOAT ULXMAP -4.99166666666667 ULYMAP -3.00833333333333 XDIM 0.0166666666666667 YDIM 0.0166666666666667

Can this information be utilised to read and write the contained data. I am presently comparing the difference between reading from an ESRI Ascii raster grid and a binary file (far smaller).

Thanks

Lester

Is this link of any help? It is a bit thin on details, but some of the links might be helpful.

https://www.loc.gov/preservation/digital/formats/fdd/fdd000283.shtml#:~:text=In%20ESRI%20applications%2C%20accompanying%20the,use%20of%20keywords%20and%20values.

BTW, I have used Google Earth to process ESRI files in order to extract the coordinates for another purpose of my software. If Google Earth can read them, then you should be able to.

Good luck! Let us know what you find! Bill

Hi Bill,

Thanks for the link, that helped to understand the structure of the ESRI BIL format. I tried the following just to attempt to read the data from the file 'faa.bil' into the allocatable variable (zdata), however it seems to fall over at the attempt to read from unit=26.

program binary_ESRI_bil

implicit none

integer :: nrows, ncols, nbands, nbits, bandrowbytes, totalrowbytes real(kind=2) :: upperleft_x, upperleft_y, xdim, ydim character(len=10) :: layout, pixeltype, byteorder real, allocatable, dimension( :,: ) :: zdata

open (25, file='faa.hdr', status='old') ! get information from header file

read(25, '(15x,A)') byteorder read(25, '(15x, A)') layout read(25, '(15x,I3)') nrows ! 120 read(25, '(15x,I3)') ncols ! 120 read(25, '(15x,I1)') nbands read(25, '(15x,I2)') nbits read(25, '(15x,I3)') bandrowbytes read(25, '(15x,I3)') totalrowbytes read(25, '(15x,A)') pixeltype read(25, '(15x,f15.6)') upperleft_x ! -4.99166666666667 read(25, '(15x,f15.6)') upperleft_y ! -3.00833333333333 read(25, '(15x,f15.6)') xdim ! 0.0166666666666667 read(25, '(15x,f15.6)') ydim ! 0.0166666666666667

open(26, file='faa.bil', form='formatted', access='sequential', status='old') read(26) zdata ! dimensions (ncols*nrows)

close(25) close(26)

end program binary_ESRI_bil

I tried various options with the open form = formatted/unformatted, access= sequential/transparent etc., but with no help. I am guessing it is the read access point, but not sure why? The data is square (120 * 120)

Thanks

Lester

If the file is binary, then a formatted read will not work. Formatted assumes the data is in ASCII.

To do an unformatted read using the parameters from the header is not trivial, and may require the use of lower-level I/O routines. I use these in a couple of my subroutines on specialized data.

Also, you have declared ZDATA but not actually allocated any space based on what the dimensions are. Use something like:

allocate(ZDATA(nrows,ncols))

to get the variable allocated.

BTW: When you get errors, there should be a description of the run-time error. Providing that should help guide assistance from others on the forum. Was the error you got from the I/O statement or was it a NULL pointer assignment error? Each is very different in where to attack!

You are on the right track, though.

Be aware that if the ESRI data is written from 'C' versus Fortran, your accessing of the data in the matrix will be different because of how the data are stored/accessed. Just sayin'.

Hi

Added the correction to allocate the dimensions.

The run-time error is:

Silverfrost 64-bit exception report on D:\Geoscience\Fortran\Test\Binary2.exe Thu Apr 29 17:32:57 2021

Unformatted record is corrupt at address 1c0089a1

Within file Binary2.exe in BINARY_ESRI_BIL at address 69e

RAX = 000000000000006e RBX = 000000000000006e RCX = 000000000000006e RDX = 0000000000000000 RBP = 000000001c000000 RSI = 000000000000006e RDI = 0000000000000000 RSP = 000000000240f580 R8 = fffff805fa571ff6 R9 = 00000000000000a2 R10 = 00000000000000a1 R11 = 000000000240f340 R12 = 0000000000000000 R13 = 000000000000e100 R14 = 0000000000000078 R15 = 0000000000000077

1c0089a1) int 9

I also set the format as unformatted - it is a fiddly one! I can verify that the file is fully readable and can be imaged in GIS.

Lester

One thing about unformatted I/O is: it can be finicky.

You may need to specify a record length. From the HELP file:

RECL=<value> <value> is an integer expression, which gives record length in bytes. For direct access files, this specifier is obligatory. As an extension to the standard, RECL may also be specified for a file opened for sequential access. This causes fixed length records to be read from or written to file and allows a BACKSPACE to be followed by a WRITE.

You could calculate the record length as 120x120x8 to read the data you wish to read.

Your open should include : ACCESS='SEQUENTIAL', RECL=115200

This might just do it! If not, I have some alternatives that you can use. Non-portable, but usable.

Your post is unclear, but if the file is a binary unformatted file, then you need to be careful how it was generated. If so, then you need to identify the record formats. Typically sequential binary (unformatted) files have a header and a trailer to identify the record size. (for reading or backspace)

FTN95 header/footer formats are different to other compilers, so you need to know how the records were generated. (FTN95 provides a 1-byte h/f for small record storage efficiency)

This makes the file usage slightly more difficult, but you can overcome this by using FTN95 transparent access and read and process the header/footer in a way that is compatible with their generation.

Typically FTN95 uses a 1-byte or 5-byte format (for records longer than 255 bytes), while most other Fortran compilers use 4-byte format for records shorter than 2GBytes. (Unsure of format for records longer than 2GBytes) You can experiment with FTN95 OPEN (..., ACCESS='TRANSPARENT',...) and start writing a routine Get_Next_Record_Size (num_bytes ) and then go from there.

An unformatted is WRITE (unit) [I/O list] so increased difficulty comes if the I/O list for generating the record mixes real and integer kinds. It is much easier to decode if the form is : WRITE (unit) n WRITE (unit) array(1:n).

It would be good if FTN95 supported a 4-byte header, via an OPEN option, which I don't think it does. Perhaps FORM='4-BYTE' We should find out what any compilers do for unformatted records larger than 2GBytes.

In 2021, with large disks and fast processing, text files are much easier and more flexible for transfering information. 'READ (unit,*) array(1:n)' will read until n values are recovered, from an unknown number of lines of data. You must get 'n' correct and don't make each line too long. Errors in the data are difficult to manage.

Note: FTN95 OPEN (..., ACCESS='TRANSPARENT',...) is very similar to F03+ OPEN (..., ACCESS='STREAM',...), which provides both sequential and direct access. Fortran fixed length record, random access binary files do not have a record length header/footer so are the same format for all Fortran compilers I have tested. These are very easy for transferring data, but must have the same fixed record length. These are OPEN (..., ACCESS='DIRECT',...) 'recl=' specifies the length of each record in processor dependent units (bytes or words!!)

I am not sure if all this is in FTN95.chm, but it should be.

Hello all,

Finally sorted out the binary read; it is a fiddly job!

The binary file does not have any obvious header within it, but must have some reference to record length. The solution that worked was to have the following:

open(26, file='faa.bil', form='unformatted', access='transparent', recl=115200, status='old')
allocate(zdata(ncols,nrows))
read(26) zdata ! dimensions (ncols*nrows)

open(27, file='faa_bil.dat', status='new')
write (27,'(f15.10)') zdata

The output in 'faa_bil.dat' is exactly what was expected, i.e. a total of 14400 values (120 x 120)

Thanks for all the guidance, got there in the end.

Lester

Regarding TRANSPARENT access, the standard conforming STREAM access is available in FTN95 v8.71 which can be downloaded via the Sticky post in the Support section of this forum.

Thanks for the update Paul. I was using ftn95 v8.70.

John, most contemporary applications do not use a record format in the traditional sense. As arctica demonstrated, you just read the whole darn thing in one gulp. This is typical of 'C' where you can point to an array of double precision values and say 'read (or write) the next 115200 bytes'. But 'C' and FORTRAN store data differently, so one must be careful and understand how the data are structured in sequential memory.

AutoCAD binary file (DWG, etc.) are another example. I much prefer to use DXF (ASCII) because of the record format nature of a DXF file. You always know what the next record will be, and its next level down is determined by what kind of record it is, ad infinitum.

Records are surely easier to accommodate logically, but they are also limiting in applications where there is a truly variable amount of data, and 'waste' is not allowed. IBM VBS (variable length block with spanning) was an attempt to get at that problem (logically), but had overhead in each record, and was actually a fixed record structure in every other way imaginable.

Paul,

Thanks very much for implementing stream I/O. It certainly provides more flexibility for accessing information from other compilers and operating systems. 'Unix' text files are another good example for stream I/O. We need an example that demonstrates random access (write then read) to records of different lengths and kinds, as a clear demonstration of positioning in a binary stream file would eliminate confusion on what syntax the standard provides for OPEN, READ, WRITE & INQUIRE. Re-reading a block of data is a typical positioning example.

Bill,

Given the speed and storage capacity of modern computers, I doubt if binary files can be justified for transferring data between 'groups'. I certainly use binary files when creating databases in a large calculations (eg FE analysis with variable length random access binary records that I started using on CDC Fortran), but a clearly defined text structure is much easier for transferring information to others.

I wish that I used text files for all the spreadsheets I have generated over the years, especially for all those Quatro-pro files and .xls files that are lists of data for old projects. Old Access databases would be much more accessible as a text file, especially for tables. Most of these are no longer readable !!

Text files for .xlsx files would certainly make auditing much easier and would not cost a lot of time to recreate. Dealing with charts and pivot tables may be more challenging? Ever tried to identify what has changed in 2 versions of the same spreadsheet with different modified dates ? ( often very little although you can never be sure )

Lester,

You are lucky you could read the file in the way you did. Check if the matrix is transposed ?

Hi John,

The data were flipped, so I had to apply a fix, think it was largely because the data started at top left, not sure if this is the best method:

sz=size(zdata,1)
zdata = zdata(sz:1:-1, :)
zdata=transpose(zdata)
zdata = zdata(:, sz:1:-1)

I could use separate software to know which way round the grid should be.

Added a call to GMT to grid up the data and check it:

call cissue@('xyz2grd faa_bil_final.dat -GBIL_test.grd -R-5/-3/-5/-3 -I0.016666666 -F -V',fail)

The record length could be defined from the separate header file as: bandrowbytes * nrows * 2

Lester

Lester, glad you were able to read the data and get it in the right form!

Good luck on the next steps of your project!

Bill