forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

HDF5

 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Thu Jun 20, 2019 11:08 am    Post subject: HDF5 Reply with quote

Do anyone succeeded to read HDF5 files from Fortran in this users group or any other Fortran groups?
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1183

PostPosted: Thu Jun 20, 2019 12:50 pm    Post subject: Reply with quote

Absoft Fortran comes with pre-built libraries for HDF4, HDF5 and NetCDF. Absoft provides example source files, headers, and a makefile to build the examples.

See https://forums.absoft.com/index.php?topic=810.0 .

HDF is also available for Gfortran/Cygwin.

Many years ago (~10 ?) I built HDF from source using Intel Fortran on Windows 7. Doing so was not easy, because HDF depends on other packages as prerequisites, and required many tweaks. The easiest route, perhaps, is to get the stuff working on Linux and then try to duplicate on Windows.

What are your specific needs?
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Thu Jun 20, 2019 1:19 pm    Post subject: Reply with quote

Thanks mecej4. I need just to read the content of hdf5 file, ideally using FTN95 /64. Its structure is simple, 10-12 quantities on 3D mesh. File size up to 0.1-10 TB. If this is not possible then using any Windows Fortran and then either make content visible to FTN95 or convert the content of HDF5 into regular ASCII.

By the way Absoft says "Further info you will get on HDF website." And this HDF website was made to confuse people to ultimate extent Smile. Ideally, to find someone who have done this with Fortran and can make simple example of reading some simple demo HDF5 file.
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1183

PostPosted: Thu Jun 20, 2019 3:02 pm    Post subject: Reply with quote

Have you looked at the NCDUMP converter, available at https://nsidc.org/data/hdfeos/hdf_to_ascii.html ? You should be able to convert one of your smaller data files and see if the conversion from HDF to ASCII produces a usable data file.

Of course, when you do such a conversion, the structure of the data is no longer contained in the data file, and your program has to have that structural information programmed by you in order to be able to process the converted ASCII data file.
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Thu Jun 20, 2019 11:09 pm    Post subject: Reply with quote

Thanks, will look.
Again Matlab shocked me. This is how things have to be done: everything ultimately convenient and simple for the people. Our student, without programming experience, just in couple hours made reading of the hdf5 file in Matlab. Before that another student in a month made the same in Python. And before before that, another guy, professor, in whole several months to a year made writing and reading hdf5 using C++. After that he was not able to repeat that to correct some errors and expand functionality Smile
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Fri Jun 21, 2019 2:36 am    Post subject: Reply with quote

Update
ncdump works with their demo hdf file but does not show me anything with my file. HdfViewer made by the same group shows content of my hdf file OK but das not have functionality to extract everything from DOS prompt command (via Windows GUI only). Probably they stopped developing ncdump in latest versions HDF5 because i do not see it there, it only supplied in the HDF version 4. The DOS version like ncdump would be possible to use in automatic regime calling it from FTN95.
So my idea now is to find anyone who made HDF5 reader in Fortran and create own analogy of ncdump.exe ...
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Fri Jun 21, 2019 11:08 pm    Post subject: Reply with quote

And guess those who do not know how much efforts is in Matlab to extract data out of HDF file?


Yes, like it should be with award winning software, just one single line
Code:

x1=h5read(file,'/X');


Do anyone knows if it is possible to make stand alone program in Matlab? Then we'd create EXE module in Matlab which will extract data off HDF and convert it to ASCII or binary and run it from FTN95
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1183

PostPosted: Sat Jun 22, 2019 12:51 am    Post subject: Reply with quote

A number of utilities for HDF5 are listed at https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Dump .

Some words of caution regarding HDF and other binary file formats: plan and investigate carefully before adopting such a file format. There are several problems and attempting to overcome them can eat up a lot of time before the user finds out that some cannot be overcome. See https://cyrille.rossant.net/moving-away-hdf5/ .
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Sat Jun 22, 2019 8:54 am    Post subject: Reply with quote

That was great help, mecej4, thanks, i found the workaround. The HDF group made HDF5 analog of HDF4 standalone extraction tool ncdump, called h5dump, i just missed it when looked first time.
Hopefully after getting experience with this tool, i some day will be able to read HDF5 files from FTN95 directly. Or may be Silverfrost will adapt HDF5 library. For now the approach of launching external converter from FTN95 will be probably OK.

Yes, there exist danger with binary or zipped storage, but we have no choice, the sizes of files PIC codes generate are enormous, 1.2 TB for example was just one of today's runs from one of 3 supercomputers we use every day.

/* 10-15 years from now this will be laughable how small it is what we today consider hell large
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1176
Location: Aerospace Valley

PostPosted: Sun Jun 23, 2019 11:15 pm    Post subject: Reply with quote

Dad,
A Fortran library is available, equivalnt to the C-routines contained in HDF5.

Presumeably this is what people use to write interfaces to HDF5 format.

https://support.hdfgroup.org/HDF5/doc/fortran/index.html

Of course there's a lot of additional reaading to do to understand first how a C-program is written to access the format .... and then to convert to a Fortran equivalent, calling the equivalent fortran routines.ut then conversion from one format to another never has been easy.

I guess it's easier if you know you'll always be getting a particular data format and content every time in the HDF file (seen using h5dump)

mcej4 , ths že article you wrote is a sobering one. It basically says 'don't ue HDF5, because ,,,, ! !

I haven't looked at the nitty-gritty of the file format description yet myself, but after reading the article my gut reaction is .... don't touch it with a bargepole ! .... except a program I use frequently is in the process of adopting it as (presumeanžby) a defacto standard. I suppose.

As I see it , instead of being more transparnt, these larger commercial programs are obsessed with efficiency and catering for mega-file sizes , which can only end up in tears imo. For users and vendors alike.
_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1176
Location: Aerospace Valley

PostPosted: Mon Jun 24, 2019 12:06 am    Post subject: Reply with quote

P.S.
a flowchart description, asuming it's done well of course) of the whole HDF5 data structure would go a long way to hlping sers get to grips with it.

A picture does a quick sketch of a million attempts to describe it in text sort of thingywotsit, as shakespeare once probably never said.

or was it Mr Gates who sang 'if a picture paints a thousand words then why can't I parse you ?' ? ....
https://www.youtube.com/watch?v=a6_efthmkAQ&hd=1

Of course it doesn't help that the vendors are catering for us as if we're chickens clucking for the technology ...


_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Mon Jun 24, 2019 9:12 am    Post subject: Reply with quote

The hdfgroup does not teach people going from simplest to complex. You will break your legs trying to understand what is what there. They even do not bother to clean up the mess in their website.

I guarantee I'd have no clue till today what is Clearwin or OpenGL or .NET or draw_text@/draw_line@ if Silverfrost had no simplest possible demo examples for all that options. Though OpenGL examples were not that simple, just one was. And i started with it and am very happy with the results. Sorry there were so little of them
Back to top
View user's profile Send private message
LitusSaxonicum



Joined: 23 Aug 2005
Posts: 2023
Location: Yateley, Hants, UK

PostPosted: Mon Jun 24, 2019 11:04 am    Post subject: Reply with quote

Quote:
just one was


And which was that?

Eddie
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2032
Location: South Pole, Antarctica

PostPosted: Tue Jun 25, 2019 3:51 am    Post subject: Reply with quote

The one which rotates the XY plot
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1176
Location: Aerospace Valley

PostPosted: Tue Jul 02, 2019 10:53 am    Post subject: Reply with quote

...there's one which does animated plots too.

... which would be nice to have in native plot form too, if it's possible

The main restriction as I see it is that example codes are 'scattered' through the documentation.
There are some in 'examples' folder, some occasionally in the descriptions od certain commands/functionalities, and of course some within the forums which Paul 'plucks out of the air' from time to time.

A repository, a repository, my kingdom for a single repository (with index !)
_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group