forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Database or NetCDF / HDF5 interfaces?
Goto page Previous  1, 2, 3, 4, 5, 6
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Tue Nov 22, 2016 1:33 pm    Post subject: Reply with quote

OK, finally i was able to reboot my computer and install RAMdisk.

And now after being happy for last few days with increased by factor of 4.5x read speeds using unformatted I/O with binary data i'm again in the big black jealousy mode.

Here are my RAMdisk speeds.



Versus what i got (~300MB/s) these speeds are 20-30x larger...The C code of this benchmark somehow reads hell faster and writes damn faster.

If you know C and have time to investigate how these speeds are tested here is link on the C source code of this benchmark (called CrystalDiskMark which is mostly used for such tests)

http://crystalmark.info/software/CrystalDiskMark/index-e.html

How come Fortran is 30x slower at unformatted I/O then C ?? No need to find the field start and end, no LOG tables, no slowness of char arrays and still such hell small speeds. Something is still rotten in the state of Denmark in its Fortran district
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Tue Nov 22, 2016 10:16 pm    Post subject: Reply with quote

Dan,

I presume that reading a sequential file at 8491 MB/s is 8.4 GigaBytes per second. These are impressive rates. The problem is you can't process the information at this rate.

You have removed Disk I/O performance as a bottleneck for your analysis with this approach.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Tue Nov 22, 2016 11:03 pm    Post subject: Reply with quote

John,
Yes, these are multi-GB per second I/O speeds. Despite of these crazy speeds the DRAM is still next slowest bottleneck of the computer after harddrives. In my fox hole at North Pole I have older generation DDR3 memory, the newer DDR4 are probably twice faster (my previous DDR2 memory computer had 6GB/s speed with the same QSoft RAMdisk). And processor's caches are more then an order of magnitude faster with L1 cache two orders faster. Given that why we can not process the information at this rate ? This 10GB/s rate is like a turtle for any processor. The rate at which Fortran codes are doing I/O is like a dead man walk

What this CrystalDiskMark benchmark doing is the same like all we do above with all our tests - it is reading/writing and measuring time. But if this test was written in Fortran it'd show the speeds not more then 500-600MB/s because Fortran itself is a bottleneck, it by some unknown reason can not do faster. This is what fast drives like RAMdisk revealed. So the question is : why taking bytes from one place of DRAM (on RAMdisk) and placing them into another place of DRAM using C takes 30x less time then with Fortran? Is Fortran calculation of indexes of an array in the DRAM where data will be placed THAT more expensive then for C to find the address of the same data placed on the RAMdisk file?

We clearly don't use all capabilities of modern hardware. The compiler developers must notice and address this huge discrepancy. Besides RAMdisks, the 3.5GB/s speeds have PCI-Express (not SATA) mainstream reasonably priced SSDs (like the ones from Samsung), all ultraportable computers have at least 1 GB/s. Compare that to the shameful speeds we read/write...

By the way if anyone think that RAMdisks lose information on reboot etc, they are way overprotective. I still have all files on my older computer RAMdisk which are probably 3-4 years old. It makes automatic backups on shutdowns and on predefined intervals and loads that on login
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4, 5, 6
Page 6 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group