forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Database or NetCDF / HDF5 interfaces?
Goto page Previous  1, 2, 3, 4, 5, 6  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Fri Nov 11, 2016 1:39 pm    Post subject: Reply with quote

Dan,

I have done some testing of both binary I/O and text I/O.
I have tested on 2 desktop pc's

i5-2300 with 8gb memory and HDD drives
i7-4790K with 32 gb memory and HDD and SSD drives

A preliminary description of the test results in in the linked word document:

https://www.dropbox.com/s/sif5ect54kqh0zn/test_results.docx?dl=0

The text I/O test program is fairly simple, (I wrote it in an hour so hopefully works correctly!!) It creates files of 130 gb, so needs a bit of disk space. see link:

https://www.dropbox.com/s/ck1etkaqxzsx2wt/test_text.f90?dl=0

These results are very fast, much faster than you appear to be complaining about.
The results have been assembled via a spreadsheet and generated with a pivot table. I shall update these links as I check the results in the next few days.

John
Back to top
View user's profile Send private message
LitusSaxonicum



Joined: 23 Aug 2005
Posts: 2388
Location: Yateley, Hants, UK

PostPosted: Fri Nov 11, 2016 2:06 pm    Post subject: Reply with quote

For those of us who nowadays only program in Fortran, Anton Kruger's 'Efficient Fortran Programming' (Wiley, 1990) contains Fortran routines for compressing and decompressing files using Huffman encoding. Chapter 10, pp 137-173.

Eddie
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sat Nov 12, 2016 12:37 am    Post subject: Reply with quote

John,
Can you please post them here in usual text spreadsheet? We have policy not to open from the net any WORD, EXCEL, PDF script containing files, run EXE etc
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sat Nov 12, 2016 4:09 am    Post subject: Reply with quote

Dan,

The word document is 8 pages with 4 pages of text and 4 pages of charts, but unfortunately no pages of tables as yet. (only a draft)
Perhaps I could link to a text file of text and tables, although the charts are fairly useful.

Alternatively you could do the text file tests by downloading test_text.f90 and run it, as it does not need any libraries, only 130gb of disk space !
ftn95 test_text.f90 /debug /link /map ( or /opt )
Import file text_io.log into excel and this has elapsed time and mb/sec stats for all file operations.
I should have stated the i5 has Win 7_64 and i7 Win 10

I'd be interested in your test results.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sat Nov 12, 2016 11:31 am    Post subject: Reply with quote

John,
OK, from my lately very damn busy fox hole in North Pole (you probably know my saying "Who is not working Saturdays does not work at all" (c) Smile ) there is something strange there. Reading Numbers takes same time no matter what size. Reading text is around 250-350KB/s from SSD Samsung 850Pro, harddrive WD Black and RAMdrive, numbers - around 30

Also, is this real or an artifact that reading numbers is 10x slower then text (?)
Please check

/* If text is indeed 10x faster then direct reading whole line of numbers then may be the faster solution will be to read first line of numbers at ones as single long text variable and then use another so called internal read to read these numbers from text variable
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1885

PostPosted: Sat Nov 12, 2016 12:37 pm    Post subject: Re: Reply with quote

DanRRight wrote:
Also, is this real or an artifact that reading numbers is 10x slower than text (?) If text is indeed 10x faster than direct reading whole line of numbers then may be the faster solution will be to read first line of numbers at ones as single long text variable and then use another so called internal read to read these numbers from text variable

Formatted I/O of numbers is going to be significantly slower than unformatted/stream I/O because it takes time to convert from/to decimal floating point to/from internal binary IEEE format.

Instead of focusing narrowly on formatted I/O alone, you could think about (i) avoiding unnecessary/repetitious I/O, (ii) what your program does with the numbers that are read and written, and tailor the I/O operations to suit.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sat Nov 12, 2016 1:13 pm    Post subject: Reply with quote

Dan,

For the read text and read number text files, I tried to create tests where the file was not in memory buffers. The performance I recorded is in Mbytes per second as:
Code:
Hardware read  read
         text  numbers
         (A80) (10F8.0)
i5-HDD     67    53
i7-HDD    155    93
i7-SSD    172    93


As expected, the reading numbers is slower than read text, as there is more processing.
With the i7, both disk types have similar performance @ 93 mb/sec. This was not expected! I assume from this that the performance is dominated by the time for converting of text to numbers, while disk I/O time is less significant. (I need to check this result and I should confirm if the conversion times are this slow)

For reading text only, this is faster, but I am surprised by the i7-HDD performance, as I would have expected similar to i5-HDD.
Also I would have expected the i7-SSD to be much faster than i7-HDD than was reported. It is as if there is some memory buffering of the reading, but I can't see it from the test approach.

Still, 50 to 67 mb/sec on i5 and 100 to 170 mb/sec on i7 is fairly fast.
You need to identify both disk I/O times and number format conversion times.

If you look at the results for binary reads, with the benefit of memory cache, the performance is much faster, at about 1 gb/sec, although reading a new file would not get this rate.

The main conclusion from this test is the estimated performance is much faster than your post of last week implied. Having available memory for cacheing is very effective.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sat Nov 12, 2016 1:35 pm    Post subject: Reply with quote

First, there probably is some error somewhere as
I have around 300/30 MB/s for text/numbers no matter what PC and disks

In my task I am reading 9 numbers per line and get 31 MB/s (this is easy also to see in Task Manager/Processes/Disk Speed)

Given the copy speed in RAMdrive is 5000-7000 MB/s this gives 200x slower as I wrote above

Will try to modify to internal read in few minutes and see what this gives...

CONTINUE:
Nope, no difference. Conversion takes too much time

EVEN MORE TRIES:
If I use /nockeck /opt the speed gets to 56 MB/s.
But if I use fixed format 9e13.7 instead of * the speed increases a bit to 75 MB/s

Better. Still not 7500 or 750 or at least 300 MB/s Sad Do I have too unreasonable demands?

Also I remember Salford had its own reading commands READF@ and READFA@ instead of READ. Are they faster ?

How about other compilers?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sun Nov 13, 2016 2:04 am    Post subject: Reply with quote

Dan,

Quote:
Given the copy speed in RAMdrive is 5000-7000 MB/s

Your claims of ramdisk etc are not practical/achievable. These rates are faster than memory transfers and would only be available for targeted transfer instructions, with no processing of the contents.

I modified the test_text program to only do the number processing only, ie no disk transfers).
Code:
      do i = 1,2
        read (11,'(A)',iostat=iostat) line
      end do
!
      do i = 1,n
        read (line,fmt='(11f8.0)',iostat=iostat) aa
        if ( iostat /= 0) exit
      end do

https://www.dropbox.com/s/std62006mih51c3/test_read.f90?dl=0

For processing numbers with a format (10F8.0) using FTN95 /link : Ver 8.05 I get:
i5-2300 80 mb/sec, which is 12.8 seconds per gigabyte
i7-4790k 141 mb/sec, which is 7.25 seconds per gigabyte

I previously reported for the i5 that read and process numbers is 53 mb/sec or 19.3 sec per gb, which means there is a delay of 6.5 seconds per gb for reading (read time is 6.5 sec plus due to buffering of the read)

For the i7, read and process is 93 mb/sec or 11.0 sec per gb, which means there is a delay of 3.75 seconds per gb for reading (read time is 3.75 sec plus due to buffering of the read)

Given the performance I reported in the previous post, these numbers look sensible.
The read text times for the I7-HDD are much faster that the I5-HDD : 155 mb/sec vs 67 mb/sec, which may be due to a faster drive and faster processor.
I was also surprised by the SSD rate of 172 mb/sec (I expected higher) but this limit is probably due to the limit of processing the read statement.
character aa*80
read (11,'(a)',iostat=iostat) aa

The performance I have reported for binary transfers are much faster than these text transfers, as they benefit from no/minimal processing of the I/O list and also disk cacheing.

It looks that extra memory (for cacheing) may be a more flexible substitute for a SSD drive, although a SSD drive does separate the memory use and disk caching of available memory from SSD transfers for files and paging.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Nov 13, 2016 6:21 am    Post subject: Reply with quote

John
Look at read and write test results of RAM drives, they all are in 5+ GB/s range. Definitely not with Fortran code. I'm pretty sure also all these tests were done with not overclocked processors and with typical ddr3 1600MHz modules or even DDR2 but they can go to 3200 MHz in extreme cases so transfer rates of RAM are larger, some also use RAID-like streams splitting.

Try RAMdrive, many are free. Or PCI Express based SSDs but they are typically more expensive. You also use few other compilers...

I do not understand what is few seconds buffering in your case and how you count it. Is this what you get?
Code:

 Opening File_1gb.txt : iostat=           0    1.000000E-04
            1 write_text     332.954     mb/sec     3.07550        13421774
 Opening File_2gb.txt : iostat=           0    3.000000E-04
            2 write_text     119.923     mb/sec     17.0776        26843547
 Opening File_4gb.txt : iostat=           0    8.000000E-04
            4 write_text     87.8678     mb/sec     46.6155        53687093
 Opening File_6gb.txt : iostat=           0    2.000000E-04
            6 write_text     115.922     mb/sec     53.0010        80530641
 Opening File_8gb.txt : iostat=           0    6.000000E-04
            8 write_text     94.4901     mb/sec     86.6969       107374185
 Opening File_10gb.txt : iostat=           0    2.200000E-03
           10 write_text     100.273     mb/sec     102.122       134217729
 Opening File_14gb.txt : iostat=           0    2.000000E-04
           14 write_text     94.2578     mb/sec     152.094       187904817
 Opening File_18gb.txt : iostat=           0    1.000000E-04
           18 write_text     93.4449     mb/sec     197.250       241591905
 Opening Number_1gb.txt : iostat=           0    2.000000E-04
            1 write_numb     30.7510     mb/sec     33.2997        13421774
 Opening Number_2gb.txt : iostat=           0    4.000000E-04
            2 write_numb     37.1232     mb/sec     55.1676        26843547
 Opening Number_4gb.txt : iostat=           0    2.000000E-04
            4 write_numb     37.1622     mb/sec     110.220        53687093
 Opening Number_6gb.txt : iostat=           0    1.000000E-04
            6 write_numb     37.1684     mb/sec     165.302        80530641
 Opening Number_8gb.txt : iostat=           0    1.000000E-04
            8 write_numb     37.0267     mb/sec     221.245       107374185
 Opening Number_10gb.txt : iostat=           0    4.000000E-04
           10 write_numb     36.8365     mb/sec     277.985       134217729
 Opening Number_14gb.txt : iostat=           0    1.000000E-04
           14 write_numb     36.9128     mb/sec     388.375       187904817
 Opening Number_18gb.txt : iostat=           0    2.000000E-04
           18 write_numb     36.8586     mb/sec     500.073       241591905
 Opening File_1gb.txt : iostat=           0    0.634000


Last edited by DanRRight on Sun Nov 13, 2016 9:02 am; edited 1 time in total
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Nov 13, 2016 8:50 am    Post subject: Reply with quote

Code:

 Opening Number_1gb.txt : iostat=           0    0.100300
            1 read_numb     92.4931     mb/sec     11.0711         1.00000        13421775
 Opening Number_2gb.txt : iostat=           0    4.420000E-02
            2 read_numb     92.5687     mb/sec     22.1241         2.00000        26843548
 Opening Number_4gb.txt : iostat=           0    3.040000E-02
            4 read_numb     92.4252     mb/sec     44.3169         4.00000        53687094
 Opening Number_6gb.txt : iostat=           0    4.260000E-02
            6 read_numb     92.7522     mb/sec     66.2410         6.00000        80530642
 Opening Number_8gb.txt : iostat=           0    3.510000E-02
            8 read_numb     92.9064     mb/sec     88.1748         8.00000       107374186
 Opening Number_10gb.txt : iostat=           0    2.750000E-02
           10 read_numb     93.3271     mb/sec     109.722         10.0000       134217730
 Opening Number_14gb.txt : iostat=           0    5.080000E-02
           14 read_numb     93.3013     mb/sec     153.653         14.0000       187904818
 Opening Number_18gb.txt : iostat=           0    1.000000E-04
           18 read_numb     92.9581     mb/sec     198.283         18.0000       241591906
 Opening Number_1gb.txt : iostat=           0    5.060000E-02
            1 process_numb     141.638     mb/sec     7.22970         1.00000        13421774           0
 Opening Number_2gb.txt : iostat=           0    1.000000E-04
            2 process_numb     141.286     mb/sec     14.4954         2.00000        26843547           0
 Opening Number_4gb.txt : iostat=           0    3.460000E-02
            4 process_numb     141.092     mb/sec     29.0307         4.00000        53687093           0
 Opening Number_6gb.txt : iostat=           0    1.000000E-04
            6 process_numb     141.251     mb/sec     43.4971         6.00000        80530641           0
 Opening Number_8gb.txt : iostat=           0    2.000000E-04
            8 process_numb     141.062     mb/sec     58.0736         8.00000       107374185           0
 Opening Number_10gb.txt : iostat=           0    2.480000E-02
           10 process_numb     140.938     mb/sec     72.6559         10.0000       134217729           0
 Opening Number_14gb.txt : iostat=           0    2.670000E-02
           14 process_numb     141.211     mb/sec     101.522         14.0000       187904817           0
 Opening Number_18gb.txt : iostat=           0    1.000000E-04
           18 process_numb     141.078     mb/sec     130.651         18.0000       241591905           0
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sun Nov 13, 2016 12:21 pm    Post subject: Reply with quote

Hi Dan,

I get similar to what you have presented on my i7, but report slower performance on my i5.

Your results are averaging:
Write text : 100 mb/sec (variable due to managing memory cacheing)
write number : 37 mb/sec (uniform)
read text : ???
read number : 92 mb/sec (uniform)
process number : 141 mb/sec (uniform)

From your results:
read numbers is 1024/92 = 11.1 seconds per gigabyte
process numbers is 1024/141 = 7.3 seconds per gigabyte
so 3.8 seconds delay for reading file.

These read tests are basically un-buffered, although the early write tests would be buffered, hence the greater variability.

I am not sure I fully understand the claims about ram drives, as ddr3 1600 mhz is a transfer rate of 1.6 ghz x packet size.
The other issue I suspect is that SSD drives use a disk driver and so are limited by the bandwidth of the pseudo device driver. Not sure if this is still a problem.

I think your practical limit for reading numbers is the 141 mb/sec, which is the processing rate; not much faster that the 92 mb/sec your disk provides. Basically about 10 seconds per gigabyte is what to expect for reading numbers from a .csv file.

I should do this test using FTN /64 to see if there is any change.
Also, I should test gFortran as it may have a problem with this; it's speed can be very poor for processing I/O lists.

Hope this provides some context to what is practical performance for large files.

John
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1885

PostPosted: Sun Nov 13, 2016 2:45 pm    Post subject: Re: Reply with quote

DanRRight wrote:
... there probably is some error somewhere as
I have around 300/30 MB/s for text/numbers no matter what PC and disks
...
How about other compilers?


As I stated earlier, I think that you should separate device I/O time from format processing time.

Here is a very different example case. The input file contained 580930 text records, each containing 58 characters, with LF as EOL. The C program is multithreaded, (i) reads the data file with a single block read() call, (ii) scans the block for LF to set up a pointer array to the individual records, (iii) sorts the records on a field, and (iv) outputs the sorted records to a file. All the files are on a Ramdisk (DataRam, not the fastest among the choices available) on a Dell laptop with i7-2720 CPU, 8 GB RAM.

Code:
PHASE                    TIME

Block read file          0.078 s
Split into records       0.015
Sort                     0.063
Output file              0.297

The block read time corresponds to a throughput of 440 MB/s, which is in the same range as what you have for "text read".

The file output was done with one call to fputs() for each record, plus an fputc('\n') for each record, and could be made faster by writing the sorted records to a buffer and doing block writes on full buffer.

Quote:
I remember Salford had its own reading commands READF@ and READFA@ instead of READ. Are they faster ?

READF does a block read of raw data. You would have to separate the data into lines and read individual lines under format control. READFA reads one line at a time, so it should give similar performance to your "text read" version.
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Nov 13, 2016 11:11 pm    Post subject: Reply with quote

The results I posted are done on harddrive. I can not use ramdrives temporally as I will need reboot for that while I'm under stressful deadline now. SSDs and all other drives are also full. As for transfer rates of DDR3 RAM - see for example Wiki. And currently more and more becoming mainstream DDR4 PC4-2400+ have 20 GB/s rates

On my own code I do not see improvement if I use text read into char variable first and then convert it into numbers with internal read

Please continue testing with other compilers and please check if READFA is heavily optimized for read as I remember this claim of Salford in the past.

Mecej4, though this could be of minor influence but still would be good if you adopt the same initial conditions for example take all sizes for ASCII read and numbers per line like in John's case. Also please do John's test for us to know how fast is your laptop on the same task at least for small file sizes


Last edited by DanRRight on Mon Nov 14, 2016 12:54 am; edited 7 times in total
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sun Nov 13, 2016 11:40 pm    Post subject: Reply with quote

Hi Dan,

I have been doing some other testing with FTN95 /64 and gFortran.
They are both much worse for writing numbers to file, but better for reading numbers, which is your original question.

Paul, you should check FTN95 test_read.f90 /64 /link, as for the write number test using FTN95 /64, as I am getting 8.6 mb/sec vs 24.1 mb/sec for /32 on my i5-HDD pc. (you may want to only test smaller files, given the performance!)

Strangely FTN95 /64 is faster for processing numbers (133 mb/sec vs 80 mb/sec), reading numbers (89 mb/sec vs 56 mb/sec) or writing text (105 mb/sec vs 80 mb/sec). Not sure why an internal read is faster with /64 and also write text (??), but write numbers is so much slower. Hopefully FTN95 /64 /opt might have some answers.

gFortran is also much worse again for write numbers at 4.3 mb/sec for writing numbers, which is a known long term weakness for gFortran.

John
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4, 5, 6  Next
Page 2 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group