View previous topic :: View next topic |
Author |
Message |
dgurok
Joined: 26 May 2011 Posts: 66
|
Posted: Tue Jan 22, 2019 10:46 am Post subject: Read and Write Performance |
|
|
I'm working with large files on a SSD drive and I wonder why read and write takes so much more time. With windows copy command 500MB file is copied in ~1 second.
If I try to read and write the same ascii file it takes ~8-9 seconds.
Does anyone have a clue? Do I need to set some flags or is it caused by overhead? |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 7927 Location: Salford, UK
|
Posted: Tue Jan 22, 2019 11:17 am Post subject: |
|
|
Do you mean READ and WRITE in a Fortran program?
Is it a formatted READ? |
|
Back to top |
|
|
dgurok
Joined: 26 May 2011 Posts: 66
|
Posted: Tue Jan 22, 2019 11:32 am Post subject: |
|
|
Yes, I mean READ and WRITE in a Fortran program and I did a simple formatted READ with ('A').
Code: | OPEN(KA1,FILE=MFILE1,STATUS='readonly',ERR=100)
OPEN(KA2,FILE=MFILE2,ERR=200)
10 READ(KA1,1000,END=100,ERR=100) TXTBUF
WRITE(KA2,1000) TXTBUF(1:LENG(TXTBUF))
GOTO 10
100 CLOSE(KA1)
200 CLOSE(KA2)
1000 FORMAT(A) |
|
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 7927 Location: Salford, UK
|
Posted: Tue Jan 22, 2019 12:38 pm Post subject: |
|
|
Basically formatted input and output will take longer because the data is converted. If you are simply copying from one file to another then you could use unformatted I/O or, better still, call a file copy routine such as CopyFile...
Code: | program main
STDCALL COPYFILE 'CopyFileA' (STRING,STRING,VAL):LOGICAL*4
logical L
integer,parameter::fail_if_exists=1
L = COPYFILE("ExistingFileName","NewFileName",fail_if_exists)
end |
|
|
Back to top |
|
|
dgurok
Joined: 26 May 2011 Posts: 66
|
Posted: Tue Jan 22, 2019 1:50 pm Post subject: |
|
|
It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds. |
|
Back to top |
|
|
mecej4
Joined: 31 Oct 2006 Posts: 1886
|
Posted: Tue Jan 22, 2019 2:25 pm Post subject: Re: |
|
|
dgurok wrote: | It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds. |
The timings look quite reasonable. When you do formatted reads with format A, the bytes read have to be scanned for line endings (CR+LF or just LF), and the count of the bytes remaining in the buffer has to be managed. |
|
Back to top |
|
|
LitusSaxonicum
Joined: 23 Aug 2005 Posts: 2388 Location: Yateley, Hants, UK
|
Posted: Tue Jan 22, 2019 2:36 pm Post subject: |
|
|
dgurok,
You haven't measured the full time taken to do the writing stage, as data are written into the disk buffer, not direct to the device. As the disk buffer is in RAM, the differences in write speeds are downplayed. The hdd is probably still writing after the last write statement for longer than the ssd.
You might get more insight into the problem by reading from the ssd and writing to the hdd, or vice versa. You might also be better doing all the reading followed by all the writing and timing each stage. Some insight into the time taken to do the formatting could be gained by comparing formatted and unformatted methods.
Read and write operations are also influenced by how full or fragmented the device is, and the relative speeds of anything in the cpu or storage devices line. The length of the records processed (and therefore how many there are) will also affect the results.
If you have an application with a Clearwin+ Windows interface, and the timings bother you, then the interface probably needs a progress bar or an animated cursor so the user knows that something time consuming is going on, and the app hasn't crashed.
Eddie |
|
Back to top |
|
|
John-Silver
Joined: 30 Jul 2013 Posts: 1520 Location: Aerospace Valley
|
Posted: Tue Jan 22, 2019 7:37 pm Post subject: |
|
|
Quote: | On normal HDD copy took ~12 seconds and read write ~30 seconds. |
so that's compared with 1 sec (copy) and 8-9 seconds read-write with the SSD.
So the copyu is clearly much faster (an order of magnitude) ... maybe not as much as you'd anticipated but clearly significantly faster.
The 'sabre rattlers' of the SSD world will tell you it will do things 100 times faster , and wash the dishes and do the ironing at the sanìme time of course, but they have different priorities (selling as many as possible.
It's a bit like the internet providers who only quote the fastest speeds and 'forget' to tell you that, although you've bought a 5Mb speed connection, if you live about 5 miles from the telephone exchange then you'll only get half a Mb ! (i.e. an order of magnitude less !)
I guess they don't publish their portfolio of test results because it would reveal the 'variations' possible, and that wouldn't be good fot (their) business
This is why people should take their laptop to the shop and ask to connct an SSD and run a pre-concocted (by you) test .....
.... what do you mean Amazon won't let you into their warehouse ??? _________________ ''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... " |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Wed Jan 23, 2019 3:36 am Post subject: |
|
|
I suggest you check the "Windows copy" performance.
You can review this copy performance using Resource Monitor (available via Task Manager).
What I suspect is that the COPY command reports complete before the write to disk is completed. This is because of the disk I/O buffering.
Assessing the performance it all gets a bit confusing between HDD and SSD due to disk buffering with free memory. I have 32 gb memory installed, so get great apparent performance with HDD.
Comparing between different PC's becomes even more confusing, as it not only depends on the reported transfer speed of the HDD or SSD, but also their controller's and the installed memory available for buffering.
I have also tried to confirm that the file is written to disk, by using a Fortran CLOSE, but the test results are still confusing.
Another problem I identified, but could not confirm, is if the file is already in disk buffers before the test starts.
If you do a Fortran test of write new file, then read file, the test may complete before any disk access actually takes place.
However if you do a test that only reads an existing (on disk) file, you can not re-do the test as the file may still be in memory buffers, or may have already been in memory for the first test.
I have done tests where the test file size exceeds the memory size to flood the buffers, but my real program doesn't have files that large, so what test result is relevant to my real case ?
Very confusing, but fortunately the performance times are becoming very fast, unless you have terabytes of data to analyse. 500MB should not be a significant issue, unless you have very old hardware and O/S, so update ! |
|
Back to top |
|
|
dgurok
Joined: 26 May 2011 Posts: 66
|
Posted: Wed Jan 23, 2019 10:59 am Post subject: |
|
|
Thanks for the feedback.
I did another test with binary read and write and this provides a much better performace. The same test results in 0.6-1.6 seconds.
We have set up a test enviroment with a 10GBit/s network and NVMe-SSDs and we were disappointed about the "poor" performance increase. Our technical people can't belive it |
|
Back to top |
|
|
John-Silver
Joined: 30 Jul 2013 Posts: 1520 Location: Aerospace Valley
|
Posted: Wed Jan 23, 2019 11:32 am Post subject: |
|
|
dgurok,
what is the make, model and size of the SSD's in the system you are using ? _________________ ''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... " |
|
Back to top |
|
|
LitusSaxonicum
Joined: 23 Aug 2005 Posts: 2388 Location: Yateley, Hants, UK
|
Posted: Wed Jan 23, 2019 11:38 am Post subject: |
|
|
John-S,
The best test of an SSD compared to a normal HDD is the speed at which Windows starts up. In that case, memory starts clear, and there's no disk buffer to confuse matters. There's a definite, and very useful, improvement. Running all programs from the SSD gives a less noticeable improvement, but one that's still useful.
Eddie |
|
Back to top |
|
|
dgurok
Joined: 26 May 2011 Posts: 66
|
Posted: Wed Jan 23, 2019 11:48 am Post subject: |
|
|
@John-Silver
Samsung 970 EVO, SSD 250 GB NVMe M.2 |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Wed Jan 23, 2019 2:21 pm Post subject: |
|
|
There are many possible limits on I/O performance.
This can also include the I/O libraries (binary vs text)
I have not been able to approach 10GBit/s (1250 MByte/sec ?), although have achieved about 300 MByte/sec with my latest SSD in some tests. (and 200 MByte/sec with HDD !) It has been a while since I did these tests.
There is clearly some benefit of memory buffers in this, with memory speed and bandwidth also being a factor.
I used Samsung 500 GB 960 EVO PCI Express M.2 SSD |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2818 Location: South Pole, Antarctica
|
Posted: Wed Jan 23, 2019 5:18 pm Post subject: |
|
|
dgurok, Find using Search the testing of I/O speed we have done in this forum 2-3 years ago. There were many demos. One option you already have tried - using unformatted I/O. Optimizing formatted I/O also helps. But dramatic increase you will get only using FTN95 specific functions READF@ and READFA@ of direct loading data into memory of your code. No other compiler has this, there we were achieving read speeds 4GB/s, your slow network will start to hiccup trying to feed your programs at such rate, and your technical people will be ashamed
UPDATE.
Here is the link
http://forums.silverfrost.com/viewtopic.php?t=3377&start=30
Last edited by DanRRight on Thu Jan 24, 2019 5:59 am; edited 1 time in total |
|
Back to top |
|
|
|