forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Read and Write Performance
Goto page 1, 2  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support
View previous topic :: View next topic  
Author Message
dgurok



Joined: 26 May 2011
Posts: 66

PostPosted: Tue Jan 22, 2019 10:46 am    Post subject: Read and Write Performance Reply with quote

I'm working with large files on a SSD drive and I wonder why read and write takes so much more time. With windows copy command 500MB file is copied in ~1 second.
If I try to read and write the same ascii file it takes ~8-9 seconds.

Does anyone have a clue? Do I need to set some flags or is it caused by overhead?
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Tue Jan 22, 2019 11:17 am    Post subject: Reply with quote

Do you mean READ and WRITE in a Fortran program?
Is it a formatted READ?
Back to top
View user's profile Send private message AIM Address
dgurok



Joined: 26 May 2011
Posts: 66

PostPosted: Tue Jan 22, 2019 11:32 am    Post subject: Reply with quote

Yes, I mean READ and WRITE in a Fortran program and I did a simple formatted READ with ('A').

Code:
      OPEN(KA1,FILE=MFILE1,STATUS='readonly',ERR=100)
      OPEN(KA2,FILE=MFILE2,ERR=200)
10    READ(KA1,1000,END=100,ERR=100) TXTBUF
      WRITE(KA2,1000) TXTBUF(1:LENG(TXTBUF))
      GOTO 10
100   CLOSE(KA1)
200   CLOSE(KA2)
1000  FORMAT(A)
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7916
Location: Salford, UK

PostPosted: Tue Jan 22, 2019 12:38 pm    Post subject: Reply with quote

Basically formatted input and output will take longer because the data is converted. If you are simply copying from one file to another then you could use unformatted I/O or, better still, call a file copy routine such as CopyFile...

Code:
program main
STDCALL COPYFILE 'CopyFileA' (STRING,STRING,VAL):LOGICAL*4
logical L
integer,parameter::fail_if_exists=1
L = COPYFILE("ExistingFileName","NewFileName",fail_if_exists)
end
Back to top
View user's profile Send private message AIM Address
dgurok



Joined: 26 May 2011
Posts: 66

PostPosted: Tue Jan 22, 2019 1:50 pm    Post subject: Reply with quote

It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds.
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1885

PostPosted: Tue Jan 22, 2019 2:25 pm    Post subject: Re: Reply with quote

dgurok wrote:
It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds.

The timings look quite reasonable. When you do formatted reads with format A, the bytes read have to be scanned for line endings (CR+LF or just LF), and the count of the bytes remaining in the buffer has to be managed.
Back to top
View user's profile Send private message
LitusSaxonicum



Joined: 23 Aug 2005
Posts: 2388
Location: Yateley, Hants, UK

PostPosted: Tue Jan 22, 2019 2:36 pm    Post subject: Reply with quote

dgurok,

You haven't measured the full time taken to do the writing stage, as data are written into the disk buffer, not direct to the device. As the disk buffer is in RAM, the differences in write speeds are downplayed. The hdd is probably still writing after the last write statement for longer than the ssd.

You might get more insight into the problem by reading from the ssd and writing to the hdd, or vice versa. You might also be better doing all the reading followed by all the writing and timing each stage. Some insight into the time taken to do the formatting could be gained by comparing formatted and unformatted methods.

Read and write operations are also influenced by how full or fragmented the device is, and the relative speeds of anything in the cpu or storage devices line. The length of the records processed (and therefore how many there are) will also affect the results.

If you have an application with a Clearwin+ Windows interface, and the timings bother you, then the interface probably needs a progress bar or an animated cursor so the user knows that something time consuming is going on, and the app hasn't crashed.

Eddie
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1520
Location: Aerospace Valley

PostPosted: Tue Jan 22, 2019 7:37 pm    Post subject: Reply with quote

Quote:
On normal HDD copy took ~12 seconds and read write ~30 seconds.


so that's compared with 1 sec (copy) and 8-9 seconds read-write with the SSD.

So the copyu is clearly much faster (an order of magnitude) ... maybe not as much as you'd anticipated but clearly significantly faster.

The 'sabre rattlers' of the SSD world will tell you it will do things 100 times faster , and wash the dishes and do the ironing at the sanìme time of course, but they have different priorities (selling as many as possible.

It's a bit like the internet providers who only quote the fastest speeds and 'forget' to tell you that, although you've bought a 5Mb speed connection, if you live about 5 miles from the telephone exchange then you'll only get half a Mb ! (i.e. an order of magnitude less !)

I guess they don't publish their portfolio of test results because it would reveal the 'variations' possible, and that wouldn't be good fot (their) business Sad

This is why people should take their laptop to the shop and ask to connct an SSD and run a pre-concocted (by you) test .....
.... what do you mean Amazon won't let you into their warehouse ???
_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Wed Jan 23, 2019 3:36 am    Post subject: Reply with quote

I suggest you check the "Windows copy" performance.
You can review this copy performance using Resource Monitor (available via Task Manager).
What I suspect is that the COPY command reports complete before the write to disk is completed. This is because of the disk I/O buffering.
Assessing the performance it all gets a bit confusing between HDD and SSD due to disk buffering with free memory. I have 32 gb memory installed, so get great apparent performance with HDD.
Comparing between different PC's becomes even more confusing, as it not only depends on the reported transfer speed of the HDD or SSD, but also their controller's and the installed memory available for buffering.
I have also tried to confirm that the file is written to disk, by using a Fortran CLOSE, but the test results are still confusing.

Another problem I identified, but could not confirm, is if the file is already in disk buffers before the test starts.
If you do a Fortran test of write new file, then read file, the test may complete before any disk access actually takes place.
However if you do a test that only reads an existing (on disk) file, you can not re-do the test as the file may still be in memory buffers, or may have already been in memory for the first test.

I have done tests where the test file size exceeds the memory size to flood the buffers, but my real program doesn't have files that large, so what test result is relevant to my real case ?

Very confusing, but fortunately the performance times are becoming very fast, unless you have terabytes of data to analyse. 500MB should not be a significant issue, unless you have very old hardware and O/S, so update !
Back to top
View user's profile Send private message
dgurok



Joined: 26 May 2011
Posts: 66

PostPosted: Wed Jan 23, 2019 10:59 am    Post subject: Reply with quote

Thanks for the feedback.

I did another test with binary read and write and this provides a much better performace. The same test results in 0.6-1.6 seconds.

We have set up a test enviroment with a 10GBit/s network and NVMe-SSDs and we were disappointed about the "poor" performance increase. Our technical people can't belive it Very Happy
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1520
Location: Aerospace Valley

PostPosted: Wed Jan 23, 2019 11:32 am    Post subject: Reply with quote

dgurok,
what is the make, model and size of the SSD's in the system you are using ?
_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
LitusSaxonicum



Joined: 23 Aug 2005
Posts: 2388
Location: Yateley, Hants, UK

PostPosted: Wed Jan 23, 2019 11:38 am    Post subject: Reply with quote

John-S,

The best test of an SSD compared to a normal HDD is the speed at which Windows starts up. In that case, memory starts clear, and there's no disk buffer to confuse matters. There's a definite, and very useful, improvement. Running all programs from the SSD gives a less noticeable improvement, but one that's still useful.

Eddie
Back to top
View user's profile Send private message
dgurok



Joined: 26 May 2011
Posts: 66

PostPosted: Wed Jan 23, 2019 11:48 am    Post subject: Reply with quote

@John-Silver

Samsung 970 EVO, SSD 250 GB NVMe M.2
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Wed Jan 23, 2019 2:21 pm    Post subject: Reply with quote

There are many possible limits on I/O performance.
This can also include the I/O libraries (binary vs text)

I have not been able to approach 10GBit/s (1250 MByte/sec ?), although have achieved about 300 MByte/sec with my latest SSD in some tests. (and 200 MByte/sec with HDD !) It has been a while since I did these tests.
There is clearly some benefit of memory buffers in this, with memory speed and bandwidth also being a factor.

I used Samsung 500 GB 960 EVO PCI Express M.2 SSD
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Wed Jan 23, 2019 5:18 pm    Post subject: Reply with quote

dgurok, Find using Search the testing of I/O speed we have done in this forum 2-3 years ago. There were many demos. One option you already have tried - using unformatted I/O. Optimizing formatted I/O also helps. But dramatic increase you will get only using FTN95 specific functions READF@ and READFA@ of direct loading data into memory of your code. No other compiler has this, there we were achieving read speeds 4GB/s, your slow network will start to hiccup trying to feed your programs at such rate, and your technical people will be ashamed

UPDATE.
Here is the link
http://forums.silverfrost.com/viewtopic.php?t=3377&start=30


Last edited by DanRRight on Thu Jan 24, 2019 5:59 am; edited 1 time in total
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> Support All times are GMT + 1 Hour
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group