Read and Write Performance

I'm working with large files on a SSD drive and I wonder why read and write takes so much more time. With windows copy command 500MB file is copied in ~1 second. If I try to read and write the same ascii file it takes ~8-9 seconds.

Does anyone have a clue? Do I need to set some flags or is it caused by overhead?

Do you mean READ and WRITE in a Fortran program? Is it a formatted READ?

Yes, I mean READ and WRITE in a Fortran program and I did a simple formatted READ with ('A').

      OPEN(KA1,FILE=MFILE1,STATUS='readonly',ERR=100)
      OPEN(KA2,FILE=MFILE2,ERR=200)
10    READ(KA1,1000,END=100,ERR=100) TXTBUF
      WRITE(KA2,1000) TXTBUF(1:LENG(TXTBUF))
      GOTO 10
100   CLOSE(KA1)
200   CLOSE(KA2)
1000  FORMAT(A) 

Basically formatted input and output will take longer because the data is converted. If you are simply copying from one file to another then you could use unformatted I/O or, better still, call a file copy routine such as CopyFile...

program main
STDCALL COPYFILE 'CopyFileA' (STRING,STRING,VAL):LOGICAL*4
logical L
integer,parameter::fail_if_exists=1
L = COPYFILE('ExistingFileName','NewFileName',fail_if_exists)
end

It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds.

Quoted from dgurok It was just a test case to figured out where we lose time. After read I want to process my data, so its not just a copy think. I would have expected a much more faster result compared to normal HDDs. On normal HDD copy took ~12 seconds and read write ~30 seconds.

The timings look quite reasonable. When you do formatted reads with format A, the bytes read have to be scanned for line endings (CR+LF or just LF), and the count of the bytes remaining in the buffer has to be managed.

dgurok,

You haven't measured the full time taken to do the writing stage, as data are written into the disk buffer, not direct to the device. As the disk buffer is in RAM, the differences in write speeds are downplayed. The hdd is probably still writing after the last write statement for longer than the ssd.

You might get more insight into the problem by reading from the ssd and writing to the hdd, or vice versa. You might also be better doing all the reading followed by all the writing and timing each stage. Some insight into the time taken to do the formatting could be gained by comparing formatted and unformatted methods.

Read and write operations are also influenced by how full or fragmented the device is, and the relative speeds of anything in the cpu or storage devices line. The length of the records processed (and therefore how many there are) will also affect the results.

If you have an application with a Clearwin+ Windows interface, and the timings bother you, then the interface probably needs a progress bar or an animated cursor so the user knows that something time consuming is going on, and the app hasn't crashed.

Eddie

I suggest you check the 'Windows copy' performance. You can review this copy performance using Resource Monitor (available via Task Manager). What I suspect is that the COPY command reports complete before the write to disk is completed. This is because of the disk I/O buffering. Assessing the performance it all gets a bit confusing between HDD and SSD due to disk buffering with free memory. I have 32 gb memory installed, so get great apparent performance with HDD. Comparing between different PC's becomes even more confusing, as it not only depends on the reported transfer speed of the HDD or SSD, but also their controller's and the installed memory available for buffering. I have also tried to confirm that the file is written to disk, by using a Fortran CLOSE, but the test results are still confusing.

Another problem I identified, but could not confirm, is if the file is already in disk buffers before the test starts. If you do a Fortran test of write new file, then read file, the test may complete before any disk access actually takes place. However if you do a test that only reads an existing (on disk) file, you can not re-do the test as the file may still be in memory buffers, or may have already been in memory for the first test.

I have done tests where the test file size exceeds the memory size to flood the buffers, but my real program doesn't have files that large, so what test result is relevant to my real case ?

Very confusing, but fortunately the performance times are becoming very fast, unless you have terabytes of data to analyse. 500MB should not be a significant issue, unless you have very old hardware and O/S, so update !

Thanks for the feedback.

I did another test with binary read and write and this provides a much better performace. The same test results in 0.6-1.6 seconds.

We have set up a test enviroment with a 10GBit/s network and NVMe-SSDs and we were disappointed about the 'poor' performance increase. Our technical people can't belive it 😄

John-S,

The best test of an SSD compared to a normal HDD is the speed at which Windows starts up. In that case, memory starts clear, and there's no disk buffer to confuse matters. There's a definite, and very useful, improvement. Running all programs from the SSD gives a less noticeable improvement, but one that's still useful.

Eddie

@John-Silver

Samsung 970 EVO, SSD 250 GB NVMe M.2

There are many possible limits on I/O performance. This can also include the I/O libraries (binary vs text)

I have not been able to approach 10GBit/s (1250 MByte/sec ?), although have achieved about 300 MByte/sec with my latest SSD in some tests. (and 200 MByte/sec with HDD !) It has been a while since I did these tests. There is clearly some benefit of memory buffers in this, with memory speed and bandwidth also being a factor.

I used Samsung 500 GB 960 EVO PCI Express M.2 SSD

dgurok, Find using Search the testing of I/O speed we have done in this forum 2-3 years ago. There were many demos. One option you already have tried - using unformatted I/O. Optimizing formatted I/O also helps. But dramatic increase you will get only using FTN95 specific functions READF@ and READFA@ of direct loading data into memory of your code. No other compiler has this, there we were achieving read speeds 4GB/s, your slow network will start to hiccup trying to feed your programs at such rate, and your technical people will be ashamed

UPDATE. Here is the link https://forums.silverfrost.com/Forum/Topic/2987&start=30

Dan,

At the rates you reported, you are not doing disk transfers but doing transfers between the disk memory buffers and the program memory. Using Resource Monitor to review memory and disk usage should demonstrate this, although the tests don't last long !!

At these apparent rates, there are so many potential bottlenecks (bandwidth limitations), including memory speeds and software conversion rates. My tests showed my software limits of 300 to 500 Mibytes / sec, so I could never demonstrate the quoted SSD performance for my test problems.

Your test results are not a problem, but it is important to be able to reproduce the disk buffering to be available in the real problem. Strategies to pre-charge the buffers could be effective, you just need to understand how. ( open the file earlier might help?)

However, for the case where you are reading large data feeds, say 10x installed memory ( 500 gb file ) disk transfer performance becomes the issue. I haven't had this problem for 10 years, when I used a pre-processor to filter out the raw data to a smaller interface file for multiple analysis (marine survey data).

It all depends on the rate at which you are receiving the data vs the rate required to process it, which is a small subset of all data processing projects. Not many data feeds are at 4 GiBy / sec and if they were you would have the budget to but a better 'disk' controller.

John, Large data files become pretty common thing. NVMe at 3GB/s speeds are also common, no RAM buffering needed, they become mainstream at < $0.25/GB. Soon many people will start asking why regular READ is so hell slow

Quoted from DanRRight Soon many people will start asking why regular READ is so hell slow

Regular reads, especially text reads, can be limited by the processing rate of the I/O library routines. As I recall, my I/O library attempts process at about 0.5 GB/sec.

At 1 gbyte/sec, this would be equivalent to 4 processor cycles per byte, which I would expect is a fast processing rate. To get more for character conversion, you could be looking at a vector (AVX registers?) or multi-thread approach (for very long strings).

Achieving 4 gbyte/sec would probably be no conversion, ie byte (binary) transfer. These would be rates that would probably exceed any data processing capacity, such as even basic calculating of mean, standard deviation or high/low limits. Consider the processing rates if also getting the median ! Dgurok, what type of data processing are you performing ?

With HDD, we were once limited to about 20-30 MByte/sec. Options of SSD or memory disk buffers have moved the goal posts, highlighting the processing rates and our data processing limitations.

FTN95 binary transfer library is useful for this type of problem.