JohnCampbell
Joined: 16 Feb 2006
Posts: 2615
Location: Sydney
|
 Posted: Thu Jan 24, 2019 7:37 am Post subject: Re: |
 |
|
| DanRRight wrote: | | Soon many people will start asking why regular READ is so hell slow |
Regular reads, especially text reads, can be limited by the processing rate of the I/O library routines. As I recall, my I/O library attempts process at about 0.5 GB/sec.
At 1 gbyte/sec, this would be equivalent to 4 processor cycles per byte, which I would expect is a fast processing rate. To get more for character conversion, you could be looking at a vector (AVX registers?) or multi-thread approach (for very long strings).
Achieving 4 gbyte/sec would probably be no conversion, ie byte (binary) transfer. These would be rates that would probably exceed any data processing capacity, such as even basic calculating of mean, standard deviation or high/low limits. Consider the processing rates if also getting the median !
Dgurok, what type of data processing are you performing ?
With HDD, we were once limited to about 20-30 MByte/sec. Options of SSD or memory disk buffers have moved the goal posts, highlighting the processing rates and our data processing limitations.
FTN95 binary transfer library is useful for this type of problem. |
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
