Can REAL*4 array store more than 4GB?

Pail,

I am trying to understand what is wrong, so I created a 'simpler' example. This uses records of 256 bytes (no header/footer) and reads a single byte from a random address.

If I:

1. write the records sequentially, without an address, then read randomly using pos=calculated_address appears to work.
2. write the records sequentially, with a calculated address, then read randomly using pos= with the same calculated address approach, it FAILS.
3. write the records randomly, with a calculated address, then read pos= with the same calculated address, it FAILS. With this random write test 3, the file appears to be truncated, possible after the last write, as I got an end of file error for a valid address in the read test with 212 records of 28 length. (1MB file)

This program example is possibly easier to follow and identify the problem(s).

Test 1 : gives hope, as the read (pos=) test appears to work, but Test 2 : write ( pos=) looks to be incorrect. Test 3 : random write could be worrying for rewriting data ? All tests appear to work with gFortran, hopefully showing my approach to stream I/O is as expected.

The calculated address: for record:byte in read > address = (record-1)*256 + byte for start of record in write > address = (record-1)*256+1 for byte read > byte value = mod (address,record_length)

https://www.dropbox.com/s/jpdu6f8ub0z9d63/stream_read.f90?dl=0

For stream I/O to be a useful approach for large files (> 4GB), pos= needs to support an 8-byte addresses.

John

Thank you for the feedback. I don't have a response at the moment. This issue is logged for investigation but other work is currently in hand.

Thanks Paul,

I was trying to understand where the problems may be.

INQUIRE ( .. pos= ) and WRITE ( .. pos= ) work differently to gFortran, while READ ( .. pos= ) appears to work.

My comparison to gFortran is as I assume this is standard compilant, I hope this is the case. If clearer comparison is required, I could remove use of RANDOM_NUMBER.

I am hoping to increase the record size and number of records in the test, combined with a 8-byte address and test much larger files. (on a better PC)

I look forward to what you find when time is available.

John

So, is there some chance any time soon that REAL*4 arrays will be unlimited size ?

Quoted from DanRRight So, is there some chance any time soon that REAL*4 arrays will be unlimited size ?

The day after we are able to buy PCs with unlimited RAM size, CPUs able to access addresses with no limit, etc.

Dan,

I am typically allocating REAL*8 vectors that are 24 GBytes in size, for a Skyline storage solver and getting reliable results, so vector indexes larger than 2 gbytes are not an issue. At the moment I am generating, using and discarding the array, but not waiting around to store the data on disk. I am considering writing the vector to disk, probably as multiple vectors smaller than 2GBytes, as I use a virtual blocking for the solver, based on the L3 cache size. This would be a useful approach to writing a large array using blocks (records) smaller than 2GBytes. At the moment I have 64-gbytes of physical memory installed, but who knows what it wil be in 2022, post covid!, which appears to be the excuse for all constraints.

It is difficult to keep up with what M.2 drive would be required to support the new 64-bit stream I/O !

John, Everyone have to keep the data for 10 years after publication in case anyone will ask. Now imagine to keep even 100 TB of data, that how much will be if keep just major selected files. If keep everything it could be 1000TB. Typically these are harddrives, and good ones

Mecej4, of course unlimited within 2^{64 space which is 2}32 times (or few billion times) larger than today FTN95 real*4 allows. Doesn't it sound like unimaginable and almost unlimited today? If space will double every 2 years, and now 16 TB is common disk size, this will require ~60 years to reach it, some young people will see it. I'd be happy with just 1000x (re-phrasing well-known saying, the '1000x will be enough for everyone' or '16 petabytes will be enough for everyone besides supercomputers') or whatever limit Microsoft and Intel restrict their current processors and OS. These two do not allow to address the entire 64-bit space just to control the market and open it bit by bit.

Dan, just the thing for you: 100 TB HDD for $40,000 . If you get one, don't drop it!

 https://nimbusdata.com/products/exadrive/pricing/

Mecej4 Yea, wrap me 3.

They are too slow because consist of regular SSD, 500 MB/s. That's ~3 days to wait when 100TB fill the disk. NVMe with PCIe 5 are 30x faster

And can you imagine 10 years from now they all will lie on a city dump ?

Quoted from DanRRight Everyone have to keep the data for 10 years after publication in case anyone will ask

This becomes quite the problem. In most cases, I try to do analysis via .bat files and I store the data files / generation information to create all these large data sets, but how do you store the program that generated this ?

I do have regular backups of the source code of the programs, but also storing the compiler and .dll's is required to regenerate the results.

I do find that, as the program evolves, some of the data rules change and the approach to 'non-typical' data also evolves. Reproducing results from only 5 years ago is difficult, as the analysis approach changes and in my case adapting the souution approach to the computer hardware.

This is my problem with fairly established analysis techniques and I am sure your requirements for reproducing results would be more difficult.

John, Good you reminded to backup sources and bat files too

By the way i listen and like also another John Campbell, he is from UK, on the Youtube who comments things about this fashionable illness the name of which better not to pronounce on the network. Be carefully listening this though there in Australia, you may get 10 years in the clink from your name twin 😃

Please delete this if not appropriate, but;

Any reference to Australia having severe restrictions about this fashionable illness are unbelievably distorted ! eg TC on Fox was ludicrous. Compare the fatality rates in Australia to Florida and you should conclude that a more accurate discussion could be helpful.

John,

I will be careful with wording but you will understand.

Thing change fast, John. Florida has the same population as AU but has currently twice less people's 'Blue Screen Of D' per day than even your totally closed AU after implementing treatment using monoclonals instead of fashionable procedure into the arm. With this it is also in the leaders today even in the US and will lead even more as another John Campbell showed recently results of naturally obtained resistance because Florida does not close or lock itself.

And TC is really interesting and funny young guy. Due to him i for first time looked at Fox despite do not watch any massmedia for 25 years. Still massmedia is massmedia, too much of obvious propaganda. What he said was too funny?

To All : by the way one senator from Florida claims these unbelievable things about England. Is this true ? Listen these couple minutes on Youtube starting from 11:45

https://youtu.be/fha3yxLIV1c?t=705

Dan,

'one senator from Florida' is not a good start.

If 67% of poputaloun are vacinated what % of deaths would suggest vaccines are not effective ? I don't think this senator gets that.

I did get the following headline from News corp / sky news, which is not a reliable source ?

COVID death risk 32 times higher for the unvaccinated, figures suggest

https://news.sky.com/story/covid-death-risk-32-times-higher-for-the-unvaccinated-figures-suggest-12457074

I do wonder what is happening to democracy in USA. Why are vaccines a political choice ?

What figure there shows with the peak at 55? What these units mean? Why in May unvacced were lower than 1 shot vacced? And why currently there is almost zero difference between them?

Dan, What is your point ? A senator is mis-quoting some medical statistics ? The statistics appear wrong ? medical science is wrong ? Do you have a point to make about how medical statistics are being reported ? Lets limit ourselves to talking rubbish about Fortran !!

Nothing could be wrong.

I just discussed the curves in the link you gave. Because it is obvious that you and the link i gave compare different things. Your number was obtained by integration of blue and green curves, when the main contribution was in Jan-Mar, while the guy tells about the part starting in May.

By the way this part of forum is about 'basically anything that takes your fancy!' What discussed is actually extremely important for fortraneers because of average age of users.

More surprises https://youtu.be/8G_8TT7xuNU?t=524

I usually make typos, errors, mistakes when i do first versions of anything and it takes me a week to gather all puzzle pieces. I got even more shocking data but may be i made a mistake. Can anyone double check last table in this report from which follows totally unbelievable. This is official data about Scotland last month. Looks like new data partially corroborates what Rick Scott mentioned for different period of time or i miss something ? Why i ask? Because Dr.John Campbell from UK (not our JC) which i still follow on the net does not mention something like this and on emails does not respond and because nobody on this planet bothers to check the numbers

https://publichealthscotland.scot/media/10091/21-11-10-covid19-publication_report.pdf