forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Can REAL*4 array store more than 4GB?
Goto page Previous  1, 2, 3, 4  Next
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7148
Location: Salford, UK

PostPosted: Fri Oct 01, 2021 6:31 pm    Post subject: Reply with quote

John

Can you post a short sample program together with a note of what you get and what you expect.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Sat Oct 02, 2021 3:20 am    Post subject: Reply with quote

Paul,

The following code is my test code for stream I/O.

(it is a work in progress for a 64-bit address, random access, stream I/O library so some variable names need updating)

The 2 FTN95 problems I am seeing are:
pos=address reports the last byte used, rather than next byte
pos=address does not support integer*8 address.

Does stream I/O support files larger than 2GBytes ?

I am using FTN95 ver 8.80 /64 and gFortran ver 11.1.0

I run it in Plato using either FTN95 or gFortran via tools>settings>miscellaneous.

It first writes 5 records using stream I/O with pos=address (I*4)
It reports the pos=address as correct for the first record, but then wrong by 1 byte for all others.
At the start of the file INQUIRE (lu, pos=address) reports the next byte to be used, but after this it reports the last byte used.
gFortran always reports the next byte to be used, which I think is correct ?

see line 117 for INQUIRE ( pos= )

My Read_Stream_i4 routine (L149) returns both the array and its size for the given record address, by reading the header, but the address previously obtained from INQUIRE in write_stream_i4 (L123) is wrong.

I am providing links for code and runs on the 2 compilers

https://www.dropbox.com/s/9ruvvl8w4ip3cuh/stream_test_f95.f90?dl=0
https://www.dropbox.com/s/1q5bmiw9arxt8zb/ftn95_run.log?dl=0
https://www.dropbox.com/s/rtqswue4eg91egl/gF_run.log?dl=0
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7148
Location: Salford, UK

PostPosted: Sat Oct 02, 2021 9:29 am    Post subject: Reply with quote

Thanks John. I have downloaded your program.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Sun Oct 03, 2021 3:00 am    Post subject: Reply with quote

Paul,

I have previously mentioned the problem with INQUIRE (pos=.

On further testing, I am finding a problem with READ (... POS=..) is not correctly positioning to the file where WRITE (... POS=..) previously did.

My test basically writes a set of records sequentially, using 3 writes, as:

if (address < 0) INQUIRE ( pos=address )
WRITE ( pos=address) data_byte_length
WRITE ( (sequentially) ) data
WRITE ( (sequentially) data_byte_length

Even if "address" is wrong by a few bytes, the start of each record should not be destroyed. I do test "address" and I do correct for the 1-byte error in INQUIRE and it appears to be correct.

I then read the records randomly, taking the "address" from a table that was assembled when the record was written, using 3 reads:

if (address < 0) INQUIRE ( pos=address )
READ ( pos=address) data_byte_length
READ ( (sequentially) ) data
READ ( (sequentially) data_byte_length

However, the first read of "data_byte_length" is not valid.
This indicates that pos=address is not working the same for WRITE then READ. The following link is for an updated test, where I write the 4 bytes of "data_byte_length", which apears to show for 1 instance the read positioning is out by 1 byte.
WRITE and READ are inconsistent in their positioning using pos=same_address

The following example may show this better.

https://www.dropbox.com/s/mr20b0mgvbt9os2/stream_test2_f95.f90?dl=0
(updated for improved diagnostic messages)

If you compile the test with gFortran, it works correctly, although they use a different random_number sequence for generating variable data.


Last edited by JohnCampbell on Mon Oct 04, 2021 4:07 am; edited 2 times in total
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7148
Location: Salford, UK

PostPosted: Sun Oct 03, 2021 7:41 am    Post subject: Reply with quote

Thanks for the feedback John.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Mon Oct 04, 2021 12:31 pm    Post subject: Reply with quote

Pail,

I am trying to understand what is wrong, so I created a "simpler" example.
This uses records of 256 bytes (no header/footer) and reads a single byte from a random address.

If I:
1) write the records sequentially, without an address, then read randomly using pos=calculated_address appears to work.
2) write the records sequentially, with a calculated address, then read randomly using pos= with the same calculated address approach, it FAILS.
3) write the records randomly, with a calculated address, then read pos= with the same calculated address, it FAILS.
With this random write test 3, the file appears to be truncated, possible after the last write, as I got an end of file error for a valid address in the read test with 2**12 records of 2**8 length. (1MB file)

This program example is possibly easier to follow and identify the problem(s).

Test 1 : gives hope, as the read (pos=) test appears to work, but
Test 2 : write ( pos=) looks to be incorrect.
Test 3 : random write could be worrying for rewriting data ?
All tests appear to work with gFortran, hopefully showing my approach to stream I/O is as expected.

The calculated address:
for record:byte in read > address = (record-1)*256 + byte
for start of record in write > address = (record-1)*256+1
for byte read > byte value = mod (address,record_length)

https://www.dropbox.com/s/jpdu6f8ub0z9d63/stream_read.f90?dl=0

For stream I/O to be a useful approach for large files (> 4GB), pos= needs to support an 8-byte addresses.
Back to top
View user's profile Send private message
PaulLaidler
Site Admin


Joined: 21 Feb 2005
Posts: 7148
Location: Salford, UK

PostPosted: Tue Oct 05, 2021 7:00 am    Post subject: Reply with quote

John

Thank you for the feedback. I don't have a response at the moment. This issue is logged for investigation but other work is currently in hand.
Back to top
View user's profile Send private message AIM Address
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Tue Oct 05, 2021 9:53 am    Post subject: Reply with quote

Thanks Paul,

I was trying to understand where the problems may be.

INQUIRE ( .. pos= ) and WRITE ( .. pos= ) work differently to gFortran, while READ ( .. pos= ) appears to work.

My comparison to gFortran is as I assume this is standard compilant, I hope this is the case. If clearer comparison is required, I could remove use of RANDOM_NUMBER.

I am hoping to increase the record size and number of records in the test, combined with a 8-byte address and test much larger files. (on a better PC)

I look forward to what you find when time is available.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2477
Location: South Pole, Antarctica

PostPosted: Sun Oct 17, 2021 10:09 pm    Post subject: Reply with quote

So, is there some chance any time soon that REAL*4 arrays will be unlimited size ?
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1582

PostPosted: Mon Oct 18, 2021 3:27 am    Post subject: Re: Reply with quote

DanRRight wrote:
So, is there some chance any time soon that REAL*4 arrays will be unlimited size ?

The day after we are able to buy PCs with unlimited RAM size, CPUs able to access addresses with no limit, etc.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Mon Oct 18, 2021 4:51 am    Post subject: Reply with quote

Dan,

I am typically allocating REAL*8 vectors that are 24 GBytes in size, for a Skyline storage solver and getting reliable results, so vector indexes larger than 2 gbytes are not an issue.
At the moment I am generating, using and discarding the array, but not waiting around to store the data on disk.
I am considering writing the vector to disk, probably as multiple vectors smaller than 2GBytes, as I use a virtual blocking for the solver, based on the L3 cache size. This would be a useful approach to writing a large array using blocks (records) smaller than 2GBytes.
At the moment I have 64-gbytes of physical memory installed, but who knows what it wil be in 2022, post covid!, which appears to be the excuse for all constraints.

It is difficult to keep up with what M.2 drive would be required to support the new 64-bit stream I/O !
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2477
Location: South Pole, Antarctica

PostPosted: Mon Oct 18, 2021 5:16 am    Post subject: Reply with quote

John,
Everyone have to keep the data for 10 years after publication in case anyone will ask. Now imagine to keep even 100 TB of data, that how much will be if keep just major selected files. If keep everything it could be 1000TB. Typically these are harddrives, and good ones

Mecej4, of course unlimited within 2^64 space which is 2^32 times (or few billion times) larger than today FTN95 real*4 allows. Doesn't it sound like unimaginable and almost unlimited today? If space will double every 2 years, and now 16 TB is common disk size, this will require ~60 years to reach it, some young people will see it. I'd be happy with just 1000x (re-phrasing well-known saying, the "1000x will be enough for everyone" or "16 petabytes will be enough for everyone besides supercomputers") or whatever limit Microsoft and Intel restrict their current processors and OS. These two do not allow to address the entire 64-bit space just to control the market and open it bit by bit.
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1582

PostPosted: Mon Oct 18, 2021 9:03 am    Post subject: Reply with quote

Dan, just the thing for you: 100 TB HDD for $40,000 . If you get one, don't drop it!

https://nimbusdata.com/products/exadrive/pricing/
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2477
Location: South Pole, Antarctica

PostPosted: Mon Oct 18, 2021 10:07 am    Post subject: Reply with quote

Mecej4
Yea, wrap me 3.

They are too slow because consist of regular SSD, 500 MB/s. That's ~3 days to wait when 100TB fill the disk. NVMe with PCIe 5 are 30x faster

And can you imagine 10 years from now they all will lie on a city dump ?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2354
Location: Sydney

PostPosted: Tue Oct 19, 2021 3:34 am    Post subject: Re: Reply with quote

DanRRight wrote:
Everyone have to keep the data for 10 years after publication in case anyone will ask

This becomes quite the problem.
In most cases, I try to do analysis via .bat files and I store the data files / generation information to create all these large data sets, but how do you store the program that generated this ?

I do have regular backups of the source code of the programs, but also storing the compiler and .dll's is required to regenerate the results.

I do find that, as the program evolves, some of the data rules change and the approach to "non-typical" data also evolves. Reproducing results from only 5 years ago is difficult, as the analysis approach changes and in my case adapting the souution approach to the computer hardware.

This is my problem with fairly established analysis techniques and I am sure your requirements for reproducing results would be more difficult.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group