 |
forums.silverfrost.com
Welcome to the Silverfrost forums
|
| View previous topic :: View next topic |
| Author |
Message |
PaulLaidler
Site Admin
Joined: 21 Feb 2005
Posts: 7927
Location: Salford, UK
|
 Posted: Thu Jul 13, 2023 5:55 pm Post subject: |
 |
|
John
Thank you for the information. I hope to get back to this issue soon. |
|
| Back to top |
|
 |
DanRRight
Joined: 10 Mar 2008
Posts: 2818
Location: South Pole, Antarctica
|
| Posted: Thu Jul 13, 2023 10:22 pm Post subject: |
|
|
The approaches have to be compatible with Intel, gFortran all others, don't you agree, John? This is what for STREAM is good for, otherwise anything proprietary will marginalize this compiler further.
/* Decently i still do not get what you are trying to accomplish and why anyone needs that. Including yourself, as you do now your parallel numbercrunching in gFortran? I have not noticed also that you are a user of Clearwin or SDBG or OpenGL or Plato. Oh, you have ones tried Plato, i am sorry  Do linking of libraries become a problem when the library file is larger than 2GB? No. You can not link Gfortran or Intel library to FTN95? You can. The FTN95 can not read gFortran or Intel stream binary files? It reads them fine, including files made with any other existing software. So what's this confusing fuss for? Or you have found some good solution for something useful here? Intel and gFortran and others read/write large files ok and do that fast. While this thread is already a year old discussing your proprietary headers nobody knows what for.
Last edited by DanRRight on Fri Jul 14, 2023 2:38 am; edited 2 times in total |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Fri Jul 14, 2023 2:18 am Post subject: Re: |
|
|
| DanRRight wrote: | | i still do not get what you are trying to accomplish |
Dan ! In my defence,
Limiting my response to this comment, There are now 3 types of Fortran unformatted file types.
# Both fixed length and stream formats are compatible between FTN95, Ifort and Gfortan.
# Unformatted sequential is not compatible, as it has a different header/footer format.
# Before this thread, both FTN95's Unformatted sequential and stream did not support records longer than 2 GBytes (due to implementation problems). ( I have not tested fixed length)
# For unformatted sequential; I have been trying to explain the header/footer formats, as they need to be modified to support longer records.
# Header/footer labeling can also enhance stream file "record" flexibility.
Header/Footer formats are key to addressing the large record support
For many years, I have based my "random access, variable length record" library by superimposing records onto Fortran "direct access, fixed length record" file type. (Based on CDC Fortran Openmp library of 1974)
My library provided a buffering for small records and did improve performance, however with improved Windows file buffering, became less significant.
During this thread, I have done testing of stream I/O to compare with my buffered fixed length record file usage. This has shown that the buffering in my fixed length record implementation is no longer needed/significant and a simpler approach based on using Stream I/O works well.
The other change in this testing, was to support records larger than 2 GBytes using an 8-byte file address (My previous library was limited to 8 GBytes, as the record address was based on 4-byte word address)
Changes would need to be made to FTN95's unformatted sequential files to support larger records or be compatible with Ifort/Gfortran (or use stream access).
As to my failure to effectively embrace "Clearwin or SDBG or OpenGL or Plato" Wooh, a bit harsh, but yes you do have a point.
Although I do use Plato as my main IDE, but my projects are based on using .bat files.
| Quote: | | Or you have found some good solution for something useful here? |
Yes, I think that Stream I/O is a more flexible solution and thanks to Paul's support, FTN95 Stream now supports "arrays > 4GB" It is a good alternative.
| Quote: | | Do linking of libraries become a problem when the library file is larger than 2GB? |
I think this is probably going to become a Windows /64 limitation, as isn't the code base limited to 2GBytes ?
edited 15/7 |
|
| Back to top |
|
|
DanRRight
Joined: 10 Mar 2008
Posts: 2818
Location: South Pole, Antarctica
|
| Posted: Sat Jul 15, 2023 11:54 pm Post subject: |
|
|
John,
Al that has zero relationship to the original post.
"FTN95 Stream now supports arrays > 4GB"
It does? Does my test above work? |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Sun Jul 16, 2023 1:53 pm Post subject: |
|
|
Dan,
"Al that has zero relationship to the original post." I do not agree !
"Does my test above work?" It does?
A modified test program that uses elapsed time and reports the compiler info is generated in Plato and produces :
| Code: | Start test 0.0000
FTN95 v8.92
64;DEBUG;ECHO_OPTIONS;ERROR_NUMBERS;IMPLICIT_NONE;INTL;LINK;LOGL;NO_BANNER;SET_ERROR_LEVEL;UNLIMITED_ERRORS;VS7;
Trying to allocate GB of RAM : 8.80000000000
Allocation success
Initialise 4.3210
Trying to save the data Method 1
Method 1 14.1577
Speed of write Method 1 = 0.625083 0.621570 GB/sec
Trying to save the data Method 2
Method 2 6.0167
Speed of write Method 2= 2.93333 1.46260 GB/sec
**** PAUSE: File LargeFile.dat created OK
Press ENTER to continue:
Comparing files LargeFile.dat and MANYRECORD.DAT
FC: no differences encountered
|
This version 8.92 was modified to test changes to stream I/O library. I am not sure if these test changes were adopted for subsequent versions.
Last edited by JohnCampbell on Sun Jul 16, 2023 2:08 pm; edited 1 time in total |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Sun Jul 16, 2023 1:55 pm Post subject: |
|
|
The enhanced code, using FTN95 Ver 8.92 is
| Code: | lation: ftn95 aaa.f90 /link /64 >z
use iso_fortran_env
!
real*4, dimension(:,:), allocatable :: Arr4
real*8 :: sec
real*4 :: GBps, SpeedGBps, t0, t1
integer*4 :: nA,nB,ierr, i
call delta_sec ( 'Start test', sec)
write (*,*) compiler_version()
write (*,*) compiler_options()
nA = 11
nB = 2.e8
!...Allocating array
Print*, 'Trying to allocate GB of RAM :', 1.d-9 * 4. * nA * nB
allocate ( Arr4 (nA, nB), stat = ierr)
if (ierr.eq.0) then
Print*,'Allocation success'
else
Pause 'Fail to allocate'
goto 1000
endif
!...Filling the array with some data
do i=1,nB
Arr4(:,i) = [1,2,3,4,5,6,7,8,9,10,11]
enddo
call delta_sec ( 'Initialise',sec)
! __ __ ____ ____ _ _ _____ ____ __
! ( \/ )( ___)(_ _)( )_( )( _ )( _ \ / )
! ) ( )__) )( ) _ ( )(_)( )(_) ) )(
! (_/\/\_)(____) (__) (_) (_)(_____)(____/ (__)
!....................................................
Print*,'Trying to save the data Method 1 '
call cpu_time(t0)
open (11, file='ManyRecord.dat', FORM='UNFORMATTED', access="STREAM", err=900)
do i=1,nB
write(11) Arr4(:,i)
enddo
close(11)
call cpu_time(t1)
call delta_sec ( 'Method 1',sec)
!...Speeed of writing method 1
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
GBps = 1.d-9 * 4. * nA * nB / (sec+1.e-10)
print*,' Speed of write Method 1 =', SpeedGBps, GBps, ' GB/sec' ! typically ~0.5 GB/s
! __ __ ____ ____ _ _ _____ ____ ___
! ( \/ )( ___)(_ _)( )_( )( _ )( _ \ (__ ! ) ( )__) )( ) _ ( )(_)( )(_) ) / _/
! (_/\/\_)(____) (__) (_) (_)(_____)(____/ (____)
!....................................................
Print*,'Trying to save the data Method 2'
call cpu_time(t0)
open (11, file='LargeFile.dat', FORM='UNFORMATTED', access="STREAM", err=900)
write(11) Arr4
close(11)
call cpu_time(t1)
call delta_sec ( 'Method 2',sec)
!...Speeed of writing Method 2
SpeedGBps = 1.d-9 * 4. * nA * nB / (t1-t0+1.e-10)
GBps = 1.d-9 * 4. * nA * nB / (sec+1.e-10)
print*,' Speed of write Method 2=', SpeedGBps,GBps, ' GB/sec' ! typically ~2.6 GB/s
pause 'File LargeFile.dat created OK'
goto 1000
!...............
!...Errors
900 Print*,'Can not open file LargeFile.dat'
goto 1000
910 Print*,'Can not save file LargeFile.dat'
1000 Continue
End
Subroutine delta_sec ( title, sec)
character title*(*)
real*8 :: sec
integer*8 :: last=-1, ticks, rate
call system_clock (ticks, rate)
if ( last < 0 ) last = ticks
sec = dble(ticks-last)/dble(rate)
last = ticks
write (*,fmt='(a,2x,f0.4)') title, sec
end Subroutine delta_sec
|
|
|
| Back to top |
|
|
DanRRight
Joined: 10 Mar 2008
Posts: 2818
Location: South Pole, Antarctica
|
| Posted: Sun Jul 16, 2023 7:24 pm Post subject: |
|
|
John,
That was resolved a year ago. And broke again. Read the whole thread with latest posts. Looks like they still in transit to Australia |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Mon Jul 17, 2023 7:10 am Post subject: |
|
|
Dan,
Trying to unpack your comments !!
I worked with Paul late last year to identify problems with stream I/O.
Paul provided me with FTN95 updates to test for errors I had found.
These addressed most problems of Stream I/O, but not reading large unformatted sequential file type records.
The latest version of FTN95, which I am using, I have labeled as FTN95_8.92.37b. FTN95.exe is 15-Nov-22, while all .dll and .lib are 24-Dec-22
If the (8.95?) version you are using is later than this 8.92 version, then Paul has not yet incorporated these test changes into the main release.
I had interpreted your description of "Al that has zero relationship to the original post" as referring to the test Posted: Mon Sep 26, 2022 11:13 am
Excuse me if I don't interpret your comments as you intended ! (I am struggling with this point)
I have now modified your latest test Posted: Wed Jul 12, 2023 11:44 am,
which I understand is to also test stream reads of large records.
This does not work with earlier versions of FTN95 Ver 8.92 I have on other pc, but does work with my 8.92_37b version.
I have also written an inefficient file comparison using stream I/O to compare the 2 different files created in this latest test.
There are many aspects to efficient read/write of binary files, to achieve high GByte per second performance you claim and lots of traps if you are not careful.
Trap 1 : don't run this test program on a notebook with only 8 GBytes of memory !
Trap 2 : don't expect all M.2 NVMe SSDs are fast, as it also depends on their internal buffer capacity. Which PCIe version is also a hardware factor.
Trap 3 : then with testing, test performance can be misleading, as OS memory buffering of 64 GBytes of installed memory can hide/obscure a lot of disk performance.
The main outcome of the new tests is FTN95 Ver 8.92_38b passes the write and read tests you were wanting to do in your "original post" or "test above"
I shall now try to post this latest test program and the test logs I got for running on a C: SSD 970 EVO Plus 500GB and an E: 2 TByte WD drive of unknown details! |
|
| Back to top |
|
|
PaulLaidler
Site Admin
Joined: 21 Feb 2005
Posts: 7927
Location: Salford, UK
|
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Mon Jul 17, 2023 7:49 am Post subject: |
|
|
The below links are to the latest test program and also to the log file that has been generated/appended.
https://www.dropbox.com/s/u16l11lcpaa6v88/read_write.f90?dl=0
https://www.dropbox.com/s/exqqpwzsb0efiwg/stream_tests.log?dl=0
The test program does not report the details of the disk being used, which can be provided at the start of the test, using the details prompt.
Performance is similar between FTN95 and Gfortran.
The test results on my 500GB SSD 970 EVO Plus on my Ryzen 5900x were fairly good and the 2 Tbyte HDD was good (due to 64 GBytes of memory buffers) These are included in the .log file.
However the tests on my I7-6700HQ notebook with a Samsung MZNLF128H (a 128 GB SSD) were much slower.
Basically the FTN95 compiler version I am using supports large record sizes, but the performance difference probably does not justify the large record choice.
Certainly 44 byte records are too small, but you don't need to go to 8 GByte records. If you can get say 64 KByte records that may be a good comprimise, which is what I have been using in my IO library for many years.
Please don't claim this latest post "zero relationship to the original post" |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Mon Jul 17, 2023 7:50 am Post subject: Re: |
|
|
I shall investigate this version and reply. |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Mon Jul 17, 2023 8:52 am Post subject: |
|
|
Paul,
I downloaded the maintenance version 8.95
Then installed the http://forums.silverfrost.com/viewtopic.php?t=4245 (Latest executables and DLLs)
This resulted in Ver 8.97.2
I compiled and ran my read_write.f90 I postecd today and it ran to completion.
###### Start of Stream file test ######
on c: Samsung SSD 960 EVO 500GB I7-8700k 32 GBytes memory
Start test 0.0000
FTN95 v8.97
64;ECHO_OPTIONS;ERROR_NUMBERS;IMPLICIT_NONE;INTL;LINK;LOGL;
Trying to allocate GB of RAM : 8.80000
Allocation success
Initialise 4.0870
============ Trying to save the data Method 1 ============
Method 1 write 24.5124
Write OK. Speed of write Method 1 = 0.360 0.359 GB/sec
24.4531 24.5124000000
============ Now read Method 1 ============
Method 1 read 19.3572
READ OK. Speed of read Method 1 = 0.462 0.455 GB/sec
19.0469 19.3572000000
============ Trying to save the data Method 2 ============
Method 2 write 7.5058
Write OK. Speed of write Method 2= 3.112 1.172 GB/sec
2.82813 7.50580000000
============ Now read Method 2 ============
Method 2 read 1.7184
READ OK. Speed of read Method 2 = 5.576 5.121 GB/sec
1.57812 1.71840000000
============ N O W C O M P A R I N G F I L E S ============
No differences found in file compare to 8800000000 bytes
Compare files 139.5964
The results appear to show the program runs without error for stream I/O for small then large records, then confirms both files are the same.
Note : The above report of "Speed of write Method 2= 3.112 1.172 GB/sec" shows 2 different transfer rate estimates. The first is based on CPU time, while the second is based on elapsed time (system_clock). Elapsed time is a better estimate of performance, as CPU time ignores any disk update delays.
Comparing the results between 960 EVO and 970 EVO Plus on the two different PC's I am using I get:
| Code: | 960 EVO 970 EVO+
write small 0.359 GB/sec 0.720 GB/sec
read small 0.455 GB/sec 0.827 GB/sec
write big R 1.172 GB/sec 2.442 GB/sec
read big R 5.121 GB/sec 2.980 GB/sec
|
It is hard to explain the performance of 5 GB/sec read of the large record using the slower SSD and only 32 GBytes of memory, but,
FTN95 Ver 8.97.2 supports stream I/O records of 8 GBytes, based on this test.
Dan, do you see any problems now ? |
|
| Back to top |
|
|
DanRRight
Joined: 10 Mar 2008
Posts: 2818
Location: South Pole, Antarctica
|
| Posted: Tue Jul 18, 2023 5:48 am Post subject: |
|
|
| With my latest demo test ( on page 4, the one which writes large file and then reads it), after successful allocation of 8 GB immediately goes "Access violation". Gfortran works ok |
|
| Back to top |
|
|
PaulLaidler
Site Admin
Joined: 21 Feb 2005
Posts: 7927
Location: Salford, UK
|
| Posted: Tue Jul 18, 2023 7:04 am Post subject: |
|
|
Dan
I have tested the program that you posted on 12 July using the latest downloads. It runs to the end for me giving the output...
Trying to allocate GB of RAM : 8.80000000000
Allocation success
Trying to save the data Method 1
Write OK. Speed of write Method 1 = 1.36699
=====================
================ N O W R E A D ====================
READ OK. Speed of read Method 1 = 3.04432
=============================
Trying to save the data Method 2
Write OK. Speed of write Method 2= 6.54884
=====================
================ N O W R E A D ==================
READ OK. Speed of read Method 2 = 1.23239
======================
**** PAUSE: File LargeFile.dat created OK
Press ENTER to continue: |
|
| Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006
Posts: 2554
Location: Sydney
|
| Posted: Tue Jul 18, 2023 10:04 am Post subject: Re: |
|
|
| DanRRight wrote: | | With my latest demo test ( on page 4, the one which writes large file and then reads it), after successful allocation of 8 GB immediately goes "Access violation". Gfortran works ok |
Dan,
I can get this "Access violation" response, if after installing a new version of FTN95, then use PLATO (from the task bar).
A check list can be:
Check that you have the correct path settings for latest FTN95,
Delete the PLATO icon from the task bar,
Pin the latest PLATO in latest FTN95 directory, to the task bar,
then try again.
This approach will work if you have a new directory for your latest FTN95 version and you had an old version of PLATO linked to the task bar.
It works for me as I keep multiple versions of FTN95 available. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
