|
forums.silverfrost.com Welcome to the Silverfrost forums
|
View previous topic :: View next topic |
Author |
Message |
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Thu Oct 30, 2014 10:18 am Post subject: |
|
|
Dan,
"declare arrays the size you want till these gazillions of heaxabytes"
What you are describing with a very large array which would take too long to analyse.
I have recently solved a surveying problem for a 75km long x 500 metre wide navigation channel which covers an area of 60 km x 50km, with results reduced to 2 metre centres. There are about 60 bytes of information for each point; 750 million virtual points, but only 9 million active points.
By mapping the active points to a virtual 2D page system, I reduce the storage to about 1% and importantly the calculation time by a similar factor.
Sparse matrix approaches still have an important role in large problems, where often the performance savings become even greater.
The luxury you are describing is not what you need !!
John
Last edited by JohnCampbell on Thu Oct 30, 2014 11:42 am; edited 1 time in total |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2815 Location: South Pole, Antarctica
|
Posted: Thu Oct 30, 2014 11:40 am Post subject: |
|
|
What is too long to analyse? It is computer which analyses Matrices are almost empty by the way. Parallelization exists too. At the end computer have to tell us just one single number or the answer Yes or No
I have to use sparsity or do not even try to solve these tasks i run. The /VC compiler switch of FTN77 allowed me in 1990 run 2GB problem suitable only for supercomputers on Intel 386 PC with 1-4 MB or RAM which was a lot at that time. Having 1000x more RAM on PC was unimaginably at that time. Having billion times more memory is unimaginable right now too. But with the 64bit compiler with virtual common we can repeat similar jump into very distant future TODAY.
So using sparsity is important. My life would be easier if Fortran allowed options to declare the arrays Arr(i,j,k) say like these Arr(10000,10000,100) which contain large 10B number of elements to be of variable bandwidth with all the savings associated with this. That is to accomodate actual number of elements of real task.
For example Array with k=1 is Arr Arr(10000,10000,1)
Array k=2 is (10000,10000,2)
Array k=3 is (10000,10000,3)
Array k=4 is only (10,10,4)
Array k=5 is only (1,1,5)
Array k=6 is larger (1000,1000,6)
Array k=7 is also (1000,1000,7)
and so on.
Actual arrays i use have 4 and ideally for convenience would have 5 dimensions if 32bit allowed me such luxury. The 32 bit limit does not allow me to run comfortably literally any task i tried last decade or two.
Instead i have to declare array of full size Arr(10000,10000,1:3) and declare another Arr2(1000,1000,4:100) to fit all numbers, break the homogeneity of the code, making it way more complex and have tons of error due to that. All these arrays are still probably 99% empty. |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Thu Oct 30, 2014 12:26 pm Post subject: |
|
|
Dan,
You could try something like the following, although I got a stack overflow with copies of test(k)%arr being placed on the stack with calls to report_size. This might be a problem for more general use of these arrays.
Code: | module test_mod
type dan_array
real*4, allocatable, dimension(:,:,:) :: arr
end type dan_array
!
type (dan_array) test(7)
!
contains
subroutine report_size ( k, arr)
integer*4 k
real*4, dimension(:,:,:) :: arr
write (*,10) ' Element ',k,' size=', size (arr), ' b1=',ubound(arr,1), ' b2=',ubound(arr,2), ' b3=',ubound(arr,3)
10 format (a,i0,a,b'z,zzz,zzz,zz#',3(a,b'zzz,zz#') )
end subroutine report_size
end module test_mod
use test_mod
!
integer k, sum_size
!
allocate ( test(1)%arr(1000,1000,1) )
allocate ( test(2)%arr(1000,1000,2) )
allocate ( test(3)%arr(100,100,3) )
allocate ( test(4)%arr(10,10,4) )
allocate ( test(5)%arr(1,1,5) )
allocate ( test(6)%arr(10000,1000,6) )
allocate ( test(7)%arr(1000,1000,7) )
!
sum_size = 0
do k = 1,7
if ( k/=6) &
call report_size ( k, test(k)%arr ) ! call fails as test(k)%arr is copied to stack
write (*,*) 'Element',k,' size=', size (test(k)%arr), ubound(test(k)%arr,1), ubound(test(k)%arr,2)
sum_size = sum_size + size (test(k)%arr)
end do
write (*,11) sum_size
11 format ('Total size = ',b'z,zzz,zzz,zz#')
!
end
|
|
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2815 Location: South Pole, Antarctica
|
Posted: Thu Oct 30, 2014 9:12 pm Post subject: |
|
|
John, Thanks for the demo. Before, like 10 years back, such Fortran-90 tricks did not work well, so i am very careful to move with them. What do you mean placed in the stack causing overflow? The same code but without allocate? Can you please also check if it overflows on Ifort/gFortran since you have tried many different Fortran compilers recently? I may use this code with the future FTNpro, it may save me on RAM |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Thu Oct 30, 2014 10:46 pm Post subject: |
|
|
The following change reports the memory address of test(k)%arr, indicating that a duplicate is being provided. gFortran reports the same address, indicating that this stack copy is not required. Code: | module test_mod
type dan_array
real*4, allocatable, dimension(:,:,:) :: arr
end type dan_array
!
type (dan_array) test(7)
!
contains
subroutine report_size ( k, arr)
integer*4 k
real*4, dimension(:,:,:) :: arr
write (*,10) ' Element ',k,' size=', size (arr), ' b1=',ubound(arr,1), ' b2=',ubound(arr,2), ' b3=',ubound(arr,3)
write (*,12) ' Start address= ',loc(arr(1,1,1)), loc(arr)
10 format (a,i0,a,b'z,zzz,zzz,zz#',3(a,b'zzz,zz#') )
12 format (a,2(b'z,zzz,zzz,zz#'))
end subroutine report_size
end module test_mod
use test_mod
!
integer k, sum_size
!
allocate ( test(1)%arr(1000,1000,1) )
allocate ( test(2)%arr(1000,1000,2) )
allocate ( test(3)%arr(100,100,3) )
allocate ( test(4)%arr(10,10,4) )
allocate ( test(5)%arr(1,1,5) )
allocate ( test(6)%arr(10000,1000,6) )
allocate ( test(7)%arr(1000,1000,7) )
!
sum_size = 0
do k = 1,7
if ( k/=6) &
call report_size ( k, test(k)%arr ) ! call fails as test(k)%arr is copied to stack
write (*,12) ' Start address= ',loc(test(k)%arr(1,1,1))
write (*,10) ' Element ',k,' size=', size (test(k)%arr), ' b1=',ubound(test(k)%arr,1), ' b2=',ubound(test(k)%arr,2)
sum_size = sum_size + size (test(k)%arr)
end do
write (*,11) sum_size
10 format (a,i0,a,b'z,zzz,zzz,zz#',3(a,b'zzz,zz#') )
11 format ('Total size = ',b'z,zzz,zzz,zz#')
12 format (a,2(b'z,zzz,zzz,zz#'))
!
end
|
After all these years of using FTN95, I still don't know how to change the stack size. can someone give an example of changing the stack. I compiled and linked with : ftn95 dan_array /link |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2815 Location: South Pole, Antarctica
|
Posted: Fri Oct 31, 2014 6:37 pm Post subject: |
|
|
Your intuition was right. That was easy, just remove the workaround condition for k=6 and link (minimum is stack:240000000 which is exactly the size of the largest k=6 array is enough with /nocheck option. With /debug /undef it has to be larger)
ftn95 arr.f95
slink arr.obj /stack:1000000000 /3gb
Why this large array goes to stack with /3gb by the way? Hope in the future 64bit compiler there will be no damn stack at all
Last edited by DanRRight on Sat Nov 01, 2014 7:49 am; edited 1 time in total |
|
Back to top |
|
|
JohnCampbell
Joined: 16 Feb 2006 Posts: 2554 Location: Sydney
|
Posted: Sat Nov 01, 2014 4:15 am Post subject: |
|
|
Dan,
I presume the stack value you gave is decimal. I could not find documentation that confirms this, as the example I found in
Win32 platform > Using the linker > Reference > Interactive mode only
gives the examples in hex.
Do you know where hex octal and decimal syntax is described.
I think I once suggested that 240m or 240000k should be supported in stead of 240000000, as I get lost in all the zeros.
Anyway, this does give an example of how to manage variable sizes and moving to 64-bit will not mean that we can ignore the memory usage.
As I have described previously, when you run out of physical memory, everything appears to stop. When it first happens, you think it is like the blue screen system crash. I shut down the PC and then had to check all the disks!
I've found that moving from 2gb to 4gb then 8gb then 16gb does not dramatically change the types of problems you can solve, although it does make it a bit easier and a bit faster.
John |
|
Back to top |
|
|
DanRRight
Joined: 10 Mar 2008 Posts: 2815 Location: South Pole, Antarctica
|
Posted: Sat Nov 01, 2014 1:05 pm Post subject: |
|
|
And imagine 9 more zeros with 64bit compiler? LOL Does Intel use stack by the way?
There are types of problems which look crippled in 32bit and ones which are not even possible with 32bits as they need minimum 16GB like some our 3D PIC plasma simulations. If you use less than that the numerical fluctuations are so large that they create fake physics reality.
Intel's Knight Landing processor with 72 64bit cores combined with the future Silverfrost 64bit FTNXX supporting OpenMP would be great home supercomputer i think |
|
Back to top |
|
|
John-Silver
Joined: 30 Jul 2013 Posts: 1520 Location: Aerospace Valley
|
Posted: Sun Nov 02, 2014 3:01 am Post subject: |
|
|
"... this is included in FTN95 today ..."
is this only available to paid-up users or is there a 'trial' personal version ? |
|
Back to top |
|
|
PaulLaidler Site Admin
Joined: 21 Feb 2005 Posts: 7924 Location: Salford, UK
|
Posted: Sun Nov 02, 2014 9:56 am Post subject: |
|
|
John
I have lost the thread. What is "this"? |
|
Back to top |
|
|
LitusSaxonicum
Joined: 23 Aug 2005 Posts: 2388 Location: Yateley, Hants, UK
|
Posted: Sun Nov 02, 2014 1:25 pm Post subject: |
|
|
"This" is the 64 bit Clearwin+ for other compilers, and to John, yes, it is bundled with the PE.
Eddie |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|