Again: 64 bit Compiler

Following the discussion about FORTRAN 2003 and 2008, Simon had a very important point: 'How many young programmers use FORTRAN?' The consequence is that the most of the older programmers – and this group might be 90% of all FORTRAN users – do not need all the new features; but their demand of fast processing and usage of a big amount of memory is very high. This is especially important for number crunching people like John and me.

I am using FORTRAN since 1969! It has been a poor language but step by step new very useful features have been added. Since times of DBOS I am using very satisfied the Salford compiler. However, users of our software want to process bigger and bigger data sets. The matrices reach in the mean time dimensions of 4.000.000 by 4.000.000 (sparse). Computation times can be sometimes 5 hours – and one has to rerun it after error elimination.

Using large memory and fast processing algorithms are the key points for me.

Imaging: in two years a workstation computer has 128 GB of RAM and 16 processors, and you can use just one processor an at most 2 GB of RAM. It’s a pain!

The libraries to process these large matrices using all processors are available. But the precondition to use it is to have the whole matrix (say 10 to 20 GB each) in RAM.

32 bit FORTRAN is useless. Two years ago I purchased an additional FORTRAN compiler: Intel.

My main number crunching program runs now three to four times faster. I.e. five hours are reduced to e.g. one and a half. One selected part of the program – a small matrix multiplication – runs 12 times faster: 25 min vs. 2 min. All these enhancements without using more then one processor!

Paul: Hurry up with your 64 bit development!

Regards, E.Kruck

BTW: All my code is now in real FORTRAN 95.

I fully agree with to this statements of EKruck.

The time for Silverfrost is running (away)!

And by the way:

In a lot of other postings I have read that compiler bugs are fixed for the next release, but the next release is announced for the end of 2012!!

Who can wait so long? I think two or three releases a year should be possible.

Still hoping for the future of Silverfrost

Detlef Pannhorst

I'm also waiting for 64-bits FTN95 and suitable functions supporting parallel processing, and I fully agree with Erwin and Detlef! If FTN95 will not be upgraded soon, I must move to the Intel compiler because customers are not willing to wait... wait... wait...

Paul wrote (https://forums.silverfrost.com/Forum/Topic/1305):

'There are no plans at the moment to produce a 64 bit compiler.' (February 2010)

'There are currently no plans to write a 64 bit compiler but the possibility is continuously under review.' (August 2011)

Regards - Wilfried Linder

Paul,

I can imagine that you don’t like this topic. But people like me are both, customers (of Silverfrost) and developers with our own customers who are pushing us. Nowadays, nearly everybody asks “does your software supports 64 bits”, “does it support multi-core processors”, “is it also available for Linux” etc. So, it becomes more and more difficult to explain why my software has none of that.

Just an example: My software can produce image mosaics. Now take 24-bit images from modern digital cameras (say 12 millions of pixels each), and take hundreds of them, then the result will often have a size of several gigabytes. It must be possible to handle such images, for instance to print them in one part instead of several sub images. So, things like an upgrade to 64 bits are not a “fashion” for me, they become necessary.

I think Silverfrost can help many of us by just giving clear answers, even if the answer is “we will never expand to 64 bits”. This is quite better than “may be” or “sometime”. Perhaps you can encourage your company to give us obligatory information about the plans and time schedules of FTN95 upgrades.

With kindly regards Wilfried Linder

Yes!! It is very very urgent!

We have the same problem with the worldwide customers of the SIEMENS simulation tool PSS(R)NETOMAC for simulation of electrical power systems and our Eigenvalue analysis tool NEVA, both written in Fortran and compiled with Silverfrost's FTN95 compiler.

See www.pss-netomac.com

Detlef

There are multiple questions here. I'm always better at questions than answers! I also have specific requirements that I know are not the same as everyone else.

(a) FTN95 doesn't produce the fastest executables. I eventually came around to thinking (as John Campbell did in another post) that it is because it uses 80x87 code, not the appropriate kind of SSE. But, as 80x87 is 80-bit and SSE is 64 bit, the roundoff could be worse using the latter. FTN did produce the fastest executables before SSE (in the DBOS era) so it has to be something simple. You could get a speed increase by profiling your program, and writing the innermost loop in assembler using SSE - if there is one loop that takes the majority of the time.

(b) FTN95 doesn't explicitly use threads (the DBOS version of FTN77 was multithreaded). But windows uses all the cores, and if you look at the performance monitor you will see them all being used. I'm prepared to believe that it is possible to produce a better scheduling system for a single application than Microsoft does, but isn't that a lot of work?

(c) There must be lots of users (like me) who don't need >2Gb arrays. For those of you who do, how many arrays of this size do you need? 1?, 2?, 4? The whole of FTN95 doesn't need to be 64-bit, does it? The way Windows 64 is organised means that the huge array has to be ALLOCATABLE (See Steve Lionel's discussion of this in the Intel Fortran support). Why don't you ask Paul to create a system where it is possible to:

(i) ALLOCATE a 64-bit addressable region of a predetermined size (ii) Post values to it (iii) Read values from it (iv) Clear it when it is finished with.

FTN95 already has INTEGER*8 for calculating the offset, and the algorithm for computing the offset into an equivalent 1-D array from any number of dimensions is pretty simple to do yourself.

... continued

... And surely, if your program is standard-conforming, you should be able to compile the production version with one of the faster, fully-64bit compilers. You can still do development and testing in FTN95/Windows.

I always liked to compile my programs with multiple compilers because they all give slightly different diagnostics even for a working program. In the mainframe era, that meant running it on different computers. Polyhedron lists 6 commercial vendors plus G95 and GFortran for Windows, so there is plenty of choice. There is even choice on the Mac and Linux.

I find it difficult to believe that one of these monumental applications is terribly dependent on Clearwin. You can still compile a program that uses Clearwin with other compilers, but you get errors for the @ in FTN95 names, and of course the object files would not be linkable.

To return to the original thread, my requirements are fully met with Fortran77 and Clearwin, and I couldn't really use executables that were 2x faster, let alone more. Most of the facilities in Fortran 90 and beyond seem to me like re-inventing the wheel rather than using it on a vehicle. Fast compilation, good diagnostics, Windows only - and Clearwin - are on my particular requirements list. However, the statistics of a single sample (i.e. me!) are always questionable, and my requirements are very limited in comparison with other people.

For what it is worth, Absoft now seems to be the leader of the pack as far as speed is concerned. I found that it did not give me the results I expected for a program that I had compiled and run with no changes on over a dozen mainframes and 6 or 7 PC compilers (I used Absoft v5.0), and as a result I was unhappy with it and never got a later version. An odd experience that I put down to me not wanting to put in the effort to make it work. I valued my time more highly than the cost of a different compiler.

Eddie

The strength of the Silverfrost compiler lies in its code checking (CHECKMATE) and debugging (SDBG) capabilities in my opinion. I would much rather have these useful tools than another 64 bit compiler with poor checking and a debugger that crashes all the time.

As LitusSaxonicum says, one can always use FTN95 for development and use a different compiler for 'faster' production code. If you want parallel programming then use OpenMP. Your programs will still compile and run with Silverfrost's FTN95 but will run sequentially. They will run in parallel on a compiler which supports OpenMP. This gives you a fully scalable fortran solution. Development time is shorter with FTN95 in charge, and your codes run and get benefit from the latest software (compilers, optimization) and hardware (multicore, multiprocessor) technologies.

I usually find that if a program uses big arrays then proper account has not been made of the structure of matrices (banded, sparse, variable row size etc.) and, for my problem space at least, I can usually manage with smaller arrays that fit comfortably in 32 bit addressable memory.

Eddie,

of course I can only speek for my applications. One example is the following: Two very large images from an aerial camera (24 bits) together with a terrain model (16 bits), all three must be completely in memory. You can test this here:

      PROGRAM test 

      IMPLICIT NONE

      character*1,allocatable::image_1(:)
      character*1,allocatable::image_2(:)
      integer*2,allocatable::terrain_model(:)
      integer*4   rtcode

      allocate(image_1(1000000000),stat=rtcode)
      if (rtcode .ne. 0) stop
      print*,'image_1 OK'
      image_1 = char(0)

      allocate(image_2(1000000000),stat=rtcode)
      if (rtcode .ne. 0) stop
      print*,'image_2 OK'
      image_2 = char(0)

      allocate(terrain_model(1000000000),stat=rtcode)
      if (rtcode .ne. 0) stop
      print*,'terrain_model OK'
      terrain_model = 0

      deallocate(image_1,stat=rtcode)
      deallocate(image_2,stat=rtcode)
      deallocate(terrain_model,stat=rtcode)
      end

Does it run on your PC? In this example, about 4 GB of RAM is needed, but there are cases were I need even more. If you have an idea how to handle this fast and easy under 32 bit Windows, please tell me.

Regards - Wilfried

Eddie,

calling for 64-bit-support and faster execution code does not mean do loose all of the advantages of the FTN95 compiler. Why shouldn't it be possibible to [u:0b5dce10e3]add[/u:0b5dce10e3] the new features?

The programs I am building are number crunching programs which need much memory and fast execution times.

As an example: our customers do Eigenvalue analysis of the whole European electrical power system, which needs execution times of 2 or 3 hours. To be 3 or 4 times faster will be a big advantage.

On the other hand: the graphical user interface of the software is built with Clearwin, which is a very good and easy solution for doing this as all Clearwin users know.

This part is not portable to other compilers. So we had to rebuild this with other solutions.

Regarding the execution times: Have a look at http://www.polyhedron.com/pb05-win32-f90bench_SBhtml Silverfrost's FTN95 is the last but one in this table nowadays. I remember times (when we had decided to work with former Salford FTN77) where the Silverfrost compiler was on top of this table.

Since moving from Salford to Silverfrost there seems to be no real development for the future of the Fortran compiler at this company.

It is hard, to mention this, but I think that is the reality of Silverfrost.

I only can hope, Paul will tell us, that there will be fast changes in the near future.

Regards,

Detlef

I have been a very interested reader of this discussion and would like to provide a few comments. Detlef and Wilfred have presented a few options for enhancement of FTN95, including support for;

• 64 bit addressing
• Mult-core capability
• Linux OS support I would add:
• Modern processor instruction set support
• More support for Fortran 2003 and 2008
• Quicker access to a beta upgrade to salflibc.dll All of these have merit, but they must be considered in the context of the resources available to Silverfrost FTN95.

I think there is general agreement that the reason most of us use FTN95 is because of it’s Checkmate and SDBG support for program development, together with the functionality of Clearwin+. FTN95 is a great compiler for software development, using Fortran 95.

I would like to know if any of the features identified have a possibility of being advanced. To address some of these:

64-bit : FTN95 now supports extension to 2gb-4gb address space. All 64-bit compilers I know only support above 4gb via ALLOCATE. I’m not sure of the change required to support addresses above 4gb (using only ALLOCATE) and to create a 64-bit .exe file. What I do not at present have access to is >4gb in-memory info and Clearwin+.

Multi Core : My experience with this is that it is a fairly complex issue which would be beyond the resources of Silverfrost to develop. A possible path is to support 3rd party solutions, like OpenMP with Intel. Support for multi-threading and these packages solutions may be the way to go. My middle ground recommendation would be to support some of the newer instruction set, such as vector calculations. I am not knowledgeable in this field; there are newer instructions available which could address some of the performance issues. Certainly my recent experience with parallel processing is that it is more multi thread processing and needs to be implemented at a higher level than the inner most DO loop to be successful.

Linux Support: I have never been a fan of Linux or Unix. FTN95 / Clearwin is a windows solution. I’d much prefer to see support for newer computing approaches in Win32/64. I am not sure what is required to be able to run a Win32/64 .exe on a Linux OS.

Fortran 2003 : Fortran is moving ahead, while I do not see the relevance with a lot of the complexity in 2008, FTN95 remains in the past. I also wonder if Silverfrost can afford the cost to develop this extensive extension to the standard ?

Access to updates. It is difficult to wait till the next upgrade. If the update is 9 months away, I have had to put on hold developments using these fixes. Support for better graphics on larger screens with DIB is a good recent example where I have a stop-work.

64-bit support is a good example of the problems we as software developers face, as although there are alternatives using 32-bit, there are much slower that the 64-bit alternative. Our 32-bit solutions are lagging behind what others are achieving and we need to meet the competition. Having a 64-bit solution also allows us to consider new ways to solve problems, expanding on what can be done. Without access to these computing advances, we are falling behind what is being done elsewhere.

We must be realistic about the development resources available to Silverfrost. However, it would be good to have a clearer indication of what changes might be possible and when.

John

Wilfred,

Try the following on Windows 7x64. I have found that I can not use an address above 2gb, but the array can extend above 2gb. It also helps to allocate the largest array first. PROGRAM test ! ! test this on Windows 7x64 ! character1,allocatable :: image_1(:) character1,allocatable :: image_2(:) integer2, allocatable :: terrain_model(:) ! integer4 rtcode, i integer4 array_size integer4, parameter :: mb = 10241024 real8 gb ! do i = 1,5 ! if (i1) array_size = 102 * mb if (i2) array_size = 512 * mb if (i3) array_size = 768 * mb if (i4) array_size = 922 * mb if (i==5) array_size = 1020 * mb ! gb = real(array_size) / 1024. / 1024. / 1024. write (*,fmt='(/a,f0.3,a)') 'Testing Array size ',gb,' gb' ! allocate (terrain_model(array_size), stat=rtcode) call report ('terrain ', terrain_model, rtcode) if (rtcode == 0) terrain_model = 0 ! allocate (image_1(array_size), stat=rtcode) call report ('image_1 ', image_1, rtcode) if (rtcode == 0) image_1 = char(0) ! allocate (image_2(array_size), stat=rtcode) call report ('image_2 ', image_2, rtcode) if (rtcode == 0) image_2 = char(0) ! deallocate (image_1, stat=rtcode) deallocate (image_2, stat=rtcode) deallocate (terrain_model, stat=rtcode) end do ! end

      subroutine report (image_name, image_1, rtcode)
!
      character image_name*(*)
      integer*1 image_1(*)
      integer*4 rtcode
!
      if (rtcode == 0) then
         write (*,1001) image_name,' OK at address', loc(image_1)
      else
         write (*,*) image_name,' could not be allocated, stat=',rtcode 
      end if
1001  format (a,a,b'zz,zzz,zzz,zz#')
      end

Here is interesting reading related to roadblock on static arrays for 64bit compilers

http://software.intel.com/en-us/articles/memory-limits-applications-windows/

Can somebody ask MS if they plan to remove this dumb req in future versions?

I'd buy 64bit compiler immediately. Even if it will be just the Fortran 95_64. Agree with people that 'no plans for 64bit compiler' is kind of shocking... chilling... killing. Same as '640kbytes are enough for everyone'. But mostly completely unexplainable -- these devs know the matter because they have made 32bit one in the 16bit era. Does really something or somebody keeps developers by the balls ?

John,

thank you for your code. It works up to 750 MB array size, meaning that in total about 3 GB can be used. This is what I know and what is working within my software. But above more or less 3 GB it seems to be not possible for 32 bits Windows.

@all

I use FTN77/95 since early DBOS times, and for me it is a very good compiler! Besides, I use really a lot of the Clearwin++ functions. Therefore, moving to another compiler would be an enormous part of work. But, as I wrote: If FTN95 will not be upgraded for 64 bits soon, I will have no alternative then to change the compiler 😦

Regards - Wilfried

Dan, To answer your post; the stack limit is not good, but is readily overcome by using allocate arrays in a module, as with my example big.f95. The 2gb limit applies to local variables and common. Code is also included in this 2gb limit, but this is not a significant restriction either. It would be good not to have the 2gb limit, but it works ok. I've run programs up to 9gb and I'm sure others have run much larger. The main rule would be to not use more than about 80% of physical memory, else you loose the efficiency of including disk paging. John

This forum is driven by software that allows polls to be conducted. Perhaps this facility could be enabled so that users could record their feelings about needs for future development?

Eddie

Wilfried,

As far as your problem goes, then clearly it is NEVER solvable under a 32-bit OS, whether it is Windows or anything else. Having dabbled in photogrammetry and DTMs, then the answer must lie (if one must do it in a 32-bit OS) in doing it in panes or tiles – I presume that the two digital camera images have a stereoscopic overlap. This straightaway means that the only parts of the two camera images that really need to be in memory at any time are in the overlap. Not only that, but the image can be processed in bands or tiles thereafter. The movie industry renders images in tiles (see Autodesk Maya and its various add-ons).

Even assuming you need to have 2 photos and a DTM simultaneously in memory, this is still a small number of huge arrays, and my suggestion as to how to get limited 64-bit addressable arrays into FTN95 still applies: it might be do-able with as little as 4 arrays, i.e. the rest of FTN95 could stay 32-bit, as long as there was a way of creating those arrays, and interacting with the memory constants – and you could only do it in 64-bit Windows. Or, all ALLOCATABLE arrays could be 64-bit: static arrays still have to fit in 2Gb, and so by definition are not 64-bit addressable.

Eddie

Eddie,

within my stereo display both images (24 bits / real colours) are shown, connected with the DTM. The mouse movement changes x and y (terrain units), then the corresponding z value is taken from the DTM, then the image positions are calculated. In this way, a real-time movement is realised. You are right, tiling the images may be a solution - hope that real-time movement will still work.

And you are right that (for me) it would be enough if up to 4 arrays could be handled in '64-bit-mode'. Nevertheless it would not only be necessary to allocate them but also to have functions like openr@, openw@, fpos@ and others available for such arrays. The 2GB limit for static arrays is no problem for me.

It would be nice if Silverfrost could realise this ...

Regards - Wilfried

John, may be i'm lost something, i see how you've allocated around 3+smth GB but that's end of the game. How did you get 9?

Dan, I got 9gb using a 64-bit compiler.

Eddie, The result of using 64-bit is that you do not have to use tiling. This allows faster and simpler access to all the data, enabling new ways to present the information. When all the data is present, it is easy to try other ways to use the data, rather than the overhead of manageing the data sharing.

John