forums.silverfrost.com

[dheller]

Hey there,

I have got two problems concerning FTN95 shared memory. Please consider the ftn95 /help page 'Extended ALLOCATE statement' for reference.


A)
After allocating a segment of shared memory in my main program, I can successfully access this memory from outside with a small test program.

However, I need to start another instance of my main program and access the memory from there. _With exactly the same_ code as in the test program, the second instance of my main program somehow fails to view the shared memory.
How is this possible?

I will include the code samples for reference:

Allocation in main program:

[PaulLaidler]

I don't know about A but B needs an include file such as

clearwin.ins or windows.ins etc

[dheller]

Okay, B is working, thanks so far.


A still open though, it would be great if anyone had an idea.

[sparge]

This is a bit advanced for me, I never used this FTN95 extension before.

I don't really understand what it is you are doing: notwithstanding your emphatic statement that you are using exactly the same code in the test program and the main program, you give three samples of code and the relevant bit is different in all three. Not differences that should make a difference, but differences nonetheless.

Is this the clue to the answer? ... as I understand it, you only have two programs: a "main" program and a "test" program. So why do you have three bits of code? Two instances of the main program should have the same code. Is your main program allocating the same shared memory twice in different bits of code? I would have thought it doubly important to use the STAT keyword to check for an allocate error if you are going to use it in this non-standard way.

[dheller]

Okay, I think I need to explain this a bit further for clarification.


I have a main program, doing some calculations for a mathematical problem. Results of these calculations should be stored in a shared segment of memory. (This is the first code sample)

Now, the main program starts another instance of itself, but with a different input file. The sub-problem which is solved in this process needs access to the shared memory data, written by the host process of the first problem.
(This is the third code sample)


Since the data transfer does not work, I have written an external test program (second code sample), successfully accessing the allocated shared memory. So the allocation in the host program seems to work, it's just that the sub-process cannot read any data; the allocation yields a nullified memory segment here.

That's why there are three code snippets, a) allocation in host-main program, b) external test program, c) access in sub-main process.

Sorry for the confusion with different syntax; essentially, all the code samples are the same:

[sparge]

I've been using FORTRAN more than 30 years and I never yet got a main program to start a new instance of itself or to share memory with another program. And I'm struggling to think why I might ever have needed to.

The documentation says one can get two different programs to share memory - but it's non-standard FORTRAN to do so. It doesn't say anything about either of those programs also starting other instances of themselves. So you are trying to build undocumented on top of non-standard.

Why are you making lfe so complicated for yourself? I cannot see why your main program needs to start another instance of itself just so it can use a different input file.

Clearly you have a computer science background even to be able to conceive of what you are trying to do. I think your problem is that you are trying to run (in a new language) before you can walk.

[dheller]

I can see your point, but, unfortunately, I have no choice.
This is not "my" program, it's an academic code written in the 1980s (heck, that's why it's Fortran ), which is still actively developed and shall be enhanced for current research interests.
As a new part of the developer team, I'm also new in Fortran.

One of these research interests is solving a "host" macro problem in the main process, which itself calls several "client" micro problems. This was already successfully implemented, but data transfer between macro and micro problems was rudimentary using temporary files written on hdd.

My task is to improve this data transfer to an efficient and fast way, since data can be several hundred MB large, where hard disc access takes way too long. Shared memory seems to be the natural choice here, and it already works perfectly using Intel Fortran in conjunction with Windows-API file mapping functions.
Unfortunately, these WinAPI functions (CreateFileMapping, MapViewOfFile, even CopyMemory) fail to work at all in FTN95.

Since it is desirable to have both working Intel & FTN95 versions, I tried to implement this behaviour using FTN95's extended allocate shared memory.

[sparge]

Hi D (what's your name?),

OK, I understand that you are constrained by inheritance I also understand now that you are dealing with a lot of data.

I must be missing an essential point here, and I'm always keen to learn ... so at the risk of hanging myself out to dry, the way you just described what the main program does didn't contain anything that sounded to me like it requires a main program to spawn copies of itself. It just sounds like it requires a main program, that runs once, with calls to subroutines to solve the micro problems. You even used the word "call" yourself.

The fact that your program does things the way it does sounds like a legacy of times when memory in the quantities you need it was still hard to come by. Why can you not just define all the stuff that needs to be transferred in a COMMON block? That's a completely standard 80s FORTRAN way for a main program to share memory with subroutines.

As regards the WinAPI functions, I don't know why the ones you want to use would work with Intel FORTRAN and not with FTN95. More information needed - I mean code snippets! How did you get them to work with the former, and how did you fail to do so with the latter?

Andy

[dheller]

Hey Andy,

name's Dominik.
Thanks for your help thus far.


The problem lies within the program conception. It is just not designed to be able to solve multiple problems at the same time, which was not necessary until now.
The program takes an input file when started, containing one problem specification which is then read and can be solved, reviewed and plotted interactively.

This "micro problem(s) in macro problem" concept is a wholly new approach, which is why it is a bit awkward to realize within the given framework.
As such, it is not possible to solve the micro problems within the same macro process.
Another motivation lies in parallelization: Spreading the micro problems upon distinct processes allows to easily solve on multiple cores with a shared memory machine.


For the WinAPI functions:
* CopyMemory doesn't seem to exist in FTN95 at all (winbase.h normally). It is effectively just another name for C's memcpy; allowing to copy relevant data segments in memory directly.
FTN95's MOVE@ seems to work as a substitute, though it takes variables instead of adresses as its arguments.
* creating File mappings in swap (they are apparently stored in RAM directly) via CreateFileMapping with INVALID_HANDLE_VALUE does not work when compiling with FTN95 (ERROR_FILE_INVALID)



Code for the test program as follows:

[sparge]

Hey Dominik,

Don't think I have been any help, except maybe to others in terms of seeking clarification of what you are attempting to do, but it's an interesting topic of conversation. I wonder why others are still notable by their silence thus far; I guess I am a fool and they are angels

Now it seems that you are striving for parallelization. That was certainly not clear (to me) from your original post. But you say it is "another" motivation. If I were you, I would be worried that my terms of reference were not clear enough.

We must not be using the same version of FTN95. My winbase.h contains this line:

[dheller]

Yea, the CopyMemory function exists in the C headers in FTN95's include, but not in the Fortran wrapper include file win32api.ins. At least that was with FTN95 version 4.xx; I will re-check this with the current version tomorrow.



CreateFileMapping, etc. exist in FTN95, but creating the mapping in system memory as described below fails, even in the simple test program above.

The first argument is a handle to the file on which the mapping object should be created. This file will be mapped in memory, to provide fast and random access.
By specifying INVALID_HANDLE_VALUE as a handle, the mapping will be created in the system's pagefile. That is, you just have opened a memory 'window' directly in the RAM which can be opened and viewed by any process specifying the correct name ('Global\Testmap' here).

See here for an example: http://msdn.microsoft.com/en-us/library/aa366551(v=vs.85).aspx

A handle to the created mapping object is returned as the return value ( http://msdn.microsoft.com/en-us/library/aa366537(v=vs.85).aspx )

[JohnCampbell]

Like Andy, I have limited experience of memory sharing.

You could search the forum as there has been past discussion on this subject, which I think was not limited to the following link:

http://forums.silverfrost.com/viewtopic.php?t=1791&start=0&postdays=0&postorder=asc&highlight=

My recent poor experience with parallel processing has shown me that managing the multiple processes yourself and using shared memory might be nearly as good. As you probably know, the suitability of this approach requires 1) being able to package up the problem and 2) being able to efficiently manage the uncertainty of when the distributed packages all finish.
Some of the techniques of managing multiple access database files need to be transferred to managing the memory.

As my experience has been limited and I have not been able to adequately address my point 1) above, I am waiting until the automatic management of parallel processing improves.

John

[dheller]

I don't want to jump to a conclusion, but with a few additional tests, this seems like a serious bug for me.

I had mentioned before that another shared memory segment (nssmap) with 4 bytes works perfectly. Now I tried this and that and have the following facts:

[li] When allocating nssmap with 88 bytes (normal size of emap) and storing my data there, I can read it in the sub-process without any problems.
[li] When I swap the SHARENAME (i.e. emap is now allocated with SHARENAME="FEAPNSSmap" and vice versa), emap now works, but nssmap does not work anymore

With these results I tried some other combinations with the same odd behaviour; and even a further influence by this code:

[PaulLaidler]

I have tested your original part A setup and can confirm that it does not work. I get "ALLOCATE was unable to obtain sufficient storage".

I did not think to test this kind of setup when I wrote the code for shared memory so I don't class this as a bug. At the moment I am not even sure this arrangement makes sense.

I will add this to the list of things to investigate to see if the sharing mechanism can be extended to this situation.

[dheller]

Thanks for your effort Paul.

Just to be clear about this, I understand that the unsupported/unintended detail of the implementation is that both processes accessing the shared memory segment are of the same executable?