Extracting part of code from the large source

One problem with using MODULEs instead of COMMON blocks is difficulty to extract part of code while keeping only relevant definitions. Modules tend to grow in size swallowing or monopolizing other modules and each and every variable of the code inside just a single major module. This is because this way it is just easier to exchange the data. With COMMON blocks you kept in subroutines and functions only relevant variables. Separating subroutines from large code with COMMON blocks were not a problem at all.

I need right now to make small and simple demo out of large code which has a problem to send it to Paul for investigation but losing again days to throw all not related variables and their initializations. Did anyone try to write the code where you set which subroutines to keep and it will automatically remove all not related variables and subroutines?

Dan,

I use /xref to get a list of variables that are referenced outside the module, using Excel and pivot tables. Once I have that list, I can comment out all variables in the module that are not required. I then use another 'fiendish' option to check I have not deleted too much.

I actually find little difference between MODULE and INCLUDE for this issue. I can have multiple COMMON in a single include (for separating integer/real from character)

MODULE has the benefit of allocated arrays, but looses some EQUIVALENCE capability.

FIND and FINDSTR are also good utilities to check this. Try the following search for variable %1 options: findstr /i /n /s /c:%1 *.f9? *.in? >> tfs.tce find /i /n '%1' *.f?? *.in? >%1.tce

Not a major problem

Thanks John, but this is still kind of manual or semi-manual way of extraction. I made with the mind crushing difficulty 5-6 manual demos during these two weeks trying to get the source of crash caused by simpleplot_redraw@ and all of them get me demos which show no crash problems. I put them back into the larger code and they crash...

I thought may be there exist fully automatic way for demo extraction. A lot of users in this forum complained at some time about difficulty to reproduce their bugs in small demo examples. May be Silverfrost or some of us would make such Demo Extractor to help people to debug their codes. You tell it the name of subroutine you want to extract and it will leave only relevant subroutines and variables removing all others

Dan,

I'd recommend you look at my recent post regarding a bug I had dumping .png files of a %gr plot. It was difficult to locate and very hard to reproduce. In the end it was most likely not a clearwin+ bug, but a problem from multiple open/close operations (this is close to a known problem with windows). ( this code was a work-around, due to unknown problems with INQUIRE 20 years ago, which had since been fixed. As you say we should report all bugs)

See: https://forums.silverfrost.com/Forum/Topic/3214&start=15

Anyway, if it is difficult to reproduce, you may be looking in the wrong place. Worth considering.

John

How I am screwed: looked at making list of variables using /XREF and on my 1.5 pages long subroutine it generated 30 pages of data 😦

Dan,

I did suggest Excel to filter the listing and ignore line numbers of the module. I disagree with JohnS's response! Perhaps he forgot to define the outcome. If you don't like a 'manual or semi-manual way of extraction' then look to code generator software. I prefer to write some code.

Regarding your and JohnS's comments, I find the hardest part of generating a small cut-down test is to regenerate the data set to apply. You may remember my equation solver test program that mecej4 identified as having lots of floating point exceptions (FPE) because I did not generate a matrix that was sufficiently 'well behaved'. Found a lot of other problems along the way that were not related to the original problem. (FTN95's new AVX routines helped a lot, which was a good outcome)

John

This has been an interesting thread, and Dan's colorful descriptions got me thinking, 'Suppose we could write the Code Exorciser that he asks for, how well would it work?' In other words, what would we gain, if we had such a tool? Even though we do not have such a tool, we could simulate (with manual labor) what it is supposed to do, and assess whether it would be worth developing.

Fortunately, I had available to me a nice test case. Here was a well-established and widely used code with a history of over 30 years. I had recently found a bug in FTN95 that showed up when applied to the code, spent quite a bit of time creating a small reproducer and reporting the bug (see https://forums.silverfrost.com/Forum/Topic/3332 ). The original code, at http://www.tampa.phys.ucl.ac.uk/rmat/src/inner/swmol3/swmol3.f, is 143 kbytes, mostly F77 and a few allocatable local arrays. My manually created reproducer is 245 bytes.

Here is what I did.

Step 1: Choose a test case (input data) for which the bug surfaced. Compile with /debug and locate the line where the program aborted, and obtain the traceback. (prog. size: 143 kb)

Step 2: Place a STOP statement above the line located in Step 1, compile with /timing, and run again. From the .TMR report that was generated, obtain a list of subprograms that were not executed during the run. Remove the STOP statement placed in this step. (prog. size: 143 kb)

Step 3: Remove the source code of the unused subprograms, and replace that by dummy subprograms that had just STOP 'Subroutine xxx was called'. Compile and run to verify that the bug is still present. (prog. size: 50 kb)

Step 4: Remove the source lines in the main program that came logically after the subroutine call that caused the abort. Compile and run to verify that the bug is still present. (prog. size: 43 kb)

Step 5: Further reduction by removing unused variables. FTN95 gave me a list of unused variables, but I did not want to spend the time to remove them manually. Instead, I used SPAG with clutter removal set at level 5. The resulting program ran, but the bug had disappeared. (prog. size: 49 kb).

Conclusions: The tool that Dan wants would have taken care of Steps 4 and 5 with little effort from the user. Unfortunately, the reduction in size (from full code to reproducer) was just 1/3 (compare to the 1/600 that resulted from manual code simplification). Furthermore, clutter removal had the unfortunate effect of making the bug disappear.

Side result: It is curious that SPAG's removing the declarations of unused local allocatable arrays caused the bug to disappear. This finding may help with fixing the bug reported in https://forums.silverfrost.com/Forum/Topic/3332 . Is FTN95 generating code to deallocate, just before the RETURN statement, arrays that, since they were never used, it decided earlier not to allocate?

It is possible to write tests as separate subroutines. With the right software (I use a python script that 'understands' Fortran) you can automatically generate Fortran source code for a program that calls a selection of (or all of) the tests. Only the subroutines and functions that need to be tested are called in the final executable. It ensures the tests are independent of each other and makes writing/testing code much easier.

I'm sure any new information will be useful, even if after the automatic Extractor the bug will disappear. This may mean that the conflict or bug lies somewhere else. The extraction will take less then 1 second. Manually that takes days full of hell. I will expect my specific code after such automatic extraction will be not 1/3 but 1% or 0.1% of initial code as the bug is associated with plotting for one specific graph. There are tons of such plots and tons of other things in the code.

I think that specifically easy to create such bug Extractor/Exorciser would be for developers of FTN95 as FTN95 is doing most detailed parsing of the source and knows about it everything.

FTN95 Source.f95 /extract sub1

where Source.f95 is initial Fortran source, /extract is appropriate switch sub1 is the subroutine name where the the bug is suspected The final source will be in source.xtr