forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Extracting part of code from the large source

 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sat Apr 28, 2018 4:44 am    Post subject: Extracting part of code from the large source Reply with quote

One problem with using MODULEs instead of COMMON blocks is difficulty to extract part of code while keeping only relevant definitions. Modules tend to grow in size swallowing or monopolizing other modules and each and every variable of the code inside just a single major module. This is because this way it is just easier to exchange the data. With COMMON blocks you kept in subroutines and functions only relevant variables. Separating subroutines from large code with COMMON blocks were not a problem at all.

I need right now to make small and simple demo out of large code which has a problem to send it to Paul for investigation but losing again days to throw all not related variables and their initializations. Did anyone try to write the code where you set which subroutines to keep and it will automatically remove all not related variables and subroutines?
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sat Apr 28, 2018 5:37 am    Post subject: Reply with quote

Dan,

I use /xref to get a list of variables that are referenced outside the module, using Excel and pivot tables.
Once I have that list, I can comment out all variables in the module that are not required.
I then use another "fiendish" option to check I have not deleted too much.

I actually find little difference between MODULE and INCLUDE for this issue. I can have multiple COMMON in a single include (for separating integer/real from character)

MODULE has the benefit of allocated arrays, but looses some EQUIVALENCE capability.

FIND and FINDSTR are also good utilities to check this. Try the following search for variable %1 options:
findstr /i /n /s /c:%1 *.f9? *.in? >> tfs.tce
find /i /n "%1" *.f?? *.in? >%1.tce

Not a major problem
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Sun Apr 29, 2018 1:28 am    Post subject: Reply with quote

Thanks John, but this is still kind of manual or semi-manual way of extraction. I made with the mind crushing difficulty 5-6 manual demos during these two weeks trying to get the source of crash caused by simpleplot_redraw@ and all of them get me demos which show no crash problems. I put them back into the larger code and they crash...

I thought may be there exist fully automatic way for demo extraction. A lot of users in this forum complained at some time about difficulty to reproduce their bugs in small demo examples. May be Silverfrost or some of us would make such Demo Extractor to help people to debug their codes. You tell it the name of subroutine you want to extract and it will leave only relevant subroutines and variables removing all others
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Sun Apr 29, 2018 6:31 am    Post subject: Reply with quote

Dan,

I'd recommend you look at my recent post regarding a bug I had dumping .png files of a %gr plot. It was difficult to locate and very hard to reproduce. In the end it was most likely not a clearwin+ bug, but a problem from multiple open/close operations (this is close to a known problem with windows). ( this code was a work-around, due to unknown problems with INQUIRE 20 years ago, which had since been fixed. As you say we should report all bugs)

See: http://forums.silverfrost.com/viewtopic.php?t=3619&start=15

Anyway, if it is difficult to reproduce, you may be looking in the wrong place. Worth considering.

John
Back to top
View user's profile Send private message
John-Silver



Joined: 30 Jul 2013
Posts: 1520
Location: Aerospace Valley

PostPosted: Sun Apr 29, 2018 9:12 pm    Post subject: Reply with quote

Good point JohnC, but the main stumbling block is two-fold (I'm stating the obvious here I know):

a) the bigger the original code the more difficult to create a minimum reproducer, not least because it could well be because of several interacting factors

b) what is considered an acceptable 'reproducer code' size ? I recently reported some %pl errors and Paul at one point suggested I create a more concise reproducer code - when my code was a variant on an FTN95 Example code in the documentation !!! I resisted the temptation to reply in the way my head was telling me to !

Dan's point is a valid one though. As soon as I saw you mention Excel JohnC I intuitively know it's not an optimum solution, not because it won't work but because of Excel's inherent 'learning curve (the method is optimum for yourself but isn't a point-n-click fool-proof method) I know from experience using Excel as standard in my line of work that it's often as counter-productive as it is productive precisely because of it's inherent flexibility. Excel should be taken to the european court of human rights for its efforts to waste time & money (10 people will produce 10 different Excel spreadsheets if given the same problem to solve and each one will declare theirs to be the best and far superior to the others when in !). But I digress.

Everything is relative though. As I mention above, you may reduce say a 50000 line program down to 1000 which may still be considred to be not 'simplified' to get the intensity of user-interaction and developer comittment necessary to pinpoint & correct the bug(s).

Of course what's needed are these

3 steps on the road to Dan's FTN bug-fixing heaven
www.youtube.com/watch?v=WaVSkh5Hy-U&pbjreload=10

1. Full source code in
2.define (simple list of) procedures to be included + I suppose only a (what's thought to be the relevant) part of main code ?)
3. Reproducer code out

... with the program being 'clever enough' to parse the routines+ main sub-code and simply retain only the module(s) entries necessary.
If I understand correctly your request Dan.

Everything is relative though. As I mention above, you may reduce say a 50000 line program down to 1000 which may still b considred to be not 'simplified' to get the intensity of user-interaction and developer comittment necessary to pinpoint & correct the bug(s).

Ahh - why not SF use their contacts to get one of those time-wasting-bar-propping-rag-week-preparing/recuperating-student-types to write one !!!!! Wink(not actually a joke).
_________________
_________________
''Computers (HAL and MARVIN excepted) are incredibly rigid. They question nothing. Especially input data.Human beings are incredibly trusting of computers and don't check input data. Together cocking up even the simplest calculation ... Smile "
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Mon Apr 30, 2018 4:05 am    Post subject: Reply with quote

How I am screwed: looked at making list of variables using /XREF and on my 1.5 pages long subroutine it generated 30 pages of data Sad
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Mon Apr 30, 2018 6:20 am    Post subject: Reply with quote

Dan,

I did suggest Excel to filter the listing and ignore line numbers of the module. I disagree with JohnS's response! Perhaps he forgot to define the outcome.
If you don't like a "manual or semi-manual way of extraction" then look to code generator software. I prefer to write some code.

Regarding your and JohnS's comments, I find the hardest part of generating a small cut-down test is to regenerate the data set to apply. You may remember my equation solver test program that mecej4 identified as having lots of floating point exceptions (FPE) because I did not generate a matrix that was sufficiently "well behaved". Found a lot of other problems along the way that were not related to the original problem. (FTN95's new AVX routines helped a lot, which was a good outcome)

John
Back to top
View user's profile Send private message
mecej4



Joined: 31 Oct 2006
Posts: 1885

PostPosted: Mon Apr 30, 2018 3:08 pm    Post subject: Reply with quote

This has been an interesting thread, and Dan's colorful descriptions got me thinking, "Suppose we could write the Code Exorciser that he asks for, how well would it work?" In other words, what would we gain, if we had such a tool? Even though we do not have such a tool, we could simulate (with manual labor) what it is supposed to do, and assess whether it would be worth developing.

Fortunately, I had available to me a nice test case. Here was a well-established and widely used code with a history of over 30 years. I had recently found a bug in FTN95 that showed up when applied to the code, spent quite a bit of time creating a small reproducer and reporting the bug (see http://forums.silverfrost.com/viewtopic.php?t=3743 ). The original code, at http://www.tampa.phys.ucl.ac.uk/rmat/src/inner/swmol3/swmol3.f, is 143 kbytes, mostly F77 and a few allocatable local arrays. My manually created reproducer is 245 bytes.

Here is what I did.

Step 1: Choose a test case (input data) for which the bug surfaced. Compile with /debug and locate the line where the program aborted, and obtain the traceback. (prog. size: 143 kb)

Step 2: Place a STOP statement above the line located in Step 1, compile with /timing, and run again. From the .TMR report that was generated, obtain a list of subprograms that were not executed during the run. Remove the STOP statement placed in this step. (prog. size: 143 kb)

Step 3: Remove the source code of the unused subprograms, and replace that by dummy subprograms that had just STOP 'Subroutine xxx was called'. Compile and run to verify that the bug is still present. (prog. size: 50 kb)

Step 4: Remove the source lines in the main program that came logically after the subroutine call that caused the abort. Compile and run to verify that the bug is still present. (prog. size: 43 kb)

Step 5: Further reduction by removing unused variables. FTN95 gave me a list of unused variables, but I did not want to spend the time to remove them manually. Instead, I used SPAG with clutter removal set at level 5. The resulting program ran, but the bug had disappeared. (prog. size: 49 kb).

Conclusions: The tool that Dan wants would have taken care of Steps 4 and 5 with little effort from the user. Unfortunately, the reduction in size (from full code to reproducer) was just 1/3 (compare to the 1/600 that resulted from manual code simplification). Furthermore, clutter removal had the unfortunate effect of making the bug disappear.

Side result: It is curious that SPAG's removing the declarations of unused local allocatable arrays caused the bug to disappear. This finding may help with fixing the bug reported in http://forums.silverfrost.com/viewtopic.php?t=3743 . Is FTN95 generating code to deallocate, just before the RETURN statement, arrays that, since they were never used, it decided earlier not to allocate?
Back to top
View user's profile Send private message
davidb



Joined: 17 Jul 2009
Posts: 560
Location: UK

PostPosted: Mon Apr 30, 2018 6:02 pm    Post subject: Reply with quote

It is possible to write tests as separate subroutines. With the right software (I use a python script that "understands" Fortran) you can automatically generate Fortran source code for a program that calls a selection of (or all of) the tests. Only the subroutines and functions that need to be tested are called in the final executable. It ensures the tests are independent of each other and makes writing/testing code much easier.
_________________
Programmer in: Fortran 77/95/2003/2008, C, C++ (& OpenMP), java, Python, Perl
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2813
Location: South Pole, Antarctica

PostPosted: Tue May 01, 2018 8:09 am    Post subject: Reply with quote

I'm sure any new information will be useful, even if after the automatic Extractor the bug will disappear. This may mean that the conflict or bug lies somewhere else. The extraction will take less then 1 second. Manually that takes days full of hell. I will expect my specific code after such automatic extraction will be not 1/3 but 1% or 0.1% of initial code as the bug is associated with plotting for one specific graph. There are tons of such plots and tons of other things in the code.

I think that specifically easy to create such bug Extractor/Exorciser would be for developers of FTN95 as FTN95 is doing most detailed parsing of the source and knows about it everything.

FTN95 Source.f95 /extract sub1

where Source.f95 is initial Fortran source,
/extract is appropriate switch
sub1 is the subroutine name where the the bug is suspected
The final source will be in source.xtr
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group