forums.silverfrost.com Forum Index forums.silverfrost.com
Welcome to the Silverfrost forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Best way to add data to the growing file
Goto page Previous  1, 2
 
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General
View previous topic :: View next topic  
Author Message
JohnHorspool



Joined: 26 Sep 2005
Posts: 270
Location: Gloucestershire UK

PostPosted: Wed May 19, 2010 10:09 pm    Post subject: Reply with quote

Dan,

It gives incorrect result on my machine (64bit AMD Phenom quadcore) running XP64.
Back to top
View user's profile Send private message Visit poster's website
DanRRight



Joined: 10 Mar 2008
Posts: 2816
Location: South Pole, Antarctica

PostPosted: Wed May 19, 2010 10:18 pm    Post subject: Reply with quote

What it gives, current clock, max clock, other? You know clock frequency is varied by OS, right? Issue may appear when different cores are clocked differently.
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Thu May 20, 2010 1:42 am    Post subject: Reply with quote

John,

As you have found, it is not guaranteed for QueryPerformanceFrequency to give the processor frequency.
QueryPerformanceCounter and CPU_Clock@ appear to be based on the QueryPerformanceFrequency and most accurately give elapsed time. On reading the documentation for these there is no guarantee what the processor manufacturer does with updating these counters.

Another puzzle to me is why so many of the other system timing counters are only updated @ 60 hz. I tried to send questions on the MSDN Forums but never got an answer as to how to increase the update frequency. CPU_Time, date_and_time, Dclock@, GetLocalTime, GetTickCount and GetProcessTimes all appear to be related to this timer that is only updated 60 times a second. I don't know:
- where the 60 cycles is defined,
- how it could be changed, and
- what overhead would result.

I have written a routine to run CPU_Clock@ for .7 seconds (timed using dclock@) and calibrate the tick count, then store this in c:\processor_mhz.txt for other routines to quickly get. (I keep calling dclock@, noting when it changes value 41 times, indicating the start time of each cycle and call QPC each cycle)

If you want to differentiate between elapsed time and process (CPU) time, then GetProcessTimes gives some better info, but is only updated @ 60hz. It can be useful to differentiate between CPU or Processor time and disk I/O time, but achieving this changes with different processors, even different Intel processors !

If anyone can provide more information on this, I would love to know.

John
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2816
Location: South Pole, Antarctica

PostPosted: Sun Sep 12, 2010 1:57 am    Post subject: Reply with quote

Related question:

As it should be expected. at some point i've got "disk full" and some crazy mess on the disk with 5000 empty tmp files. Great that more nasty things did not happen so i'd lose all the data gathered during last 4-5 months. That's tick data for all trades ever happened with each and every stock

Smart code has to predict that of course. What is the best way to find the free space left on specified disk drive using library functions ?

I'd do that by quasi-portable way using one or another OS commands built into all latest Fortran compilers like this

OScommand ='R:>dir >zzzOut '
call cissue@(OScommand,iOk)

and then analyzing last line in the created file zzzOut which usually end like this
13 File(s) 3,234,554,325 bytes
4 Dir(s) 4,454,872,832 bytes free

Any other way of doing that by invoking this compiler's huge library or WinAPI ?

(Let me know if you need something for your research or curiosity, no problem to sponsor our fellow fortraneers)
Back to top
View user's profile Send private message
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Tue Sep 14, 2010 2:10 am    Post subject: Reply with quote

Dan,

I wrote a simple program to do what you suggested. One problem, this uses a temproary file so you will need to check when the disk is full that the file might not be created. What would cissue@ do in this case ? Also, are there other possible failures for cissue@ that are not due to a full disk.

Code:
subroutine list_file (file_name, free)
character file_name*(*)
integer*8 free
character line*80
integer*4 iostat, n
real*8    x
!
free = 0   ! assume no space
!
open (unit=11, file=file_name,iostat=iostat)
if (iostat /= 0) then
  write (*,*) 'Error',iostat,' opening file ',trim(file_name)
  return
else
  write (*,*) 'File : ',trim(file_name),' opened'
end if
!
do n = 1,huge(n)
   read (11,fmt='(a)',iostat=iostat) line
   if (iostat /= 0) exit
   write (*,*) trim (line)
   call get_free (line, free)
end do
!
close (unit=11, status='delete')
!
x = dble(free)/1024./1024./1024.        ! I still have a lot of disk space
write (*,*) n-1,' lines'
write (*,fmt='(i0,a,f0.2,a)') free, ' free space (',x,'gb)'
!
end

subroutine get_free (line, free)
character line*(*)
integer*8 free
!
integer*4 i,j
integer*8 :: ten  = 10
integer*8 c
!
i = index (line, 'bytes free')
if (i < 1) return
!
j = index (line, ')')
write (*,*)' Free =', line(j+1:i-1)
free = 0
do i = j+1,i-1
   c = ichar(line(i:i)) - ichar('0')
    if (c < 0) cycle
    if (c > 9) cycle
   free = free * ten + c
end do
end

John
Back to top
View user's profile Send private message
IanLambley



Joined: 17 Dec 2006
Posts: 490
Location: Sunderland

PostPosted: Tue Sep 14, 2010 12:58 pm    Post subject: Reply with quote

Dan,
Are you trying to find the free space, then try this program which has a couple of methods.
Code:

winapp
include <windows.ins>
logical*4 status
integer*8 BytesAvailableToCaller, TotalBytes, FreeBytes
integer*8 lpSectorsPerCluster8, lpBytesPerSector8, lpNumberOfFreeClusters8, lpTotalNumberOfClusters8

STDCALL GETDISKFREESPACEEX 'GetDiskFreeSpaceExA' (STRING,REF,REF,REF):LOGICAL*4

! First version for small disks < 2GB
status = GETDISKFREESPACE('c:\',lpSectorsPerCluster,    &
                               lpBytesPerSector,       &
                               lpNumberOfFreeClusters, &
                               lpTotalNumberOfClusters)
print *, status,lpSectorsPerCluster,    &
                lpBytesPerSector,       &
                lpNumberOfFreeClusters, &
                lpTotalNumberOfClusters
lpSectorsPerCluster8     = lpSectorsPerCluster
lpBytesPerSector8        = lpBytesPerSector
lpTotalNumberOfClusters8 = lpTotalNumberOfClusters
lpNumberOfFreeClusters8  = lpNumberOfFreeClusters             
TotalBytes=lpSectorsPerCluster8*lpBytesPerSector8*lpTotalNumberOfClusters8
print *,'Totalbytes',TotalBytes
FreeBytes=lpSectorsPerCluster8*lpBytesPerSector8*lpNumberOfFreeClusters8
print *,'Freebytes ',FreeBytes

!alternative version for large disks > 2GB requires STDCALL declaration above
status = GETDISKFREESPACEEX('c:\',BytesAvailableToCaller, &
                                  TotalBytes,             &
                                  FreeBytes)
print *, status,BytesAvailableToCaller, &
                TotalBytes,             &
                FreeBytes
end

Regards
Ian
Back to top
View user's profile Send private message Send e-mail
DanRRight



Joined: 10 Mar 2008
Posts: 2816
Location: South Pole, Antarctica

PostPosted: Wed Sep 15, 2010 1:01 am    Post subject: Reply with quote

Thanks John and Ian for ideas and solutions, they are very different valuable approaches.

As to the cissue, John, it is not completely clear to me yet when it issues an error status and when it does not. Definitely though when i heavily debugged it some time ago i've seen one case, i forgot right now exact details, will edit this post when remember, when i'd expect it to issue an error but it did not. So the idea is to get rid of it and use all other alternatives, may be even all of them for reliability
Back to top
View user's profile Send private message
DanRRight



Joined: 10 Mar 2008
Posts: 2816
Location: South Pole, Antarctica

PostPosted: Thu Sep 16, 2010 9:56 am    Post subject: Reply with quote

Ian, do i have to use both functions for arbitrary large partitions or just the second is enough?
Back to top
View user's profile Send private message
IanLambley



Joined: 17 Dec 2006
Posts: 490
Location: Sunderland

PostPosted: Thu Sep 16, 2010 12:00 pm    Post subject: Reply with quote

Just the second
Back to top
View user's profile Send private message Send e-mail
JohnCampbell



Joined: 16 Feb 2006
Posts: 2554
Location: Sydney

PostPosted: Fri Sep 17, 2010 3:04 am    Post subject: Reply with quote

Ian,

The first call will give details for a disk of up to 8 terrabytes ! I think that is still a big disk.
Viewing the files will be an issue as Notepad struggles with files approaching 1gb. My line editor reads much quicker but tops out at 1.6gb (win32 again!!)

Incidently, your programing example sent me looking through the include files in the ftn95\include directory. Looking at the .ins files has driven me to writing a converter from 72 character fixed to 132 character free format so I can see the routine names and aliases as lists. It makes what looked like machine code look more readable.

John
Back to top
View user's profile Send private message
DrTip



Joined: 01 Aug 2006
Posts: 74
Location: Manchester

PostPosted: Fri Nov 19, 2010 12:44 pm    Post subject: Reply with quote

I know this one is a bit old now but it struck me at the time a more abstracted way of doing this would be to use a real database management system for data storage.

You would then use your favourite database communication technology to talk to the database.

The advantages are this. The people who have built the DBMS (microsoft, oracle) whoever have done lots of research into optimal database read write stuff, data integrity allow multi user access to data etc, etc.

The database can be put on a dedicated server and 1GB is a tiny database so will DBMS will make mince meat of it. eg SQL Server Express (free software) allows databases up to 12GB in size. DB2 is similar I believe. Don't know about MySQL.

Just a thought

Carl
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    forums.silverfrost.com Forum Index -> General All times are GMT + 1 Hour
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group