samedi 22 août 2015

Datetime/Timestamp If-Comparision, handling of Datetime ranges

I am setting up a monitoring project, where i receive CSV-files for three years. I read, check and process them in an infinite loop.

In the first try: except: statements I am checking if there are previously processed files ( in case of rerun).

Then I am starting an inifinite loop (while True) that can be interrupted by Contrl-c (_try: except:), where FIRST csv files are read and a DataFrame with 'time' as the index is produced. (this resuts in Timestamps). This works fine.

SECONDLY, check if the start time of the file already read data.index[0] is in the list of before processedfiles. If True the file is moved to the DUPLICATEARCHIVE and not processed, ELSE:start, endtime and filename are added to the Dataframe Processedfiledaten and are appended to the disk DatafileTable.csv.

PROBLEM 1: The if data.index[0] in processedfiles.startzeit: is False for equal Date&Time processes the else: part.

I consulted the docs, but it is becoming more and more confusing.

PROBLEM 2: Is there an easy way to check if Data is missing (time range)?

Help? Suggestions? Thanks.

try:
    processedfiles = pd.read_table('DatafileTable.csv', sep=',', header=0, parse_dates=['startzeit', 'endzeit']) # , index_col='startzeit'
    print('The previously proccessed data will be used:')
except:
    print('No previously analyseed data was found')
    processedfiles= DataFrame.from_dict({'startzeit': [], 'endzeit': [], 'Archivname': []}, orient='columns')    
    processedfiles.to_csv('DatafileTable.csv', mode='a', sep=',', header= True, index=False, line_terminator='\n')


try:
    while True:
        for file in os.listdir(dataIndir):
            if file.endswith(".CSV"):
                # reads .CSV files from dataIndir
                data = pd.read_table(dataIndir+'/'+file, sep=';', skiprows=7, 
                         decimal =',', usecols= range(6),
                         header=None, names =datanames, date_parser =date_converter_CLUM_v1,
                         parse_dates=[0], index_col='time')

        if data.index[0] in processedfiles.startzeit:
                #move CSV file to directory dataDuplikat
                archivefile=Archive_original_Datafiles(file,dataIndir, dataDuplikat) 
                print('This data was already analyzed')
            else:
                # Function that zip(s) and moves the files to 'dataOriginal'-Directory
                archivefile=Archive_original_Datafiles(file,dataIndir, dataOriginal) 

                # saves _Starttime, Endtime, Archivfilename_ in _processedfileDaten_, appends to _processedfiles_ and appends on diskfile _DatafileTable.csv_
                processedfileDaten= DataFrame.from_dict({'startzeit': [data.index[0]], 'endzeit': [data.index[-1]], 'Archivname': [archivefile]}, orient='columns')
                processedfileDaten.to_csv('DatafileTable.csv', mode='a', sep=',', header= False, index=False, line_terminator='\n')
                processedfiles = processedfiles.append(processedfileDaten, ignore_index=True)

        if not os.listdir(dataIndir):
            print('Waiting for Datafiles..., to stop Program press Control-c')
            time.sleep(10)
except KeyboardInterrupt:
    print('interrupted!')

when i look at the data

In[39]:processedfiles.startzeit.dtype
Out[39]: dtype('<M8[ns]')

In[40]:data.index.dtype
Out[40]: dtype('<M8[ns]')

In[36]:data.head(3)
Out[36]: 
                     wind_v  wind_Phi  Temp    DMS1    DMS2
time                                                           
2014-11-07 13:09:19.000     2.9       184  25.4 -0.0009  4.9884
2014-11-07 13:09:19.010     2.9       184  25.4  0.0037  4.9866
2014-11-07 13:09:19.020     2.9       184  25.4 -0.0006  4.9854

In[37]:data.dtypes
Out[37]: 
wind_v      float64
wind_Phi      int64
Temp        float64
DMS1        float64
DMS2        float64
dtype: object


In[38]:processedfiles.startzeit
Out[38]: 
0   2014-11-07 13:09:19
1   2014-11-07 13:09:19
2   2014-11-07 10:35:43
3   2014-11-07 10:35:43
4   2014-11-07 10:35:43
5   2014-11-07 10:35:43
Name: startzeit, dtype: datetime64[ns]

Aucun commentaire:

Enregistrer un commentaire