I am setting up a monitoring project, where i receive CSV-files for three years. I read, check and process them in an infinite loop.
In the first try: except: statements I am checking if there are previously processed files ( in case of rerun).
Then I am starting an inifinite loop (while True) that can be interrupted by Contrl-c (_try: except:), where FIRST csv files are read and a DataFrame with 'time' as the index is produced. (this resuts in Timestamps). This works fine.
SECONDLY, check if the start time of the file already read data.index[0] is in the list of before processedfiles. If True the file is moved to the DUPLICATEARCHIVE and not processed, ELSE:start, endtime and filename are added to the Dataframe Processedfiledaten and are appended to the disk DatafileTable.csv.
PROBLEM 1: The if data.index[0] in processedfiles.startzeit: is False for equal Date&Time processes the else: part.
I consulted the docs, but it is becoming more and more confusing.
PROBLEM 2: Is there an easy way to check if Data is missing (time range)?
Help? Suggestions? Thanks.
try:
processedfiles = pd.read_table('DatafileTable.csv', sep=',', header=0, parse_dates=['startzeit', 'endzeit']) # , index_col='startzeit'
print('The previously proccessed data will be used:')
except:
print('No previously analyseed data was found')
processedfiles= DataFrame.from_dict({'startzeit': [], 'endzeit': [], 'Archivname': []}, orient='columns')
processedfiles.to_csv('DatafileTable.csv', mode='a', sep=',', header= True, index=False, line_terminator='\n')
try:
while True:
for file in os.listdir(dataIndir):
if file.endswith(".CSV"):
# reads .CSV files from dataIndir
data = pd.read_table(dataIndir+'/'+file, sep=';', skiprows=7,
decimal =',', usecols= range(6),
header=None, names =datanames, date_parser =date_converter_CLUM_v1,
parse_dates=[0], index_col='time')
if data.index[0] in processedfiles.startzeit:
#move CSV file to directory dataDuplikat
archivefile=Archive_original_Datafiles(file,dataIndir, dataDuplikat)
print('This data was already analyzed')
else:
# Function that zip(s) and moves the files to 'dataOriginal'-Directory
archivefile=Archive_original_Datafiles(file,dataIndir, dataOriginal)
# saves _Starttime, Endtime, Archivfilename_ in _processedfileDaten_, appends to _processedfiles_ and appends on diskfile _DatafileTable.csv_
processedfileDaten= DataFrame.from_dict({'startzeit': [data.index[0]], 'endzeit': [data.index[-1]], 'Archivname': [archivefile]}, orient='columns')
processedfileDaten.to_csv('DatafileTable.csv', mode='a', sep=',', header= False, index=False, line_terminator='\n')
processedfiles = processedfiles.append(processedfileDaten, ignore_index=True)
if not os.listdir(dataIndir):
print('Waiting for Datafiles..., to stop Program press Control-c')
time.sleep(10)
except KeyboardInterrupt:
print('interrupted!')
when i look at the data
In[39]:processedfiles.startzeit.dtype
Out[39]: dtype('<M8[ns]')
In[40]:data.index.dtype
Out[40]: dtype('<M8[ns]')
In[36]:data.head(3)
Out[36]:
wind_v wind_Phi Temp DMS1 DMS2
time
2014-11-07 13:09:19.000 2.9 184 25.4 -0.0009 4.9884
2014-11-07 13:09:19.010 2.9 184 25.4 0.0037 4.9866
2014-11-07 13:09:19.020 2.9 184 25.4 -0.0006 4.9854
In[37]:data.dtypes
Out[37]:
wind_v float64
wind_Phi int64
Temp float64
DMS1 float64
DMS2 float64
dtype: object
In[38]:processedfiles.startzeit
Out[38]:
0 2014-11-07 13:09:19
1 2014-11-07 13:09:19
2 2014-11-07 10:35:43
3 2014-11-07 10:35:43
4 2014-11-07 10:35:43
5 2014-11-07 10:35:43
Name: startzeit, dtype: datetime64[ns]
Aucun commentaire:
Enregistrer un commentaire