dimanche 24 mai 2015

Nested if loop with DataFrame is very,very slow

I have 10 million rows to go through and it will take many hours to process, I must be doing something wrong

I converted the names of my df variables for ease in typing

Close=df['Close']
eqId=df['eqId']
date=df['date']
IntDate=df['IntDate']
expiry=df['expiry']
delta=df['delta']
ivMid=df['ivMid']
conf=df['conf']

The below code works fine, just ungodly slow, any suggestions?

print(datetime.datetime.now().time())
for i in range(2,1000):
if delta[i]==90:
    if delta[i-1]==50:
        if delta[i-2]==10:
            if expiry[i]==expiry[i-2]:
                df.Skew[i]=ivMid[i]-ivMid[i-2]

print(datetime.datetime.now().time())


14:02:11.014396
14:02:13.834275


df.head(100)
Close   eqId    date    IntDate expiry  delta   ivMid   conf    Skew
0   37.380005   7   2008-01-02  39447   1   50  0.3850  0.8663  
1   37.380005   7   2008-01-02  39447   1   90  0.5053  0.7876  
2   36.960007   7   2008-01-03  39448   1   50  0.3915  0.8597  
3   36.960007   7   2008-01-03  39448   1   90  0.5119  0.7438  
4   35.179993   7   2008-01-04  39449   1   50  0.4055  0.8454  
5   35.179993   7   2008-01-04  39449   1   90  0.5183  0.7736  
6   33.899994   7   2008-01-07  39452   1   50  0.4464  0.8400  
7   33.899994   7   2008-01-07  39452   1   90  0.5230  0.7514  
8   31.250000   7   2008-01-08  39453   1   10  0.4453  0.7086  
9   31.250000   7   2008-01-08  39453   1   50  0.4826  0.8246  
10  31.250000   7   2008-01-08  39453   1   90  0.5668  0.6474  0.1215
11  30.830002   7   2008-01-09  39454   1   10  0.4716  0.7186  
12  30.830002   7   2008-01-09  39454   1   50  0.4963  0.8479  
13  30.830002   7   2008-01-09  39454   1   90  0.5735  0.6704  0.1019
14  31.460007   7   2008-01-10  39455   1   10  0.4254  0.6737  
15  31.460007   7   2008-01-10  39455   1   50  0.4929  0.8218  
16  31.460007   7   2008-01-10  39455   1   90  0.5902  0.6411  0.1648
17  30.699997   7   2008-01-11  39456   1   10  0.4868  0.7183  
18  30.699997   7   2008-01-11  39456   1   50  0.4965  0.8411  
19  30.639999   7   2008-01-14  39459   1   10  0.5117  0.7620  
20  30.639999   7   2008-01-14  39459   1   50  0.4989  0.8804  
21  30.639999   7   2008-01-14  39459   1   90  0.5887  0.6845  0.077
22  29.309998   7   2008-01-15  39460   1   10  0.4956  0.7363  
23  29.309998   7   2008-01-15  39460   1   50  0.5054  0.8643  
24  30.080002   7   2008-01-16  39461   1   10  0.4983  0.6646  

At this rate it will take 7.77 hrs to process

Aucun commentaire:

Enregistrer un commentaire