Pandas中Apply函数加速百倍的技巧

import pandas as pdimport numpy as npdf = pd.DataFrame(np.random.randint(0, 11, size=(1000000, 5)), columns=('a','b','c','d','e'))def func(a,b,c,d,e):    if e == 10:        return c*d    elif (e < 10) and (e>=5):        return c+d    elif e < 5:        return a+b%%timedf['new'] = df.apply(lambda x: func(x['a'], x['b'], x['c'], x['d'], x['e']), axis=1)CPU times: user 17.9 s, sys: 301 ms, total: 18.2 sWall time: 18.4 s

%%time# !pip install swifterimport swifterdf['new'] = df.swifter.apply(lambda x : func(x['a'],x['b'],x['c'],x['d'],x['e']),axis=1)HBox(children=(HTML(value='Dask Apply'), FloatProgress(value=0.0, max=16.0), HTML(value='')))
CPU times: user 329 ms, sys: 240 ms, total: 569 msWall time: 7.67 s

• for循环；
• 列表处理；
• apply等操作

%%timedf['new'] = df['c'] * df['d'] #default case e = =10mask = df['e'] < 10df.loc[mask,'new'] = df['c'] + df['d']mask = df['e'] < 5df.loc[mask,'new'] = df['a'] + df['b']CPU times: user 134 ms, sys: 149 ms, total: 283 msWall time: 421 ms

for col in ('a','b','c','d'):    df[col] = df[col].astype(np.int16) %%timedf['new'] = df['c'] * df['d'] #default case e = =10mask = df['e'] < 10df.loc[mask,'new'] = df['c'] + df['d']mask = df['e'] < 5df.loc[mask,'new'] = df['a'] + df['b']CPU times: user 71.3 ms, sys: 42.5 ms, total: 114 msWall time: 116 ms

• 此处先转化为.values等价于转化为numpy，这样我们的向量化操作会更加快捷。

%%timedf['new'] = df['c'].values * df['d'].values #default case e = =10mask = df['e'].values < 10df.loc[mask,'new'] = df['c'] + df['d']mask = df['e'].values < 5df.loc[mask,'new'] = df['a'] + df['b']CPU times: user 64.5 ms, sys: 12.5 ms, total: 77 msWall time: 74.9 ms

• Apply: 18.4 s
• Apply + Swifter: 7.67 s
• Pandas vectorizatoin: 421 ms
• Pandas vectorization + data types: 116 ms
• Pandas vectorization + values + data types: 74.9ms