[关闭]
@hainingwyx 2016-11-27T13:50:57.000000Z 字数 16837 阅读 1288

Pandas入门

Python pandas


Getting started with pandas

  1. from pandas import Series, DataFrame
  2. import pandas as pd
  1. from __future__ import division
  2. from numpy.random import randn
  3. import numpy as np
  4. import os
  5. import matplotlib.pyplot as plt
  6. np.random.seed(12345)
  7. plt.rc('figure', figsize=(10, 6))
  8. from pandas import Series, DataFrame
  9. import pandas as pd
  10. np.set_printoptions(precision=4)
  1. %pwd
u'C:\\Users\\WangYixin\\Desktop\\pydata-book-master'

Introduction to pandas data structures

Series

由一组数据和一组与之相关的数据标签组成,仅由一组数据即可产生最简单的Series

  1. obj = Series([4, 7, -5, 3])
  2. obj
0    4
1    7
2   -5
3    3
dtype: int64
  1. obj.values
  2. obj.index
RangeIndex(start=0, stop=4, step=1)
  1. obj2 = Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
  2. obj2
d    4
b    7
a   -5
c    3
dtype: int64
  1. obj2.index
Index([u'd', u'b', u'a', u'c'], dtype='object')
  1. obj2['a']#索引
-5
  1. obj2['d'] = 6
  2. obj2[['c', 'a', 'd']]
c    3
a   -5
d    6
dtype: int64
  1. obj2[obj2 > 0]
d    6
b    7
c    3
dtype: int64
  1. obj2 * 2#保留Numpy数组运算
d    12
b    14
a   -10
c     6
dtype: int64
  1. np.exp(obj2)
d     403.428793
b    1096.633158
a       0.006738
c      20.085537
dtype: float64
  1. 'b' in obj2#索引值到数据值的映射
True
  1. 'e' in obj2
False
  1. sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}#可以通过字典直接创建Series
  2. obj3 = Series(sdata)
  3. obj3
Ohio      35000
Oregon    16000
Texas     71000
Utah       5000
dtype: int64
  1. states = ['California', 'Ohio', 'Oregon', 'Texas']
  2. obj4 = Series(sdata, index=states)
  3. obj4
California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64
  1. pd.isnull(obj4)#检测数据缺失
California     True
Ohio          False
Oregon        False
Texas         False
dtype: bool
  1. pd.notnull(obj4)
California    False
Ohio           True
Oregon         True
Texas          True
dtype: bool
  1. obj4.isnull()
California     True
Ohio          False
Oregon        False
Texas         False
dtype: bool
  1. obj3
Ohio      35000
Oregon    16000
Texas     71000
Utah       5000
dtype: int64
  1. obj4
California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64
  1. obj3 + obj4#数据对齐
California         NaN
Ohio           70000.0
Oregon         32000.0
Texas         142000.0
Utah               NaN
dtype: float64
  1. obj4.name = 'population'#对象和索引的name属性
  2. obj4.index.name = 'state'
  3. obj4
state
California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
Name: population, dtype: float64
  1. obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']
  2. obj
Bob      4
Steve    7
Jeff    -5
Ryan     3
dtype: int64

DataFrame

表格型数据结构,含有一组有序的列。既有行索引也有列索引

  1. data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],
  2. 'year': [2000, 2001, 2002, 2001, 2002],
  3. 'pop': [1.5, 1.7, 3.6, 2.4, 2.9]}
  4. frame = DataFrame(data)
  1. frame
  1. DataFrame(data, columns=['year', 'state', 'pop'])#重新排列
  1. frame2 = DataFrame(data, columns=['year', 'state', 'pop', 'debt'],
  2. index=['one', 'two', 'three', 'four', 'five'])#缺失值
  3. frame2
  1. frame2.columns
Index([u'year', u'state', u'pop', u'debt'], dtype='object')
  1. frame2['state']#返回Series
one        Ohio
two        Ohio
three      Ohio
four     Nevada
five     Nevada
Name: state, dtype: object
  1. frame2.year#返回Series
one      2000
two      2001
three    2002
four     2001
five     2002
Name: year, dtype: int64
  1. frame2.ix['three']#索引行字段
year     2002
state    Ohio
pop       3.6
debt      NaN
Name: three, dtype: object
  1. frame2['debt'] = 16.5#修改列字段
  2. frame2
  1. frame2['debt'] = np.arange(5.)
  2. frame2
  1. val = Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])
  2. frame2['debt'] = val#匹配索引,空位置NA
  3. frame2
  1. frame2['eastern'] = frame2.state == 'Ohio'
  2. frame2
  1. del frame2['eastern']#删除列
  2. frame2.columns
Index([u'year', u'state', u'pop', u'debt'], dtype='object')
  1. pop = {'Nevada': {2001: 2.4, 2002: 2.9},
  2. 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}#嵌套字典,外层作为列,内层作为嵌套索引
  1. frame3 = DataFrame(pop)
  2. frame3
  1. frame3.T#转置
  1. frame3
  1. DataFrame(pop, index=[2001, 2002, 2003])
  1. pdata = {'Ohio': frame3['Ohio'][:-1],
  2. 'Nevada': frame3['Nevada'][:2]}#最后一行不要
  3. DataFrame(pdata)
  1. frame3.index.name = 'year'; frame3.columns.name = 'state'#index和columns的name属性
  2. frame3
  1. frame3.values#values属性
array([[ nan,  1.5],
       [ 2.4,  1.7],
       [ 2.9,  3.6]])
  1. frame2
  1. frame2.values
array([[2000L, 'Ohio', 1.5, nan],
       [2001L, 'Ohio', 1.7, -1.2],
       [2002L, 'Ohio', 3.6, nan],
       [2001L, 'Nevada', 2.4, -1.5],
       [2002L, 'Nevada', 2.9, -1.7]], dtype=object)

Index objects

  1. obj = Series(range(3), index=['a', 'b', 'c'])
  2. index = obj.index
  3. index
Index([u'a', u'b', u'c'], dtype='object')
  1. index[1:]
Index([u'b', u'c'], dtype='object')
  1. index[1] = 'd'#index对象不可以修改
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-65-c44a2554ac58> in <module>()
----> 1 index[1] = 'd'#index对象不可以修改


C:\Users\WangYixin\Anaconda2\lib\site-packages\pandas\indexes\base.pyc in __setitem__(self, key, value)
   1235 
   1236     def __setitem__(self, key, value):
-> 1237         raise TypeError("Index does not support mutable operations")
   1238 
   1239     def __getitem__(self, key):


TypeError: Index does not support mutable operations
  1. index = pd.Index(np.arange(3))
  2. obj2 = Series([1.5, -2.5, 0], index=index)
  3. obj2.index is index
True
  1. frame3
  1. 'Ohio' in frame3.columns
True
  1. 2003 in frame3.index
False

index的方法和属性
append:连接另一个index对象,产生新的index
diff:计算差集。并得到一个index
intersection:计算交集
union:计算并集
isin:计算一个指示各值是否都包含在参数集合中的布尔型数组
delete:删除索引i处的元素,并得到新的Index
drop:删除传入的值,并得到新的Index
insert:将元素插入索引i处,并得到新的Index
is_monotonic:各元素均大于等于前一个元素时,返回true
is_unique:当index没有重复值时,返回true
unique:计算Index中唯一值的数组

Essential functionality

Reindexing

  1. obj = Series([4.5, 7.2, -5.3, 3.6], index=['d', 'b', 'a', 'c'])
  2. obj
d    4.5
b    7.2
a   -5.3
c    3.6
dtype: float64
  1. obj2 = obj.reindex(['a', 'b', 'c', 'd', 'e'])
  2. obj2
a   -5.3
b    7.2
c    3.6
d    4.5
e    NaN
dtype: float64
  1. obj.reindex(['a', 'b', 'c', 'd', 'e'], fill_value=0)
a   -5.3
b    7.2
c    3.6
d    4.5
e    0.0
dtype: float64
  1. obj3 = Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])
  2. obj3.reindex(range(6), method='ffill')#前向值填充
0      blue
1      blue
2    purple
3    purple
4    yellow
5    yellow
dtype: object
  1. frame = DataFrame(np.arange(9).reshape((3, 3)), index=['a', 'c', 'd'],
  2. columns=['Ohio', 'Texas', 'California'])
  3. frame
  1. frame2 = frame.reindex(['a', 'b', 'c', 'd'])
  2. frame2
  1. states = ['Texas', 'Utah', 'California']
  2. frame.reindex(columns=states)#重新索引列
  1. frame.reindex(index=['a', 'b', 'c', 'd'], method='ffill',
  2. columns=states)#同时行列索引
  1. frame.ix[['a', 'b', 'c', 'd'], states]#重新索引

reindex的method选项
ffill/pad 前向填充或搬运值
bfill或backfill 后向填充或搬运值
reindex函数的参数
index 用作索引的新序列
method 填充方式
fill_value 重新索引的过程中,需要引入缺失值时使用的替代值
limit 前向或后向填充时最大的填充量
level MultiIndex指定级别上匹配简单索引,否则选取子集
copy 默认为true,复制。false,新旧相等就不复制

Dropping entries from an axis

  1. obj = Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])
  2. new_obj = obj.drop('c')
  3. new_obj
a    0.0
b    1.0
d    3.0
e    4.0
dtype: float64
  1. obj.drop(['d', 'c'])
a    0.0
b    1.0
e    4.0
dtype: float64
  1. data = DataFrame(np.arange(16).reshape((4, 4)),
  2. index=['Ohio', 'Colorado', 'Utah', 'New York'],
  3. columns=['one', 'two', 'three', 'four'])
  1. data.drop(['Colorado', 'Ohio'])
  1. data.drop('two', axis=1)
  1. data.drop(['two', 'four'], axis=1)

Indexing, selection, and filtering

  1. obj = Series(np.arange(4.), index=['a', 'b', 'c', 'd'])
  2. obj['b']
1.0
  1. obj[1]
1.0
  1. obj[2:4]
c    2.0
d    3.0
dtype: float64
  1. obj[['b', 'a', 'd']]
b    1.0
a    0.0
d    3.0
dtype: float64
  1. obj[[1, 3]]
b    1.0
d    3.0
dtype: float64
  1. obj[obj < 2]
a    0.0
b    1.0
dtype: float64
  1. obj['b':'c']#末端包含
b    1.0
c    2.0
dtype: float64
  1. obj['b':'c'] = 5
  2. obj
a    0.0
b    5.0
c    5.0
d    3.0
dtype: float64
  1. data = DataFrame(np.arange(16).reshape((4, 4)),
  2. index=['Ohio', 'Colorado', 'Utah', 'New York'],
  3. columns=['one', 'two', 'three', 'four'])
  4. data
  1. data['two']
Ohio         1
Colorado     5
Utah         9
New York    13
Name: two, dtype: int32
  1. data[['three', 'one']]
  1. data[:2]
  1. data[data['three'] > 5]
  1. data < 5
  1. data[data < 5] = 0
  1. data.ix['Colorado', ['two', 'three']]
two      5
three    6
Name: Colorado, dtype: int32
  1. data.ix[['Colorado', 'Utah'], [3, 0, 1]]
  1. data.ix[2]
one       8
two       9
three    10
four     11
Name: Utah, dtype: int32
  1. data.ix[:'Utah', 'two']
Ohio        0
Colorado    5
Utah        9
Name: two, dtype: int32
  1. data.ix[data.three > 5, :3]

DataFrame的索引选项
obj[val] 选取单列或者一组列
obj.ix[val] 选取单行或者一组行
obj.ix[:,val] 选取单列或者列子集
obj.ix[val1, val2] 同时选取行和列
reindex方法 将一个或多个轴匹配到新索引
xs方法 根据标签选取单行或单列,返回Series
icol、irow方法 根据整数位置选取单行或单列,并返回一个Series
get_value、set_value方法 根据航标签和列表前选取单个值

Arithmetic and data alignment

  1. s1 = Series([7.3, -2.5, 3.4, 1.5], index=['a', 'c', 'd', 'e'])
  2. s2 = Series([-2.1, 3.6, -1.5, 4, 3.1], index=['a', 'c', 'e', 'f', 'g'])
  1. s1
a    7.3
c   -2.5
d    3.4
e    1.5
dtype: float64
  1. s2
a   -2.1
c    3.6
e   -1.5
f    4.0
g    3.1
dtype: float64
  1. s1 + s2
a    5.2
c    1.1
d    NaN
e    0.0
f    NaN
g    NaN
dtype: float64
  1. df1 = DataFrame(np.arange(9.).reshape((3, 3)), columns=list('bcd'),
  2. index=['Ohio', 'Texas', 'Colorado'])
  3. df2 = DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),
  4. index=['Utah', 'Ohio', 'Texas', 'Oregon'])
  5. df1
  1. df2
  1. df1 + df2

Arithmetic methods with fill values

  1. df1 = DataFrame(np.arange(12.).reshape((3, 4)), columns=list('abcd'))
  2. df2 = DataFrame(np.arange(20.).reshape((4, 5)), columns=list('abcde'))
  3. df1
  1. df2
  1. df1 + df2
  1. df1.add(df2, fill_value=0)
  1. df1.reindex(columns=df2.columns, fill_value=0)

Operations between DataFrame and Series

  1. arr = np.arange(12.).reshape((3, 4))
  2. arr
array([[  0.,   1.,   2.,   3.],
       [  4.,   5.,   6.,   7.],
       [  8.,   9.,  10.,  11.]])
  1. arr[0]
array([ 0.,  1.,  2.,  3.])
  1. arr - arr[0]
array([[ 0.,  0.,  0.,  0.],
       [ 4.,  4.,  4.,  4.],
       [ 8.,  8.,  8.,  8.]])
  1. frame = DataFrame(np.arange(12.).reshape((4, 3)), columns=list('bde'),
  2. index=['Utah', 'Ohio', 'Texas', 'Oregon'])
  3. series = frame.ix[0]#第一行
  4. frame
  1. series
b    0.0
d    1.0
e    2.0
Name: Utah, dtype: float64
  1. frame - series
  1. series2 = Series(range(3), index=['b', 'e', 'f'])
  2. frame + series2
  1. series3 = frame['d']
  2. frame
  1. series3
Utah       1.0
Ohio       4.0
Texas      7.0
Oregon    10.0
Name: d, dtype: float64
  1. frame.sub(series3, axis=0)#匹配行索引进行广播

Function application and mapping

  1. frame = DataFrame(np.random.randn(4, 3), columns=list('bde'),
  2. index=['Utah', 'Ohio', 'Texas', 'Oregon'])
  1. frame
  1. np.abs(frame)
  1. f = lambda x: x.max() - x.min()
  1. frame.apply(f)#默认是按照列
b    1.802165
d    1.684034
e    2.689627
dtype: float64
  1. frame.apply(f, axis=1)#按照行
Utah      0.998382
Ohio      2.521511
Texas     0.676115
Oregon    2.542656
dtype: float64
  1. def f(x):
  2. return Series([x.min(), x.max()], index=['min', 'max'])
  3. frame.apply(f)
  1. format = lambda x: '%.2f' % x
  2. frame.applymap(format)#dataFrame
  1. frame['e'].map(format)#Series
Utah      -0.52
Ohio       1.39
Texas      0.77
Oregon    -1.30
Name: e, dtype: object

Sorting and ranking

  1. obj = Series(range(4), index=['d', 'a', 'b', 'c'])
  2. obj.sort_index()
a    1
b    2
c    3
d    0
dtype: int64
  1. frame = DataFrame(np.arange(8).reshape((2, 4)), index=['three', 'one'],
  2. columns=['d', 'a', 'b', 'c'])
  3. frame.sort_index()
  1. frame.sort_index(axis=1)#columns排序
  1. frame.sort_index(axis=1, ascending=False)#降序
  1. obj = Series([4, 7, -3, 2])
  2. obj.order()
C:\Users\WangYixin\Anaconda2\lib\site-packages\ipykernel\__main__.py:2: FutureWarning: order is deprecated, use sort_values(...)
  from ipykernel import kernelapp as app





2   -3
3    2
0    4
1    7
dtype: int64
  1. obj = Series([4, np.nan, 7, np.nan, -3, 2])#Series排序,缺失值放到末尾
  2. obj.order()
C:\Users\WangYixin\Anaconda2\lib\site-packages\ipykernel\__main__.py:2: FutureWarning: order is deprecated, use sort_values(...)
  from ipykernel import kernelapp as app





4   -3.0
5    2.0
0    4.0
2    7.0
1    NaN
3    NaN
dtype: float64
  1. frame = DataFrame({'b': [4, 7, -3, 2], 'a': [0, 1, 0, 1]})
  2. frame
  1. frame.sort_index(by='b')#根据某一列的值进行排序
C:\Users\WangYixin\Anaconda2\lib\site-packages\ipykernel\__main__.py:1: FutureWarning: by argument to sort_index is deprecated, pls use .sort_values(by=...)
  if __name__ == '__main__':
  1. frame.sort_index(by=['a', 'b'])#根据多个列进行排序
C:\Users\WangYixin\Anaconda2\lib\site-packages\ipykernel\__main__.py:1: FutureWarning: by argument to sort_index is deprecated, pls use .sort_values(by=...)
  if __name__ == '__main__':
  1. obj = Series([7, -5, 7, 4, 2, 0, 4])
  2. obj.rank()#为各组分配一个平均排名
0    6.5
1    1.0
2    6.5
3    4.5
4    3.0
5    2.0
6    4.5
dtype: float64
  1. obj.rank(method='first')#根据值在元数据中出现的顺序
0    6.0
1    1.0
2    7.0
3    4.0
4    3.0
5    2.0
6    5.0
dtype: float64
  1. obj.rank(ascending=False, method='max')#降序
0    2.0
1    7.0
2    2.0
3    4.0
4    5.0
5    6.0
6    4.0
dtype: float64
  1. frame = DataFrame({'b': [4.3, 7, -3, 2], 'a': [0, 1, 0, 1],
  2. 'c': [-2, 5, 8, -2.5]})
  3. frame
  1. frame.rank(axis=1)

排序时破坏平级关系的method选项
average 为各个值分配平均排名
min 使用整个分组的最小排名
max 使用整个分组的最大排名
first 按值在原始数据中出现顺序分配排名

Axis indexes with duplicate values

  1. obj = Series(range(5), index=['a', 'a', 'b', 'b', 'c'])#重复索引
  2. obj
a    0
a    1
b    2
b    3
c    4
dtype: int64
  1. obj.index.is_unique
False
  1. obj['a']
a    0
a    1
dtype: int64
  1. obj['c']
4
  1. df = DataFrame(np.random.randn(4, 3), index=['a', 'a', 'b', 'b'])
  2. df
  1. df.ix['b']

Summarizing and computing descriptive statistics

  1. df = DataFrame([[1.4, np.nan], [7.1, -4.5],
  2. [np.nan, np.nan], [0.75, -1.3]],
  3. index=['a', 'b', 'c', 'd'],
  4. columns=['one', 'two'])
  5. df
  1. df.sum()#NA自动排除
one    9.25
two   -5.80
dtype: float64
  1. df.sum(axis=1)
a    1.40
b    2.60
c    0.00
d   -0.55
dtype: float64
  1. df.mean(axis=1, skipna=False)
a      NaN
b    1.300
c      NaN
d   -0.275
dtype: float64
  1. df.idxmax()#每列最大值的id
one    b
two    d
dtype: object
  1. df.cumsum()#按列累积求和
  1. df.describe()
  1. obj = Series(['a', 'a', 'b', 'c'] * 4)
  2. obj.describe()
count     16
unique     3
top        a
freq       8
dtype: object
  1. obj
0     a
1     a
2     b
3     c
4     a
5     a
6     b
7     c
8     a
9     a
10    b
11    c
12    a
13    a
14    b
15    c
dtype: object

描述和汇总统计
count 非NA值的数量
describe 针对Series或个DataFrame列计算汇总统计
min、max 计算最小值和最大值
argmin、argmax 计算最大值和最小值的索引位置
idxmin、idxmax 计算最小值和最大值的索引值
quantile 计算样本的分位数(0-1)
sum 值的总和
mean 值得平均数
median 算数中位数
mad 平均绝对离差
var 方差
std 标准差
skew 样本值的偏度
kurt 峰度
cumsum 累计和
cummin\cummax 累计最大值和累计最小值
cumprod 累计积
diff 一阶差分
pct_change 百分数变化

Correlation and covariance

  1. import pandas.io.data as web
  2. all_data = {}
  3. for ticker in ['AAPL', 'IBM', 'MSFT', 'GOOG']:
  4. all_data[ticker] = web.get_data_yahoo(ticker)
  5. price = DataFrame({tic: data['Adj Close']
  6. for tic, data in all_data.iteritems()})
  7. volume = DataFrame({tic: data['Volume']
  8. for tic, data in all_data.iteritems()})
C:\Users\WangYixin\Anaconda2\lib\site-packages\pandas\io\data.py:35: FutureWarning: 
The pandas.io.data module is moved to a separate package (pandas-datareader) and will be removed from pandas in a future version.
After installing the pandas-datareader package (https://github.com/pydata/pandas-datareader), you can change the import ``from pandas.io import data, wb`` to ``from pandas_datareader import data, wb``.
  FutureWarning)
  1. returns = price.pct_change()#价格的百分数变化
  2. returns.tail()
  1. returns.MSFT.corr(returns.IBM)#计算两个Series的相关系数
0.49616105806910621
  1. returns.MSFT.cov(returns.IBM)#计算协方差
8.7745727843692117e-05
  1. returns.corr()#DataFrame的相关系数矩阵
  1. returns.cov()#DataFrame的协方差矩阵
  1. returns.corrwith(returns.IBM)#计算列或行与另一个SeriesDataFrame之间的相关系数
AAPL    0.383470
GOOG    0.401322
IBM     1.000000
MSFT    0.496161
dtype: float64
  1. returns.corrwith(volume)
AAPL   -0.073558
GOOG   -0.007108
IBM    -0.202749
MSFT   -0.092586
dtype: float64

Unique values, value counts, and membership

  1. obj = Series(['c', 'a', 'd', 'a', 'a', 'b', 'b', 'c', 'c'])
  1. uniques = obj.unique()#唯一值数组
  2. uniques
array(['c', 'a', 'd', 'b'], dtype=object)
  1. obj.value_counts()#各值出现的频率
c    3
a    3
b    2
d    1
dtype: int64
  1. pd.value_counts(obj.values, sort=False)#降序排列
a    3
c    3
b    2
d    1
dtype: int64
  1. mask = obj.isin(['b', 'c'])
  2. mask
0     True
1    False
2    False
3    False
4    False
5     True
6     True
7     True
8     True
dtype: bool
  1. obj[mask]
0    c
5    b
6    b
7    c
8    c
dtype: object
  1. data = DataFrame({'Qu1': [1, 3, 4, 3, 4],
  2. 'Qu2': [2, 3, 1, 2, 3],
  3. 'Qu3': [1, 5, 2, 4, 4]})
  4. data
  1. result = data.apply(pd.value_counts).fillna(0)
  2. result

isin 计算一个表示Series各值是否包含于传入的值序列中的布尔型数组
unique 计算Series中的唯一值数组,按发现的顺序返回
value_counts 返回一个Series,其索引为唯一值,值为频率

Handling missing data

  1. string_data = Series(['aardvark', 'artichoke', np.nan, 'avocado'])
  2. string_data
0     aardvark
1    artichoke
2          NaN
3      avocado
dtype: object
  1. string_data.isnull()
0    False
1    False
2     True
3    False
dtype: bool
  1. string_data[0] = None
  2. string_data.isnull()
0     True
1    False
2     True
3    False
dtype: bool

NA处理方法
dropna 根据各标签的值中是否存在缺失数据对轴标签进行过滤
fillna 用指定值或者插值的办法填充缺失数据
isnull 返回一个含有布尔值的对象,指示哪些值是缺失值
notnull isnull的否定形式

Filtering out missing data

  1. from numpy import nan as NA
  2. data = Series([1, NA, 3.5, NA, 7])
  3. data.dropna()
0    1.0
2    3.5
4    7.0
dtype: float64
  1. data[data.notnull()]
0    1.0
2    3.5
4    7.0
dtype: float64
  1. data = DataFrame([[1., 6.5, 3.], [1., NA, NA],
  2. [NA, NA, NA], [NA, 6.5, 3.]])
  3. cleaned = data.dropna()
  4. data
  1. cleaned
  1. data.dropna(how='all')
  1. data[4] = NA
  2. data
  1. data.dropna(axis=1, how='all')
  1. df = DataFrame(np.random.randn(7, 3))
  2. df.ix[:4, 1] = NA; df.ix[:2, 2] = NA# 注意行是包含的
  3. df
  1. df.dropna(thresh=2)#每一行至少两个不为缺失值

Filling in missing data

  1. df.fillna(0)
  1. df.fillna({1: 0.5, 2: -1})#不同的列填充不同的值
  1. # always returns a reference to the filled object
  2. _ = df.fillna(0, inplace=True)
  3. df
  1. df = DataFrame(np.random.randn(6, 3))
  2. df.ix[2:, 1] = NA; df.ix[4:, 2] = NA
  3. df
  1. df.fillna(method='ffill')
  1. df.fillna(method='ffill', limit=2)#最多两个
  1. data = Series([1., NA, 3.5, NA, 7])
  2. data.fillna(data.mean())
0    1.000000
1    3.833333
2    3.500000
3    3.833333
4    7.000000
dtype: float64

fillna函数的参数
value 用于填充缺失值的标量值或者字典对象
method 插值方式
axis 待填充的轴
inplace 修改调用者对象而不产生副本
limit 可以连续填充的最大数量

Hierarchical indexing/层次化索引

在一个轴上拥有多个索引级别,低维度处理高维度数据

  1. data = Series(np.random.randn(10),
  2. index=[['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'd', 'd'],
  3. [1, 2, 3, 1, 2, 3, 1, 2, 2, 3]])
  4. data
a  1    0.029610
   2    0.795253
   3    0.118110
b  1   -0.748532
   2    0.584970
   3    0.152677
c  1   -1.565657
   2   -0.562540
d  2   -0.032664
   3   -0.929006
dtype: float64
  1. data.index
MultiIndex(levels=[[u'a', u'b', u'c', u'd'], [1, 2, 3]],
           labels=[[0, 0, 0, 1, 1, 1, 2, 2, 3, 3], [0, 1, 2, 0, 1, 2, 0, 1, 1, 2]])
  1. data['b']
1   -0.748532
2    0.584970
3    0.152677
dtype: float64
  1. data['b':'c']
b  1   -0.748532
   2    0.584970
   3    0.152677
c  1   -1.565657
   2   -0.562540
dtype: float64
  1. data.ix[['b', 'd']]
b  1   -0.748532
   2    0.584970
   3    0.152677
d  2   -0.032664
   3   -0.929006
dtype: float64
  1. data[:, 2]#内层索引
a    0.795253
b    0.584970
c   -0.562540
d   -0.032664
dtype: float64
  1. data.unstack()#组成DataFrame
  1. data.unstack().stack()
a  1    0.029610
   2    0.795253
   3    0.118110
b  1   -0.748532
   2    0.584970
   3    0.152677
c  1   -1.565657
   2   -0.562540
d  2   -0.032664
   3   -0.929006
dtype: float64
  1. frame = DataFrame(np.arange(12).reshape((4, 3)),
  2. index=[['a', 'a', 'b', 'b'], [1, 2, 1, 2]],
  3. columns=[['Ohio', 'Ohio', 'Colorado'],
  4. ['Green', 'Red', 'Green']])
  5. frame
  1. frame.index.names = ['key1', 'key2']
  2. frame.columns.names = ['state', 'color']
  3. frame
  1. frame['Ohio']

MultiIndex.from_arrays([['Ohio', 'Ohio', 'Colorado'], ['Green', 'Red', 'Green']],
names=['state', 'color'])#可以单独创建MultiIndex

Reordering and sorting levels

  1. frame.swaplevel('key1', 'key2')
  1. frame
  1. frame.sortlevel(1)
  1. frame.swaplevel(0, 1).sortlevel(0)

Summary statistics by level

  1. frame.sum(level='key2')
  1. frame.sum(level='color', axis=1)

Using a DataFrame's columns

  1. frame = DataFrame({'a': range(7), 'b': range(7, 0, -1),
  2. 'c': ['one', 'one', 'one', 'two', 'two', 'two', 'two'],
  3. 'd': [0, 1, 2, 0, 1, 2, 3]})
  4. frame
  1. frame2 = frame.set_index(['c', 'd'])#将列转换为行索引
  2. frame2
  1. frame.set_index(['c', 'd'], drop=False)#不去掉索引列
  1. frame2.reset_index()#层次化索引回到列里面

Other pandas topics

Integer indexing/整数索引

  1. ser = Series(np.arange(3.))
  2. ser.iloc[-1]
2.0
  1. ser
0    0.0
1    1.0
2    2.0
dtype: float64
  1. ser2 = Series(np.arange(3.), index=['a', 'b', 'c'])#非整数索引,比较好
  2. ser2[-1]
2.0
  1. ser.ix[:1]#这里为索引,所以取到1
0    0.0
1    1.0
dtype: float64
  1. ser3 = Series(range(3), index=[-5, 1, 3])
  2. ser3.iloc[2]
2
  1. frame = DataFrame(np.arange(6).reshape((3, 2)), index=[2, 0, 1])
  2. frame.iloc[1]
0    2
1    3
Name: 0, dtype: int32

Panel data

三维版的DataFrame

  1. import pandas.io.data as web
  2. pdata = pd.Panel(dict((stk, web.get_data_yahoo(stk))
  3. for stk in ['AAPL', 'GOOG', 'MSFT', 'DELL']))
  1. pdata
<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 1758 (major_axis) x 6 (minor_axis)
Items axis: AAPL to MSFT
Major_axis axis: 2010-01-04 00:00:00 to 2016-11-25 00:00:00
Minor_axis axis: Open to Adj Close
  1. pdata = pdata.swapaxes('items', 'minor')
  2. pdata['Adj Close']
  1. pdata.ix[:, '6/1/2012', :]
  1. stacked.to_panel()#to_frame的逆运算
<class 'pandas.core.panel.Panel'>
Dimensions: 6 (items) x 1145 (major_axis) x 4 (minor_axis)
Items axis: Open to Adj Close
Major_axis axis: 2012-05-30 00:00:00 to 2016-11-25 00:00:00
Minor_axis axis: AAPL to MSFT
添加新批注
在作者公开此批注前,只有你和作者可见。
回复批注