Essential Basic Functionality

Here we discuss a lot of the essential functionality common to the pandas data structures. Here’s how to create some of the objects used in the examples from the previous section:

In [1]: index = pd.date_range('1/1/2000', periods=8)
In [2]: s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
In [3]: df = pd.DataFrame(np.random.randn(8, 3), index=index,
   ...:                   columns=['A', 'B', 'C'])
   ...: 
In [4]: wp = pd.Panel(np.random.randn(2, 5, 4), items=['Item1', 'Item2'],
   ...:               major_axis=pd.date_range('1/1/2000', periods=5),
   ...:               minor_axis=['A', 'B', 'C', 'D'])
   ...:

Head and Tail

To view a small sample of a Series or DataFrame object, use the head() and tail() methods. The default number of elements to display is five, but you may pass a custom number.

In [5]: long_series = pd.Series(np.random.randn(1000))
In [6]: long_series.head()
Out[6]: 
0   -2.211372
1    0.974466
2   -2.006747
3   -0.410001
4   -0.078638
dtype: float64
In [7]: long_series.tail(3)
Out[7]: 
997   -0.196166
998    0.380733
999   -0.275874
dtype: float64

Attributes and Underlying Data

pandas objects have a number of attributes enabling you to access the metadata

• shape: gives the axis dimensions of the object, consistent with ndarray
• Axis labels
  • Series: index (only axis)
  • DataFrame: index (rows) and columns
  • Panel: items, major_axis, and minor_axis

Note, these attributes can be safely assigned to!

In [8]: df[:2]
Out[8]: 
                   A         B         C
2000-01-01 -0.173215  0.119209 -1.044236
2000-01-02 -0.861849 -2.104569 -0.494929
In [9]: df.columns = [x.lower() for x in df.columns]
In [10]: df
Out[10]: 
                   a         b         c
2000-01-01 -0.173215  0.119209 -1.044236
2000-01-02 -0.861849 -2.104569 -0.494929
2000-01-03  1.071804  0.721555 -0.706771
2000-01-04 -1.039575  0.271860 -0.424972
2000-01-05  0.567020  0.276232 -1.087401
2000-01-06 -0.673690  0.113648 -1.478427
2000-01-07  0.524988  0.404705  0.577046
2000-01-08 -1.715002 -1.039268 -0.370647

Pandas objects (Index, Series, DataFrame) can be thought of as containers for arrays, which hold the actual data and do the actual computation. For many types, the underlying array is a numpy.ndarray. However, pandas and 3rd party libraries may extend NumPy’s type system to add support for custom arrays (see dtypes).

To get the actual data inside a Index or Series, use the .array property

In [11]: s.array
Out[11]: 
<PandasArray>
[ 0.46911229990718628, -0.28286334432866328,  -1.5090585031735124,
  -1.1356323710171934,   1.2121120250208506]
Length: 5, dtype: float64
In [12]: s.index.array
Out[12]: 
<PandasArray>
['a', 'b', 'c', 'd', 'e']
Length: 5, dtype: object

array will always be an ExtensionArray. The exact details of what an ExtensionArray is and why pandas uses them is a bit beyond the scope of this introduction. See dtypes for more.

If you know you need a NumPy array, use to_numpy() or numpy.asarray().

In [13]: s.to_numpy()
Out[13]: array([ 0.4691, -0.2829, -1.5091, -1.1356,  1.2121])
In [14]: np.asarray(s)
Out[14]: array([ 0.4691, -0.2829, -1.5091, -1.1356,  1.2121])

When the Series or Index is backed by an ExtensionArray, to_numpy() may involve copying data and coercing values. See dtypes for more.

to_numpy() gives some control over the dtype of the resulting numpy.ndarray. For example, consider datetimes with timezones. NumPy doesn’t have a dtype to represent timezone-aware datetimes, so there are two possibly useful representations:

1. An object-dtype numpy.ndarray with Timestamp objects, each with the correct tz
2. A datetime64[ns] -dtype numpy.ndarray, where the values have been converted to UTC and the timezone discarded

Timezones may be preserved with dtype=object

In [15]: ser = pd.Series(pd.date_range('2000', periods=2, tz="CET"))
In [16]: ser.to_numpy(dtype=object)
Out[16]: 
array([Timestamp('2000-01-01 00:00:00+0100', tz='CET', freq='D'),
       Timestamp('2000-01-02 00:00:00+0100', tz='CET', freq='D')], dtype=object)

Or thrown away with dtype='datetime64[ns]'

In [17]: ser.to_numpy(dtype="datetime64[ns]")
Out[17]: array(['1999-12-31T23:00:00.000000000', '2000-01-01T23:00:00.000000000'], dtype='datetime64[ns]')

Getting the “raw data” inside a DataFrame is possibly a bit more complex. When your DataFrame only has a single data type for all the columns, DataFrame.to_numpy() will return the underlying data:

In [18]: df.to_numpy()
Out[18]: 
array([[-0.1732,  0.1192, -1.0442],
       [-0.8618, -2.1046, -0.4949],
       [ 1.0718,  0.7216, -0.7068],
       [-1.0396,  0.2719, -0.425 ],
       [ 0.567 ,  0.2762, -1.0874],
       [-0.6737,  0.1136, -1.4784],
       [ 0.525 ,  0.4047,  0.577 ],
       [-1.715 , -1.0393, -0.3706]])

If a DataFrame or Panel contains homogeneously-typed data, the ndarray can actually be modified in-place, and the changes will be reflected in the data structure. For heterogeneous data (e.g. some of the DataFrame’s columns are not all the same dtype), this will not be the case. The values attribute itself, unlike the axis labels, cannot be assigned to.

::: tip Note When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all of the data involved. For example, if strings are involved, the result will be of object dtype. If there are only floats and integers, the resulting array will be of float dtype. :::

In the past, pandas recommended Series.values or DataFrame.values for extracting the data from a Series or DataFrame. You’ll still find references to these in old code bases and online. Going forward, we recommend avoiding .values and using .array or .to_numpy(). .values has the following drawbacks:

1. When your Series contains an extension type, it’s unclear whether Series.values returns a NumPy array or the extension array. Series.array will always return an ExtensionArray, and will never copy data. Series.to_numpy() will always return a NumPy array, potentially at the cost of copying / coercing values.
2. When your DataFrame contains a mixture of data types, DataFrame.values may involve copying data and coercing values to a common dtype, a relatively expensive operation. DataFrame.to_numpy(), being a method, makes it clearer that the returned NumPy array may not be a view on the same data in the DataFrame.

Accelerated operations

pandas has support for accelerating certain types of binary numerical and boolean operations using the numexpr library and the bottleneck libraries.

These libraries are especially useful when dealing with large data sets, and provide large speedups. numexpr uses smart chunking, caching, and multiple cores. bottleneck is a set of specialized cython routines that are especially fast when dealing with arrays that have nans.

Here is a sample (using 100 column x 100,000 row DataFrames):

You are highly encouraged to install both libraries. See the section Recommended Dependencies for more installation info.

These are both enabled to be used by default, you can control this by setting the options:

New in version 0.20.0.

pd.set_option('compute.use_bottleneck', False)
pd.set_option('compute.use_numexpr', False)

Flexible binary operations

With binary operations between pandas data structures, there are two key points of interest:

• Broadcasting behavior between higher- (e.g. DataFrame) and lower-dimensional (e.g. Series) objects.
• Missing data in computations.

We will demonstrate how to manage these issues independently, though they can be handled simultaneously.

Matching / broadcasting behavior

DataFrame has the methods add(), sub(), mul(), div() and related functions radd(), rsub(), … for carrying out binary operations. For broadcasting behavior, Series input is of primary interest. Using these functions, you can use to either match on the index or columns via the axis keyword:

In [19]: df = pd.DataFrame({
   ....:     'one': pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
   ....:     'two': pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
   ....:     'three': pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
   ....: 
In [20]: df
Out[20]: 
        one       two     three
a  1.400810 -1.643041       NaN
b -0.356470  1.045911  0.395023
c  0.797268  0.924515 -0.007090
d       NaN  1.553693 -1.670830
In [21]: row = df.iloc[1]
In [22]: column = df['two']
In [23]: df.sub(row, axis='columns')
Out[23]: 
        one       two     three
a  1.757280 -2.688953       NaN
b  0.000000  0.000000  0.000000
c  1.153738 -0.121396 -0.402113
d       NaN  0.507782 -2.065853
In [24]: df.sub(row, axis=1)
Out[24]: 
        one       two     three
a  1.757280 -2.688953       NaN
b  0.000000  0.000000  0.000000
c  1.153738 -0.121396 -0.402113
d       NaN  0.507782 -2.065853
In [25]: df.sub(column, axis='index')
Out[25]: 
        one  two     three
a  3.043851  0.0       NaN
b -1.402381  0.0 -0.650888
c -0.127247  0.0 -0.931605
d       NaN  0.0 -3.224524
In [26]: df.sub(column, axis=0)
Out[26]: 
        one  two     three
a  3.043851  0.0       NaN
b -1.402381  0.0 -0.650888
c -0.127247  0.0 -0.931605
d       NaN  0.0 -3.224524

Furthermore you can align a level of a MultiIndexed DataFrame with a Series.

In [27]: dfmi = df.copy()
In [28]: dfmi.index = pd.MultiIndex.from_tuples([(1, 'a'), (1, 'b'),
   ....:                                         (1, 'c'), (2, 'a')],
   ....:                                        names=['first', 'second'])
   ....: 
In [29]: dfmi.sub(column, axis=0, level='second')
Out[29]: 
                   one       two     three
first second                              
1     a       3.043851  0.000000       NaN
      b      -1.402381  0.000000 -0.650888
      c      -0.127247  0.000000 -0.931605
2     a            NaN  3.196734 -0.027789

With Panel, describing the matching behavior is a bit more difficult, so the arithmetic methods instead (and perhaps confusingly?) give you the option to specify the broadcast axis. For example, suppose we wished to demean the data over a particular axis. This can be accomplished by taking the mean over an axis and broadcasting over the same axis:

In [30]: major_mean = wp.mean(axis='major')
In [31]: major_mean
Out[31]: 
      Item1     Item2
A -0.378069  0.675929
B -0.241429 -0.018080
C -0.597702  0.129006
D  0.204005  0.245570
In [32]: wp.sub(major_mean, axis='major')
Out[32]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 2000-01-01 00:00:00 to 2000-01-05 00:00:00
Minor_axis axis: A to D

And similarly for axis="items" and axis="minor".

::: tip Note I could be convinced to make the axis argument in the DataFrame methods match the broadcasting behavior of Panel. Though it would require a transition period so users can change their code… :::

Series and Index also support the divmod() builtin. This function takes the floor division and modulo operation at the same time returning a two-tuple of the same type as the left hand side. For example:

In [33]: s = pd.Series(np.arange(10))
In [34]: s
Out[34]: 
0    0
1    1
2    2
3    3
4    4
5    5
6    6
7    7
8    8
9    9
dtype: int64
In [35]: div, rem = divmod(s, 3)
In [36]: div
Out[36]: 
0    0
1    0
2    0
3    1
4    1
5    1
6    2
7    2
8    2
9    3
dtype: int64
In [37]: rem
Out[37]: 
0    0
1    1
2    2
3    0
4    1
5    2
6    0
7    1
8    2
9    0
dtype: int64
In [38]: idx = pd.Index(np.arange(10))
In [39]: idx
Out[39]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')
In [40]: div, rem = divmod(idx, 3)
In [41]: div
Out[41]: Int64Index([0, 0, 0, 1, 1, 1, 2, 2, 2, 3], dtype='int64')
In [42]: rem
Out[42]: Int64Index([0, 1, 2, 0, 1, 2, 0, 1, 2, 0], dtype='int64')
We can also do elementwise divmod():
In [43]: div, rem = divmod(s, [2, 2, 3, 3, 4, 4, 5, 5, 6, 6])
In [44]: div
Out[44]: 
0    0
1    0
2    0
3    1
4    1
5    1
6    1
7    1
8    1
9    1
dtype: int64
In [45]: rem
Out[45]: 
0    0
1    1
2    2
3    0
4    0
5    1
6    1
7    2
8    2
9    3
dtype: int64

Missing data / operations with fill values

In Series and DataFrame, the arithmetic functions have the option of inputting a fill_value, namely a value to substitute when at most one of the values at a location are missing. For example, when adding two DataFrame objects, you may wish to treat NaN as 0 unless both DataFrames are missing that value, in which case the result will be NaN (you can later replace NaN with some other value using fillna if you wish).

In [46]: df
Out[46]: 
        one       two     three
a  1.400810 -1.643041       NaN
b -0.356470  1.045911  0.395023
c  0.797268  0.924515 -0.007090
d       NaN  1.553693 -1.670830
In [47]: df2
Out[47]: 
        one       two     three
a  1.400810 -1.643041  1.000000
b -0.356470  1.045911  0.395023
c  0.797268  0.924515 -0.007090
d       NaN  1.553693 -1.670830
In [48]: df + df2
Out[48]: 
        one       two     three
a  2.801620 -3.286083       NaN
b -0.712940  2.091822  0.790046
c  1.594536  1.849030 -0.014180
d       NaN  3.107386 -3.341661
In [49]: df.add(df2, fill_value=0)
Out[49]: 
        one       two     three
a  2.801620 -3.286083  1.000000
b -0.712940  2.091822  0.790046
c  1.594536  1.849030 -0.014180
d       NaN  3.107386 -3.341661

Flexible Comparisons

Series and DataFrame have the binary comparison methods eq, ne, lt, gt, le, and ge whose behavior is analogous to the binary arithmetic operations described above:

In [50]: df.gt(df2)
Out[50]: 
     one    two  three
a  False  False  False
b  False  False  False
c  False  False  False
d  False  False  False
In [51]: df2.ne(df)
Out[51]: 
     one    two  three
a  False  False   True
b  False  False  False
c  False  False  False
d   True  False  False

These operations produce a pandas object of the same type as the left-hand-side input that is of dtype bool. These boolean objects can be used in indexing operations, see the section on Boolean indexing.

Boolean Reductions

You can apply the reductions: empty, any(), all(), and bool() to provide a way to summarize a boolean result.

In [52]: (df > 0).all()
Out[52]: 
one      False
two      False
three    False
dtype: bool
In [53]: (df > 0).any()
Out[53]: 
one      True
two      True
three    True
dtype: bool

You can reduce to a final boolean value.

In [54]: (df > 0).any().any()
Out[54]: True

You can test if a pandas object is empty, via the empty property.

In [55]: df.empty
Out[55]: False
In [56]: pd.DataFrame(columns=list('ABC')).empty
Out[56]: True

To evaluate single-element pandas objects in a boolean context, use the method bool():

In [57]: pd.Series([True]).bool()
Out[57]: True
In [58]: pd.Series([False]).bool()
Out[58]: False
In [59]: pd.DataFrame([[True]]).bool()
Out[59]: True
In [60]: pd.DataFrame([[False]]).bool()
Out[60]: False

::: Warning Warning

You might be tempted to do the following:

>>> if df:
...     pass

Or

>>> df and df2

These will both raise errors, as you are trying to compare multiple values.:

ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().

:::

See gotchas for a more detailed discussion.

Comparing if objects are equivalent

Often you may find that there is more than one way to compute the same result. As a simple example, consider df + df and df * 2. To test that these two computations produce the same result, given the tools shown above, you might imagine using (df + df == df * 2).all(). But in fact, this expression is False:

In [61]: df + df == df * 2
Out[61]: 
     one   two  three
a   True  True  False
b   True  True   True
c   True  True   True
d  False  True   True
In [62]: (df + df == df * 2).all()
Out[62]: 
one      False
two       True
three    False
dtype: bool

Notice that the boolean DataFrame df + df == df * 2 contains some False values! This is because NaNs do not compare as equals:

In [63]: np.nan == np.nan
Out[63]: False

So, NDFrames (such as Series, DataFrames, and Panels) have an equals() method for testing equality, with NaNs in corresponding locations treated as equal.

In [64]: (df + df).equals(df * 2)
Out[64]: True

Note that the Series or DataFrame index needs to be in the same order for equality to be True:

In [65]: df1 = pd.DataFrame({'col': ['foo', 0, np.nan]})
In [66]: df2 = pd.DataFrame({'col': [np.nan, 0, 'foo']}, index=[2, 1, 0])
In [67]: df1.equals(df2)
Out[67]: False
In [68]: df1.equals(df2.sort_index())
Out[68]: True

Comparing array-like objects

You can conveniently perform element-wise comparisons when comparing a pandas data structure with a scalar value:

In [69]: pd.Series(['foo', 'bar', 'baz']) == 'foo'
Out[69]: 
0     True
1    False
2    False
dtype: bool
In [70]: pd.Index(['foo', 'bar', 'baz']) == 'foo'
Out[70]: array([ True, False, False], dtype=bool)

Pandas also handles element-wise comparisons between different array-like objects of the same length:

In [71]: pd.Series(['foo', 'bar', 'baz']) == pd.Index(['foo', 'bar', 'qux'])
Out[71]: 
0     True
1     True
2    False
dtype: bool
In [72]: pd.Series(['foo', 'bar', 'baz']) == np.array(['foo', 'bar', 'qux'])
Out[72]: 
0     True
1     True
2    False
dtype: bool

Trying to compare Index or Series objects of different lengths will raise a ValueError:

In [55]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo', 'bar'])
ValueError: Series lengths must match to compare
In [56]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo'])
ValueError: Series lengths must match to compare

Note that this is different from the NumPy behavior where a comparison can be broadcast:

In [73]: np.array([1, 2, 3]) == np.array([2])
Out[73]: array([False,  True, False], dtype=bool)

or it can return False if broadcasting can not be done:

In [74]: np.array([1, 2, 3]) == np.array([1, 2])
Out[74]: False

Combining overlapping data sets

A problem occasionally arising is the combination of two similar data sets where values in one are preferred over the other. An example would be two data series representing a particular economic indicator where one is considered to be of “higher quality”. However, the lower quality series might extend further back in history or have more complete data coverage. As such, we would like to combine two DataFrame objects where missing values in one DataFrame are conditionally filled with like-labeled values from the other DataFrame. The function implementing this operation is combine_first(), which we illustrate:

In [75]: df1 = pd.DataFrame({'A': [1., np.nan, 3., 5., np.nan],
   ....:                     'B': [np.nan, 2., 3., np.nan, 6.]})
   ....: 
In [76]: df2 = pd.DataFrame({'A': [5., 2., 4., np.nan, 3., 7.],
   ....:                     'B': [np.nan, np.nan, 3., 4., 6., 8.]})
   ....: 
In [77]: df1
Out[77]: 
     A    B
0  1.0  NaN
1  NaN  2.0
2  3.0  3.0
3  5.0  NaN
4  NaN  6.0
In [78]: df2
Out[78]: 
     A    B
0  5.0  NaN
1  2.0  NaN
2  4.0  3.0
3  NaN  4.0
4  3.0  6.0
5  7.0  8.0
In [79]: df1.combine_first(df2)
Out[79]: 
     A    B
0  1.0  NaN
1  2.0  2.0
2  3.0  3.0
3  5.0  4.0
4  3.0  6.0
5  7.0  8.0

General DataFrame Combine

The combine_first() method above calls the more general DataFrame.combine(). This method takes another DataFrame and a combiner function, aligns the input DataFrame and then passes the combiner function pairs of Series (i.e., columns whose names are the same).

So, for instance, to reproduce combine_first() as above:

In [80]: def combiner(x, y):
   ....:     return np.where(pd.isna(x), y, x)
   ....:

Descriptive statistics

There exists a large number of methods for computing descriptive statistics and other related operations on Series, DataFrame, and Panel. Most of these are aggregations (hence producing a lower-dimensional result) like sum(), mean(), and quantile(), but some of them, like cumsum() and cumprod(), produce an object of the same size. Generally speaking, these methods take an axis argument, just like ndarray .{sum, std, …}, but the axis can be specified by name or integer:

• Series: no axis argument needed
• DataFrame: “index” (axis=0, default), “columns” (axis=1)
• Panel: “items” (axis=0), “major” (axis=1, default), “minor” (axis=2)

For example:

In [81]: df
Out[81]: 
        one       two     three
a  1.400810 -1.643041       NaN
b -0.356470  1.045911  0.395023
c  0.797268  0.924515 -0.007090
d       NaN  1.553693 -1.670830
In [82]: df.mean(0)
Out[82]: 
one      0.613869
two      0.470270
three   -0.427633
dtype: float64
In [83]: df.mean(1)
Out[83]: 
a   -0.121116
b    0.361488
c    0.571564
d   -0.058569
dtype: float64

All such methods have a skipna option signaling whether to exclude missing data (True by default):

In [84]: df.sum(0, skipna=False)
Out[84]: 
one           NaN
two      1.881078
three         NaN
dtype: float64
In [85]: df.sum(axis=1, skipna=True)
Out[85]: 
a   -0.242232
b    1.084464
c    1.714693
d   -0.117137
dtype: float64

Combined with the broadcasting / arithmetic behavior, one can describe various statistical procedures, like standardization (rendering data zero mean and standard deviation 1), very concisely:

In [86]: ts_stand = (df - df.mean()) / df.std()
In [87]: ts_stand.std()
Out[87]: 
one      1.0
two      1.0
three    1.0
dtype: float64
In [88]: xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0)
In [89]: xs_stand.std(1)
Out[89]: 
a    1.0
b    1.0
c    1.0
d    1.0
dtype: float64

Note that methods like cumsum() and cumprod() preserve the location of NaN values. This is somewhat different from expanding() and rolling(). For more details please see this note.

In [90]: df.cumsum()
Out[90]: 
        one       two     three
a  1.400810 -1.643041       NaN
b  1.044340 -0.597130  0.395023
c  1.841608  0.327385  0.387933
d       NaN  1.881078 -1.282898

Here is a quick reference summary table of common functions. Each also takes an optional level parameter which applies only if the object has a hierarchical index.

Note that by chance some NumPy methods, like mean, std, and sum, will exclude NAs on Series input by default:

In [91]: np.mean(df['one'])
Out[91]: 0.6138692844180106
In [92]: np.mean(df['one'].to_numpy())
Out[92]: nan

Series.nunique() will return the number of unique non-NA values in a Series:

In [93]: series = pd.Series(np.random.randn(500))
In [94]: series[20:500] = np.nan
In [95]: series[10:20] = 5
In [96]: series.nunique()
Out[96]: 11

Summarizing data: describe

There is a convenient describe() function which computes a variety of summary statistics about a Series or the columns of a DataFrame (excluding NAs of course):

In [97]: series = pd.Series(np.random.randn(1000))
In [98]: series[::2] = np.nan
In [99]: series.describe()
Out[99]: 
count    500.000000
mean      -0.020695
std        1.011840
min       -2.683763
25%       -0.709297
50%       -0.070211
75%        0.712856
max        3.160915
dtype: float64
In [100]: frame = pd.DataFrame(np.random.randn(1000, 5),
   .....:                      columns=['a', 'b', 'c', 'd', 'e'])
   .....: 
In [101]: frame.iloc[::2] = np.nan
In [102]: frame.describe()
Out[102]: 
                a           b           c           d           e
count  500.000000  500.000000  500.000000  500.000000  500.000000
mean     0.026515    0.022952   -0.047307   -0.052551    0.011210
std      1.016752    0.980046    1.020837    1.008271    1.006726
min     -3.000951   -2.637901   -3.303099   -3.159200   -3.188821
25%     -0.647623   -0.593587   -0.709906   -0.691338   -0.689176
50%      0.047578   -0.026675   -0.029655   -0.032769   -0.015775
75%      0.723946    0.771931    0.603753    0.667044    0.652221
max      2.740139    2.752332    3.004229    2.728702    3.240991

You can select specific percentiles to include in the output:

In [103]: series.describe(percentiles=[.05, .25, .75, .95])
Out[103]: 
count    500.000000
mean      -0.020695
std        1.011840
min       -2.683763
5%        -1.641337
25%       -0.709297
50%       -0.070211
75%        0.712856
95%        1.699176
max        3.160915
dtype: float64

By default, the median is always included.

For a non-numerical Series object, describe() will give a simple summary of the number of unique values and most frequently occurring values:

In [104]: s = pd.Series(['a', 'a', 'b', 'b', 'a', 'a', np.nan, 'c', 'd', 'a'])
In [105]: s.describe()
Out[105]: 
count     9
unique    4
top       a
freq      5
dtype: object

Note that on a mixed-type DataFrame object, describe() will restrict the summary to include only numerical columns or, if none are, only categorical columns:

In [106]: frame = pd.DataFrame({'a': ['Yes', 'Yes', 'No', 'No'], 'b': range(4)})
In [107]: frame.describe()
Out[107]: 
              b
count  4.000000
mean   1.500000
std    1.290994
min    0.000000
25%    0.750000
50%    1.500000
75%    2.250000
max    3.000000

This behavior can be controlled by providing a list of types as include/exclude arguments. The special value all can also be used:

In [108]: frame.describe(include=['object'])
Out[108]: 
          a
count     4
unique    2
top     Yes
freq      2
In [109]: frame.describe(include=['number'])
Out[109]: 
              b
count  4.000000
mean   1.500000
std    1.290994
min    0.000000
25%    0.750000
50%    1.500000
75%    2.250000
max    3.000000
In [110]: frame.describe(include='all')
Out[110]: 
          a         b
count     4  4.000000
unique    2       NaN
top     Yes       NaN
freq      2       NaN
mean    NaN  1.500000
std     NaN  1.290994
min     NaN  0.000000
25%     NaN  0.750000
50%     NaN  1.500000
75%     NaN  2.250000
max     NaN  3.000000

That feature relies on select_dtypes. Refer to there for details about accepted inputs.

Index of Min/Max Values

The idxmin() and idxmax() functions on Series and DataFrame compute the index labels with the minimum and maximum corresponding values:

In [111]: s1 = pd.Series(np.random.randn(5))
In [112]: s1
Out[112]: 
0   -0.068822
1   -1.129788
2   -0.269798
3   -0.375580
4    0.513381
dtype: float64
In [113]: s1.idxmin(), s1.idxmax()
Out[113]: (1, 4)
In [114]: df1 = pd.DataFrame(np.random.randn(5, 3), columns=['A', 'B', 'C'])
In [115]: df1
Out[115]: 
          A         B         C
0  0.333329 -0.910090 -1.321220
1  2.111424  1.701169  0.858336
2 -0.608055 -2.082155 -0.069618
3  1.412817 -0.562658  0.770042
4  0.373294 -0.965381 -1.607840
In [116]: df1.idxmin(axis=0)
Out[116]: 
A    2
B    2
C    4
dtype: int64
In [117]: df1.idxmax(axis=1)
Out[117]: 
0    A
1    A
2    C
3    A
4    A
dtype: object

When there are multiple rows (or columns) matching the minimum or maximum value, idxmin() and idxmax() return the first matching index:

In [118]: df3 = pd.DataFrame([2, 1, 1, 3, np.nan], columns=['A'], index=list('edcba'))
In [119]: df3
Out[119]: 
     A
e  2.0
d  1.0
c  1.0
b  3.0
a  NaN
In [120]: df3['A'].idxmin()
Out[120]: 'd'

::: tip Note idxmin and idxmax are called argmin and argmax in NumPy. :::

Value counts (histogramming) / Mode

The value_counts() Series method and top-level function computes a histogram of a 1D array of values. It can also be used as a function on regular arrays:

In [121]: data = np.random.randint(0, 7, size=50)
In [122]: data
Out[122]: 
array([6, 4, 1, 3, 4, 4, 4, 6, 5, 2, 6, 1, 0, 4, 3, 2, 5, 3, 4, 0, 5, 3, 0,
       1, 5, 0, 1, 5, 3, 4, 1, 2, 3, 2, 4, 6, 1, 4, 3, 5, 2, 1, 2, 4, 1, 6,
       3, 6, 3, 3])
In [123]: s = pd.Series(data)
In [124]: s.value_counts()
Out[124]: 
4    10
3    10
1     8
6     6
5     6
2     6
0     4
dtype: int64
In [125]: pd.value_counts(data)
Out[125]: 
4    10
3    10
1     8
6     6
5     6
2     6
0     4
dtype: int64

Similarly, you can get the most frequently occurring value(s) (the mode) of the values in a Series or DataFrame:

In [126]: s5 = pd.Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])
In [127]: s5.mode()
Out[127]: 
0    3
1    7
dtype: int64
In [128]: df5 = pd.DataFrame({"A": np.random.randint(0, 7, size=50),
   .....:                     "B": np.random.randint(-10, 15, size=50)})
   .....: 
In [129]: df5.mode()
Out[129]: 
   A  B
0  0 -9

Discretization and quantiling

Continuous values can be discretized using the cut() (bins based on values) and qcut() (bins based on sample quantiles) functions:

In [130]: arr = np.random.randn(20)
In [131]: factor = pd.cut(arr, 4)
In [132]: factor
Out[132]: 
[(1.27, 2.31], (0.231, 1.27], (-0.809, 0.231], (-1.853, -0.809], (1.27, 2.31], ..., (0.231, 1.27], (-0.809, 0.231], (-1.853, -0.809], (1.27, 2.31], (0.231, 1.27]]
Length: 20
Categories (4, interval[float64]): [(-1.853, -0.809] < (-0.809, 0.231] < (0.231, 1.27] < (1.27, 2.31]]
In [133]: factor = pd.cut(arr, [-5, -1, 0, 1, 5])
In [134]: factor
Out[134]: 
[(1, 5], (0, 1], (-1, 0], (-5, -1], (1, 5], ..., (1, 5], (-1, 0], (-5, -1], (1, 5], (0, 1]]
Length: 20
Categories (4, interval[int64]): [(-5, -1] < (-1, 0] < (0, 1] < (1, 5]]

qcut() computes sample quantiles. For example, we could slice up some normally distributed data into equal-size quartiles like so:

In [135]: arr = np.random.randn(30)
In [136]: factor = pd.qcut(arr, [0, .25, .5, .75, 1])
In [137]: factor
Out[137]: 
[(-2.219, -0.669], (-0.669, 0.00453], (0.367, 2.369], (0.00453, 0.367], (0.367, 2.369], ..., (0.00453, 0.367], (0.367, 2.369], (0.00453, 0.367], (-0.669, 0.00453], (0.367, 2.369]]
Length: 30
Categories (4, interval[float64]): [(-2.219, -0.669] < (-0.669, 0.00453] < (0.00453, 0.367] <
                                    (0.367, 2.369]]
In [138]: pd.value_counts(factor)
Out[138]: 
(0.367, 2.369]       8
(-2.219, -0.669]     8
(0.00453, 0.367]     7
(-0.669, 0.00453]    7
dtype: int64

We can also pass infinite values to define the bins:

In [139]: arr = np.random.randn(20)
In [140]: factor = pd.cut(arr, [-np.inf, 0, np.inf])
In [141]: factor
Out[141]: 
[(0.0, inf], (-inf, 0.0], (-inf, 0.0], (-inf, 0.0], (-inf, 0.0], ..., (-inf, 0.0], (-inf, 0.0], (0.0, inf], (-inf, 0.0], (-inf, 0.0]]
Length: 20
Categories (2, interval[float64]): [(-inf, 0.0] < (0.0, inf]]

Function application

To apply your own or another library’s functions to pandas objects, you should be aware of the three methods below. The appropriate method to use depends on whether your function expects to operate on an entire DataFrame or Series, row- or column-wise, or elementwise.

1. Tablewise Function Application: pipe()
2. Row or Column-wise Function Application: apply()
3. Aggregation API: agg() and transform()
4. Applying Elementwise Functions: applymap()

Tablewise Function Application

DataFrames and Series can of course just be passed into functions. However, if the function needs to be called in a chain, consider using the pipe() method. Compare the following

# f, g, and h are functions taking and returning ``DataFrames``
>>> f(g(h(df), arg1=1), arg2=2, arg3=3)

with the equivalent

>>> (df.pipe(h)
...    .pipe(g, arg1=1)
...    .pipe(f, arg2=2, arg3=3))

Pandas encourages the second style, which is known as method chaining. pipe makes it easy to use your own or another library’s functions in method chains, alongside pandas’ methods.

In the example above, the functions f, g, and h each expected the DataFrame as the first positional argument. What if the function you wish to apply takes its data as, say, the second argument? In this case, provide pipe with a tuple of (callable, data_keyword). .pipe will route the DataFrame to the argument specified in the tuple.

For example, we can fit a regression using statsmodels. Their API expects a formula first and a DataFrame as the second argument, data. We pass in the function, keyword pair (sm.ols, 'data') to pipe:

In [142]: import statsmodels.formula.api as sm
In [143]: bb = pd.read_csv('data/baseball.csv', index_col='id')
In [144]: (bb.query('h > 0')
   .....:    .assign(ln_h=lambda df: np.log(df.h))
   .....:    .pipe((sm.ols, 'data'), 'hr ~ ln_h + year + g + C(lg)')
   .....:    .fit()
   .....:    .summary()
   .....:  )
   .....: 
Out[144]: 
<class 'statsmodels.iolib.summary.Summary'>
"""
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                     hr   R-squared:                       0.685
Model:                            OLS   Adj. R-squared:                  0.665
Method:                 Least Squares   F-statistic:                     34.28
Date:                Tue, 12 Mar 2019   Prob (F-statistic):           3.48e-15
Time:                        22:38:35   Log-Likelihood:                -205.92
No. Observations:                  68   AIC:                             421.8
Df Residuals:                      63   BIC:                             432.9
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
===============================================================================
                  coef    std err          t      P>|t|      [0.025      0.975]
-------------------------------------------------------------------------------
Intercept   -8484.7720   4664.146     -1.819      0.074   -1.78e+04     835.780
C(lg)[T.NL]    -2.2736      1.325     -1.716      0.091      -4.922       0.375
ln_h           -1.3542      0.875     -1.547      0.127      -3.103       0.395
year            4.2277      2.324      1.819      0.074      -0.417       8.872
g               0.1841      0.029      6.258      0.000       0.125       0.243
==============================================================================
Omnibus:                       10.875   Durbin-Watson:                   1.999
Prob(Omnibus):                  0.004   Jarque-Bera (JB):               17.298
Skew:                           0.537   Prob(JB):                     0.000175
Kurtosis:                       5.225   Cond. No.                     1.49e+07
==============================================================================
Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.49e+07. This might indicate that there are
strong multicollinearity or other numerical problems.
"""