# What’s new in 0.25.0 (July 18, 2019)

Warning

Starting with the 0.25.x series of releases, pandas only supports Python 3.5.3 and higher. See Plan for dropping Python 2.7 (opens new window) for more details.

Warning

The minimum supported Python version will be bumped to 3.6 in a future release.

Warning

Panel has been fully removed. For N-D labeled data structures, please use xarray (opens new window)

Warning

read_pickle() and read_msgpack() are only guaranteed backwards compatible back to pandas version 0.20.3 (GH27082 (opens new window))

These are the changes in pandas 0.25.0. See Release Notes for a full changelog including other versions of pandas.

# Enhancements

# Groupby aggregation with relabeling

Pandas has added special groupby behavior, known as “named aggregation”, for naming the output columns when applying multiple aggregation functions to specific columns (GH18366 (opens new window), GH26512 (opens new window)).

In [1]: animals = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],
   ...:                         'height': [9.1, 6.0, 9.5, 34.0],
   ...:                         'weight': [7.9, 7.5, 9.9, 198.0]})
   ...: 

In [2]: animals
Out[2]: 
  kind  height  weight
0  cat     9.1     7.9
1  dog     6.0     7.5
2  cat     9.5     9.9
3  dog    34.0   198.0

[4 rows x 3 columns]

In [3]: animals.groupby("kind").agg(
   ...:     min_height=pd.NamedAgg(column='height', aggfunc='min'),
   ...:     max_height=pd.NamedAgg(column='height', aggfunc='max'),
   ...:     average_weight=pd.NamedAgg(column='weight', aggfunc=np.mean),
   ...: )
   ...: 
Out[3]: 
      min_height  max_height  average_weight
kind                                        
cat          9.1         9.5            8.90
dog          6.0        34.0          102.75

[2 rows x 3 columns]

Pass the desired columns names as the **kwargs to .agg. The values of **kwargs should be tuples where the first element is the column selection, and the second element is the aggregation function to apply. Pandas provides the pandas.NamedAgg namedtuple to make it clearer what the arguments to the function are, but plain tuples are accepted as well.

In [4]: animals.groupby("kind").agg(
   ...:     min_height=('height', 'min'),
   ...:     max_height=('height', 'max'),
   ...:     average_weight=('weight', np.mean),
   ...: )
   ...: 
Out[4]: 
      min_height  max_height  average_weight
kind                                        
cat          9.1         9.5            8.90
dog          6.0        34.0          102.75

[2 rows x 3 columns]

Named aggregation is the recommended replacement for the deprecated “dict-of-dicts” approach to naming the output of column-specific aggregations (Deprecate groupby.agg() with a dictionary when renaming).

A similar approach is now available for Series groupby objects as well. Because there’s no need for column selection, the values can just be the functions to apply

In [5]: animals.groupby("kind").height.agg(
   ...:     min_height="min",
   ...:     max_height="max",
   ...: )
   ...: 
Out[5]: 
      min_height  max_height
kind                        
cat          9.1         9.5
dog          6.0        34.0

[2 rows x 2 columns]

This type of aggregation is the recommended alternative to the deprecated behavior when passing a dict to a Series groupby aggregation (Deprecate groupby.agg() with a dictionary when renaming).

See Named aggregation (opens new window) for more.

# Groupby Aggregation with multiple lambdas

You can now provide multiple lambda functions to a list-like aggregation in pandas.core.groupby.GroupBy.agg (opens new window) (GH26430 (opens new window)).

In [6]: animals.groupby('kind').height.agg([
   ...:     lambda x: x.iloc[0], lambda x: x.iloc[-1]
   ...: ])
   ...: 
Out[6]: 
      <lambda_0>  <lambda_1>
kind                        
cat          9.1         9.5
dog          6.0        34.0

[2 rows x 2 columns]

In [7]: animals.groupby('kind').agg([
   ...:     lambda x: x.iloc[0] - x.iloc[1],
   ...:     lambda x: x.iloc[0] + x.iloc[1]
   ...: ])
   ...: 
Out[7]: 
         height                weight           
     <lambda_0> <lambda_1> <lambda_0> <lambda_1>
kind                                            
cat        -0.4       18.6       -2.0       17.8
dog       -28.0       40.0     -190.5      205.5

[2 rows x 4 columns]

Previously, these raised a SpecificationError.

# Better repr for MultiIndex

Printing of MultiIndex (opens new window) instances now shows tuples of each row and ensures that the tuple items are vertically aligned, so it’s now easier to understand the structure of the MultiIndex. (GH13480 (opens new window)):

The repr now looks like this:

In [8]: pd.MultiIndex.from_product([['a', 'abc'], range(500)])
Out[8]: 
MultiIndex([(  'a',   0),
            (  'a',   1),
            (  'a',   2),
            (  'a',   3),
            (  'a',   4),
            (  'a',   5),
            (  'a',   6),
            (  'a',   7),
            (  'a',   8),
            (  'a',   9),
            ...
            ('abc', 490),
            ('abc', 491),
            ('abc', 492),
            ('abc', 493),
            ('abc', 494),
            ('abc', 495),
            ('abc', 496),
            ('abc', 497),
            ('abc', 498),
            ('abc', 499)],
           length=1000)

Previously, outputting a MultiIndex (opens new window) printed all the levels and codes of the MultiIndex, which was visually unappealing and made the output more difficult to navigate. For example (limiting the range to 5):

In [1]: pd.MultiIndex.from_product([['a', 'abc'], range(5)])
Out[1]: MultiIndex(levels=[['a', 'abc'], [0, 1, 2, 3]],
   ...:            codes=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]])

In the new repr, all values will be shown, if the number of rows is smaller than options.display.max_seq_items (default: 100 items). Horizontally, the output will truncate, if it’s wider than options.display.width (default: 80 characters).

# Shorter truncated repr for Series and DataFrame

Currently, the default display options of pandas ensure that when a Series or DataFrame has more than 60 rows, its repr gets truncated to this maximum of 60 rows (the display.max_rows option). However, this still gives a repr that takes up a large part of the vertical screen estate. Therefore, a new option display.min_rows is introduced with a default of 10 which determines the number of rows showed in the truncated repr:

  • For small Series or DataFrames, up to max_rows number of rows is shown (default: 60).
  • For larger Series of DataFrame with a length above max_rows, only min_rows number of rows is shown (default: 10, i.e. the first and last 5 rows).

This dual option allows to still see the full content of relatively small objects (e.g. df.head(20) shows all 20 rows), while giving a brief repr for large objects.

To restore the previous behaviour of a single threshold, set pd.options.display.min_rows = None.

# Json normalize with max_level param support

json_normalize() normalizes the provided input dict to all nested levels. The new max_level parameter provides more control over which level to end normalization (GH23843 (opens new window)):

The repr now looks like this:

In [9]: from pandas.io.json import json_normalize

In [10]: data = [{
   ....:     'CreatedBy': {'Name': 'User001'},
   ....:     'Lookup': {'TextField': 'Some text',
   ....:                'UserField': {'Id': 'ID001', 'Name': 'Name001'}},
   ....:     'Image': {'a': 'b'}
   ....: }]
   ....: 

In [11]: json_normalize(data, max_level=1)
Out[11]: 
  CreatedBy.Name Lookup.TextField                    Lookup.UserField Image.a
0        User001        Some text  {'Id': 'ID001', 'Name': 'Name001'}       b

[1 rows x 4 columns]

# Series.explode to split list-like values to rows

Series (opens new window) and DataFrame (opens new window) have gained the DataFrame.explode() (opens new window) methods to transform list-likes to individual rows. See section on Exploding list-like column (opens new window) in docs for more information (GH16538 (opens new window), GH10511 (opens new window))

Here is a typical usecase. You have comma separated string in a column.

In [12]: df = pd.DataFrame([{'var1': 'a,b,c', 'var2': 1},
   ....:                    {'var1': 'd,e,f', 'var2': 2}])
   ....: 

In [13]: df
Out[13]: 
    var1  var2
0  a,b,c     1
1  d,e,f     2

[2 rows x 2 columns]

Creating a long form DataFrame is now straightforward using chained operations

In [14]: df.assign(var1=df.var1.str.split(',')).explode('var1')
Out[14]: 
  var1  var2
0    a     1
0    b     1
0    c     1
1    d     2
1    e     2
1    f     2

[6 rows x 2 columns]

# Other enhancements

# Backwards incompatible API changes

# Indexing with date strings with UTC offsets

Indexing a DataFrame (opens new window) or Series (opens new window) with a DatetimeIndex (opens new window) with a date string with a UTC offset would previously ignore the UTC offset. Now, the UTC offset is respected in indexing. (GH24076 (opens new window), GH16785 (opens new window))

In [15]: df = pd.DataFrame([0], index=pd.DatetimeIndex(['2019-01-01'], tz='US/Pacific'))

In [16]: df
Out[16]: 
                           0
2019-01-01 00:00:00-08:00  0

[1 rows x 1 columns]

Previous behavior:

In [3]: df['2019-01-01 00:00:00+04:00':'2019-01-01 01:00:00+04:00']
Out[3]:
                           0
2019-01-01 00:00:00-08:00  0

New behavior:

In [17]: df['2019-01-01 12:00:00+04:00':'2019-01-01 13:00:00+04:00']
Out[17]: 
                           0
2019-01-01 00:00:00-08:00  0

[1 rows x 1 columns]

# MultiIndex constructed from levels and codes

Constructing a MultiIndex (opens new window) with NaN levels or codes value < -1 was allowed previously. Now, construction with codes value < -1 is not allowed and NaN levels’ corresponding codes would be reassigned as -1. (GH19387 (opens new window))

Previous behavior:

In [1]: pd.MultiIndex(levels=[[np.nan, None, pd.NaT, 128, 2]],
   ...:               codes=[[0, -1, 1, 2, 3, 4]])
   ...:
Out[1]: MultiIndex(levels=[[nan, None, NaT, 128, 2]],
                   codes=[[0, -1, 1, 2, 3, 4]])

In [2]: pd.MultiIndex(levels=[[1, 2]], codes=[[0, -2]])
Out[2]: MultiIndex(levels=[[1, 2]],
                   codes=[[0, -2]])

New behavior:

In [18]: pd.MultiIndex(levels=[[np.nan, None, pd.NaT, 128, 2]],
   ....:               codes=[[0, -1, 1, 2, 3, 4]])
   ....: 
Out[18]: 
MultiIndex([(nan,),
            (nan,),
            (nan,),
            (nan,),
            (128,),
            (  2,)],
           )

In [19]: pd.MultiIndex(levels=[[1, 2]], codes=[[0, -2]])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-225a01af3975> in <module>
----> 1 pd.MultiIndex(levels=[[1, 2]], codes=[[0, -2]])

/pandas/pandas/util/_decorators.py in wrapper(*args, **kwargs)
    206                 else:
    207                     kwargs[new_arg_name] = new_arg_value
--> 208             return func(*args, **kwargs)
    209 
    210         return wrapper

/pandas/pandas/core/indexes/multi.py in __new__(cls, levels, codes, sortorder, names, dtype, copy, name, verify_integrity, _set_identity)
    270 
    271         if verify_integrity:
--> 272             new_codes = result._verify_integrity()
    273             result._codes = new_codes
    274 

/pandas/pandas/core/indexes/multi.py in _verify_integrity(self, codes, levels)
    348                 raise ValueError(
    349                     "On level {level}, code value ({code})"
--> 350                     " < -1".format(level=i, code=level_codes.min())
    351                 )
    352             if not level.is_unique:

ValueError: On level 0, code value (-2) < -1

# Groupby.apply on DataFrame evaluates first group only once

The implementation of DataFrameGroupBy.apply() previously evaluated the supplied function consistently twice on the first group to infer if it is safe to use a fast code path. Particularly for functions with side effects, this was an undesired behavior and may have led to surprises. (GH2936 (opens new window), GH2656 (opens new window), GH7739 (opens new window), GH10519 (opens new window), GH12155 (opens new window), GH20084 (opens new window), GH21417 (opens new window))

Now every group is evaluated only a single time.

In [20]: df = pd.DataFrame({"a": ["x", "y"], "b": [1, 2]})

In [21]: df
Out[21]: 
   a  b
0  x  1
1  y  2

[2 rows x 2 columns]

In [22]: def func(group):
   ....:     print(group.name)
   ....:     return group
   ....:

Previous behavior:

In [3]: df.groupby('a').apply(func)
x
x
y
Out[3]:
   a  b
0  x  1
1  y  2

New behavior:

In [23]: df.groupby("a").apply(func)
x
y
Out[23]: 
   a  b
0  x  1
1  y  2

[2 rows x 2 columns]

# Concatenating sparse values

When passed DataFrames whose values are sparse, concat() (opens new window) will now return a Series (opens new window) or DataFrame (opens new window) with sparse values, rather than a SparseDataFrame (GH25702 (opens new window)).

In [24]: df = pd.DataFrame({"A": pd.SparseArray([0, 1])})

Previous behavior:

In [2]: type(pd.concat([df, df]))
pandas.core.sparse.frame.SparseDataFrame

New behavior:

In [25]: type(pd.concat([df, df]))
Out[25]: pandas.core.frame.DataFrame

This now matches the existing behavior of concat (opens new window) on Series with sparse values. concat() (opens new window) will continue to return a SparseDataFrame when all the values are instances of SparseDataFrame.

This change also affects routines using concat() (opens new window) internally, like get_dummies() (opens new window), which now returns a DataFrame (opens new window) in all cases (previously a SparseDataFrame was returned if all the columns were dummy encoded, and a DataFrame (opens new window) otherwise).

Providing any SparseSeries or SparseDataFrame to concat() (opens new window) will cause a SparseSeries or SparseDataFrame to be returned, as before.

# The .str-accessor performs stricter type checks

Due to the lack of more fine-grained dtypes, Series.str (opens new window) so far only checked whether the data was of object dtype. Series.str (opens new window) will now infer the dtype data within the Series; in particular, 'bytes'-only data will raise an exception (except for Series.str.decode() (opens new window), Series.str.get() (opens new window), Series.str.len() (opens new window), Series.str.slice() (opens new window)), see GH23163 (opens new window), GH23011 (opens new window), GH23551 (opens new window).

Previous behavior:

In [1]: s = pd.Series(np.array(['a', 'ba', 'cba'], 'S'), dtype=object)

In [2]: s
Out[2]:
0      b'a'
1     b'ba'
2    b'cba'
dtype: object

In [3]: s.str.startswith(b'a')
Out[3]:
0     True
1    False
2    False
dtype: bool

New behavior:

In [26]: s = pd.Series(np.array(['a', 'ba', 'cba'], 'S'), dtype=object)

In [27]: s
Out[27]: 
0      b'a'
1     b'ba'
2    b'cba'
Length: 3, dtype: object

In [28]: s.str.startswith(b'a')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-ac784692b361> in <module>
----> 1 s.str.startswith(b'a')

/pandas/pandas/core/strings.py in wrapper(self, *args, **kwargs)
   1840                     )
   1841                 )
-> 1842                 raise TypeError(msg)
   1843             return func(self, *args, **kwargs)
   1844 

TypeError: Cannot use .str.startswith with values of inferred dtype 'bytes'.

# Categorical dtypes are preserved during groupby

Previously, columns that were categorical, but not the groupby key(s) would be converted to object dtype during groupby operations. Pandas now will preserve these dtypes. (GH18502 (opens new window))

In [29]: cat = pd.Categorical(["foo", "bar", "bar", "qux"], ordered=True)

In [30]: df = pd.DataFrame({'payload': [-1, -2, -1, -2], 'col': cat})

In [31]: df
Out[31]: 
   payload  col
0       -1  foo
1       -2  bar
2       -1  bar
3       -2  qux

[4 rows x 2 columns]

In [32]: df.dtypes
Out[32]: 
payload       int64
col        category
Length: 2, dtype: object

Previous Behavior:

In [5]: df.groupby('payload').first().col.dtype
Out[5]: dtype('O')

New Behavior:

In [33]: df.groupby('payload').first().col.dtype
Out[33]: CategoricalDtype(categories=['bar', 'foo', 'qux'], ordered=True)

# Incompatible Index type unions

When performing Index.union() (opens new window) operations between objects of incompatible dtypes, the result will be a base Index (opens new window) of dtype object. This behavior holds true for unions between Index (opens new window) objects that previously would have been prohibited. The dtype of empty Index (opens new window) objects will now be evaluated before performing union operations rather than simply returning the other Index (opens new window) object. Index.union() (opens new window) can now be considered commutative, such that A.union(B) == B.union(A) (GH23525 (opens new window)).

Previous behavior:

In [1]: pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))
...
ValueError: can only call with other PeriodIndex-ed objects

In [2]: pd.Index([], dtype=object).union(pd.Index([1, 2, 3]))
Out[2]: Int64Index([1, 2, 3], dtype='int64')

New behavior:

In [34]: pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))
Out[34]: Index([1991-09-05, 1991-09-06, 1, 2, 3], dtype='object')

In [35]: pd.Index([], dtype=object).union(pd.Index([1, 2, 3]))
Out[35]: Index([1, 2, 3], dtype='object')

Note that integer- and floating-dtype indexes are considered “compatible”. The integer values are coerced to floating point, which may result in loss of precision. See Set operations on Index objects (opens new window) for more.

# DataFrame groupby ffill/bfill no longer return group labels

The methods ffill, bfill, pad and backfill of DataFrameGroupBy previously included the group labels in the return value, which was inconsistent with other groupby transforms. Now only the filled values are returned. (GH21521 (opens new window))

In [36]: df = pd.DataFrame({"a": ["x", "y"], "b": [1, 2]})

In [37]: df
Out[37]: 
   a  b
0  x  1
1  y  2

[2 rows x 2 columns]

Previous behavior:

In [3]: df.groupby("a").ffill()
Out[3]:
   a  b
0  x  1
1  y  2

New behavior:

In [38]: df.groupby("a").ffill()
Out[38]: 
   b
0  1
1  2

[2 rows x 1 columns]

# DataFrame describe on an empty categorical / object column will return top and freq

When calling DataFrame.describe() (opens new window) with an empty categorical / object column, the ‘top’ and ‘freq’ columns were previously omitted, which was inconsistent with the output for non-empty columns. Now the ‘top’ and ‘freq’ columns will always be included, with numpy.nan in the case of an empty DataFrame (opens new window) (GH26397 (opens new window))

In [39]: df = pd.DataFrame({"empty_col": pd.Categorical([])})

In [40]: df
Out[40]: 
Empty DataFrame
Columns: [empty_col]
Index: []

[0 rows x 1 columns]

Previous behavior:

In [3]: df.describe()
Out[3]:
        empty_col
count           0
unique          0

New behavior:

In [41]: df.describe()
Out[41]: 
       empty_col
count          0
unique         0
top          NaN
freq         NaN

[4 rows x 1 columns]

# __str__ methods now call __repr__ rather than vice versa

Pandas has until now mostly defined string representations in a Pandas objects’s __str__/__unicode__/__bytes__ methods, and called __str__ from the __repr__ method, if a specific __repr__ method is not found. This is not needed for Python3. In Pandas 0.25, the string representations of Pandas objects are now generally defined in __repr__, and calls to __str__ in general now pass the call on to the __repr__, if a specific __str__ method doesn’t exist, as is standard for Python. This change is backward compatible for direct usage of Pandas, but if you subclass Pandas objects and give your subclasses specific __str__/__repr__ methods, you may have to adjust your __str__/__repr__ methods (GH26495 (opens new window)).

# Indexing an IntervalIndex with Interval objects

Indexing methods for IntervalIndex (opens new window) have been modified to require exact matches only for Interval (opens new window) queries. IntervalIndex methods previously matched on any overlapping Interval. Behavior with scalar points, e.g. querying with an integer, is unchanged (GH16316 (opens new window)).

In [42]: ii = pd.IntervalIndex.from_tuples([(0, 4), (1, 5), (5, 8)])

In [43]: ii
Out[43]: 
IntervalIndex([(0, 4], (1, 5], (5, 8]],
              closed='right',
              dtype='interval[int64]')

The in operator (__contains__) now only returns True for exact matches to Intervals in the IntervalIndex, whereas this would previously return True for any Interval overlapping an Interval in the IntervalIndex.

Previous behavior:

In [4]: pd.Interval(1, 2, closed='neither') in ii
Out[4]: True

In [5]: pd.Interval(-10, 10, closed='both') in ii
Out[5]: True

New behavior:

In [44]: pd.Interval(1, 2, closed='neither') in ii
Out[44]: False

In [45]: pd.Interval(-10, 10, closed='both') in ii
Out[45]: False

The get_loc() (opens new window) method now only returns locations for exact matches to Interval queries, as opposed to the previous behavior of returning locations for overlapping matches. A KeyError will be raised if an exact match is not found.

Previous behavior:

In [6]: ii.get_loc(pd.Interval(1, 5))
Out[6]: array([0, 1])

In [7]: ii.get_loc(pd.Interval(2, 6))
Out[7]: array([0, 1, 2])

New behavior:

In [6]: ii.get_loc(pd.Interval(1, 5))
Out[6]: 1

In [7]: ii.get_loc(pd.Interval(2, 6))
---------------------------------------------------------------------------
KeyError: Interval(2, 6, closed='right')

Likewise, get_indexer() (opens new window) and get_indexer_non_unique() will also only return locations for exact matches to Interval queries, with -1 denoting that an exact match was not found.

These indexing changes extend to querying a Series (opens new window) or DataFrame (opens new window) with an IntervalIndex index.

In [46]: s = pd.Series(list('abc'), index=ii)

In [47]: s
Out[47]: 
(0, 4]    a
(1, 5]    b
(5, 8]    c
Length: 3, dtype: object

Selecting from a Series or DataFrame using [] (__getitem__) or loc now only returns exact matches for Interval queries.

Previous behavior:

In [8]: s[pd.Interval(1, 5)]
Out[8]:
(0, 4]    a
(1, 5]    b
dtype: object

In [9]: s.loc[pd.Interval(1, 5)]
Out[9]:
(0, 4]    a
(1, 5]    b
dtype: object

New behavior:

In [48]: s[pd.Interval(1, 5)]
Out[48]: 'b'

In [49]: s.loc[pd.Interval(1, 5)]
Out[49]: 'b'

Similarly, a KeyError will be raised for non-exact matches instead of returning overlapping matches.

Previous behavior:

In [9]: s[pd.Interval(2, 3)]
Out[9]:
(0, 4]    a
(1, 5]    b
dtype: object

In [10]: s.loc[pd.Interval(2, 3)]
Out[10]:
(0, 4]    a
(1, 5]    b
dtype: object

New behavior:

In [6]: s[pd.Interval(2, 3)]
---------------------------------------------------------------------------
KeyError: Interval(2, 3, closed='right')

In [7]: s.loc[pd.Interval(2, 3)]
---------------------------------------------------------------------------
KeyError: Interval(2, 3, closed='right')

The overlaps() (opens new window) method can be used to create a boolean indexer that replicates the previous behavior of returning overlapping matches.

New behavior:

In [50]: idxr = s.index.overlaps(pd.Interval(2, 3))

In [51]: idxr
Out[51]: array([ True,  True, False])

In [52]: s[idxr]
Out[52]: 
(0, 4]    a
(1, 5]    b
Length: 2, dtype: object

In [53]: s.loc[idxr]
Out[53]: 
(0, 4]    a
(1, 5]    b
Length: 2, dtype: object

# Binary ufuncs on Series now align

Applying a binary ufunc like numpy.power() now aligns the inputs when both are Series (opens new window) (GH23293 (opens new window)).

In [54]: s1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])

In [55]: s2 = pd.Series([3, 4, 5], index=['d', 'c', 'b'])

In [56]: s1
Out[56]: 
a    1
b    2
c    3
Length: 3, dtype: int64

In [57]: s2
Out[57]: 
d    3
c    4
b    5
Length: 3, dtype: int64

Previous behavior

In [5]: np.power(s1, s2)
Out[5]:
a      1
b     16
c    243
dtype: int64

New behavior

In [58]: np.power(s1, s2)
Out[58]: 
a     1.0
b    32.0
c    81.0
d     NaN
Length: 4, dtype: float64

This matches the behavior of other binary operations in pandas, like Series.add() (opens new window). To retain the previous behavior, convert the other Series to an array before applying the ufunc.

In [59]: np.power(s1, s2.array)
Out[59]: 
a      1
b     16
c    243
Length: 3, dtype: int64

# Categorical.argsort now places missing values at the end

Categorical.argsort() now places missing values at the end of the array, making it consistent with NumPy and the rest of pandas (GH21801 (opens new window)).

In [60]: cat = pd.Categorical(['b', None, 'a'], categories=['a', 'b'], ordered=True)

Previous behavior

In [2]: cat = pd.Categorical(['b', None, 'a'], categories=['a', 'b'], ordered=True)

In [3]: cat.argsort()
Out[3]: array([1, 2, 0])

In [4]: cat[cat.argsort()]
Out[4]:
[NaN, a, b]
categories (2, object): [a < b]

New behavior

In [61]: cat.argsort()
Out[61]: array([2, 0, 1])

In [62]: cat[cat.argsort()]
Out[62]: 
[a, b, NaN]
Categories (2, object): [a < b]

# Column order is preserved when passing a list of dicts to DataFrame

Starting with Python 3.7 the key-order of dict is guaranteed (opens new window). In practice, this has been true since Python 3.6. The DataFrame (opens new window) constructor now treats a list of dicts in the same way as it does a list of OrderedDict, i.e. preserving the order of the dicts. This change applies only when pandas is running on Python>=3.6 (GH27309 (opens new window)).

In [63]: data = [
   ....:     {'name': 'Joe', 'state': 'NY', 'age': 18},
   ....:     {'name': 'Jane', 'state': 'KY', 'age': 19, 'hobby': 'Minecraft'},
   ....:     {'name': 'Jean', 'state': 'OK', 'age': 20, 'finances': 'good'}
   ....: ]
   ....:

Previous Behavior:

The columns were lexicographically sorted previously,

In [1]: pd.DataFrame(data)
Out[1]:
   age finances      hobby  name state
0   18      NaN        NaN   Joe    NY
1   19      NaN  Minecraft  Jane    KY
2   20     good        NaN  Jean    OK

New Behavior:

The column order now matches the insertion-order of the keys in the dict, considering all the records from top to bottom. As a consequence, the column order of the resulting DataFrame has changed compared to previous pandas verisons.

In [64]: pd.DataFrame(data)
Out[64]: 
   name state  age      hobby finances
0   Joe    NY   18        NaN      NaN
1  Jane    KY   19  Minecraft      NaN
2  Jean    OK   20        NaN     good

[3 rows x 5 columns]

# Increased minimum versions for dependencies

Due to dropping support for Python 2.7, a number of optional dependencies have updated minimum versions (GH25725 (opens new window), GH24942 (opens new window), GH25752 (opens new window)). Independently, some minimum supported versions of dependencies were updated (GH23519 (opens new window), GH25554 (opens new window)). If installed, we now require:

Package Minimum Version Required
numpy 1.13.3 X
pytz 2015.4 X
python-dateutil 2.6.1 X
bottleneck 1.2.1
numexpr 2.6.2
pytest (dev) 4.0.2

For optional libraries (opens new window) the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

Package Minimum Version
beautifulsoup4 4.6.0
fastparquet 0.2.1
gcsfs 0.2.2
lxml 3.8.0
matplotlib 2.2.2
openpyxl 2.4.8
pyarrow 0.9.0
pymysql 0.7.1
pytables 3.4.2
scipy 0.19.0
sqlalchemy 1.1.4
xarray 0.8.2
xlrd 1.1.0
xlsxwriter 0.9.8
xlwt 1.2.0

See Dependencies (opens new window) and Optional dependencies (opens new window) for more.

# Other API changes

# Deprecations

# Sparse subclasses

The SparseSeries and SparseDataFrame subclasses are deprecated. Their functionality is better-provided by a Series or DataFrame with sparse values.

Previous way

In [65]: df = pd.SparseDataFrame({"A": [0, 0, 1, 2]})

In [66]: df.dtypes
Out[66]: 
A    Sparse[int64, nan]
Length: 1, dtype: object

New way

In [67]: df = pd.DataFrame({"A": pd.SparseArray([0, 0, 1, 2])})

In [68]: df.dtypes
Out[68]: 
A    Sparse[int64, 0]
Length: 1, dtype: object

The memory usage of the two approaches is identical. See Migrating (opens new window) for more (GH19239 (opens new window)).

# msgpack format

The msgpack format is deprecated as of 0.25 and will be removed in a future version. It is recommended to use pyarrow for on-the-wire transmission of pandas objects. (GH27084 (opens new window))

# Other deprecations

# Removal of prior version deprecations/changes

# Performance improvements

# Bug fixes

# Categorical

# Datetimelike

# Timedelta

# Timezones

# Numeric

# Conversion

# Strings

# Interval

# Indexing

# Missing

# MultiIndex

# I/O

# Plotting

# Groupby/resample/rolling

# Reshaping

# Sparse

# Build Changes

# ExtensionArray

# Other

# Contributors