Frequently Asked Questions (FAQ)¶
DataFrame memory usage¶
As of pandas version 0.15.0, the memory usage of a dataframe (including
the index) is shown when accessing the info method of a dataframe. A
configuration option, display.memory_usage (see Options and Settings),
specifies if the dataframe’s memory usage will be displayed when
invoking the df.info() method.
For example, the memory usage of the dataframe below is shown
when calling df.info():
In [1]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
...: 'complex128', 'object', 'bool']
...:
In [2]: n = 5000
In [3]: data = dict([ (t, np.random.randint(100, size=n).astype(t))
...: for t in dtypes])
...:
In [4]: df = pd.DataFrame(data)
In [5]: df['categorical'] = df['object'].astype('category')
In [6]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
bool 5000 non-null bool
complex128 5000 non-null complex128
datetime64[ns] 5000 non-null datetime64[ns]
float64 5000 non-null float64
int64 5000 non-null int64
object 5000 non-null object
timedelta64[ns] 5000 non-null timedelta64[ns]
categorical 5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 284.1+ KB
The + symbol indicates that the true memory usage could be higher, because
pandas does not count the memory used by values in columns with
dtype=object.
New in version 0.17.1.
Passing memory_usage='deep' will enable a more accurate memory usage report,
that accounts for the full usage of the contained objects. This is optional
as it can be expensive to do this deeper introspection.
In [7]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
bool 5000 non-null bool
complex128 5000 non-null complex128
datetime64[ns] 5000 non-null datetime64[ns]
float64 5000 non-null float64
int64 5000 non-null int64
object 5000 non-null object
timedelta64[ns] 5000 non-null timedelta64[ns]
categorical 5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 401.2 KB
By default the display option is set to True but can be explicitly
overridden by passing the memory_usage argument when invoking df.info().
The memory usage of each column can be found by calling the memory_usage
method. This returns a Series with an index represented by column names
and memory usage of each column shown in bytes. For the dataframe above,
the memory usage of each column and the total memory usage of the
dataframe can be found with the memory_usage method:
In [8]: df.memory_usage()
Out[8]:
Index 72
bool 5000
complex128 80000
datetime64[ns] 40000
float64 40000
int64 40000
object 40000
timedelta64[ns] 40000
categorical 5800
dtype: int64
# total memory usage of dataframe
In [9]: df.memory_usage().sum()
Out[9]: 290872
By default the memory usage of the dataframe’s index is shown in the
returned Series, the memory usage of the index can be suppressed by passing
the index=False argument:
In [10]: df.memory_usage(index=False)
Out[10]:
bool 5000
complex128 80000
datetime64[ns] 40000
float64 40000
int64 40000
object 40000
timedelta64[ns] 40000
categorical 5800
dtype: int64
The memory usage displayed by the info method utilizes the
memory_usage method to determine the memory usage of a dataframe
while also formatting the output in human-readable units (base-2
representation; i.e., 1KB = 1024 bytes).
See also Categorical Memory Usage.
Byte-Ordering Issues¶
Occasionally you may have to deal with data that were created on a machine with a different byte order than the one on which you are running Python. To deal with this issue you should convert the underlying NumPy array to the native system byte order before passing it to Series/DataFrame/Panel constructors using something similar to the following:
In [11]: x = np.array(list(range(10)), '>i4') # big endian
In [12]: newx = x.byteswap().newbyteorder() # force native byteorder
In [13]: s = pd.Series(newx)
See the NumPy documentation on byte order for more details.