Frequently Asked Questions (FAQ)¶
DataFrame memory usage¶
The memory usage of a DataFrame
(including the index) is shown when calling
the info()
. A configuration option, display.memory_usage
(see the list of options), specifies if the
DataFrame
’s memory usage will be displayed when invoking the df.info()
method.
For example, the memory usage of the DataFrame
below is shown
when calling info()
:
In [1]: dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',
...: 'complex128', 'object', 'bool']
...:
In [2]: n = 5000
In [3]: data = dict([(t, np.random.randint(100, size=n).astype(t))
...: for t in dtypes])
...:
In [4]: df = pd.DataFrame(data)
In [5]: df['categorical'] = df['object'].astype('category')
In [6]: df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
int64 5000 non-null int64
float64 5000 non-null float64
datetime64[ns] 5000 non-null datetime64[ns]
timedelta64[ns] 5000 non-null timedelta64[ns]
complex128 5000 non-null complex128
object 5000 non-null object
bool 5000 non-null bool
categorical 5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 289.1+ KB
The +
symbol indicates that the true memory usage could be higher, because
pandas does not count the memory used by values in columns with
dtype=object
.
Passing memory_usage='deep'
will enable a more accurate memory usage report,
accounting for the full usage of the contained objects. This is optional
as it can be expensive to do this deeper introspection.
In [7]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 8 columns):
int64 5000 non-null int64
float64 5000 non-null float64
datetime64[ns] 5000 non-null datetime64[ns]
timedelta64[ns] 5000 non-null timedelta64[ns]
complex128 5000 non-null complex128
object 5000 non-null object
bool 5000 non-null bool
categorical 5000 non-null category
dtypes: bool(1), category(1), complex128(1), datetime64[ns](1), float64(1), int64(1), object(1), timedelta64[ns](1)
memory usage: 425.6 KB
By default the display option is set to True
but can be explicitly
overridden by passing the memory_usage
argument when invoking df.info()
.
The memory usage of each column can be found by calling the
memory_usage()
method. This returns a Series
with an index
represented by column names and memory usage of each column shown in bytes. For
the DataFrame
above, the memory usage of each column and the total memory
usage can be found with the memory_usage
method:
In [8]: df.memory_usage()
Out[8]:
Index 80
int64 40000
float64 40000
datetime64[ns] 40000
timedelta64[ns] 40000
complex128 80000
object 40000
bool 5000
categorical 10920
dtype: int64
# total memory usage of dataframe
In [9]: df.memory_usage().sum()