Using PyArrow dtypes in Python Pandas isn’t always faster: %timeit s_pyarr.mean() # 21.5ms%timeit s_np.mean() # 47.3ms %timeit s_pyarr.nlargest(10) # 667ms%timeit s_np.nlargest(10) # 663ms The post PyArrow dtypes in Pandas aren’t always faster than NumPy appeared first on LernerPython.

PyArrow strings in Python Pandas are smaller than Python strings. But they’re also far faster: %timeit s_pyarr.str.len() # 106 µs%timeit s_py.str.len() # 1.6 ms In many examples, PyArrow was far […] The post PyArrow strings in pandas: Faster and smaller than object dtype appeared first on LernerPyth...

Want PyArrow dtypes in your Python Pandas data frame? The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones. Note: This is still experimental… but it’s also the […] The post PyArrow dtypes in pandas: Reading CSVs with dtype_backend appeared first on LernerPyt...

Using Python Pandas 3? Strings use PyArrow, not Pandas 2’s Python strings (dtype “object”): The post Pandas 3 string dtype: PyArrow strings cut memory use appeared first on LernerPython.

In Python Pandas 3, you can use PyArrow dtypes — which are nullable: What is s? 0 101 202 <NA> # pd.NA, not np.nan3 40dtype: int64[pyarrow] # see? Not float! The post PyArrow dtypes in Pandas 3: Missing values without float conversion appeared first on LernerPython.

Assign either np.nan or pd.NA to a Python Pandas series with a NumPy dtype, and it’ll be np.nan, a float. Which means the entire series has a dtype of float: […] The post Pandas missing values: Why assigning pd.NA gives you float64 appeared first on LernerPython.

If your dtype is too small, operations on your Python Pandas series will fail: Returns: 0 1101 1202 -126 # 🤯dtype: int8 This does give an error: s + 500OverflowError: […] The post Pandas int8 overflow: Why small dtypes wrap around silently appeared first on LernerPython.

Reading a CSV into a Python Pandas data frame? Use the “dtype” keyword arg and a dict to specify dtypes, and avoid the int64/float64/str defaults: The post Pandas read_csv dtype: Setting column types to save memory appeared first on LernerPython.

Want to change the dtype of a Python Pandas series? You can’t — at least, not by assigning: Instead, use “astype” to get a new series, and replace the old […] The post Changing a Pandas column dtype: Use astype, not assignment appeared first on LernerPython.

Get the dtype of a Python Pandas series with dtype: Get the dtypes of all columns in a data frame with “dtypes”, which returns a series: df.dtypes How many of […] The post Pandas dtypes: Checking column types with dtype and value_counts appeared first on LernerPython.

Want to get a quick look at a Python Pandas data frame you just created? Option 1, use df.head(n) to look at the first n rows: Option 2, use df.sample(n) […] The post Pandas data frame preview: df.head and df.sample appeared first on LernerPython.

If you want to get better at Pandas, the hard part isn’t finding tutorials. It’s finding problems worth solving. Most exercises hand you a tidy little table of five rows […] The post Free real-world Pandas exercises, with solutions appeared first on LernerPython.

Want to read a zipped CSV file into a Python Pandas data frame: Just pass the filename to read_csv: The file can contain a single CSV file, with an extension […] The post Pandas read_csv reads zipped CSV files without unzipping appeared first on LernerPython.

What sheets are in an Excel document you’re about to read into Python Pandas? You can check with: You’ll get a list of Python strings back. The post Excel sheet names in Pandas: Listing them with pd.ExcelFile appeared first on LernerPython.

Which is faster in Python Pandas, | or isin? I prefer isin; it’s clearer to write and read. Plus, fewer worries about parentheses. But is it faster? Depends on the […] The post Pandas isin vs | for filtering: which is faster appeared first on LernerPython.

Want rows in a Python Pandas data frame that might have several values? Another way (besides the | I showed yesterday) is the “isin” method: We’ll compare isin vs. | […] The post Pandas isin: Filtering rows that match multiple values appeared first on LernerPython.

Want rows in a Python Pandas data frame that might have several values? You can use the | operator, but be sure to use () to avoid precedence issues: The post Pandas OR conditions: Filtering rows with | and parentheses appeared first on LernerPython.

Stacking loc to filter a Python Pandas dataframe? Order can matter: The post Pandas loc filter order: Put the most selective filter first appeared first on LernerPython.

Filtering rows in a Python Pandas data frame? Use .loc: pd.col refers to the previous line’s returned data frame. So we can stack them: The post Filtering rows in Pandas with .loc and pd.col appeared first on LernerPython.

How much memory does a Python list use? sys.getsizeof will tell you, sort of: It reports the memory used by the list, but not its elements: The post Python list memory: Why sys.getsizeof ignores the elements appeared first on LernerPython.

Want stderr in your Python program to go somewhere else? Use redirect_stderr in contextlib: The post Redirecting stderr in Python: Using contextlib.redirect_stderr appeared first on LernerPython.

Writing a bunch to non-stdout in Python? You might want to use redirect_stdout from contextlib: The post redirect_stdout in Python: Sending print output to a file appeared first on LernerPython.

Want to print to stderr, not stdout, in a Python program? Use sys.stderr and the “file” kwarg: stdout and stderr often look identical in a terminal — but they aren’t! The post Python stderr: Printing with sys.stderr and the file kwarg appeared first on LernerPython.

Normally, “print” in Python writes to stdout. Write elsewhere with the “file” keyword argument: The post Python print to a file: Using the file keyword argument appeared first on LernerPython.

Want stdout from your Python program to go elsewhere? Assign sys.stdout to a writeable file. (Don’t forget to keep the original around!) Better: print’s file kwarg. The post Redirecting sys.stdout: Assigning print output to a file appeared first on LernerPython.

After you import a Python module, its repr (printed representation) shows its name and the loaded file. But some modules don’t have files; they were “frozen” into Python. They have […] The post Python frozen modules: Why they have no __file__ attribute appeared first on LernerPython.

Once you import a Python module, a second import won’t reload it. That’s usually good. But what if you want to? (Common in development and debugging.) This forces the reload. The post importlib.reload: Reimporting a Python module during development appeared first on LernerPython.

You import a Python module with the variable you’ll define, not a filename. “import mymod” tells Python to find mymod.py in each dir in sys.path. The first directory is ” […] The post Python import: How a module name becomes a file on sys.path appeared first on LernerPython.

Want Python to import modules in non-default dirs? You could do this: But: It doesn’t scale across many programs. And what if the dir changes? Better: Set the PYTHONPATH environment […] The post PYTHONPATH: A scalable alternative to sys.path.insert appeared first on LernerPython.

sys.path tells Python where to look when importing a module. Think of it as a list of directories. But it also handles zipfiles. A zipfile in sys.path is treated as […] The post sys.path supports zipfiles: Importing modules from an archive appeared first on LernerPython.