F42 Change Proposal: NumPy 2 (System-Wide)

NumPy2

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Update the numpy package from 1.x to 2.x (1.26.4 to 2.1.3 currently).

:link: Owner

:link: Detailed Description

Updating numpy from the 1.x to 2.x series will Just Work for some packages. Those using the C API will need to be rebuilt. Additionally, many packages are pinned to numpy < 2, and these will need to be loosened.

:link: Feedback

I’ve experimented with this in copr, as well as tracked updates to a few packages in preparation for this. One example is scipy, which was recently updated to a version supporting numpy2, and this will reduce the number of impacted packages.

:link: Benefit to Fedora

In addition to tracking upstream and receiving more up-to-date fixes and security updates, numpy2 brings many improvements to the library:

https://numpy.org/devdocs/release/2.0.0-notes.html

:link: Scope

  • Proposal owners:

Update numpy2 package to 2.1.x in f43 immediately after f42 mass branching. Help rebuild and patch impacted packages as needed.

  • Other developers:

Rebuild and patch impacted packages. Most will only require a pin adjustment and/or rebuild.

  • Release engineering: #Releng issue number

  • Policies and guidelines: N/A (not needed for this Change)

  • Trademark approval: N/A (not needed for this Change)

  • Alignment with the Fedora Strategy:

:link: Upgrade/compatibility impact[edit]

Successfully rebuilt packages should not have any compatibility issues.

:link: Early Testing (Optional)

Do you require ‘QA Blueprint’ support? Y/N

:link: How To Test

Run applications that use numpy or one of it’s dependencies.

:link: User Experience

End users should not notice any changes. Python developers will need to rebuild C API code against numpy2, and can then take advantage of new features.

:link: Dependencies

apbs

APLpy

arbor

astrometry

asv

aubio

blender

boost

bout++

cantera

coin-or-HiGHS

Cython

DisplayCAL

dolfin

espresso

flatbuffers

gau2grid

gdal

gdl

ginga

gnuradio

gr-air-modes

grass

h5py

ipython

lammps

libarrow

libcint

libfreenect

libxc

mathgl

Mayavi

mlir

mlir17

mlpack

moose

mopac

morphio

mypaint

myst-nb

nanovna-saver

netcdf4-python

nfsometer

NLopt

ocaml-pyml

onnx

onnxruntime

opencv

openmeeg

openms

openshadinglanguage

openvdb

paraview

PDAL

petsc

pgzero

plplot

ProDy

psi4

pybind11

pygame

pygrib

pygsl

PyMca

pymol

pyplane

pyproj

pytest

python-adjustText

python-animatplot

python-annarchy

python-array-api-strict

python-arviz

python-ase

python-astroML

python-astroplan

python-astropy

python-astropy-healpix

python-astroquery

python-astroscrappy

python-autograd

python-basemap

python-bioframe

python-biopython

python-bioread

python-blosc

python-blosc2

python-bluepyopt

python-Bottleneck

python-boutdata

python-boututils

python-branca

python-brian2

python-brukerapi

python-cairocffi

python-calcephpy

python-cartopy

python-ccdproc

python-cclib

python-cftime

python-chaospy

python-cma

python-cmake-build-extension

python-cmap

python-collada

python-colormath2

python-colorspacious

python-contextualbandits

python-contourpy

python-cramjam

python-crick

python-cro

python-cyipopt

python-dask

python-deap

python-deepdiff

python-dijitso

python-dipy

python-distributed

python-ducc0

python-duecredit

python-earthpy

python-eccodes

python-efel

python-elasticsearch

python-elephant

python-emcee

python-ephyviewer

python-exdir

python-ezdxf

python-fastavro

python-fast-simplification

python-ffc

python-fiat

python-fiona

python-fireflyalgorithm

python-fisx

python-fitsio

python-folium

python-formulaic

python-fsleyes

python-fsleyes-props

python-fsleyes-widgets

python-fslpy

python-fsspec

python-geomdl

python-geopandas

python-geotiler

python-glymur

python-gradunwarp

python-graph-tool

python-gsd

python-h5io

python-hamcrest

python-hdf5storage

python-hdmf

python-healpy

python-hypothesis

python-igraph

python-imagecodecs

python-imageio

python-imbalanced-learn

python-img2pdf

python-indexed_gzip

python-inkex

python-inspyred

python-intern

python-joblib

python-jsonpickle

python-klusta

python-lazyarray

python-lazy-loader

python-lazy-ops

python-lfpy

python-lfpykit

python-libNeuroML

python-libpysal

python-llama-cpp-python

python-logdetective

python-lqrt

python-lsp-server

python-mapbox-earcut

python-mapclassify

python-matplotlib

python-meautility

python-meshio

python-missingno

python-mizani

python-mmtf

python-mne

python-mne-bids

python-mockito

python-mrcfile

python-multiecho

python-music21

python-nashpy

python-ndindex

python-neatdend

python-neo

python-netpyne

python-neurodsp

python-neurom

python-neurosynth

python-niaaml

python-niaarm

python-niaclass

python-niapy

python-nibabel

python-nifti-mrs

python-nilearn

python-nipy

python-nixio

python-nmrglue

python-nptyping

python-numcodecs

python-numexpr

python-numpoly

python-numpy-stl

python-openctm

python-opfunu

python-optking

python-optuna

python-opytimark

python-opytimizer

python-orjson

python-pandas

python-pandas-datareader

python-partd

python-periodictable

python-photutils

python-pillow

python-pingouin

python-pint

python-plotnine

python-probeinterface

python-pyABF

python-pyairvisual

python-pybbi

python-pybids

python-pybv

python-pydantic-core

python-pydapsys

python-pydicom

python-pyedflib

python-pyelectro

python-pyemd

python-pyerfa

python-pyface

python-pygmtools

python-pykdtree

python-pymapvbvd

python-pymatreader

python-pynn

python-pynwb

python-pyongc

python-pyopencl

python-pyopengl

python-pypet

python-pyphi

python-pyqtgraph

python-pyriemann

python-pyscf

python-pyshtools

python-pyside6

python-pyspike

python-pytest7

python-pytest-arraydiff

python-pytest-doctestplus

python-pytest-randomly

python-pytest-regressions

python-python-fcl

python-pytools

python-pyunicorn

python-pywt

python-pyxdf

python-qcelemental

python-quantities

python-qudida

python-rapidfuzz

python-rasterio

python-ratinabox

python-redis

python-reproject

python-Rtree

python-SALib

python-scikit-image

python-scikit-learn

python-scikit-misc

python-seaborn

python-sep

python-sgp4

python-shapely

python-simframe

python-sklearn-genetic

python-sklearn-genetic-opt

python-sklearn-nature-inspired-algorithms

python-smart_open

python-snuggs

python-sortedcontainers

python-sphinx-gallery

python-sport-activities-features

python-spyking-circus

python-stackprinter

python-statsmodels

python-steps

python-stochastic

python-succulent

python-superqt

python-supersmoother

python-tables

python-tabulate

python-textdistance

python-tifffile

python-tinygrad

python-toml

python-torch

python-torchvision

python-tqdm

python-Traits

python-traitsui

python-transforms3d

python-trimesh

python-trx-python

python-tvb-gdist

python-typish

python-uranium

python-vedo

python-vhacdx

python-wavio

python-wordcloud

python-wxpython4

python-xarray

python-xarray-einstats

python-yfinance

python-zarr

python-zmq

python-zoidberg

pythran

quantum-espresso

root

R-reticulate

scipy

setzer

shybrid

SoapySDR

spec2nii

spglib

stats-collect

sympy

uhd

urh

veusz

vigra

wult

xcfun

xtensor

xtensor-python

:link: Contingency Plan

  • Contingency mechanism: Depending on scope of packages not fixed, Gwyn will either help fix those packages, or create a compat package and patch packages to use that.
  • Contingency deadline: Beta Freeze
  • Blocks release? No

:link: Documentation

Links provided above to upstream release notes. Document compat package if applicable.

:link: Release Notes

Last edited by @amoloney 2024-11-12T22:03:29Z

Last edited by @amoloney 2024-11-12T22:03:29Z

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

Just to clarify: What is the role of the branching when you want to bring this to F42? Or is this a F43 change proposal? Or do you want to experiment on rawhide (F43) after F42 branched, then bring this to F42? Don’t think the latter :wink:

This is intended for f43.

2 Likes

@amoloney FYI

I assume you are planning to introduce numpy2 and have that following upstream releases for 2.x?

Has the reverse option, updating numpy to 2.x and introducing a compatibility package for NumPy 1.x, also been evaluated? The latter seems to be the more common approach.

I really think we should aim for F42. More and more actively maintained packages are supporting NumPy 2.x. Holding all those back until F43 or having to patch out numpy >= 2 in a lot of packages seems less ideal.

There likely will be a need for a compatibility package for NumPy 1.x for some time for various reasons. So, if we are going to have that anyway, I feel we may as well get the ball rolling for the NumPy transition as soon as possible and aim for a sunset date for the compatibility package.

Updating numpy. At first glance, a compat package would either need to implicitly conflict with numpy, or consuming packages would need all their imports patched.

I have no objection to f42, if FESCO is amenable.

1 Like

For the users like me, hearing the first time of NumPy:

NumPy (Numerical Python) is an open source Python library that’s widely used in science and engineering. The NumPy library contains multidimensional array data structures, such as the homogeneous, N-dimensional ndarray, and a large library of functions that operate efficiently on these data structures. Learn more about NumPy at What is NumPy, and if you have comments or suggestions, please reach out!

I freed the dependency list from blank lines using the “grep” command: grep -v '^$'

== Dependencies 357 Total ==
apbs
APLpy
arbor
astrometry
asv
aubio
blender
boost
bout++
cantera
coin-or-HiGHS
Cython
DisplayCAL
dolfin
espresso
flatbuffers
gau2grid
gdal
gdl
ginga
gnuradio
gr-air-modes
grass
h5py
ipython
lammps
libarrow
libcint
libfreenect
libxc
mathgl
Mayavi
mlir
mlir17
mlpack
moose
mopac
morphio
mypaint
myst-nb
nanovna-saver
netcdf4-python
nfsometer
NLopt
ocaml-pyml
onnx
onnxruntime
opencv
openmeeg
openms
openshadinglanguage
openvdb
paraview
PDAL
petsc
pgzero
plplot
ProDy
psi4
pybind11
pygame
pygrib
pygsl
PyMca
pymol
pyplane
pyproj
pytest
python-adjustText
python-animatplot
python-annarchy
python-array-api-strict
python-arviz
python-ase
python-astroML
python-astroplan
python-astropy
python-astropy-healpix
python-astroquery
python-astroscrappy
python-autograd
python-basemap
python-bioframe
python-biopython
python-bioread
python-blosc
python-blosc2
python-bluepyopt
python-Bottleneck
python-boutdata
python-boututils
python-branca
python-brian2
python-brukerapi
python-cairocffi
python-calcephpy
python-cartopy
python-ccdproc
python-cclib
python-cftime
python-chaospy
python-cma
python-cmake-build-extension
python-cmap
python-collada
python-colormath2
python-colorspacious
python-contextualbandits
python-contourpy
python-cramjam
python-crick
python-cro
python-cyipopt
python-dask
python-deap
python-deepdiff
python-dijitso
python-dipy
python-distributed
python-ducc0
python-duecredit
python-earthpy
python-eccodes
python-efel
python-elasticsearch
python-elephant
python-emcee
python-ephyviewer
python-exdir
python-ezdxf
python-fastavro
python-fast-simplification
python-ffc
python-fiat
python-fiona
python-fireflyalgorithm
python-fisx
python-fitsio
python-folium
python-formulaic
python-fsleyes
python-fsleyes-props
python-fsleyes-widgets
python-fslpy
python-fsspec
python-geomdl
python-geopandas
python-geotiler
python-glymur
python-gradunwarp
python-graph-tool
python-gsd
python-h5io
python-hamcrest
python-hdf5storage
python-hdmf
python-healpy
python-hypothesis
python-igraph
python-imagecodecs
python-imageio
python-imbalanced-learn
python-img2pdf
python-indexed_gzip
python-inkex
python-inspyred
python-intern
python-joblib
python-jsonpickle
python-klusta
python-lazyarray
python-lazy-loader
python-lazy-ops
python-lfpy
python-lfpykit
python-libNeuroML
python-libpysal
python-llama-cpp-python
python-logdetective
python-lqrt
python-lsp-server
python-mapbox-earcut
python-mapclassify
python-matplotlib
python-meautility
python-meshio
python-missingno
python-mizani
python-mmtf
python-mne
python-mne-bids
python-mockito
python-mrcfile
python-multiecho
python-music21
python-nashpy
python-ndindex
python-neatdend
python-neo
python-netpyne
python-neurodsp
python-neurom
python-neurosynth
python-niaaml
python-niaarm
python-niaclass
python-niapy
python-nibabel
python-nifti-mrs
python-nilearn
python-nipy
python-nixio
python-nmrglue
python-nptyping
python-numcodecs
python-numexpr
python-numpoly
python-numpy-stl
python-openctm
python-opfunu
python-optking
python-optuna
python-opytimark
python-opytimizer
python-orjson
python-pandas
python-pandas-datareader
python-partd
python-periodictable
python-photutils
python-pillow
python-pingouin
python-pint
python-plotnine
python-probeinterface
python-pyABF
python-pyairvisual
python-pybbi
python-pybids
python-pybv
python-pydantic-core
python-pydapsys
python-pydicom
python-pyedflib
python-pyelectro
python-pyemd
python-pyerfa
python-pyface
python-pygmtools
python-pykdtree
python-pymapvbvd
python-pymatreader
python-pynn
python-pynwb
python-pyongc
python-pyopencl
python-pyopengl
python-pypet
python-pyphi
python-pyqtgraph
python-pyriemann
python-pyscf
python-pyshtools
python-pyside6
python-pyspike
python-pytest7
python-pytest-arraydiff
python-pytest-doctestplus
python-pytest-randomly
python-pytest-regressions
python-python-fcl
python-pytools
python-pyunicorn
python-pywt
python-pyxdf
python-qcelemental
python-quantities
python-qudida
python-rapidfuzz
python-rasterio
python-ratinabox
python-redis
python-reproject
python-Rtree
python-SALib
python-scikit-image
python-scikit-learn
python-scikit-misc
python-seaborn
python-sep
python-sgp4
python-shapely
python-simframe
python-sklearn-genetic
python-sklearn-genetic-opt
python-sklearn-nature-inspired-algorithms
python-smart_open
python-snuggs
python-sortedcontainers
python-sphinx-gallery
python-sport-activities-features
python-spyking-circus
python-stackprinter
python-statsmodels
python-steps
python-stochastic
python-succulent
python-superqt
python-supersmoother
python-tables
python-tabulate
python-textdistance
python-tifffile
python-tinygrad
python-toml
python-torch
python-torchvision
python-tqdm
python-Traits
python-traitsui
python-transforms3d
python-trimesh
python-trx-python
python-tvb-gdist
python-typish
python-uranium
python-vedo
python-vhacdx
python-wavio
python-wordcloud
python-wxpython4
python-xarray
python-xarray-einstats
python-yfinance
python-zarr
python-zmq
python-zoidberg
pythran
quantum-espresso
root
R-reticulate
scipy
setzer
shybrid
SoapySDR
spec2nii
spglib
stats-collect
sympy
uhd
urh
veusz
vigra
wult
xcfun
xtensor
xtensor-python

Should I actualize the list in the main topic @limb ? Yours is twice as long and not in a pre formatted text block </> what makes it quite long to scroll.

But the topic title is F42.

However, I agree with @gui1ty; we should not delay this any further and target F42. NumPy themselves already went through and attempted to ensure the ecosystem was ready:

and I think that is sufficient for us, unless there are some important packages you are aware of that don’t work.

Additionally, wheels built against NumPy 2 work with NumPy 1, but not vice-versa. Many Python packages have thus pinned NumPy 2 as a build requirement, even though their runtime requirement is lower. Several Fedora package maintainers are now holding back updating because they think they need to wait for NumPy 2. Short of pointing this out in every case I come across, this would be far simpler if NumPy was just updated instead.

Please post it here, so we can see what the issues are.

1 Like

It seems the announcement of this change has not found its way to the devel mailing list (yet). I checked my mailbox as well as the archives. I notified infrastructure to check why and fix if possible.

This post is automatically generated from the wiki page (linked above). This topic is mainly meant to discuss the change itself. The formatting is less important as long as it’s readable. Your text block requires scrolling, which has its own drawbacks.

Or rather they are reluctant to drop the version restriction, falling back to NumPy 1.x. Honestly, if upstream has pinned it to >= 2, I think we should follow suite.

From my own testing a while ago, most issues are due to use of deprecated functions / methods that have now been dropped. Quite often upstream is either aware and working on it or has already released an update fixing the issue. Some are trivial replacements. The one package that stood out as failing with NumPy 2.x was boost. But I made no effort trying to analyze.

Of course, there is also quite some bootstrap entanglement in the dependency stack.

No, I am speaking about cases where upstream did not pin runtime to >=2; they pinned build requirements because they were producing wheels that they wanted to work in all cases. The runtime requirements were still >=1.x because NumPy made those ABI guarantees. This is how NumPy intended wheel compatibility to work:
https://numpy.org/doc/stable/dev/depending_on_numpy.html#numpy-2-0-specific-advice

I was speaking more broadly. What I get is the feedback from a scratch build telling me: Nothing provides python3dist(numpy) >= 2. That could be either case - build or install. But I’m reluctant to look into it.

Infrastructure was unable to find out why the message did not arrive on the devel list. It should have, since devel is subscribed to devel-announce. Infrastructure resubscribed devel to devel-announce hoping this might fix the problem. @amoloney could you resend or retrigger the announcement? Or would that trigger the entire process including FESCo and Discussion etc. again?

I’m perfectly fine with f42 if FESCO approves. I’d already deleted my test copr but have submitted a new one: limb/numpy Copr

No no, thats just a resend. The process started when the proposal was announced first. I’ll bump it on the devel list now. Thanks!

It looks like a good chunk of these were blocked by the pinning in pandas, and another large chunk fails tests because they were built with NumPy 2, but their other dependencies were built with NumPy 1, which can’t be mixed.

So while it does appear at first glance that the majority (218 / 355 = 61.4%) are failing, the true number may be much lower.

At least for the second case, a second build may fix those packages (if all their dependencies were successfully rebuilt), but I don’t know if there are any loops in there.

1 Like

This change proposal has now been submitted to FESCo with ticket #3295 for voting.

To find out more, please visit our Changes Policy documentation.