What’s the best way to ship a python code to run as a background service on Linux (CentOS/RHEL)

Ask Fedora
, ,
1

Hi all,

I’ve developed a Python-based background service that performs periodic data uploads and file processing. It’s built with multiple modules/packages and runs persistently in the background on CentOS/RHEL systems.

Adding additional info:

I’m using py3.8 and my service should run as root to track some activities.
And, some of our Rhel/centos servers doesn’t support py3.8 by default, it can install upto py3.6 using package managers, so we are manually installing python3.8 using the zip files and setting up the env for testing it.

We would like to deploy it as a bg service which should run always, what’s your recommendation ?

Question: What is the recommended approach for packaging and deploying a Python code on CentOS/RHEL?

If I use the RPM build file (which is my Python package), do I still need to set up the Python environment on the target machine? To run the RPM file?

AFAIK, it’s Yes, can it be installed and used as a standalone service? That is, I don’t want to perform any Python env setup on my target machine, move the build file, install and run it

2

Welcome! I recommend you come over here.

What versions of rhel/centos are you targeting?

Is your code pure python?
What python 3rd party packages do you need?
Are they available from EPEL?

For background; I run a number of python programs as services on my systems.
In production I have supported large scale deployments on Centos with python code in RPMs.

3

Hey Barry,

We are targeting Rhel7.x and cent os 4.x

Yes, it only contains python (No C/C++).The python version is 3.8 for Rhel Systems, and for centos it might be either 3.4 or 2.7 (we are yet to decide foe cent os)

3rd party packages: Requests, Pydantic, stevedore

I’m not sure whether these packages are available in EPEL (I have to Google it).

4

You are saying that you setup the python env and run your rpms in it even in large scale deployment?

The problem for me is,

  • A few systems don’t have internet connections.
  • And, Rhel 7.x pkg manager has python up to 3.6, But our code is developed using py3.8, If we go with RPM route, then we have to create a venv with py3.8 to run the service which is complex. Am I correct?
  • That’s why, I’m looking for a option to deploy the python code as a single file to the target machines and install that package and run it
5

We poackaged everything as RPMs and installed into site-packages for packages and into app folder for the apps themselves. No need for venv’s.

RHEL 7 has 3.8 and 2.7 I recall.

RHEL 4! I have not seen as old a system. It was bad enough on Centos6.
I’m guessing you can only realistically use python 2.7.
But does RHEL 4 have 2.7? Or is it an older python 2?

Build for unsupported RHEL is going to be hard.

There is a mock build env for RHEL 7, but all the others are 8+.

Does you organisation have a plan to get on to supported OS?

There was a requests for 2.7 not sure about Pydantic, stevedore.

If you are connecting to the internet then using old verison of python and requrest may put your system at risk.

6

Oh okay, So you install your rpm file into the existing python’s site-packages and use it.

7

Let me ask a different question:

Assume you have a py package developed usig py3.8, but your rhel has py3.6, What would you do to use your py3.8 package without modifying it? And, Your system doesn’t have internet connection

Will you deploy it as a application? Will building the package with PyInstaller/pex be helpful?

8

Or, are you suggesting to deploy the python package in the OS py site-pacakges?

If yes, why?

I’m new to this type of distribution on Linux system:(
Kindly share your recommendations

9

depending on the package it may work. But if the package depends on feaures in 3.8 then you will need to patch it.

No. On linux this is not useful to do in a case like yours.
You have python installed and working so you just need to run your apps code as .py files.

its the easy way to use packages. ANd you will only need to install into site-packages becuase there is not an existing package already installed.

So there is not reason not to do this.

Your biggest issue is that you have an OS that is very old. Especially the RHEL 4 ones.

10

The alternative is to put all your code into /usr/local/share/ or /opt/.
Then use a shell script to setup PYTHONPATH and start your APP code.

11

Thanks Barry, for the explanation

Sure, I’ll go with deploying python packages itself in a separate environment,
These are the steps I’m going to do:

  1. create whl files for all my packages
  2. zip all the packages including requests, pydantic and so on (as the server doesn’t have a internet connection)
  3. one run_my_app.sh file to run my python package
  4. one my_app.service file
  5. create a install.sh file which performs:
  • mkdir in usr/local/my_app
  • create venv
  • installing all the py dependencies in that venv
  • moving my service file to /etc/systemd/system
  • move this run_my_app.sh file to /etc/profile
  • changing the permissions of sh file
  • systemctl reload , start and enable my service

I hope this works!

12

To use venv and the packages like requests don’t you need a python 3?
I had assumed that you cannot get a python 3 for REHL 4.

13

Oh, I was talking about Rhel7.x, not older versions.

For older OS versions, CentOS4.x, we are planning to build the same package in py2.

Let’s focus on Rhel7.x versions, not older versions!

But thanks! Barry, I’ll use the above approach in older versions of OS (maybe except venv ..)

14

Good idea get the RHEL 7’s done as they are the easier to solve.

15

So, still the same approach?
No plan to package it as app, is it?

NOTE: No internet connection for RHEL7 as well

16

Usually you get as much of the packages from RHEL and EPEL then
add your code. The “app” is not that much use on a unix system.

It’s something that I would use to target macOS or Windows,
but not RHEL.

1 Like
17

Maybe, I share a scenario,
Our App is a background service which must run all the time eventhough the python3 is uninstalled accidentally, Or, if someone upgrades the python3 version,

If we go with source code approach, this scenario will fail, right?
How to mitigate this scenario?

18

The issue you have seems to be social not technical.
Why are you allowing people to have privs to remove software required to run the system?

I am assuming you are working for a business to implement this feature.
Surely you have rules about who can and cannot admin systems?

At the end of the day if you have a user with admin they can defeat any technical measure you build.

19

If i have the priviliges or if the systems are owned by our company, then I would have upgraded the OS and just run it as a py program itself.
Since We are providing service to other companies, that’s where the issue is, they aren’t willing to upgrade their OS to the latest one :frowning:
That’s why, we are planning to deliver it with what they currently have

20

I see why you are thinking about the self contained app deployment.

Maybe you create a venv that has copies and not symlinks is the way to go?
Then deploy the whole venv as one blob.

In which case I would also put the system unit files into the blob and add a script to do the setup of systemd from the unit files in the venv.