I have an Ansible playbook that sets up a handful of servers. As part of the setup, it needs to configure the EPEL repo and install packages from it. However, when deploying on AWS, I get random “HTTP Error 403: Forbidden”, often on 3 out of 5 servers, which fails the entire play.
I suspect the downloads are rate-limited, even though I throttle the playbook to execute on one server at a time.
Is that the case, and if so, how can I overcome this?
td211
(td211)
October 2, 2023, 4:11pm
2
I’m not sure about the details, but in your case, why don’t you setup some sort of local cache to avoid having to download the same thing multiple times?
I expected you are seeing failures because that depends on the mirror chosen.
Check the error message you get and see if the problem relates to a specific mirror.
The complete error messages is {"changed": false, "msg": "Failure downloading https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm, HTTP Error 403: Forbidden"}
, no mention of a specific mirror unfortunately.
I’d prefer not to modify the workflow that used to work pretty recently. I’ll consider a mirror of course if it comes to that
Do you have a proxy or firewall between your code and the repo?
When this fails can you ssh into the instance and run curl -v on the URL in the error to get full details of the error?
I would expect the response status and headers to be informative.
Are the headers different when you do the same from your host?
kevin
(Kevin Fenzi)
October 2, 2023, 5:34pm
7
dl.fedoraproject.org are our master mirrors.
There’s 3 servers in that group, and it turns out one of them was
throwing 401’s on access to those files. I have restarted it’s httpd and
it has the right config now and should be working again.
So, I think you will not see any more errors from it, but let me know if
you do.
3 Likes
maztaim
(Tim Bosse)
October 2, 2023, 5:43pm
8
Taking a look at things:
❯ dig dl.fedoraproject.org
; <<>> DiG 9.18.19 <<>> dl.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20518
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;dl.fedoraproject.org. IN A
;; ANSWER SECTION:
dl.fedoraproject.org. 135 IN A 38.145.60.24
dl.fedoraproject.org. 135 IN A 38.145.60.22
dl.fedoraproject.org. 135 IN A 38.145.60.23
;; Query time: 21 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Oct 02 13:34:36 EDT 2023
;; MSG SIZE rcvd: 97
Only the 38.145.60.24 host returns a 403. The other two return 200. I think may be an infrastructure ticket ? Not sure.
Really, I don’t think it is recommended to use dl.fedoraproject.org for larger-scale use. It would be best to use one of the many mirrors and consider mirroring from those locally for your internal reuse.
1 Like
Thanks for looking into this. Still fails (intermittently):
[cloud-user@ip-172-31-15-61 ~]$ curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to dl.fedoraproject.org port 443 (#0)
* Trying 38.145.60.22...
* Connected to dl.fedoraproject.org (38.145.60.22) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.fedoraproject.org,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* common name: *.fedoraproject.org
* issuer: CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dl.fedoraproject.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 02 Oct 2023 17:42:17 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Last-Modified: Sat, 04 Sep 2021 17:49:17 GMT
< ETag: "3cf8-5cb2f08ee02ec"
< Accept-Ranges: bytes
< Content-Length: 15608
< AppTime: D=523
< X-Fedora-AppServer: dl01.iad2.fedoraproject.org
< Content-Type: application/x-rpm
<
{ [data not shown]
100 15608 100 15608 0 0 68231 0 --:--:-- --:--:-- --:--:-- 69368
* Connection #0 to host dl.fedoraproject.org left intact
[cloud-user@ip-172-31-15-61 ~]$ curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to dl.fedoraproject.org port 443 (#0)
* Trying 38.145.60.24...
* Connected to dl.fedoraproject.org (38.145.60.24) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.fedoraproject.org,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* common name: *.fedoraproject.org
* issuer: CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dl.fedoraproject.org
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Mon, 02 Oct 2023 17:42:24 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Content-Length: 199
< Content-Type: text/html; charset=iso-8859-1
<
{ [data not shown]
100 199 100 199 0 0 969 0 --:--:-- --:--:-- --:--:-- 970
* Connection #0 to host dl.fedoraproject.org left intact
maztaim
(Tim Bosse)
October 2, 2023, 5:44pm
10
I’m sooo slow…Thanks @kevin !
Thanks for looking into this. I’ll talk to the maintainers of our playbooks about using mirrors.
kevin
(Kevin Fenzi)
October 2, 2023, 6:02pm
12
I’m looking at that 03 server now. ;(
If I can’t figure out whats going on with it, I will take it out of dns
at least.
kevin
(Kevin Fenzi)
October 2, 2023, 6:23pm
13
ok. dl03 should be fixed finally now.
Thanks for noticing the issue!
3 Likes
Thank you for your help. Works for me now too.
luislavena
(Luis Lavena)
October 5, 2023, 11:02am
15
Hello!
It seems the issue is back, but this time consistently for me:
# curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 38.145.60.24:443...
* Connected to dl.fedoraproject.org (38.145.60.24) port 443 (#0)
* ALPN: offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2783 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [589 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
* subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.fedoraproject.org
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* subjectAltName: host "dl.fedoraproject.org" matched cert's "*.fedoraproject.org"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
* SSL certificate verify ok.
* using HTTP/1.1
} [5 bytes data]
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> Host: dl.fedoraproject.org
> User-Agent: curl/8.0.1
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 403 Forbidden
< Date: Thu, 05 Oct 2023 10:44:21 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Content-Length: 199
< Content-Type: text/html; charset=iso-8859-1
<
{ [199 bytes data]
100 199 100 199 0 0 542 0 --:--:-- --:--:-- --:--:-- 543
* Connection #0 to host dl.fedoraproject.org left intact
luislavena
(Luis Lavena)
October 5, 2023, 12:01pm
16
It appears 2 of the servers are reporting 403 forbidden:
# dig dl.fedoraproject.org
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.1 <<>> dl.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53025
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dl.fedoraproject.org. IN A
;; ANSWER SECTION:
dl.fedoraproject.org. 300 IN A 38.145.60.24
dl.fedoraproject.org. 300 IN A 38.145.60.22
dl.fedoraproject.org. 300 IN A 38.145.60.23
;; Query time: 32 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Thu Oct 05 11:59:40 UTC 2023
;; MSG SIZE rcvd: 97
# curl --resolve dl.fedoraproject.org:443:38.145.60.24 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 403 Forbidden
Date: Thu, 05 Oct 2023 12:00:40 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1
# curl --resolve dl.fedoraproject.org:443:38.145.60.23 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 403 Forbidden
Date: Thu, 05 Oct 2023 12:00:51 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1
Only 22 is responding correctly:
# curl --resolve dl.fedoraproject.org:443:38.145.60.22 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 200 OK
Date: Thu, 05 Oct 2023 12:00:45 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Last-Modified: Sat, 04 Sep 2021 17:49:17 GMT
ETag: "3cf8-5cb2f08ee02ec"
Accept-Ranges: bytes
Content-Length: 15608
AppTime: D=511
X-Fedora-AppServer: dl01.iad2.fedoraproject.org
Content-Type: application/x-rpm
Here is the relevant infrastructure issue, which was resolved earlier today.
https://pagure.io/fedora-infrastructure/issue/11564
1 Like