I have an Ansible playbook that sets up a handful of servers. As part of the setup, it needs to configure the EPEL repo and install packages from it. However, when deploying on AWS, I get random “HTTP Error 403: Forbidden”, often on 3 out of 5 servers, which fails the entire play.
I suspect the downloads are rate-limited, even though I throttle the playbook to execute on one server at a time.
Is that the case, and if so, how can I overcome this?
I’m not sure about the details, but in your case, why don’t you setup some sort of local cache to avoid having to download the same thing multiple times?
I expected you are seeing failures because that depends on the mirror chosen.
Check the error message you get and see if the problem relates to a specific mirror.
The complete error messages is {"changed": false, "msg": "Failure downloading https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm, HTTP Error 403: Forbidden"}, no mention of a specific mirror unfortunately.
I’d prefer not to modify the workflow that used to work pretty recently. I’ll consider a mirror of course if it comes to that
Do you have a proxy or firewall between your code and the repo?
When this fails can you ssh into the instance and run curl -v on the URL in the error to get full details of the error?
I would expect the response status and headers to be informative.
Are the headers different when you do the same from your host?
dl.fedoraproject.org are our master mirrors.
There’s 3 servers in that group, and it turns out one of them was
throwing 401’s on access to those files. I have restarted it’s httpd and
it has the right config now and should be working again.
So, I think you will not see any more errors from it, but let me know if
you do.
Taking a look at things:
❯ dig dl.fedoraproject.org
; <<>> DiG 9.18.19 <<>> dl.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20518
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;dl.fedoraproject.org. IN A
;; ANSWER SECTION:
dl.fedoraproject.org. 135 IN A 38.145.60.24
dl.fedoraproject.org. 135 IN A 38.145.60.22
dl.fedoraproject.org. 135 IN A 38.145.60.23
;; Query time: 21 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Oct 02 13:34:36 EDT 2023
;; MSG SIZE rcvd: 97
Only the 38.145.60.24 host returns a 403. The other two return 200. I think may be an infrastructure ticket? Not sure.
Really, I don’t think it is recommended to use dl.fedoraproject.org for larger-scale use. It would be best to use one of the many mirrors and consider mirroring from those locally for your internal reuse.
Thanks for looking into this. Still fails (intermittently):
[cloud-user@ip-172-31-15-61 ~]$ curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to dl.fedoraproject.org port 443 (#0)
* Trying 38.145.60.22...
* Connected to dl.fedoraproject.org (38.145.60.22) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.fedoraproject.org,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* common name: *.fedoraproject.org
* issuer: CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dl.fedoraproject.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 02 Oct 2023 17:42:17 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Last-Modified: Sat, 04 Sep 2021 17:49:17 GMT
< ETag: "3cf8-5cb2f08ee02ec"
< Accept-Ranges: bytes
< Content-Length: 15608
< AppTime: D=523
< X-Fedora-AppServer: dl01.iad2.fedoraproject.org
< Content-Type: application/x-rpm
<
{ [data not shown]
100 15608 100 15608 0 0 68231 0 --:--:-- --:--:-- --:--:-- 69368
* Connection #0 to host dl.fedoraproject.org left intact
[cloud-user@ip-172-31-15-61 ~]$ curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to dl.fedoraproject.org port 443 (#0)
* Trying 38.145.60.24...
* Connected to dl.fedoraproject.org (38.145.60.24) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.fedoraproject.org,O="Red Hat, Inc.",L=Raleigh,ST=North Carolina,C=US
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* common name: *.fedoraproject.org
* issuer: CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1,O=DigiCert Inc,C=US
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dl.fedoraproject.org
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Mon, 02 Oct 2023 17:42:24 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Content-Length: 199
< Content-Type: text/html; charset=iso-8859-1
<
{ [data not shown]
100 199 100 199 0 0 969 0 --:--:-- --:--:-- --:--:-- 970
* Connection #0 to host dl.fedoraproject.org left intact
Thanks for looking into this. I’ll talk to the maintainers of our playbooks about using mirrors.
I’m looking at that 03 server now. ;(
If I can’t figure out whats going on with it, I will take it out of dns
at least. 
ok. dl03 should be fixed finally now.
Thanks for noticing the issue!
Thank you for your help. Works for me now too.
Hello!
It seems the issue is back, but this time consistently for me:
# curl -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -o /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 38.145.60.24:443...
* Connected to dl.fedoraproject.org (38.145.60.24) port 443 (#0)
* ALPN: offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2783 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [589 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted http/1.1
* Server certificate:
* subject: C=US; ST=North Carolina; L=Raleigh; O=Red Hat, Inc.; CN=*.fedoraproject.org
* start date: Jan 16 00:00:00 2023 GMT
* expire date: Jan 16 23:59:59 2024 GMT
* subjectAltName: host "dl.fedoraproject.org" matched cert's "*.fedoraproject.org"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G3 TLS ECC SHA384 2020 CA1
* SSL certificate verify ok.
* using HTTP/1.1
} [5 bytes data]
> GET /pub/epel/epel-release-latest-7.noarch.rpm HTTP/1.1
> Host: dl.fedoraproject.org
> User-Agent: curl/8.0.1
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 403 Forbidden
< Date: Thu, 05 Oct 2023 10:44:21 GMT
< Server: Apache
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< Content-Security-Policy: default-src 'none'; img-src 'self'
< Strict-Transport-Security: max-age=31536000; preload
< Content-Length: 199
< Content-Type: text/html; charset=iso-8859-1
<
{ [199 bytes data]
100 199 100 199 0 0 542 0 --:--:-- --:--:-- --:--:-- 543
* Connection #0 to host dl.fedoraproject.org left intact
It appears 2 of the servers are reporting 403 forbidden:
# dig dl.fedoraproject.org
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.1 <<>> dl.fedoraproject.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53025
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dl.fedoraproject.org. IN A
;; ANSWER SECTION:
dl.fedoraproject.org. 300 IN A 38.145.60.24
dl.fedoraproject.org. 300 IN A 38.145.60.22
dl.fedoraproject.org. 300 IN A 38.145.60.23
;; Query time: 32 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Thu Oct 05 11:59:40 UTC 2023
;; MSG SIZE rcvd: 97
# curl --resolve dl.fedoraproject.org:443:38.145.60.24 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 403 Forbidden
Date: Thu, 05 Oct 2023 12:00:40 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1
# curl --resolve dl.fedoraproject.org:443:38.145.60.23 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 403 Forbidden
Date: Thu, 05 Oct 2023 12:00:51 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1
Only 22 is responding correctly:
# curl --resolve dl.fedoraproject.org:443:38.145.60.22 -I -X GET https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
HTTP/1.1 200 OK
Date: Thu, 05 Oct 2023 12:00:45 GMT
Server: Apache
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Content-Security-Policy: default-src 'none'; img-src 'self'
Strict-Transport-Security: max-age=31536000; preload
Last-Modified: Sat, 04 Sep 2021 17:49:17 GMT
ETag: "3cf8-5cb2f08ee02ec"
Accept-Ranges: bytes
Content-Length: 15608
AppTime: D=511
X-Fedora-AppServer: dl01.iad2.fedoraproject.org
Content-Type: application/x-rpm
Here is the relevant infrastructure issue, which was resolved earlier today.