Get a list of every Layer, in every Service, in every Folder in an ArcGIS REST endpoint

I have two ArcGIS REST endpoints for which I am trying to get a list of every layer:

https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services https://services1.arcgis.com/RLQu0rK7h4kbsBq5/ArcGIS/rest/services

These are not my organization's endpoints so I don't have access to them internally. At each of these endpoints there can be folders, services, and layers, or just services and layers.

My goal is to get a list of all layers. So far I have tried:

endpoints=(["https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services",
"https://services1.arcgis.com/RLQu0rK7h4kbsBq5/ArcGIS/rest/services"])

for item in endpoints:
    reqs = requests.get(item, verify=False) 
    # used this verify because otherwise I get an SSL error for endpoints[0]
    soup =BeautifulSoup(reqs.text, 'html.parser')

    layers = []
    for link in soup.find_all('a'):
        print(link.get('href'))
        layers.append(link)
     

However this doesn't account for the variable nested folders/services/layers or services/layer schemas, and it doesn't seem to be fully appending to my layers list.

I'm thinking I could also go the JSON route and append ?f=psjon . So for example:

https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/?f=pjson would get me the folders https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/broadband/?f=pjson would get me all the services in the broadband folder and https://rdgdwe.sc.egov.usda.gov/arcgis/rest/services/broadband/CDC_5yr_OpioidOverDoseDeaths_2016/MapServer?f=pjson would get me the CDC_OverDoseDeathsbyCounty2016_5yr layer in the first service (CDC_5yr_OpioidOverDoseDeaths_2016) in the broadband folder.

Any help is appreciated. I put this here vs in the GIS stack exchange as it seems a more python question than geospatial.

from Recent Questions - Stack Overflow https://ift.tt/3G3xg1i
https://ift.tt/eA8V8J