jeudi 26 mars 2020

Appending dataframes using a conditional loop

I have a number of data tables that I am pulling from a REST API using the GET function in the httr package.

I am having no issues pulling my tables with the following code:


Tables <- c("table1",
            "table2",
            "table3",
            "table4")


for(i in Tables) {

Odata_Request_Variables <- c("Var1","Var2","Var3")

Response <- GET(paste0(`url`,`i`,"?$select=",Odata_Request_Variables),
                  authenticate(username, password, "ntlm"),
                  add_headers(.headers = c('Prefer' = 'odata.include-annotations="OData.Community.Display.V1.FormattedValue"')))


  DF <- rawToChar(Response$content) %>%
    jsonlite::fromJSON(., flatten = TRUE) %>%
    as.data.frame(.)

What I get in the end is dataframe called DF.

The issue I am having is that the API limits retrieval of 5000 records and some of my tables exceed that limit. If there are more then 5000 records, a variable is generated in the dataframe called X.odata.nextLink. This contains a url to pass back to the API to retrieve the next 5000 records.

What I need to be able to do is create a loop that looks for the existence of that variable and if it exists to extract the first row (which contains the encoded url), so that I can pass it back into a GET request to retrieve another dataframe. I will then be able to append them together for my complete table.

I am at a loss how to do this, so any help would be greatly appreciated.

Aucun commentaire:

Enregistrer un commentaire