How to Access AdForm API with R

AdForm is a danish digital advertising media company. It has advertiser/agency side and publisher side.

The post is about the publisher side, but most of the techniques are similar in the advertiser/agency (client) side.

We go through how to access the AdForm API and dowload the data with the help of R.

Background of the AdForm API

You van access to the data in several ways:

• Client Credentials Flow
• Authorization Code Flow
• Implicit Flow

This post focuses on the first flow, because this is suitable for the automated data download. Here you can read more about the flows – https://api.adform.com/help/guides/getting-started/authorization-guide

To use any of them you need an oAuth token. For the token, first you need a client id and a client secret. To get these, you have to register your app. Send an email to AdForm API support (technical@adform.com) and add the following information:

• short description of your application (Adform’s API use case)
• authorization flow
• a list of needed scopes for application.

The first one is easy, just give a short description 🙂

The authorization flow is listed above. In our case the right flow is Client Credentials. If the output of your app is not an automated report, you should chose another one.

The scopes are in our case the following:

• https://api.adform.com/scope/eapi
• https://api.adform.com/scope/api.publishers.readonly
• https://api.adform.com/scope/api.placements.read

If you would like to modify the setup, you should chose more scopes.

So if you registered your app (sent an email to AdForm and add the description) you’ll get shortly the client id, which is looks like an email and the client secret, which is a random string, just like almost any other secret 🙂

Let’s start the authetication flow. In order to get the data we should go through the following steps:

• Get the token with a POST request
• Extract the token from the response
• Add the header the token string
• Run the right (POST/GET/etc) request with the right paramterts

Basically you have to create 2 HTTP request. One for receive the toke, this step authenticate your second request. The second one contains the token and the parameters you would like to retrieve.

If you have the data, you have to format it so you and most importantly the language you write can process/understand the data.

AdForm uses JSON format. You have to send the requests in JSON encoded and the output of the request is JSON, too.

Let’s start and create generate the access toke with the help of R.

Get the AdForm Access Token

After you have described the porpuse of your app and specified the scopes you get shortly the client id and client secret.

Do not confuse that you get these information in this format:

PASSWORD DETAILS
Title: [the_titleAdForm_generates]
Username: [user_name_that_looks_like_an_email]
Password: [random_string_as_a_password]

The title is no important you can ignore it, it is just for easier identification.

The username is the client id and the password the clien secret.

The HTTP POST request should looks like this:

POST https://id.adform.com/sts/connect/token HTTP/1.1
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&
client_id=<CLIENT_ID>&
client_secret=<CLIENT_SECRET>&
scope=<SCOPES>

Only the last 3 lines have to be modified. The first 3 are fixed. If you use authorization flow other than Clien Credentials Flow the HTTP POST request changes.

The response should be:

{
  "access_token": "<ACCESS_TOKEN>",
  "expires_in": <EXPIRATION_TIME>,
  "token_type": "Bearer"
}

The response is in JSON and the most important line is the first one. Without the token you can not send and request without the token. It fails (without token) because it is not authorized.

Get the token with R

To receive the AdForm token the most useful library is the ‘httr’. It handle all the requests. So, let’s load the library.

library(httr)

If you haven’t installed yet:

install.packages('httr')

For more info read the documentation of the package: https://www.rdocumentation.org/packages/httr/versions/1.4.2

For the http request the httr package is enough. The response is JSON formatted which is easier to read with the jsonlite library:

install.packages('jsonlite')

library(jsonlite)

Use variables

After you have loaded the libs get the token. Save the client id, client secret and the scopes in a variable.

client_id_adform <- 'your_adform_clien_id'
client_secret_adform <- 'your_adform_client_secret'
scopes_adform <- 'https://api.adform.com/scope/eapi https://api.adform.com/scope/api.publishers.readonly https://api.adform.com/scope/api.placements.read'

If you have more than one scopes, just save it in a string with space between. httr can read it and add all the scopes to the post request.

Post request for the token

res <- POST('https://id.adform.com/sts/connect/token',
            encode = 'form',
            body = list(grant_type = 'client_credentials', 
                        client_id = client_id_adform,
                        client_secret = client_secret_adform,
                        scope = scopes_adform
            )
)

It is similar to the post request from AdFrom, but has some differencies.

The post request has 3 different part. It should be more, but here we only user 3:

• URL – this is the url you would like to POST, here it is https://id.adform.com/sts/connect/token
• encode – it specifies the content type. In httr form means application/x-www-form-urlencoded
• the body – this part of the request contains all the necessary info
  • grant_type – it is fixed, every AdForm API call (that uses the same flow) contain the same value: clien_credentials
  • client_id – from the variable you declared
  • client_secret – from the variable you declared
  • scope – from the variable you declared

The result

If everything works fine the result is a json file:

{
  "access_token": "<ACCESS_TOKEN>",
  "expires_in": <EXPIRATION_TIME>,
  "token_type": "Bearer"
}

The access token is the most important element here. The token expires in 1 hour you your have to refresh the token every hour (just run the command above). The token type is Bearer, it is fixed.

To get the access token from the json just use the jsonlite library to retrieve the data.

accessToken <- fromJSON(rawToChar(res[["content"]]))[["access_token"]]

The res variable contains the whole response, the json encoded information. So, inside the ‘content’ we would like to know the value of the access token.

First of all we have to unwrap the content of the response. It is raw, so we have to call ‘rawToChar’ to make it “readable”. ‘fromJson’ create a list about the readable json. Every element of the json switches to a list.

We got the list, just retrieve the access_token element.

This is the list:

listContainsToken <- fromJSON(rawToChar(res[["content"]]))

From the list we access to the access_token element:

theToken <- listContainsToken[["access_token"]]

The 2 code snippet above is the same that generates ‘accessToken’ in one step.

We have the token, now we can dowload the data.

Access to the data

Now we have the token, use it to get the data. We have to create a new post request. But first get all the available dimensions and metrics.

All dimensions

sortListArgs <- list(dimensions='')

rawAllDimensions <- POST('https://api.adform.com/v1/reportingstats/publisher/metadata/dimensions',
                         body = toJSON(sortListArgs, auto_unbox = TRUE),
                         add_headers(Authorization = paste('Bearer', accessToken, sep = " ") ),
                         content_type("application/json")
)

The URL of the post request is: ttps://api.adform.com/v1/reportingstats/publisher/metadata/dimensions.

The most important line is where we attach the access token to the request. It autenticates the whole process.

To get the data, you have to add the token as a header element. So we have to use ‘add_headers’ parameter in the post request. It is a Bearer type authorization token we use in the OAuth 2.0 header.

It means we have to add the word ‘Bearer’ and the toke we received together. The ‘paste’ helps to make it:

paste('Bearer', accessToken, sep = " ")

Don’t forget to add the ‘sep’ parameter with space.

The content type is json.

The body

The body contains the dimensions, metrics and the filters, all the information you would like to receive.

The format of the body is json. In this example, we would like to get all the available dimensions. In order to do it we have to send an “empty” json.

// To retrieve a list of all available dimensions
{
   "dimensions": [ ]
}

If you have the token and created the json like above, just execute the code and retrieve the dimensions.

To make the response readable for R you have to encode it:

dimensions_data_all <- fromJSON(rawToChar(rawAllDimensions[["content"]]))[["dimensionsMetadata"]]

All metrics

To get all the metrics you have to make an almost similar post request like before.

sortListArgs <- list(metrics='')

rawAllMetrics <- POST('https://api.adform.com/v1/reportingstats/publisher/metadata/metrics',
                      body = toJSON(sortListArgs, auto_unbox = TRUE),
                      add_headers(Authorization = paste('Bearer', accessToken, sep = " ") ),
                      content_type("application/json")
)

metrics_data_all <- fromJSON(rawToChar(rawAllMetrics[["content"]]))[["metricsMetadata"]]

Of course the url and the body changed but the method is same.

Get the data

Now, we have all the dimensions and metrics, tome to get the data we would like to use.

Here you can find the official documentation: https://api.adform.com/help/guides/how-to-manage-a-sell-side/reporting

Although the table said only the metrics and filters are required, without dimension(s) you get error. So to have a valid request and at least one dimension, metric and filter.

First of all, here it is the code:

rawYesterdayData <- POST('https://api.adform.com/v1/reportingstats/publisher/reportdata',
                         body = upload_file("script/yesterdayData.json")  ,
                         encode = "json",
                         add_headers(Authorization = paste('Bearer', accessToken, sep = " "), Accept = "application/json" ),
                         content_type("application/json"),
                         verbose()
)

yesterdayDataAdForm <- fromJSON(rawToChar(rawYesterdayData[["content"]]))
yesterdayDataAdForm

It is the similar to we saw before. It’s a post request, there is a body that contains the dimensions, metrics and filters (in JSON format).

We have bearer access token inside the header. The output is ‘applications/json’.

And of course the api endpoint, the url has changed. All the reporting related requests have to send to ‘https://api.adform.com/v1/reportingstats/publisher/reportdata’

We have a last item: verbose(). It gives us feedback about the request. Print to the console the data you send and receive during the post request. I use it, so I left it there, but if you don’t check the console, just delete it.

One last step

You have the data, now you have to convert it to a dataframe. First, lets check the yesterdayDataAdForm variable.

It contains few lists. What we would like to use here is the columnheaders or the columns >> key. These contain the header of the data frame we woul like to create.

The columns >> rows contains the actual data.

So the last step is to add header to the data (rows):

yesterDataDF <- yesterdayDataAdForm[["reportData"]][["rows"]] %>% as.data.frame
      
      colnames(yesterDataDF) <- yesterdayDataAdForm[["reportData"]][["columnHeaders"]]

The firs line creates the data frame based on the data we have in rows.

The second one gives yesterdayDataDF header based on the columnHeaders list. The output is a data frame you can use in R.

Now you have the data, just use it 🙂