Import Event-Data to BigQuery using Server Side Google Tagmanager | Part 1

In this guide I will show, how to send data from any client or server to Server Side Google Tagmanager and handle the incoming data, so that it can be written to a BigQuery table immediately.
Yes I know, there are many ways to import data to Google Cloud BigQuery. And I am also aware, that this topic has been around quite some time now.
Generally speaking, the advantage of developing a REST endpoint to write data to BigQuery is obvious: you can handle multiple data streams in one sGTM Container while your developers to not have to study complex documentations for other APIs or even GA4 Measurement Protocol.

While Measurement Protocol offers a significant amount of advantages for cases in which you want to send additional data to GA4 directly, there is no real benefit in using MP, if you just want to send data to your server container and do other stuff with it. In this case, complying with MP schema is basically over-engineering a simple data stream.

Let me give you an example:
Your company’s website has a monitoring system (like Instana), that checks availability to several services (e.g. is the checkout available).
So in order to have up-to-date data for your checkout and thus being able to give context on dropping conversion-rates in your reports and dashboards, you want the availability data in your BigQuery Data Warehouse. You could do that by querying the monitoring tool’s API (in case it has one). But that would require you to query the API extremely often and deduplicate every incident in a probably complex way.
Another possibility is to use in-built webhooks, which most of these monitoring tools have. These allow you to create a request every time a specific incident condition is met.
For these you could use MP – but what would you want with arbitrary client_id, session_ids and so on, when you only want data on a technical incident.
Instead, you can build your own sGTM Client, that claims incoming requests, when the conditions, that you configured are met.

Here is an example on what this Client could look like (please find the .tpl-file in my GitHub):

const claimRequest = require('claimRequest');
const getRequestHeader = require('getRequestHeader');
const getRequestPath = require('getRequestPath');
const logToConsole = require('logToConsole');
const returnResponse = require('returnResponse');
const setResponseBody = require('setResponseBody');
const setResponseHeader = require('setResponseHeader');
const setResponseStatus = require('setResponseStatus');
const getRequestBody = require('getRequestBody');
const JSON = require('JSON');
const getType = require('getType');
const runContainer = require('runContainer');

const requestPath = getRequestPath();
const secHeader = getRequestHeader('sgtm_auth');

//function to build response
const sendResponse = (response, headers, statusCode) => {
    setResponseStatus(statusCode);
    setResponseBody(response);
    for (const key in headers) {
        if (['expires', 'date'].indexOf(key) === -1) setResponseHeader(key, headers[key]);
    }
    returnResponse();
};

//check for request path to claim
if (requestPath === data.path) {
    claimRequest();
    const header_success = {'status':'success'};
    const header_failed = {'status':'failed'};
    //check for security header
    if (secHeader === data.sec_header) {
        let eventContainer = [];
        const requestBody = getRequestBody();
        //logic to handle single or bulk object import
        const event = (getType(requestBody) == "object" || getType(requestBody) == "array")?requestBody:JSON.parse(requestBody);
        if (getType(event) === "array") {
            eventContainer = event;
        } else {
            eventContainer.push(event);
        }
        eventContainer.forEach((events) => {
            events.event_name = "generic_event";
            runContainer(events, () => logToConsole(events));
        });
        sendResponse("Data Ingestion was successful", header_success, 200);
    } else {
        sendResponse("Authentication for data ingestion failed", header_failed, 401);
    }
}

This client checks for both the request path and a request header (in this case “sgtm_auth”) to align with the configurations you made, when setting up the template.
If both are met, the client parses the request body, checks for single or bulk event import (JSON vs. Array) and runs the container.

This way you would only need to tell your developers the sGTM’s endpoint (“subdomain.mydomain.com”), the request path (e.g.: “/generic-data-import”) as well as the required request header (e.g.: “sgtm_auth”) and they’d be able to send you all the incident data, that you need.

I hope this was an interesting read for you. In the second part (coming soon), I will try to show you, how to import the data, claimed by the above client, to BigQuery.