Why Call A Webhook?

In my previous blog post (Start a Qlik Sense Cloud Reload from a Webhook) I described how to create a webhook using Qlik Automations which can start a reload of any app in your Qlik Cloud tenant. One of the places you might want to trigger this from is from a Qlik load script. This way you can call a reload of one or more Sense apps at the end of another script, perhaps dependant on data or status.

Triggering a Qlik Automation is one example of why you might want to fire a webhook, but in this post I want to share another example – raising an alert when things are broken.

How Do You Notify When Your Notifications Are Broken?

Something we always set up when building customer environments is sufficient notifications to make us aware when anything is wrong. On Qlik Sense Client Managed this is essential, as there is very little out of the box, and on Cloud it needs to be set up space by space, so some things can be missed. Sometimes we are working on a secure site where SMTP relaying from within the site is not permitted. Other times it can be that communication between the Qlik server and the SMTP server is down due to network issues, so the route to let us know there is an issue is broken along with the rest of the network. In these cases we need a way to get a message out, when nothing else is working.

Something that is likely to work in Sense, even when all else is failing, is calling a URL to extract data. If that URL is a webhook then that call can then trigger an action. What we have is a an applet set up in IFFTT (If This Then That) which sends an email via GMail containing values from the parameters of the webhook called. I won’t go into setting up that applet here, as it will be slightly different if you want to use a different mail client or call a different service, but I have given details on this in a previous blog post on Inserting Qlik Capacity Usage Into a Spreadsheet.

Once you have a webhook set up that can send the notification you are ready to build an app that triggers that webhook when there is a problem.

Raise a Notification When Files Go Stale

The most basic measure of everything being up and working is that files that are used by your Qlik environment are all up to date. If the applications you wish to monitor do not currently create any files then it is a simple job to add a STORE statement to write a small text file that can be checked for. The application we shall build here then has an inline table (which will work even if all else is broken) with a list of file locations and how old those files can go before we want to be told about them.

Before building the load script for this app you will need to set up a REST connector that we can pass parameters in to. To do this build a new REST connector, set it to a placeholder URL (I tend to use https://jsonplaceholder.typicode.com/posts), set the type to GET, ensure that Allow WITH CONNECTION is ticked and give it a name of Generic GET.

Once that connection is in place you can build the load script.

First you will need to set up some variables for use later in the script. Obviously the webhook parameter will need to be changed to match the webhook that you want to call.

let vEnv = 'My Qlik Sense Tenant';
let vLib = 'DataSpace:';
let vGET = '$(vLib)Generic GET';
let vQVDs = 'lib://$(vLib)DataFiles/';
let vTimestamps = '$(vQVDs)Timestamp/';
let vWebhook = 'https://maker.ifttt.com/trigger/error/with/key/your-webhook-key-here';

let vLastLoad = now();

Then you need to build the table of files that you want to be checking. These could be QVD files or small files that are written as part of a load script to indicate that the load script has completed. The table requires the full path to each file to be checked, how many hours old the file can get before an alert is raised and what label is to be used in the alert when that file becomes old.

tmp:
LOAD
    File,Threshold,Name
INLINE [
File,Threshold,Name
$(vQVDs)Sales.qvd,1,Sales Application
$(vTimestamps)QVDGenerateCompleted.csv,3,QVD Generation Application
];

That inline table can then be transformed to convert the hour given in the load script into a time delta to come off the current time, which can then be taken off the current time to work out the time that the file should be newer than. The actual date of the file also needs to be looked up. These two dates can then be compared to find the delta and to flag whether the file is old.

FileAges:
LOAD
    *,
    if(FileDate < Target, 'Yes', 'No') as Old,
    num((24)*(FileDate - Target), '0.00') as Difference
    ;
LOAD
    File,
    Threshold,
    Name,
    Date(alt(ConvertToLocalTime(FileTime(File), 'London'),0), 'DD MMM YYYY hh:mm') as FileDate,
    Date((LocalTime('London') - ((1/24)*Threshold)), 'DD MMM YYYY hh:mm') as Target
RESIDENT tmp;

DROP TABLE tmp;

This table then needs to be aggregated to find out how many of the files checked are older than they should be. That value needs to be Peeked out into a variable.

Old:
LOAD sum(1) as Old RESIDENT FileAges WHERE Old = 'Yes';
let vOld = alt(Peek('Old', -1, 'Old'), 0);
DROP TABLE Old;

Now we get to the actual point of the app and the demonstration. If there are any old files, we need to do the call to the webhook endpoint along with a payload that tells us which environment is affected and how many of the files are stale.

if alt(vOld, 0) > 0 then
    SET errormode = 0;
    LIB CONNECT TO [$(vGET)];
    SQL SELECT "col_1" FROM CSV (header off, delimiter ",", quote """") "CSV_source"
    WITH CONNECTION (URL "$(vWebhook)?value1=$(vEnv)&value2=$(vOld)%20Files%20Are%20Older%20Than%20They%20Should%20Be");
    SET errormode = 1;
end if

TRACE There are $(vOld) files older than expected;

To wrap up, because we are tidy Qlik developers, we clear down the variables that we have used.

let vEnv =;
let vWebhook =;
let vLib =;
let vGET =;
let vQVDs =;
let vTimestamps =;

Exit Script;

That is it for the load script, and that is all you need to make this work. What you might want to do though, as the list of files checked is in a table in the data model along with their status, is to add a table to a sheet in the app listing the six columns in the FileAges table, with some conditional formatting to show when files are old. I would also add a button, which can then refresh the app without going into the load script.

But What If Even This App Cannot Run?

If you are running Sense on a Client Managed server it is possible (although very unlikely) that even this application will not be able to run and call the webhook, perhaps if someone has unplugged the server in order to recharge their mobile phone. In this case we need an external service to monitor what is going on. If the server is (as is most likely) behind a firewall that external service will not be able to reach in and check the status. In this case we want a service outside of the network which is expecting to hear from the server at a regular beat which can then raise an alert if it doesn’t hear anything.

This is exactly what I am going to cover in my next blog post, pulling together elements of this post and the previous post. This solution will use a webhook on a Sense Cloud tenant which calls a reload of a script that writes a to a QVD. There is also a separate application that checks the date of this QVD and notifies if that has not been updated in a specified window. The history of the webhook being called is persisted, so a history of when the calling server was up and able to reach the outside world is recorded, giving an accurate account of uptime on the server.

If you have any comments on anything you would like to see covered in that blog post, or anything related to this blog post, please use the comments box below to get in touch.

Other Uses For Calling A Webhook From The Qlik Load Script

There is literally no end to what you can achieve by calling webhooks, particularly if you use a tool such as IFTTT.

Another use you might want to consider, if you are using Qlik Sense Cloud, is kicking off a reload using a webhook. In my previous blog post I described how a Qlik Automation can be set up to be triggered from a webhook to do a reload. If at the end of the load script for one app you call the webhook with the correct parameter to kick off the next reload in a chain this reload chain will work however the reload is started, interactively or from a schedule. This gets around the limitation that Qlik Sense Cloud reload chaining from the hub presently has, where the next reload in the chain only takes place if the previous reload was started from a particular schedule. This reload chaining approach also allows for logic to be applied around whether the next reload should take place or not, perhaps the front end should only reload if there has been a change to the backend data?

I hope this post has given you plenty of inspiration of how you can link different services together within, and beyond, your Sense environment.

The post Starting External Services From Qlik Sense Load Scripts appeared first on .

In June 2026 Qlik are changing how tokens must be sent to Qlik Automations. This means that the token must be passed as a HTTP Header, rather than a URL parameter.
To find out more about this read this blog post.

What are Webhooks?

Webhooks are a simple way of getting disparate systems to talk to each other. They allow the triggering of events simply by calling a URL, this could be from a browser bookmark, a link on a page or in code. I’ve previously blogged on a use of webhooks in the article Using IFTTT and WebHooks To Send Data From Qlik Sense.

In a Sense context the webhooks can be called by creating a Link in the Sense Hub, adding a link to a Button object, a value in a table set to HTML or by calling the webhook from within the load script (using a REST connector to make the call). Triggering from the load script allows the building of reload chains that are not dependant on the Sense scheduler, and for reloads that are data dependant – for instance calling a reload of another app only when certain data are found in the current app.

What Are Qlik Automations?

You may have noticed the button Automations on the left hand side of the Sense Hub. Under here is a real treasure trove of components that can be used to build complex event driven automations. A lot of the components integrate with your Sense environment, for instance sending Qlik reports to recipients based on loading addresses from an data source. There are also a number of components to integrate with services outside of the Qlik environment, such as Teams, Slack, Hubspot etc. etc.. The automations that you create here don’t necessarily even need to have a Qlik component involved, for instance you could have an automation that posts to Teams when a new row appears in a Google Sheet. There is a whole host of no-code, drag and drop, opportunities to explore here. For now we are going to create a simple webhook that receives the GUID of an application and reloads it.

Creating the Automation

From the left hand menu in the Sense Hub select Automations, and then click the Create New Automation button at the top right. If you don’t see these options then you don’t have the permissions to create Automations, and will need to get this granted to you by an administrator via the Sense Console.

You will now see a large number of pre-canned templates for automations (which you can explore later), for now though select Create Without Template at the bottom left of the dialog. You will now be prompted to give your new webhook a name and description, and choose where to store it:

You will be presented with an almost blank workspace, just with a single Start block in the middle. The properties for this block are on the right hand side. Change the Run Mode to Triggered. You will then see the code, including the execution key you will need to trigger the automation.

Next you need to add an Input block, to receive the GUID for the app you wish to reload. On the left hand side search for Inputs then drag the inputs component immediately under your Start block, you will see a green dot when it is in place and when you drop the new block should be connected by a straight line. Set the Label to app and the Input Type to Data.

At this point you need to inject a value into the app parameter, so that it can be used in the next step. First you will need to save the automation using the Save button at the top of the page. Then, from the POST example on the Start block (pictured above), copy the text from https:// through to the end of the execution token (in this case guV1). Paste this whole string into a new browser tab and make the following edits: add a ? after the word execute, delete from there up to X-, replace the colon and space with an equals symbol, then add to the end of the URL &app=x. So, given the example pictured above the complete URL would be (with the * characters replaced with your details):

https://****.**.qlikcloud.com/api/v1/automations/****-****-****-****/actions/execute?X-Execution-Token=***********&app=x

Click enter to call the URL, and you will simply see [] returned in your browser. Leave that browser tab open (you will need it shortly) and return to the Automation editor.

The next bock you require is the one that actually triggers the reload. On the left hand side click on Qlik Cloud Services and search for the Do Reload component. Drag this under the Inputs component. In the properties panel drop down under the App Id field and select Inputs (this relates to the previous node) then click History and select app: x from under Inputs (this is what you passed in during the previous step). Set the Run Mode to be Start reload and continue.

Finally you need to add an Output node, so search for this on the left hand panel and then drop it after the Do Reload node. In the Data to output properties, you should see that it has the Do Reload node as the output it will send as a result. If it does not, drop down here and select Output from Do Reload and then Do Reload.

That is the Automation completed, so click save up the top of the sheet to finish.

Now you will need the GUID of an application that you want to reload. The GUID can be found in the URL when you have an application open, you will find it between app/ and /overview or /state. It will look a bit like this: 0e316df1-fd5c-4cd0-bac1-2f66a03b4c3d. Copy this to your clipboard and return to the tab where you called the webhook previously. Replace the x at the end of the URL with the GUID and return again. This time you should get some JSON returned, confirming that the reload has started. Check the details of the app that you have chosen to reload in the Hub and you should be able to see that a reload has started.

Calling The Webhook

As you have seen, you can call your new webhook simply by pasting it into the URL of a browser. It can also be bookmarked, for ease of use. You can create different bookmarks for each of the different apps you want to allow people to reload and most browsers will allow you to export bookmarks to give to someone else to import into their browser. The Sense hub now allows you to add links to the hub, so you can create links in there. Buttons in Sense allow you to call URLs, so that is somewhere else you can call the link.

The resultant webpage is a bit messy, so you might want to have a page that calls the webhook with javascript and parses the output and gives a friendlier results page, or perhaps just triggers a back action to take the user back to where they came from.

To call from a Qlik load script you will need a REST connector created which has the webhook as the URL. If you have a lot of apps you wish to reload in this way you will want to have a placeholder URL in your rest connection and then replace the URL in the script using a WITH CONNECTION statement. I describe how to do this in the blog post I referenced at the top of this article: Using IFTTT and WebHooks To Send Data From Qlik Sense.

Hopefully that gives you what you need to be kicking off reloads from various different events and in various different ways. If you have used similar techniques please feel free to share them in the comments below.

The post Start a Qlik Sense Cloud Reload from a Webhook appeared first on .

Data Without a Source

Typically a Qlik application will load its data from a data source, such as a database, API or flat file. Sometimes you just want to create something on the fly. The AUTOGENERATE function is a great way of doing this, and many of the applications on my Qlik Examples page use just this functionality. The function gives a specified number of rows of data that you can then work from. To create a table with a single field with values from 1 to 1,000:

LOAD
    RowNo() as Value
AUTOGENERATE(1000);

This may be useful in and of itself, but it is more likely you will need to do more. Read on to find out some of the things you can do.

Well, That’s A Bit Random

Often you will want to create some random values, and for this the rand() function is what you require. This function will return a number which falls somewhere between 0 and 1. Whilst at first this sounds a bit limited, with some very basic maths you can create many kinds of values.

So, to get a random integer between 0 and 999 you could do the following:

LOAD
    floor(rand()*1000) as Value
AUTOGENERATE(1000);

If you wanted to get a relatively random number of dimension values from a list you can multiply the random value by the number of values you have, round up to the nearest integer and then use a pick() function to choose which dimension value to pick:

LOAD
    pick(ceil(rand()*5),'Ben','Ian','Tom','Paul','Olly') as Person
AUTOGENERATE(1000);

Note with this approach though you will get roughly the same number of rows for each of the values, due to how random numbers work. If you want to have less regular number of rows you will need to introduce some bias.

If you want to generate random codes that look like they could be real (but are not) you can use the fact that the chr() function picks an ascii character based on a number. To get a code that looks like ABC123 you would have:

LOAD
    chr(65+floor(rand()*26))
        & chr(65+floor(rand()*26))
        & chr(65+floor(rand()*26))
        & num(ceil(rand()*999), '000') as PersonID
AUTOGENERATE(1000);

Given the number of possible values that this could generate you would most likely get all unique values. If you would prefer to guarantee that there are no duplicates you could use rowno() instead of rand(). If you wanted to cause more duplicates you could reduce the number of characters and the multipliers on each of the random values, perhaps reducing the letters to A to D and the digits to all be multiples of 9:

LOAD
    chr(65+floor(rand()*4))
        & chr(65+floor(rand()*4))
        & num(ceil(rand()*99)*9, '000') as DupedID
AUTOGENERATE(1000);

If you want to have data shown on a line chart you can generate dates, using the fact that dates are just numbers under the bonnet and we can therefore do maths on them. Again, if you want one row per date then use the rowno() function, or if you want more randomness around the rows per day use rand():

LOAD
    date(today()-(rowno()-1), 'DD MMM YYYY') as Date,
    date(today()-floor(rand()*90), 'DD MMM YYYY') as [Date Rolling 3m]
AUTOGENERATE(1000);

Creating dates in this way has been particularly useful in some of the demo apps I’ve created, as a simple reload will make the demo look fresh and current as all dates are in a period leading up to the current date.

That’s just a few examples of generating test data using AUTOGENERATE, but I am sure you can think of many more.

But Does It All Correlate?

I mentioned at the top that I had a recent requirement to test the correl() function, and generated some data to do this. For those who have note come across the statement before, it returns the aggregated correlation coefficient for two data sets. Or put into English, a value is returned which is close to 1 if there is a very strong positive correlation, close to -1 if there is a strong negative correlation and close to zero if there is very little correlation between two sets of values.

The function is called like this:

correl(aggr(sum(X), Dim), aggr(sum(Y1), Dim))

This is where Dim is a dimension value in the data and the two aggr statements create two datasets with a row for each dimension value. The sum(X) and sum(Y) expressions would usually be more complex calculations, but in this case it is simply the numbers generated in the load script.

The load script to make this expression work is as follows:

LOAD
    'Row ' & num(RowNo(), '#,##0') as Dim,
    RowNo() as X,
    RowNo() as Y1
AUTOGENERATE(1000);

Because we are using the row number for both the X and Y1 values there is an absolute positive correlation and the function returns 1, as it should.

For all subsequent tests we keep the Dim dimension and the X value, which always gives us a point along the bottom of the chart. For a negative correlation the value of Y can be calculated as:

1000 - RowNo() as Y2

So now when X is 0 Y is 1000 and vice versa. This is an absolute negative correlation, and the correl function returns -1, again as it should.

The rest of the values for Y which I tested for where as follows:

Rand() * 1000 as Y3

A totally random value, which gives a correlation coefficient that is very close to zero and a plot that is full of randomly placed dots.

RowNo() * Rand() as Y4

A random value between 0 and 1 multiplied by the X value, so that the Y value is random, but never greater than Y. This gives a correlation coefficient that is around 0.5 and a chart where the bottom right half is filled. This could be inverted, but I did not try that value.

(RowNo() * (Rand()-0.5)) + 500

A value that deviates from the centre of the chart on the Y axis by a random number that is plus or minus half of the value of Y. As the values fall above and below the line equally the correlation is very close to zero.

RowNo() + ((Rand()-0.5)*(RowNo()/1)) as Y6

This value is similar to the X=Y chart we started with, but with a random deviation from the line in either direction, up to half of the Y value. The divide by 1 is left in there as I was experimenting with having the deviation greater or lesser by dividing the row number. Correlation coefficient is approaching 1, but is not there.

500 + ((rand()*10)-5) as Y7

This value has a similar Y value for every value of X. I started off having it exactly the same for each X value, but the correl function requires some variance in the number or the spread will be zero causing a divide by zero and no correlation coefficient number. A random deviation of up to 5 in either direction provides this difference, but that deviation is small enough that the correlation coefficient is very close to zero.

500 + ((rand()*10)-5) as X2

The final expression is a bit different to all the others, as I wanted values from 1 to 1000 for all Y values in order (I use Y1 from above for this) and then have a calculation for the X value that is close to the same value, but off by a small margin. You will note the expression for X2 is identical to Y7, and I could have used the same field in the correl function, but it felt wrong to have two values labeled Y in the chart.

The resultant scatter charts and correlation coefficients for all of the derived values above can be seen in this screenshot (click to expand).

The entire load script for the Correl application looks like this:

LOAD
    'Row ' & num(RowNo(), '#,##0') as Dim,
    RowNo() as X,
    RowNo() as Y1,
    1000 - RowNo() as Y2,
    Rand() * 1000 as Y3,
    RowNo() * Rand() as Y4,
    (RowNo() * (Rand()-0.5)) + 500 as Y5,
    RowNo() + ((Rand()-0.5)*(RowNo()/1)) as Y6,
    500 + ((rand()*10)-5) as Y7,
    500 + ((rand()*10)-5) as X2
AUTOGENERATE(1000);

And the entire application can be downloaded from Qlik Community here:
https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Create-Test-Data-For-Correl-Function-Using/ta-p/2139164

Hopefully that has given you some inspiration what you can do when you need to generate some test data. I would be very interested to hear of other thoughts you might have on the topic, so please do use the comments box below to let me know.

The post Generating Test Data In Qlik Load Scripts appeared first on .

The Challenge

The question was posed on Qlik Community how to overlay two 12 month measures in one chart. Using set analysis it is straight forward to have two measures which cover the correct date ranges, but when the Month dimension is used it will sort by calendar month, rather in the order required for rolling periods. You can see the question and my response in this post.

The response uses a technique I have used a number of times before but it occurred to me that I have not documented it on the blog, so here it goes.

Creating Rolling Month and Year Dimensions

This code can either sit in your master calendar code, should you use one, or you can place it directly in the load statement from source data. Typically I will go for the latter approach.

The first thing that we need to do in the load script is to create a sequential integer for each month. This then allows us to work out the number of calendar months between any two dates. This technique has many more uses than just rolling twelve month analysis and is an incredibly useful dimension to have in your data model.

If we do (Year(Date)*100)+Month(Date) we will get a sequential integer value within a year (e.g. as at now it will give 202305), but the deltas break when you go across a year (e.g. comparing 202212 to 202301). In order to fix this we can just multiply the year by 12 and add the month. So, again taking May 23 as an example you will get 24,281. This is obviously not at all useful as a human readable number (just as telling someone that the date today is 45051 is only useful if they are a machine), but it means that the numbers for Dec 22 (24,276) and Jan 23 (24,277) are now just one integer apart.

In the load script we can put this code in to create a new field called MonthNo like this:

(Year(Date) * 12) + Month(Date) as MonthNo,

Now we have that value we need to work out how many months back any given month is from today. The code that we used on the Date field can also be done on the today() function, giving a comparable figure for the current date. Using a preceding load, we can compare that to the MonthNo field to get a value for Months Back, like this:

(Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]

This new field in itself may be useful in your charts and filters within Sense, for instance it could make your set analysis for a chart showing the last twelve months much easier as you do not need to deal with date formats in the set code.

Jumping up another level of preceding load we can now use the Months Back value to create the Rolling Month and Rolling Year values.

The Month function gives us a dual data-typed value with the three letter month name and the integer order for the month from 1 to 12. The text part of that dual datatype is great for us but the number does not give us what we want.

We can use the Dual function though to build our own dual data-typed value with the correct sequential number. Using the output of the Month function as the first parameter of the Dual statement will only take the text part of that value. The correct sequential number can be calculated by working out how many months back the start of the year was and then taking away the months back for the date in question.

The whole of that code looks like this:

Dual(Month(Month), (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],

We can get the number of rolling years back from our Months Back value by taking the integer part of the number of months back divided by 12. In order to turn that into text to show as a dimension we want to suffix it with the text “Years Back”. As having “0 Years Back” looks a bit naff, we need to have anything in the past 12 months showing as “Current Year”. Having that text means that sorting on the field breaks (as it would if you have more than a decade of data), so we need to use the dual function again.

Putting all of that together gives us this statement:

Dual(if([Months Back] < 12, 'Current Year',
    Floor([Months Back]/12) & ' Years Back'),
    Floor([Months Back]/12)) as [Rolling Year]

We then have the two dimensions that we can use in a line chart:

Putting It All Together

This code generates a whole bunch of dates and then applies the date code described above in a resident load. You can copy and paste this to a new app and see the code working in isolation, or your can take parts of it to apply to your own data.

tmpDate:
LOAD
    Date(today()-RowNo(), 'DD MMM YYYY') as Date
AUTOGENERATE(2000);

Dates:
LOAD
    *,
    Dual(Month(Month),
        (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],
    Dual(if([Months Back] < 12, 'Current Year',
        Floor([Months Back]/12) & ' Years Back'),
        Floor([Months Back]/12)) as [Rolling Year]
    ;
LOAD
    *,
    (Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]
    ;
LOAD
    Rand() * 1000 as Value,
    Date,
    Date(MonthStart(Date), 'MMM YYYY') as Month,
    (Year(Date) * 12) + Month(Date) as MonthNo,
    Month(Date) as [Month Name]
RESIDENT tmpDate
;

DROP TABLE tmpDate;

Something else you might want to deal with when looking at rolling months is a run rate value for the current month, which takes into account how far through the month we are and what the value will be if extrapolated to the number of days in the month. I have covered this in a previous post.

Not only does this code give you dimensions that you can use in your charts, the same dimensions can be used in set analysis also. To create the variance to prior rolling twelve months gauge at the top of the chart you can now just use this code:

(Sum({<[Rolling Year]*={'1 Years Back'}>}Value) /
Sum({<[Rolling Year]*={'Current Year'}>}Value)) -1

An example QVF file including this code and some example charts can be downloaded from the Qlik Community post.

I would be very interested to hear from you on this technique and whether you have used similar or can suggest other ways it could be used. Please use the comments below if you would like to add your thoughts.

The post Compare Rolling Twelve Month Period in Qlik appeared first on .

Why Use GMail For Sending and Receiving From Qlik?

There are a wide number of features within the Qlik ecosystem that require interaction with email. QlikView Enterprise allows sending of alerts on task failures, NPrinting is all about distributing reports and Qlik Sense SaaS has multiple features for sending mails, from task failure notifications to subscriptions to charts or sheets. Beyond the obvious use cases there are many more reasons why you may interact with a mail server, such as Sending Notifications From The Load Script or even using a mail server as a data source, such as in this example of loading router logs.

Many companies use Google Workspace for their internal systems, so using a Workspace account for sending and receiving mail makes sense, and for individuals learning Sense spinning up a GMail account for this purpose is a no cost way of achieving the same result.

Why Allow Access For Less Secure Apps?

Google, and general security best practice, recommend using 2FA for authentication. Google Accounts implement this via Google Prompts (on a signed in phone), text message, the Google Authenticator app or a physical device. Having a third party application (or, ironically, Google’s own email migrate tools) connect to your account via SMTP, IMAP or POP3 is deemed less secure as it inherently uses a username as password only to connect.

The solution for this, for the past couple of years, has been to set a switch on the Google Account that is being used in this way to say that you are comfortable with username and password access to the account in this way. Recently though, a new message has appeared on the dialog where this switch is set:

It advises that this access will be revoked from the 30th of May, and the Find Out More link gives a bit more background, but not quite enough. The long and the short is though, that if you are using GMail servers after this date, without an App Password, they will no longer function correctly.

So, How Do I Fix This?

At the bottom of that article there is a link to information on Signing In With App Passwords, this is a much more helpful document. The steps to resolve the issue are outlined here though.

Head to Account Settings and Security for the account you wish to use

Turn on 2FA for the account you wish to use

Select to create an App Password for this account

Select Other from the app list and enter Qlik Sense or similar

Copy the generated password to your clipboard

Use the App Password as you would your normal password

You can come back into the App Password settings to manage old passwords and remove them when you no longer use them. Note that when you change the password on the account any App Passwords will be removed, so you will need to recreate these and plumb them back in when this happens. Note that you can not recover an old App Password, so make sure you enter it everywhere you need to before you lose it from your clipboard (or put it in your password safe).

Once all of this has been done your Qlik apps and servers can interact with your Workspace or GMail account without issue, and they will continue to do so after the date which Google removes support for less secure apps.

I hope this post helps you avoid any nasty surprises next month.

The post Google SMTP and IMAP to break in Qlik Apps Next Month appeared first on .

API Connections in Qlik Sense Client Managed

Straight of the box with a clean install Sense has a number of REST Connections created for you to explore your environment. It’s well worth exploring what these connections offer, with regards to looking at users and usage.

For this post though we are going to be looking at the monitor_apps_REST_app connection.

Create a new app and go to the Data Load Editor. If you can see the monitor_apps_REST_app connection we are good to go. If it’s not there you will need to grant yourself rights in the QMC under the Data Connections settings, find the correct connector and select Security Rules.

From here you can grant yourself access to the one connection by your name or user id:

Or alternatively, if you are a RootAdmin on the QMC you may want to grant the RootAdmin role access to all connections on the server. This will mean that administrators can all explore all of the default connections and keep an eye on what users are creating also. Do this by replacing the GUID of the connection in the Resource Filter with an asterisk. Obviously, you can tailor the permissions to be whatever you need.

When you go back to the Data Load Editor in another tab the connections will not have appeared automatically. You will have to refresh the browser with a Shift-F5, or click the refresh button in the browser toolbar.

Loading Data From REST API Connection

Now you have access to the correct REST connection, you can use the Select Data icon next to the monitor_apps_REST_app connection to look at the data. Alternatively, you can use the script here to pull the information that you need.

Before we connect to the data though, we are going to set up some variables which describe where the apps currently reside and where we want to back them up to. The apps will be in your repository, the location of which will have been set when Sense was installed. There will be a share, most likely on the Sense server, pointing to this location. In here find the Apps folder, which contains a file for each app, with a GUID for a name and no extension. The destination is simply where you want to copy to. Here I create a dated folder for the backup:

// Set up locations of Sense Repository and a backup location
// Used in the backup script field
let vSrcFolder = 'c:\QlikSenseRepository\Apps\';
let vDestFolder = 'c:\SenseBackups\AppBackup' & date(today(), 'YYYYMMDD') & '\';

A connection then needs to be made to the REST API in order to load data from it:

// Connect to the REST connection
LIB CONNECT TO 'monitor_apps_REST_app';

With this in place we can then pull the data from the REST connection, using this code:

// Load raw JSON Data
RawRESTData:
SQL SELECT
"id" AS "id_u1",
"createdDate",
"modifiedDate",
"modifiedByUserName",
"name" AS "name_u1",
"publishTime",
"published",
"description",
"fileSize",
"lastReloadTime",
"thumbnail",
"savedInProductVersion",
"availabilityStatus",
"schemaPath",
"__KEY_root",
(SELECT
"userId",
"userDirectory",
"name",
"__FK_owner"
FROM "owner" FK "__FK_owner"),
(SELECT
"name" AS "name_u0",
"__FK_stream"
FROM "stream" FK "__FK_stream")
FROM JSON (wrap on) "root" PK "__KEY_root";

As with loading data from just about any REST connection, the data will come down in a single table, even though there are three logical tables in the data model. Each row of data will have values in columns in one of the three sets (Application Details, User Details and Stream), values in the other columns will be null. Along with these values are primary and foreign keys to join things correctly.

Using Select Data the Qlik engine will create some code which will load three separate tables into the end data model. In this case however we know there is a one-to-one relationship between an Application and its Owner and the Stream it is in (if the application is published). We can therefore create a cleaner and flatter data model with ApplyMap.

There is much more we can do also, to make it easier to select the apps we want to back up, and do further analysis on apps. The same data feed can be used to help use identify apps which are not being maintained or are bloated.

Cleaning Up The Application Table

Now we have all of the data in a temporary table we need to clean it up and augment it. This will be done with a number of RESIDENT loads and a couple of Preceding Loads.

First we are going to create some ApplyMap statements to bring in the Owner name, Owner login and Stream. These use the foreign keys in the flat table.

// Create mapping tables from parts of the REST result
Map_Owner:
MAPPING LOAD
[__FK_owner],
[name]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_OwnerLogin:
MAPPING LOAD
[__FK_owner],
[userDirectory] & '\' & [userId] as Login
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_Stream:
MAPPING LOAD
[__FK_stream],
[name_u0]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_stream]);

Then some variables which act as functions in the subsequent RESIDENT load. These all utilise the fact that a SET statement does not evaluate the code that it puts into a variable (rather it copies code verbatim) and the functionality of passing parameters into variables.

// Create parameterised variables for repeated tasks
set vTime = Date(if(Date#(left($1, 10), 'YYYY-MM-DD') < 0, null(),
Date#(left(replace($1, 'T', ' '), 19), 'YYYY-MM-DD hh:mm:ss')),
'DD MMM YYYY hh:mm:ss') as [$2 Date Time];
set vDP = Date(DayStart([$1 Date Time]), 'DD MMM YYYY') as [$1 Date],
Date(MonthStart([$1 Date Time]), 'MMM-YYYY') as [$1 Month],
Month([$1 Date Time]) as [$1 Month Name],
Year([$1 Date Time]) as [$1 Year],
WeekDay([$1 Date Time]) as [$1 Day],
today() - DayStart([$1 Date Time]) as [Days Since $1];
set vNS = if($1 = '', 'Not Specified', $1);

The first function here receives the name of a date field and checks whether it has a value, if it does it returns that converted to a date and formatted correctly. The second takes the date time field created by the previous variable (thanks to a preceding load) and creates various date parts for these dates. The final variable checks a given field name and returns Not Specified if the value is empty, otherwise it returns the value.

You will see all of these variables used in the next block of code.

We can then do our final RESIDENT load to bring through all of the original data, with the various transformations applied.

Applications:
LOAD
1 as ApplicationCount,
*,
RangeMin([Days Since Created],
[Days Since Modified],
[Days Since Published],
[Days Since Last Reload]) as [Days Since Last Touch],
// Export this column to create a script to backup your apps from the Repository
'copy $(vSrcFolder)' & GUID &
' "$(vDestFolder)' & Stream & ' -- ' &
Application & '.qvf"' as [App Backup Script]
;
LOAD
*,
// Create all required date parts
$(vDP(Created)),
$(vDP(Modified)),
$(vDP(Published)),
$(vDP(Last Reload))
;
LOAD
id_u1 as GUID,
// Convert all required dates
$(vTime(createdDate, Created)),
$(vTime(modifiedDate, Modified)),
$(vTime(lastReloadTime, Last Reload)),
$(vTime(publishTime, Published)),
[modifiedByUserName] as [Modified By],
[name_u1] as Application,
[published] as [Is Published],
$(vNS(description)) as [Application Description],
fileSize / 1000000 as [File Size],
$(vNS(thumbnail)) as Thumbnail,
[savedInProductVersion] as [Product Version],
[availabilityStatus] as Availability,
ApplyMap('Map_Owner', [__KEY_root], 'Not Known') as Owner,
ApplyMap('Map_OwnerLogin', [__KEY_root], 'Not Known') as [Owner Login],
ApplyMap('Map_Stream', [__KEY_root], 'None') as Stream
RESIDENT RawRESTData
WHERE NOT IsNull([__KEY_root]);

Then, to tidy things up and avoid any strange joins, we can remove the temporary table which was created when the data was pulled from the REST connection.

DROP TABLE RawRESTData;

And that is our load script done.

Unpacking The Application Back Up Code

There is quite a lot going on in that last step of the code. Hopefully you can work out what is going on with most, if not all, of it. Much of it is to give us some extra fields to do some analysis of the applications on our server (more on that later), but for now just to focus on the creation of the App Backup Script field.

'copy $(vSrcFolder)' & GUID & ' "$(vDestFolder)' &
Stream & ' -- ' & Application & '.qvf"' as [App Backup Script]

Here we are starting with the copy statement we wish to use (you can replace this with XCopy or add parameters if you wish), adding in the two variables we set up right at the top of the script and pilling out the GUID which is the source filename for our script. There is then the building of the output filename and the placing this in a fieldname.

The construction of the output filename and path you can also alter if you wish, I have gone for the stream name (which is set to None for items that are not published) a double hyphen (as apps and streams may have hyphens in the name already) the application name and finally the QVF extension.

This then builds code that we can use in a Command Prompt on our Qlik Sense server to move the apps to their backup location.

Creating and Using The Back Up Script

Now our script is complete, click Load Data to bring all the data in from the REST connection. Once this has finished loading click Analyse Sheet and change to Edit mode. Grab a Table object from the Charts and drop it on the sheet. Click to Add Dimension and add in the App Backup Script dimension.

Come out of Edit mode and you can then right click to export the table to Excel. Open this spreadsheet and copy and paste the code into a new Notepad (or other text editor) window and then save the script as a .bat file. This file can then be copied onto the server and executed to perform the backup.

There is a slight risk of one of the files being corrupted if it is copied whilst a write is in progress, so it is best practice to stop all Sense services before taking the backup. In practice, I have not ever hit a snag with this though and you could simply check the date stamp on the files and see if there were any that were updated at around the time you took your backup and perhaps copy them again.

So there you have a relatively quick and painless way of getting QVFs of all of your Qlik Sense Client Managed Server applications.

I hope you find the technique useful. Please let us know how you get on in the comments below.

Further Analysis of Your Application Data

As well as the the information required to filter and export the backup script there were various other data points brought in about the Applications on the Sense Server which you may want to analyse. Finding which applications have not been touched in over a year, for example.

Fortunately we have your back with this, with our Instant Sense Application. This is an application template which ships with a number of pre-written load scripts (including the one above) and metadata files so that you can re-purpose the application to use your own data sources. Out of the box you can view a number of KPIs over these application data:

It also allows you to break the various measures down over different dimensions and plot those different measures against each other, by a selected dimension:

Finally there is the user defined table, where the user can select a number of dimensions and measures and view these in a table which can then be exported as required:

Everything in the application is driven from a metadata spreadsheet, this means that you can configure the app so it can be used over whatever data source you like:

All in all there are twelve sheets of user configurable visualisations, just waiting for you to bring your own data. The best thing is that it is free to download and use.

Find out more about the application, and download it, from here:

https://www.quickintelligence.co.uk/instant-qlik-sense-app/

More technical details, including how the application is built can be found in this Instant Sense Application blog post.

If you have any questions or comments about anything above, please do drop us a comment below.

If you would like to share a load script and metadata spreadsheet for the Instant Sense Application then we would love to hear from you.

The post Back Up Qlik Sense Enterprise Client Managed Applications appeared first on .

So, Why Use IFTTT With Qlik Sense?

In my previous blog post I showed how to create an app to monitor Qlik analyzer capacity licence usage in Qlik Cloud. My favourite response to the last post was from Tom Wittig, who said he had five clients needing the app.

That got me thinking, what if you have many sites to keep an eye on in Qlik SaaS? You would want to take the information from each site, place it into a common repository and then build an app over that. But how?

There are plenty of methods for storing files out, but this would require a shared DropBox or Google Drive, between clients, or many clients’ accounts linked to by the partner. Neither of these approaches feels like a good idea. Pushing just the data that is required via a one way service that can then store the data would be ideal.

Using a combination of the REST connector, Webhooks, If This Then That and Google Sheets it is simple to achieve this.

I did a previous post on using Webhooks to start a Sonos playlist when a new customer is obtained, but this is a more practical application of the same technique.

Signing Up For An IFTTT Account

If This Then That allows you to connect all of the various services you have and get them talking to each other. It’s well worth taking a look through the various example applets that they have available. If you have an account already you will know this. If not, it is simple and quick to create one and get started.

Head along to https://ifttt.com/, here you can sign up with an email address or an Apple, Google or Facebook account. As we’re going to be using Google Sheets I’m going to use a Google account, you can take your pick though.

Creating the IFTTT Applet

Once you have signed up or signed in you will get to the main IFTTT page. You may want to take the time to take a look around, or you can just crack on and click the Create button.

As the name suggests, basic apps all consist of a trigger (If This) and an action (Then That), click the Add button to set up the If

First you need to chose the service that you will use for the trigger. Search for and select Webhooks

Many of the services will have lots of different options here, but Webhooks just have one simple action – receive a web request. Click this.

Pretty much every service will require a connection to that service, this is true of Webhooks, so click the Connect button to do this. Most services will then require credentials (as you will see when we connect to Google Sheets in a moment), but Webhooks will just create the connection for you.

You will then be prompted to create a trigger, which you need to name. This name will be used when you call the Webhook, and it will be available when we create the event caused by the Webhook. Make sure you take a note of this name.

When the name has been provided click the Create Trigger button.

With the trigger created you can now click the Add button for the action.

Now you need to search for and select Google Sheets.

From here select Add Row To Spreadsheet. Again you will need to connect to the service, do this by clicking the Connect button. A popup will appear where you can sign into your Google account and authenticate IFTTT to work with it.

There are lots of other endpoints that you can use to write to, such as Slack, Twitter and Gmail. Choose and configure the one that best suits you.

For a Google Sheet you need to provide the name of the spreadsheet (clear the default one first) enter the values to enter into each cell of the spreadsheet (defaults here are fine) and specify which folder in your Google Drive you would like the spreadsheet to reside. Click Create Action when done.

That has now created the applet and you can click Continue.

Using The New Applet

After clicking continue you will be shown the default name for the applet. This will be a description of what it does. You can swap this for a more useful name if you like. Click the Finish button when you are happy with the name.

The applet is now present and working, but the issue is that you won’t know how to call it. Click on the Webhooks icon above the to the configuration screen.

From that screen click the Documentation button at the top right of the screen. This will take you to a screen where you can test your Webhook.

Replace the text {event} with the event name that you used when setting up the trigger. Enter some values into the three value text boxes and click Test It.

You will need to copy the Trigger URL to a safe place to be able to use it later, this is the one that looks like this:

https://maker.ifttt.com/trigger/capacity_usage/with/key/ABCDE1234567890

Once the test has been fired you should be able to go and find the spreadsheet, but I have noticed it can take a few minutes to appear.

Plumbing The Webhook Into The App

To use this Webhook with the Capacity Licence checking app I made available last week you will need to download it from here. To plumb it into another app you will need to put the values you want to send into variables in your load script.

To call the URL you will need a REST connection set up, as described in the previous blog post. This can point to any URL, as we are going to change it using WITH CONNECTION, I typically use https://jsonplaceholder.typicode.com/posts for this purpose. Give the connection the name GenericGET.

You then just need to add the following code after the point where the variables are set:

// Send details to a Webhook
LIB CONNECT TO 'GenericGET';
SET errormode = 0;
SendResults:
SQL SELECT
"col_1"
FROM CSV (header off, delimiter ",", quote """") "CSV_source"
WITH CONNECTION (
URL "https://maker.ifttt.com/trigger/capacity_usage/with/key/ABCDE1234567890?value1=$(vUsed)&value2=$(vRunRate)&value3=$(vAvailable)");
SET errormode = 1;

Obviously the key part of the URL will need to change to be your own personal Webhook key that you copied above. You will also need to ensure that the trigger name matches the one you gave when creating the trigger. The URL copied in the test step above didn’t have any parameters, as these were passed in the body of the GET request. You will need to manually type the three parameters to the URL and inject variable values created earlier in the script.

Once the load script has run you can check the result from the call of the Webhook by checking the col_1 field in Analyze mode in Sense.

You should now have a new row in your spreadsheet with the values you specified in:

As you can see you can insert a row which has headers and new rows will simply be appended to the end. If you have multiple Webhooks appending to the same spreadsheet you just need to ensure that you have a value in a column which shows which numbers have come from where.

Webhooks are a great way to ping small amounts of data between different services, and it is something which is going to be more closely integrated into Sense in future with blendr.io. You can start using them straight away though, from your load scripts.

Please let me know how you get on with them in the comments below.

The post Use IFTTT and Webhooks to Send Data from Qlik Sense appeared first on .

What is Capacity Licencing?

Purchased in packs of 1,000 minutes per calendar month, a capacity licence is a subscription which can be consumed by any user for access to any app in six minute time slices. This flexibility makes the capacity licence an excellent choice for getting large numbers of infrequent users using Qlik apps.

Once you have these licences though, it is crucial that you monitor things to ensure that you are not burning through your capacity too quickly. Just looking at how many minutes have been consumed against the total is not enough, you need to be aware of how far through the month you are. Looking at current usage against a run-rate of available minutes allows this. If usage is exceeding the minutes available then someone needs to be notified, so licences can be reallocated or an additional capacity licence pack be added.

This post describes how to set up this monitoring and alerting on Qlik Cloud (SaaS). A similar approach can be taken for client managed also.

Connecting To The Qlik API

Information about your Qlik Cloud tenant is exposed through a number of API Endpoints. In order to read from these endpoints API Keys need to be enabled under Management Console Settings, a nominated user must have Developer rights in the Users section of the Management Console, and that user needs to generate an API key in the API Keys section of their Profile Settings. This key is then used in the load script of the app.

This is quite a convoluted process, but it is well described in the documentation for the App Analyzer for Qlik SaaS. The analyzer app itself is one you may want to considering in your SaaS tenant.

Once you have an API key, paste it into a safe place as you will need it later.

To implement a similar solution on Qlik Sense Enterprise for Windows (Client Managed) you can use the provided REST connection, monitor_apps_REST_license_overview.

Rather than setting up a connection to each API that is needed in your SaaS environment, it makes more sense to create just one and then modify it using WITH CONNECTION. To enable this, set up a new REST connection, pointing to a placeholder (e.g. https://jsonplaceholder.typicode.com/posts) with the type set to GET and everything else as defaults. Give this connection the name GenericGET.

Sending Alerts From The Qlik Load Script

It is possible to use Qlik Alerting in Qlik Cloud, but for this purpose I would suggest simply sending directly from the load script. The SMTP connector in Qlik Cloud allows for this.

Create a new SMTP connection, populating it with the details of the mail server you wish to send via. You may want to set up an email account, such as a free GMail account, specifically for this purpose. Give this connection the default name of SMTP.

This will allow you to send custom emails from your Qlik load script. In order to do the same in Qlik Sense Enterprise for Windows see this previous blog post on Sending Data from the Qlik Load Script.

Building The Capacity Usage App

As with any Qlik app, most of the magic happens in the load script. Here we talk through the code you need to build an alerts app.

First of all some environmental variables need to be set, specifying where your tenant can be found and what email settings you wish to use:

// Name or alias of your Qlik Sense Cloud instance
let vInstance = 'instance.region.qlikcloud.com';

// Email configuration fields
let vTo = 'recipient@domain.com';
let vFrom = 'sender@domain.com';
let vFromName = 'Sense Capacity Alert';

// The API key to use - generated under your Profile settings
let vAPIKey = 'ABC123456789';

Note that you will need to paste the API key that you obtained above into this code.

Now some options to configure. Firstly at what percentage of the time-of-month available minutes do you wish to alert, 1 is anything over 100% of usage – you can increase or decrease this as you wish. You can also specify how frequently you want to be reminded you are over threshold (if usage keeps rising) or ask to be reminded every time the app reloads:

// Decimal at which run-rate percent to send alert - 1 = 100%
let vAlertAt = 1;

// Days in which to repeat send of alert - can be a decimal fraction of a day
let vRepeatIn = 0.25;

// Binary flag indicating whether you wish to ignore increase and repeat thresholds
let vSendAll = 0;

Note that the repeat value of 0.25 is equivalent to every six hours. You can adjust as you see fit.

Now we need to connect to the REST connection. This will check your placeholder is present, but do nothing else at this point:

// Connect to the REST connector
LIB CONNECT TO 'GenericGET';

Note that if you have created your REST connection in a managed space, you will want to prefix the name with the space name, e.g. ‘MySpace:GenericGET’;.

Most of the heavy lifting of working out where we are in the month is done in this next statement. It connects to the API, gives the credentials defined above, calculates the run rate and formats the output:

// Retrieve data and calculate run-rates and percentages
CapacityUsed:
LOAD
*,
num([Used Minutes] / [Available Run Rate], '#,##0.0%') as [Percent Used Run Rate]
;
LOAD
name as [Licence Type],
usageClass as [Usage Type],
units as [Total Minutes],
unitsUsed as [Used Minutes],
num(unitsUsed / units, '#,##0.0%') as [Percent Used],
// Work out a run rate by working out how far through the month we are in days,
// divided by the total number of days in the month
num(units *
((now() - MonthStart(today())) /
Day(MonthEnd(today()))), '#,##0.0') as [Available Run Rate],
overage
WHERE name = 'analyzer_time'
;
SQL SELECT
(SELECT
"name",
"usageClass",
"units",
"unitsUsed",
"overage"
FROM "allotments" FK "__FK_allotments")
FROM JSON (wrap on) "root" PK "__KEY_root"
WITH CONNECTION (
URL "https://$(vInstance)/api/v1/licenses/overview",
HTTPHEADER "Authorization" "Bearer $(vAPIKey)"
)
;

In order to be able to check values and send them in the load script we need to peek them into variables:

// Retrieve values from the data set
let vRunRate = num(peek('Percent Used Run Rate', -1, 'CapacityUsed'), '#,##0.0%');
let vTotal = num(peek('Total Minutes', -1, 'CapacityUsed'), '#,##0');
let vUsed = num(peek('Used Minutes', -1, 'CapacityUsed'), '#,##0');
let vPercent = num(peek('Percent Used', -1, 'CapacityUsed'), '#,##0.0%');
let vAvailable = num(peek('Available Run Rate', -1, 'CapacityUsed'), '#,##0.0');

Then to give information back to the user running the script we can use these variables in a set of trace statements.

// Notify of current status to load stream
TRACE ;
TRACE $(vUsed) Minutes of $(vTotal) consumed.;
TRACE $(vPercent) of Total and $(vRunRate) of Run Rate minutes.;

Capacity subscriptions reset at the end of each calendar month. We are only interested in being notified if the previous percentage usage has been exceeded, but we need to reset that if we are in a new month. That is done here:

// Reset last run rate if we are in a new month
if Month(alt(vLastRun, 0)) < > month(today()) then
let vLastRunRate = 0;
end if

We then need to decide whether the current run-rate exceeds the value at which we want to alert, and whether other criteria such as used percentage increased and interval time elapsed. This is all in one big if statement:

// Check if we have an alert situation and whether repeat and increase criteria met
if vRunRate > vAlertAt and (vSendAll < > 0
or (vRunRate > alt(vLastRunRate, 0) and now() > alt(vLastAlert + vRepeatIn, 0))) then

Now we have to set a couple of variables, to record what level we are alerting at, and when we are doing it. We are also writing to the load script progress window:

 TRACE Sending alert to $(vTo);

// Set variables to log last time an alert sent
let vLastRunRate = vRunRate;
let vLastAlert = now();

Sending emails from the Qlik SaaS load script requires a connection to the SMTP connection to be made:

// Connect to SMTP connection
LIB CONNECT TO 'SMTP';

As with the REST connection, the connection name should be prefixed with the space name and a colon if the connection is in a managed space.

Then the sending of an email is simply a LOAD call, with the recipient, sender, subject and body of the email all included in the SELECT statement:

 // Send email and retrieve result of send
Email:
LOAD
status as [Email Status],
result as [Email Result]
;
SELECT
status,
result
FROM SendEmail
WITH PROPERTIES (
to='$(vTo)',
subject='Capacity Usage Exceeding Expected For Time Of Month ($(vRunRate))',
message='

The capacity licence usage on $(vInstance) is currently $(vUsed) minutes out of $(vTotal).
This is $(vPercent) of the total capacity, and $(vRunRate) of the minutes pro-rata for the time of month.',
html='true',
fromName='$(vFromName)',
fromEmail='$(vFrom)',
cc='',
bcc=''
);

Note that you can place HTML into the body of the message here (due to the html=’true’ flag). The code above will be missing the line break and bold tags, you can simply add them back in as required.

The response from the email server is kept in the Email table. We are not looking into that here, but you can check the values in these fields in the Analyze part of your app.

We then need to end if the IF statement we started above, and clear down the variables we have used and no longer have a need of:

end if

// Clean up after ourselves
let vInstance = ;
let vTo = ;
let vFrom = ;
let vFromName = ;
let vAPIKey = ;
let vAlertAt = ;
let vRepeatIn = ;
let vSendAll = ;

So, all being well that should be that. Your tenant’s API will be polled, capacity usage will be loaded (note other licences are in that endpoint also, just excluded by a WHERE), run rates will be calculated and emails will be sent.

Seeing the data is in a Sense app though, it would be a shame not to have a chart or two…

Building a Front End for the Application

This application will only have one row of data in it and a handful of fields that can be used for measures. Master measures can be made for these, as follows:

Total Capacity: Sum([Total Minutes])
Used Minutes: Sum([Used Minutes])
Capacity Run Rate: Sum([Available Run Rate])
Percent of Total Used: Sum([Used Minutes]) / Sum([Total Minutes])
Percent of Run Rate Used: Sum([Used Minutes]) / Sum([Available Run Rate])

You might then chose to place these onto a sheet a bit like this:

If after all of this you would rather just download an app that already has all of the script and front end in it, then you are in luck. I have uploaded the completed app to Qlik Community and you can find it here:
https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Monitor-Capacity-Licences-on-Qlik-Cloud-and/ta-p/1805633

I would be very interested to hear how you get on with this application. Please feel free to contact me via the comments if you wish to get in touch.

Expanding The Solution

If you want to take this solution further, and write the capacity usage as at each reload to an online source, you may want to read my next blog post also:

Use IFTTT and Webhooks to Send Data from Qlik Sense

The post Monitor and Notify Sense Capacity Licence Usage in Qlik Cloud appeared first on .

Introducing The CSV Data

Recently I was asked to be involved with the Data On The Earth project. This is an initiative to get data that can help everyone, from policy makers to concerned individuals, make more informed decisions about environmental issues. Naturally, I was very happy to help.

The first thing I wanted to do was to present some ideas over some data that had relevance to the job in hand. The site Our World In Data seemed like a good place to start.

There I found data which related to Co2 emissions, hosted on GitHub, you can find more about these data here. The data are by Country and Year, so a bit of extra metadata on the countries would allow some demographic filtering. I found this in two further CSV files, also conveniently hosted on GitHub.

The data are not perfect, as some of the figures are not reported by all countries in every year and countries change over time. There are things that could be done to fix this up a bit (rolling forward last known figures, for instance), but I have not done that here. If you are planning to lobby a government over their emissions (or something) please check your facts and don’t rely on what is presented in these data alone!

Creating a Connection

The code in this article is for the load script of Qlik Sense. You can do similar with the REST connector in QlikView, but the syntax is a bit different, this post should give you pointers should you wish to build your own load script.

As with any load script in Sense you first need to have your connection created. Rather than create a separate connection for each data source I load I tend to create a generic connection of each type required and modify it using WITH CONNECTION. You may already have the required connection set up if you have previously followed my tutorials on Loading Sense Session Logs or Loading RSS Feeds, if not go to the Data Load Editor and click Create New Connection.

Give your connection the name GenericGET and point it to a placeholder, such as https://jsonplaceholder.typicode.com/posts, ensure the Method is set to GET and leave everything else as defaults.

Click on Test Connection, then Close and finally (if all is good) Create.

Building the Load Script

Now, onto building the load script. First we need to do some set up, to connect to our connection and set up some variables:

LIB CONNECT TO 'GenericGET';

let vCompleteYears = 'Y'; // Y/N for all years or only 1990 - 2016 where data is quite complete
let vExcludeYears = if(upper(left(vCompleteYears, 1)) = 'Y', 'AND (year >= 1990 AND year <= 2016)', '');

set vFix = if([$1] = '', '$3', [$1]) as [$2];

let v1B = 1000000000;
let v1M = 1000000;

The first variable allows us to turn on or off data for years which are not that complete, the second puts the code to implement that into a variable if switched on. Next we have a variable which includes parameters that we will use later on. If you have not used them before you should see this post on Using Parameters with Qlik Variables. Finally a couple of variables to remove the risk of typing the wrong number of zeros later on.

Now we are set up, we can do the main part of the load.

Emissions:
LOAD
iso_code & year as CountryYear,
iso_code as [Country ISO],
country as Country,
year as Year,
co2 as CO2,
consumption_co2 as [CO2 Consumption],
trade_co2 as [CO2 Trade],
co2_per_unit_energy as [CO2 Per Unit Energy],
cement_co2 as [CO2 Cement],
coal_co2 as [CO2 Coal],
flaring_co2 as [CO2 Flaring],
gas_co2 as [CO2 Gas],
oil_co2 as [CO2 Oil],
total_ghg as [Total GHG],
methane as Methane,
nitrous_oxide as [Nitrous Oxide],
population as Population,
gdp as GDP
WHERE iso_code <> '' and index(iso_code, '_') = 0 // Remove rolled up values
$(vExcludeYears)
;
SQL SELECT
"iso_code",
"country",
"year",
"co2",
"consumption_co2",
"trade_co2",
"co2_per_unit_energy",
"cement_co2",
"coal_co2",
"flaring_co2",
"gas_co2",
"oil_co2",
"total_ghg",
"methane",
"nitrous_oxide",
"population",
"gdp"
FROM CSV
(header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/owid/co2-data/master/owid-co2-data.csv")
;

As with a database load this load will use the last opened connection. The CSV on the FROM line is specifying the basic format that we are loading from this connection, the exact format of the CSV is specified in the brackets – just as it would be if you were loading the CSV from a Folder Connection. The CSV_Source relates to which data source at the connection you wish to load – with a CSV this will only ever be CSV_source, but with JSON data there may be many named sets of data for you to choose from. The WITH CONNECTION then defines the URL we are loading from.

To explore these options further create another connection, pointing to that URL and look at the Select Data dialog for that connection.

Atop of the REST load we have a Preceding Load, to tidy the fieldnames and remove data we don’t want. There are rows with data rolled up to regional levels, but we are going to do that ourselves in Qlik so need to remove the roll ups to avoid double counting. Note that when doing a WHERE in a preceding load all of the data still has to come down from the REST source first, so you may want to look if you can pass parameters to the URL you are loading from to drop some data, rather than using a WHERE statement.

So, that is the data we require from our main data source. We are going to continue though, and augment and categorise these data some more.

Bringing in more CSV Data

The next table of data is a simple table of country demographics, provided by GitHub, keyed on the ISO code for the country.

Region:
LOAD
[ISO3166-1-Alpha-3] as [Country ISO],
$(vFix(Developed / Developing Countries, Developed, Not Known)),
$(vFix(Region Name, Region, Other)),
$(vFix(Sub-region Name, Sub Region, Other))
WHERE EXISTS ([Country ISO], [ISO3166-1-Alpha-3])
;
SQL SELECT
"ISO3166-1-Alpha-3",
"Developed / Developing Countries",
"Region Name",
"Sub-region Name"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv");

This file has many more columns than we need, but we are cherry picking the few that we want. As you can see, the code for connecting to a CSV source is almost identical to above – just the URL is changing. In the Preceding load the vFix variable we created at the start is being used to replace any missing values with Not Known or Other. A WHERE EXISTS ensures we only get rows which relate to countries we have emissions data for.

Using a lookup table like this from a repository like GitHub in your Qlik apps means that your application will benefit from improvements to that dataset over time, rather than having a CSV or Excel spreadsheet languishing on your server which will never see an update.

Where In The World Is That?

As we are looking at country data it would be great to plot that on a map, and another GitHub source has just that information available for us. These data are provided and maintained by Tadas Tamošauskas in a personal repo.

Here we are going to LEFT JOIN the data onto the existing region table, bringing the latitude/longitude data into the same table as the regions. You will see why in a bit.

LEFT JOIN (Region)
LOAD
*,
GeoMakePoint(lat,long) as [Country Point]
;
LOAD
purgechar([Alpha-3 code], ' "') as [Country ISO],
purgechar([Latitude (average)], ' "') as lat,
purgechar([Longitude (average)], ' "') as long
WHERE EXISTS ([Country ISO], purgechar([Alpha-3 code], ' "'))
;
SQL SELECT
"Alpha-3 code",
"Latitude (average)",
"Longitude (average)"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://gist.githubusercontent.com/tadast/8827699/raw/f5cac3d42d16b78348610fc4ec301e9234f82821/countries_codes_and_coordinates.csv");

This now lets us plot emissions data on a map by country. Our second data source though gave us Region and Sub Region names. It would be good if we could also plot by those regions. There is probably a data source out there which has SVG data for each region, but that could take a while to find and will not necessarily map to our existing region names.

We can apply a simple hack to generating points at each of the regions though.

RegionPoint:
LOAD
Region,
GeoMakePoint(avg(lat),avg(long)) as [Region Point]
RESIDENT Region
GROUP BY Region;

SubRegionPoint:
LOAD
[Sub Region],
GeoMakePoint(avg(lat),avg(long)) as [Sub Region Point]
RESIDENT Region
GROUP BY [Sub Region];

Here, a couple of RESIDENT loads from the existing Region table can be grouped to give one row per Region and Sub Region. For each of these rows the average latitude and average longitude can be calculated. These two values can then be changed to a point that can be easily used on a map. Note that some points may appear in the ocean, but that is the joy of using averages.

Finding Latest Data and Grouping Figures

The figures from our initial dataset are great for trending over time as they are provided on an annual basis. They are not so great for categorical data and providing filters though, as a country could move from one bucket to another year to year. For these filters we only want to use the latest figures. Here we hit another minor snag though, in that the last year in which data is available differs by country.

We can deal with this by finding the latest year for each country, using a RESIDENT, GROUP BY and MAX to create a composite key.

LatestPop:
LOAD
[Country ISO] & Year as LatestPop
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE Population <> ''
GROUP BY [Country ISO];

That composite key can then be used in the WHERE EXISTS of a subsequent RESIDENT load to only bring the latest data.

Population:
LOAD
Country,
Population as [Population Latest],
dual(num(Round(Population, 100000) / $(v1M), '#,##0.0') & 'M', Round(Population, 100000)) as [Population 100K],
dual(num(Round(Population, $(v1M)) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M))) as [Population 1M],
dual(num(Round(Population, $(v1M)0) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M)0)) as [Population 10M]
RESIDENT Emissions
WHERE EXISTS (LatestPop, CountryYear);

This is a useful technique, and one I use frequently in live projects. As well as getting the exact number for the latest year for each country we can also put things into buckets. This can be done with a massive IF statement or for equal buckets an aggregation. Here I have used a Round statement, but you could use a Floor or Ceil if you prefer. The Round function in Qlik is more flexible than in most languages as, rather than just rounding to a number of decimal places, you can specify what grouping you wish to go to, whether that is more or less than one.

The same technique can then be used for GDP, then there is just a little bit of housekeeping to do at the end of our script.

LatestGDP:
LOAD
[Country ISO] & Year as LatestGDP
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE GDP <> ''
GROUP BY [Country ISO];

GDP:
LOAD
Country,
GDP as [GDP Latest],
dual(num(Round(GDP, $(v1B)), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B))) as [GDP 1B],
dual(num(Round(GDP, $(v1B)0), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)0)) as [GDP 10B],
dual(num(Round(GDP, $(v1B)00), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)00)) as [GDP 100B]
RESIDENT Emissions
WHERE EXISTS (LatestGDP, CountryYear);

DROP FIELDS lat, long, LatestPop, LatestGDP;

So, that is it. Put all those bits of script together and you should have a working load script to bring in emissions and demographic data from three separate online CSV sources.

Now you just need a quick way to analyse those data. Fortunately we have your back there.

Viewing The Data In The Instant Sense Application

To make it quick for us at Quick Intelligence to produce Proof Of Concepts for potential clients I created the Instant Sense Application. This application allows you to load any data source, and a spreadsheet of metadata about that data source, and then explore the data over twelve sheets of user configurable visualisations and tables.

We have made this application available to all, at no cost, to make it quick and easy for others to showcase what Qlik Sense is capable of.

The application can be downloaded from here:

The Quick Intelligence Instant Sense Application

Details of how the application has been built can be found here:

How To Build A User Configurable Sense Application

If you already have the application, or if you just want the script above, you can download the script and meta-data here:

Global Emissions Data By Year with Metadata

If you use the application and build more expressions that you feel will be useful for others, please share them in the comments below. As ever, I’m happy to receive questions and comments from readers, and will get back to these as soon as I am able.

The post Loading Data from Online CSV Sources into Qlik appeared first on .

Logging on Qlik Sense Enterprise

With the default settings on logs in Sense Enterprise get written to two locations. One for the log file currently being written to and then an archive, where the current log is moved to periodically. By loading from both of those sources you get a good picture of what is going on with usage on your site. Connections are created to these two locations, named ServerLogFolder and ArchivedLogsFolder. There is a reasonable chance that you won’t see these connections in the Load Script Editor, so you will need to first go into the QMC and grant yourself rights – be careful not to grant more rights than you intend to when doing this!

Once you have access to these connections you can dig around and find out quite a lot about your system and how it is performing. This post, however, just focuses on the most important information. Who has done what and when.

Sense Session Log Load Script

Once you have permissions to the connections set up you can enter the Data Load Editor and start building your script.

First you need to set a couple of variables, which form part of the path to the session files. The first of these is generally the machine name, so can be left as is. The second tends to be a combination of the machine name and the local domain. You will need to click on the Select Data button underneath the connection name to confirm the path if its not what is expected.

// Set variables which identify this server and form part of the log names
// Check in the ServerLogFolder and ArchivedLogsFolder to find the values for your installation
let vServerName = ComputerName();
let vFullServerName = 'servername.domain.local'; // Usually FQDN

The next set of variables are then constructed from the connection name (which is usually the same in each installation) and the two variables you set above.

// Build out file paths and file masks based on the variables above
let vCurLog = 'lib://ServerLogFolder/';
let vLogFolder = 'lib://ArchivedLogsFolder/$(vFullServerName)/';
let vCurFile = '$(vCurLog)Engine/Trace/$(vServerName)_Session_Engine.txt';
let vArchiveFiles = '$(vLogFolder)Engine/Trace/$(vServerName)_Session_Engine_*.log';

As you would expect, there are a number of dates in the log file. Because the log is in a text file, rather than a database table, the dates are in plain text. This means they each need to be converted to dual-datatype dates so they can be used. The code for each of the conversions would need to be repeated, so here is a parameterised variable to do the job. If you have not used variables with parameters before you may want to see this previous blog post.

// Create parameterised variables for repeated tasks
set vDT = Date(Date#(left(replace($1, 'T', ' '), 15), 'YYYYMMDD hhmmss'), 'DD MMM YYYY hh:mm:ss');

The filename for the current log file has been built into a variable. Just in case we are between log files at this point we are going to check first that it exists. If it does we can bring the fields that we want from the file. If you are a completest you may want to look to the source file and bring through the fields I have dropped. This is pretty much everything that is interesting though.

Note that I am excluding sessions relating to reloads by the Sense Distribution service. If you are looking for log-jams on CPU use at specific times of day you may want to include these rows, but for most analysis they are not useful.

// Load the raw data from the current file
if alt(FileSize('$(vCurFile)'), 0) > 0 then
Temp_Sessions:
LOAD
ActiveUserId,
[App Title],
[Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser,
ActiveUserDirectory,
Timestamp,
[Exe Version],
[Server Started],
AppId,
[Doc Timestamp],
[Authenticated User],
[Secure Protocol],
[Session Duration],
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
FROM [$(vCurFile)]
(txt, utf8, embedded labels, delimiter is '\t', msq)
// Ignoring reload sessions. Remove next line if you want all sessions
WHERE ActiveUserId <> 'sa_scheduler' and ActiveUserId <> 'sa_repository'
;
end if

For the archive log files we need to enumerate around all of the files in the archive folder. Again, we already have the file-mask in a variable, so can just use this. You may be aware that a load can be done directly from a filename with an asterisk in it, to load all matching files. Personally, I would always use a for each loop, as it makes debugging and testing easier. You can also exclude files (such as files with a tilde when loading Excel files) and work with the CROSSTABLE statement, which fails when used with an asterisk in the file name.

// Concatenate onto the temporary table the raw data from all of the archive files
// Strongly suggest building an archive of QVD files for historical logs and
// concatenate these at the end of the process - this is not done in this example though
for each vFile in FileList('$(vArchiveFiles)')
Temp_Sessions:
LOAD
ActiveUserId,
[App Title],
[Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser,
ActiveUserDirectory,
Timestamp,
[Exe Version],
[Server Started],
AppId,
[Doc Timestamp],
[Authenticated User],
[Secure Protocol],
[Session Duration],
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
FROM [$(vFile)]
(txt, utf8, embedded labels, delimiter is '\t', msq)
// Ignoring reload sessions. Remove next line if you want all sessions
WHERE ActiveUserId <> 'sa_scheduler' and ActiveUserId <> 'sa_repository'
;
next

You should now have a single table with all of the log entries that are needed for analysis. There are a number of transformations and general tidying up that is required. This is done by doing a RESIDENT load from our raw-data temporary table.

// Format all raw values, calculate new values and rename fields
Sessions:
LOAD
1 as SessionCount,
*,
Date(DayStart([Session Start]), 'DD MMM YYYY') as Date,
Date(MonthStart([Session Start]), 'MMM YYYY') as Month,
Year([Session Start]) as Year,
Month([Session Start]) as [Month Name],
Hour([Session Start]) as Hour,
WeekDay([Session Start]) as Day,
Time(Round(
[Session Start] - DayStart([Session Start]),
(1/288)), 'hh:mm') as Time, // nearest 5min
Floor([Session Start] - [Server Started]) as [Days Since Server Start],
Floor(([Session End] - [Document Reloaded]) * 24) as [Hours Since Document Reload],
Round([Bytes Received] * 1024, 1) as [KB Received],
Round([Bytes Sent] * 1024, 1) as [KB Sent]
;
LOAD
ActiveUserId as User,
[App Title] as Application,
$(vDT([Session Start])) as [Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser as [Service User],
ActiveUserDirectory as [User Directory],
$(vDT(Timestamp)) as [Session End],
[Exe Version] as [Server Version],
$(vDT([Server Started])) as [Server Started],
AppId,
$(vDT([Doc Timestamp])) as [Document Reloaded],
[Authenticated User],
[Secure Protocol],
rangemin([Session Duration], 0.5) as [Session Duration], // clip as error can cause incorrect duration
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
RESIDENT Temp_Sessions
;

There’s quite a bit going on in the script above, so to take it one bit at a time… Field names are being renamed to make them more ‘friendly’ for end-users. The date variable created above is being applied to all date fields. I found the session duration occasionally has duff values (if a session looks to be resumed), so those are capped at 12 hours. A number of new fields are added in the preceding load, including a counter (something I always to to make things more efficient), all of the date parts and some derived values for further calculations.

As should always be done at the end of load scripts, there is then some tidying up of tables and variables to do.

// Drop the temporary table
DROP TABLE Temp_Sessions;

// Clean up all the variables we have used
let vServerName = ;
let vFullServerName = ;
let vCurLog = ;
let vLogFolder = ;
let vCurFile = ;
let vArchiveFiles = ;
let vCurFile = ;
let vFile = ;

And there you have it. Your Sense Enterprise log file, loaded, cleaned and ready to analyse.

So, How Do I Analyse My Sense Logs?

Obviously, you can build your own charts and tables over these data, but wouldn’t it be simpler if there was a pre-canned dashboard you could use? Well, we have your back on that one. Our Instant Sense Application can load any data-set and some metadata about those data and then presents it in a user-configurable dashboard. There is a short video of me presenting the application on YouTube, here: https://www.youtube.com/watch?v=jN3Vj93v9x4.

The application is driven by a spreadsheet which defines the fields to be used for dimensions (in filters, charts and tables) and also the measures to calculate over those dimensions.

If you already have the latest version of the Instant Sense Application you can simply download the zip file with the load script and metadata spreadsheet from here:

https://www.quickintelligence.co.uk/SenseSessions.zip

If you have yet to download the app you can do so for free from here:

https://www.quickintelligence.co.uk/isa

Once you have downloaded the app you will need to upload it to your Sense server (via the QMC). At this point you can rename it to Sense Session Logs or similar. Open the app in your Work area and go to the data load editor. Now find the SenseSesssions folder under DataSets in the zip file. Open the load script text file and copy and paste the script over the script in the Main Data section of the load script. The SessionLogLookups.xlsx file needs to be copied to a folder you have a connection to in Sense, and then the past to that needs to be set in the variable on the Main sheet of the load script.

Once this has been done it should just be a case of reloading and you have your Sense Session Log Analyser.

There are a number of other example scripts included with the Instant Sense Application which you might like to explore. If you are interested in how the application was built, that is all explained in one of our previous blog posts.

If you have any comments or suggestions related to this post please post in the comments below.

The post Analysing Sense Session Logs appeared first on .