Why Call A Webhook?

In my previous blog post (Start a Qlik Sense Cloud Reload from a Webhook) I described how to create a webhook using Qlik Automations which can start a reload of any app in your Qlik Cloud tenant. One of the places you might want to trigger this from is from a Qlik load script. This way you can call a reload of one or more Sense apps at the end of another script, perhaps dependant on data or status.

Triggering a Qlik Automation is one example of why you might want to fire a webhook, but in this post I want to share another example – raising an alert when things are broken.

How Do You Notify When Your Notifications Are Broken?

Something we always set up when building customer environments is sufficient notifications to make us aware when anything is wrong. On Qlik Sense Client Managed this is essential, as there is very little out of the box, and on Cloud it needs to be set up space by space, so some things can be missed. Sometimes we are working on a secure site where SMTP relaying from within the site is not permitted. Other times it can be that communication between the Qlik server and the SMTP server is down due to network issues, so the route to let us know there is an issue is broken along with the rest of the network. In these cases we need a way to get a message out, when nothing else is working.

Something that is likely to work in Sense, even when all else is failing, is calling a URL to extract data. If that URL is a webhook then that call can then trigger an action. What we have is a an applet set up in IFFTT (If This Then That) which sends an email via GMail containing values from the parameters of the webhook called. I won’t go into setting up that applet here, as it will be slightly different if you want to use a different mail client or call a different service, but I have given details on this in a previous blog post on Inserting Qlik Capacity Usage Into a Spreadsheet.

Once you have a webhook set up that can send the notification you are ready to build an app that triggers that webhook when there is a problem.

Raise a Notification When Files Go Stale

The most basic measure of everything being up and working is that files that are used by your Qlik environment are all up to date. If the applications you wish to monitor do not currently create any files then it is a simple job to add a STORE statement to write a small text file that can be checked for. The application we shall build here then has an inline table (which will work even if all else is broken) with a list of file locations and how old those files can go before we want to be told about them.

Before building the load script for this app you will need to set up a REST connector that we can pass parameters in to. To do this build a new REST connector, set it to a placeholder URL (I tend to use https://jsonplaceholder.typicode.com/posts), set the type to GET, ensure that Allow WITH CONNECTION is ticked and give it a name of Generic GET.

Once that connection is in place you can build the load script.

First you will need to set up some variables for use later in the script. Obviously the webhook parameter will need to be changed to match the webhook that you want to call.

let vEnv = 'My Qlik Sense Tenant';
let vLib = 'DataSpace:';
let vGET = '$(vLib)Generic GET';
let vQVDs = 'lib://$(vLib)DataFiles/';
let vTimestamps = '$(vQVDs)Timestamp/';
let vWebhook = 'https://maker.ifttt.com/trigger/error/with/key/your-webhook-key-here';

let vLastLoad = now();

Then you need to build the table of files that you want to be checking. These could be QVD files or small files that are written as part of a load script to indicate that the load script has completed. The table requires the full path to each file to be checked, how many hours old the file can get before an alert is raised and what label is to be used in the alert when that file becomes old.

tmp:
LOAD
    File,Threshold,Name
INLINE [
File,Threshold,Name
$(vQVDs)Sales.qvd,1,Sales Application
$(vTimestamps)QVDGenerateCompleted.csv,3,QVD Generation Application
];

That inline table can then be transformed to convert the hour given in the load script into a time delta to come off the current time, which can then be taken off the current time to work out the time that the file should be newer than. The actual date of the file also needs to be looked up. These two dates can then be compared to find the delta and to flag whether the file is old.

FileAges:
LOAD
    *,
    if(FileDate < Target, 'Yes', 'No') as Old,
    num((24)*(FileDate - Target), '0.00') as Difference
    ;
LOAD
    File,
    Threshold,
    Name,
    Date(alt(ConvertToLocalTime(FileTime(File), 'London'),0), 'DD MMM YYYY hh:mm') as FileDate,
    Date((LocalTime('London') - ((1/24)*Threshold)), 'DD MMM YYYY hh:mm') as Target
RESIDENT tmp;

DROP TABLE tmp;

This table then needs to be aggregated to find out how many of the files checked are older than they should be. That value needs to be Peeked out into a variable.

Old:
LOAD sum(1) as Old RESIDENT FileAges WHERE Old = 'Yes';
let vOld = alt(Peek('Old', -1, 'Old'), 0);
DROP TABLE Old;

Now we get to the actual point of the app and the demonstration. If there are any old files, we need to do the call to the webhook endpoint along with a payload that tells us which environment is affected and how many of the files are stale.

if alt(vOld, 0) > 0 then
    SET errormode = 0;
    LIB CONNECT TO [$(vGET)];
    SQL SELECT "col_1" FROM CSV (header off, delimiter ",", quote """") "CSV_source"
    WITH CONNECTION (URL "$(vWebhook)?value1=$(vEnv)&value2=$(vOld)%20Files%20Are%20Older%20Than%20They%20Should%20Be");
    SET errormode = 1;
end if

TRACE There are $(vOld) files older than expected;

To wrap up, because we are tidy Qlik developers, we clear down the variables that we have used.

let vEnv =;
let vWebhook =;
let vLib =;
let vGET =;
let vQVDs =;
let vTimestamps =;

Exit Script;

That is it for the load script, and that is all you need to make this work. What you might want to do though, as the list of files checked is in a table in the data model along with their status, is to add a table to a sheet in the app listing the six columns in the FileAges table, with some conditional formatting to show when files are old. I would also add a button, which can then refresh the app without going into the load script.

But What If Even This App Cannot Run?

If you are running Sense on a Client Managed server it is possible (although very unlikely) that even this application will not be able to run and call the webhook, perhaps if someone has unplugged the server in order to recharge their mobile phone. In this case we need an external service to monitor what is going on. If the server is (as is most likely) behind a firewall that external service will not be able to reach in and check the status. In this case we want a service outside of the network which is expecting to hear from the server at a regular beat which can then raise an alert if it doesn’t hear anything.

This is exactly what I am going to cover in my next blog post, pulling together elements of this post and the previous post. This solution will use a webhook on a Sense Cloud tenant which calls a reload of a script that writes a to a QVD. There is also a separate application that checks the date of this QVD and notifies if that has not been updated in a specified window. The history of the webhook being called is persisted, so a history of when the calling server was up and able to reach the outside world is recorded, giving an accurate account of uptime on the server.

If you have any comments on anything you would like to see covered in that blog post, or anything related to this blog post, please use the comments box below to get in touch.

Other Uses For Calling A Webhook From The Qlik Load Script

There is literally no end to what you can achieve by calling webhooks, particularly if you use a tool such as IFTTT.

Another use you might want to consider, if you are using Qlik Sense Cloud, is kicking off a reload using a webhook. In my previous blog post I described how a Qlik Automation can be set up to be triggered from a webhook to do a reload. If at the end of the load script for one app you call the webhook with the correct parameter to kick off the next reload in a chain this reload chain will work however the reload is started, interactively or from a schedule. This gets around the limitation that Qlik Sense Cloud reload chaining from the hub presently has, where the next reload in the chain only takes place if the previous reload was started from a particular schedule. This reload chaining approach also allows for logic to be applied around whether the next reload should take place or not, perhaps the front end should only reload if there has been a change to the backend data?

I hope this post has given you plenty of inspiration of how you can link different services together within, and beyond, your Sense environment.

The post Starting External Services From Qlik Sense Load Scripts appeared first on .

What are Webhooks?

Webhooks are a simple way of getting disparate systems to talk to each other. They allow the triggering of events simply by calling a URL, this could be from a browser bookmark, a link on a page or in code. I’ve previously blogged on a use of webhooks in the article Using IFTTT and WebHooks To Send Data From Qlik Sense.

In a Sense context the webhooks can be called by creating a Link in the Sense Hub, adding a link to a Button object, a value in a table set to HTML or by calling the webhook from within the load script (using a REST connector to make the call). Triggering from the load script allows the building of reload chains that are not dependant on the Sense scheduler, and for reloads that are data dependant – for instance calling a reload of another app only when certain data are found in the current app.

What Are Qlik Automations?

You may have noticed the button Automations on the left hand side of the Sense Hub. Under here is a real treasure trove of components that can be used to build complex event driven automations. A lot of the components integrate with your Sense environment, for instance sending Qlik reports to recipients based on loading addresses from an data source. There are also a number of components to integrate with services outside of the Qlik environment, such as Teams, Slack, Hubspot etc. etc.. The automations that you create here don’t necessarily even need to have a Qlik component involved, for instance you could have an automation that posts to Teams when a new row appears in a Google Sheet. There is a whole host of no-code, drag and drop, opportunities to explore here. For now we are going to create a simple webhook that receives the GUID of an application and reloads it.

Creating the Automation

From the left hand menu in the Sense Hub select Automations, and then click the Create New Automation button at the top right. If you don’t see these options then you don’t have the permissions to create Automations, and will need to get this granted to you by an administrator via the Sense Console.

You will now see a large number of pre-canned templates for automations (which you can explore later), for now though select Create Without Template at the bottom left of the dialog. You will now be prompted to give your new webhook a name and description, and choose where to store it:

You will be presented with an almost blank workspace, just with a single Start block in the middle. The properties for this block are on the right hand side. Change the Run Mode to Triggered. You will then see the code, including the execution key you will need to trigger the automation.

Next you need to add an Input block, to receive the GUID for the app you wish to reload. On the left hand side search for Inputs then drag the inputs component immediately under your Start block, you will see a green dot when it is in place and when you drop the new block should be connected by a straight line. Set the Label to app and the Input Type to Data.

At this point you need to inject a value into the app parameter, so that it can be used in the next step. First you will need to save the automation using the Save button at the top of the page. Then, from the POST example on the Start block (pictured above), copy the text from https:// through to the end of the execution token (in this case guV1). Paste this whole string into a new browser tab and make the following edits: add a ? after the word execute, delete from there up to X-, replace the colon and space with an equals symbol, then add to the end of the URL &app=x. So, given the example pictured above the complete URL would be (with the * characters replaced with your details):

https://****.**.qlikcloud.com/api/v1/automations/****-****-****-****/actions/execute?X-Execution-Token=***********&app=x

Click enter to call the URL, and you will simply see [] returned in your browser. Leave that browser tab open (you will need it shortly) and return to the Automation editor.

The next bock you require is the one that actually triggers the reload. On the left hand side click on Qlik Cloud Services and search for the Do Reload component. Drag this under the Inputs component. In the properties panel drop down under the App Id field and select Inputs (this relates to the previous node) then click History and select app: x from under Inputs (this is what you passed in during the previous step). Set the Run Mode to be Start reload and continue.

Finally you need to add an Output node, so search for this on the left hand panel and then drop it after the Do Reload node. In the Data to output properties, you should see that it has the Do Reload node as the output it will send as a result. If it does not, drop down here and select Output from Do Reload and then Do Reload.

That is the Automation completed, so click save up the top of the sheet to finish.

Now you will need the GUID of an application that you want to reload. The GUID can be found in the URL when you have an application open, you will find it between app/ and /overview or /state. It will look a bit like this: 0e316df1-fd5c-4cd0-bac1-2f66a03b4c3d. Copy this to your clipboard and return to the tab where you called the webhook previously. Replace the x at the end of the URL with the GUID and return again. This time you should get some JSON returned, confirming that the reload has started. Check the details of the app that you have chosen to reload in the Hub and you should be able to see that a reload has started.

Calling The Webhook

As you have seen, you can call your new webhook simply by pasting it into the URL of a browser. It can also be bookmarked, for ease of use. You can create different bookmarks for each of the different apps you want to allow people to reload and most browsers will allow you to export bookmarks to give to someone else to import into their browser. The Sense hub now allows you to add links to the hub, so you can create links in there. Buttons in Sense allow you to call URLs, so that is somewhere else you can call the link.

The resultant webpage is a bit messy, so you might want to have a page that calls the webhook with javascript and parses the output and gives a friendlier results page, or perhaps just triggers a back action to take the user back to where they came from.

To call from a Qlik load script you will need a REST connector created which has the webhook as the URL. If you have a lot of apps you wish to reload in this way you will want to have a placeholder URL in your rest connection and then replace the URL in the script using a WITH CONNECTION statement. I describe how to do this in the blog post I referenced at the top of this article: Using IFTTT and WebHooks To Send Data From Qlik Sense.

Hopefully that gives you what you need to be kicking off reloads from various different events and in various different ways. If you have used similar techniques please feel free to share them in the comments below.

The post Start a Qlik Sense Cloud Reload from a Webhook appeared first on .

Data Without a Source

Typically a Qlik application will load its data from a data source, such as a database, API or flat file. Sometimes you just want to create something on the fly. The AUTOGENERATE function is a great way of doing this, and many of the applications on my Qlik Examples page use just this functionality. The function gives a specified number of rows of data that you can then work from. To create a table with a single field with values from 1 to 1,000:

LOAD
    RowNo() as Value
AUTOGENERATE(1000);

This may be useful in and of itself, but it is more likely you will need to do more. Read on to find out some of the things you can do.

Well, That’s A Bit Random

Often you will want to create some random values, and for this the rand() function is what you require. This function will return a number which falls somewhere between 0 and 1. Whilst at first this sounds a bit limited, with some very basic maths you can create many kinds of values.

So, to get a random integer between 0 and 999 you could do the following:

LOAD
    floor(rand()*1000) as Value
AUTOGENERATE(1000);

If you wanted to get a relatively random number of dimension values from a list you can multiply the random value by the number of values you have, round up to the nearest integer and then use a pick() function to choose which dimension value to pick:

LOAD
    pick(ceil(rand()*5),'Ben','Ian','Tom','Paul','Olly') as Person
AUTOGENERATE(1000);

Note with this approach though you will get roughly the same number of rows for each of the values, due to how random numbers work. If you want to have less regular number of rows you will need to introduce some bias.

If you want to generate random codes that look like they could be real (but are not) you can use the fact that the chr() function picks an ascii character based on a number. To get a code that looks like ABC123 you would have:

LOAD
    chr(65+floor(rand()*26))
        & chr(65+floor(rand()*26))
        & chr(65+floor(rand()*26))
        & num(ceil(rand()*999), '000') as PersonID
AUTOGENERATE(1000);

Given the number of possible values that this could generate you would most likely get all unique values. If you would prefer to guarantee that there are no duplicates you could use rowno() instead of rand(). If you wanted to cause more duplicates you could reduce the number of characters and the multipliers on each of the random values, perhaps reducing the letters to A to D and the digits to all be multiples of 9:

LOAD
    chr(65+floor(rand()*4))
        & chr(65+floor(rand()*4))
        & num(ceil(rand()*99)*9, '000') as DupedID
AUTOGENERATE(1000);

If you want to have data shown on a line chart you can generate dates, using the fact that dates are just numbers under the bonnet and we can therefore do maths on them. Again, if you want one row per date then use the rowno() function, or if you want more randomness around the rows per day use rand():

LOAD
    date(today()-(rowno()-1), 'DD MMM YYYY') as Date,
    date(today()-floor(rand()*90), 'DD MMM YYYY') as [Date Rolling 3m]
AUTOGENERATE(1000);

Creating dates in this way has been particularly useful in some of the demo apps I’ve created, as a simple reload will make the demo look fresh and current as all dates are in a period leading up to the current date.

That’s just a few examples of generating test data using AUTOGENERATE, but I am sure you can think of many more.

But Does It All Correlate?

I mentioned at the top that I had a recent requirement to test the correl() function, and generated some data to do this. For those who have note come across the statement before, it returns the aggregated correlation coefficient for two data sets. Or put into English, a value is returned which is close to 1 if there is a very strong positive correlation, close to -1 if there is a strong negative correlation and close to zero if there is very little correlation between two sets of values.

The function is called like this:

correl(aggr(sum(X), Dim), aggr(sum(Y1), Dim))

This is where Dim is a dimension value in the data and the two aggr statements create two datasets with a row for each dimension value. The sum(X) and sum(Y) expressions would usually be more complex calculations, but in this case it is simply the numbers generated in the load script.

The load script to make this expression work is as follows:

LOAD
    'Row ' & num(RowNo(), '#,##0') as Dim,
    RowNo() as X,
    RowNo() as Y1
AUTOGENERATE(1000);

Because we are using the row number for both the X and Y1 values there is an absolute positive correlation and the function returns 1, as it should.

For all subsequent tests we keep the Dim dimension and the X value, which always gives us a point along the bottom of the chart. For a negative correlation the value of Y can be calculated as:

1000 - RowNo() as Y2

So now when X is 0 Y is 1000 and vice versa. This is an absolute negative correlation, and the correl function returns -1, again as it should.

The rest of the values for Y which I tested for where as follows:

Rand() * 1000 as Y3

A totally random value, which gives a correlation coefficient that is very close to zero and a plot that is full of randomly placed dots.

RowNo() * Rand() as Y4

A random value between 0 and 1 multiplied by the X value, so that the Y value is random, but never greater than Y. This gives a correlation coefficient that is around 0.5 and a chart where the bottom right half is filled. This could be inverted, but I did not try that value.

(RowNo() * (Rand()-0.5)) + 500

A value that deviates from the centre of the chart on the Y axis by a random number that is plus or minus half of the value of Y. As the values fall above and below the line equally the correlation is very close to zero.

RowNo() + ((Rand()-0.5)*(RowNo()/1)) as Y6

This value is similar to the X=Y chart we started with, but with a random deviation from the line in either direction, up to half of the Y value. The divide by 1 is left in there as I was experimenting with having the deviation greater or lesser by dividing the row number. Correlation coefficient is approaching 1, but is not there.

500 + ((rand()*10)-5) as Y7

This value has a similar Y value for every value of X. I started off having it exactly the same for each X value, but the correl function requires some variance in the number or the spread will be zero causing a divide by zero and no correlation coefficient number. A random deviation of up to 5 in either direction provides this difference, but that deviation is small enough that the correlation coefficient is very close to zero.

500 + ((rand()*10)-5) as X2

The final expression is a bit different to all the others, as I wanted values from 1 to 1000 for all Y values in order (I use Y1 from above for this) and then have a calculation for the X value that is close to the same value, but off by a small margin. You will note the expression for X2 is identical to Y7, and I could have used the same field in the correl function, but it felt wrong to have two values labeled Y in the chart.

The resultant scatter charts and correlation coefficients for all of the derived values above can be seen in this screenshot (click to expand).

The entire load script for the Correl application looks like this:

LOAD
    'Row ' & num(RowNo(), '#,##0') as Dim,
    RowNo() as X,
    RowNo() as Y1,
    1000 - RowNo() as Y2,
    Rand() * 1000 as Y3,
    RowNo() * Rand() as Y4,
    (RowNo() * (Rand()-0.5)) + 500 as Y5,
    RowNo() + ((Rand()-0.5)*(RowNo()/1)) as Y6,
    500 + ((rand()*10)-5) as Y7,
    500 + ((rand()*10)-5) as X2
AUTOGENERATE(1000);

And the entire application can be downloaded from Qlik Community here:
https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Create-Test-Data-For-Correl-Function-Using/ta-p/2139164

Hopefully that has given you some inspiration what you can do when you need to generate some test data. I would be very interested to hear of other thoughts you might have on the topic, so please do use the comments box below to let me know.

The post Generating Test Data In Qlik Load Scripts appeared first on .

Background to Using the API

The Qlik SaaS API can be used for retrieving information regarding your SaaS environment and also for making changes in your environment, such as allocating permissions. The endpoints available to end-users are the same endpoints that Qlik use to build the product. You can interact with these via REST connectors, Qlik Automations and third party tools; such as Göran Sander’s Ctrl+Q. I have blogged previously on using these APIs in the Sense load script, and describe the process of setting up and calling them in my post on Monitoring and Notifying Sense Capacity Licence Usage in Qlik Cloud. If you are not yet using APIs in your own applications then you might want to start there.

Required Changes to Fix Broken Loads

Quick Intelligence provide a free download of what we call the Instant Qlik Sense Application. This is a Sense application template, which ships with a number of example load scripts and configuration files. Two of these example scripts are for monitoring Qlik SaaS usage and sessions. Both of these scripts have been broken by the change to the API.

Fortunately the change to the API endpoint is relatively minor and the fix simple. The revised documentation for the endpoint can be found here https://qlik.dev/apis/rest/licenses.

If you wish to modify the script in a an app you already have running you simply need to find the Licences part of the script and change references to userId and UserId to subject and Subject respectively, and then remove references to name and User Name.

There is a further gotcha that I have spotted whilst changing this over though, and that is that the case of the subject field on the Users endpoint does not always match the case of the same field on the Licences endpoint. You should therefore add an upper statement around Subject in the creation of the mapping table and where it is used. If you are using the endpoints and associating the tables rather than mapping you will need to match the case in the tables themselves.

If you use these endpoints in your own scripts, or have downloaded an application that uses them, then you are likely going to have to make similar changes to that load script.

If you don’t yet use the Instant Qlik Sense Application then you can find out more about it, and download it from here:

https://www.quickintelligence.co.uk/instant-qlik-sense-app/

If you would just like the script for using the Sessions or Usage endpoints of the API these can be downloaded here:

https://www.quickintelligence.co.uk/SaaSUsage.txt
https://www.quickintelligence.co.uk/SaaSSessions.txt

With the SaaS usage script and configuration file, Sense users are coloured by their licence types and if you have a mix of both analyser and capacity licencing you can ensure that the users who use the system the most are the ones who are holding analyser licences whilst occasional users can use capacity.

If you have any questions or comments on using the API or the Instant Qlik Sense Application then please use the comments section below.

The post Change to Qlik SaaS Licences API Endpoint appeared first on .

The Challenge

The question was posed on Qlik Community how to overlay two 12 month measures in one chart. Using set analysis it is straight forward to have two measures which cover the correct date ranges, but when the Month dimension is used it will sort by calendar month, rather in the order required for rolling periods. You can see the question and my response in this post.

The response uses a technique I have used a number of times before but it occurred to me that I have not documented it on the blog, so here it goes.

Creating Rolling Month and Year Dimensions

This code can either sit in your master calendar code, should you use one, or you can place it directly in the load statement from source data. Typically I will go for the latter approach.

The first thing that we need to do in the load script is to create a sequential integer for each month. This then allows us to work out the number of calendar months between any two dates. This technique has many more uses than just rolling twelve month analysis and is an incredibly useful dimension to have in your data model.

If we do (Year(Date)*100)+Month(Date) we will get a sequential integer value within a year (e.g. as at now it will give 202305), but the deltas break when you go across a year (e.g. comparing 202212 to 202301). In order to fix this we can just multiply the year by 12 and add the month. So, again taking May 23 as an example you will get 24,281. This is obviously not at all useful as a human readable number (just as telling someone that the date today is 45051 is only useful if they are a machine), but it means that the numbers for Dec 22 (24,276) and Jan 23 (24,277) are now just one integer apart.

In the load script we can put this code in to create a new field called MonthNo like this:

(Year(Date) * 12) + Month(Date) as MonthNo,

Now we have that value we need to work out how many months back any given month is from today. The code that we used on the Date field can also be done on the today() function, giving a comparable figure for the current date. Using a preceding load, we can compare that to the MonthNo field to get a value for Months Back, like this:

(Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]

This new field in itself may be useful in your charts and filters within Sense, for instance it could make your set analysis for a chart showing the last twelve months much easier as you do not need to deal with date formats in the set code.

Jumping up another level of preceding load we can now use the Months Back value to create the Rolling Month and Rolling Year values.

The Month function gives us a dual data-typed value with the three letter month name and the integer order for the month from 1 to 12. The text part of that dual datatype is great for us but the number does not give us what we want.

We can use the Dual function though to build our own dual data-typed value with the correct sequential number. Using the output of the Month function as the first parameter of the Dual statement will only take the text part of that value. The correct sequential number can be calculated by working out how many months back the start of the year was and then taking away the months back for the date in question.

The whole of that code looks like this:

Dual(Month(Month), (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],

We can get the number of rolling years back from our Months Back value by taking the integer part of the number of months back divided by 12. In order to turn that into text to show as a dimension we want to suffix it with the text “Years Back”. As having “0 Years Back” looks a bit naff, we need to have anything in the past 12 months showing as “Current Year”. Having that text means that sorting on the field breaks (as it would if you have more than a decade of data), so we need to use the dual function again.

Putting all of that together gives us this statement:

Dual(if([Months Back] < 12, 'Current Year',
    Floor([Months Back]/12) & ' Years Back'),
    Floor([Months Back]/12)) as [Rolling Year]

We then have the two dimensions that we can use in a line chart:

Putting It All Together

This code generates a whole bunch of dates and then applies the date code described above in a resident load. You can copy and paste this to a new app and see the code working in isolation, or your can take parts of it to apply to your own data.

tmpDate:
LOAD
    Date(today()-RowNo(), 'DD MMM YYYY') as Date
AUTOGENERATE(2000);

Dates:
LOAD
    *,
    Dual(Month(Month),
        (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],
    Dual(if([Months Back] < 12, 'Current Year',
        Floor([Months Back]/12) & ' Years Back'),
        Floor([Months Back]/12)) as [Rolling Year]
    ;
LOAD
    *,
    (Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]
    ;
LOAD
    Rand() * 1000 as Value,
    Date,
    Date(MonthStart(Date), 'MMM YYYY') as Month,
    (Year(Date) * 12) + Month(Date) as MonthNo,
    Month(Date) as [Month Name]
RESIDENT tmpDate
;

DROP TABLE tmpDate;

Something else you might want to deal with when looking at rolling months is a run rate value for the current month, which takes into account how far through the month we are and what the value will be if extrapolated to the number of days in the month. I have covered this in a previous post.

Not only does this code give you dimensions that you can use in your charts, the same dimensions can be used in set analysis also. To create the variance to prior rolling twelve months gauge at the top of the chart you can now just use this code:

(Sum({<[Rolling Year]*={'1 Years Back'}>}Value) /
Sum({<[Rolling Year]*={'Current Year'}>}Value)) -1

An example QVF file including this code and some example charts can be downloaded from the Qlik Community post.

I would be very interested to hear from you on this technique and whether you have used similar or can suggest other ways it could be used. Please use the comments below if you would like to add your thoughts.

The post Compare Rolling Twelve Month Period in Qlik appeared first on .

Instant Sense Application Live Demos

It is sometimes to hard to describe what the Instant Sense Application is or how it functions. The simplest way to get your head around it is to see it working, with data that means something. In order to achieve that I set up auto-authentication on a SaaS tenant to allow anonymous usage of some ISA demos.

My thanks go to Daniel Pilla at Qlik who helped me get this working. The instructions to set this up can be found on Qlik Dev here: https://qlik.dev/embed/embed-quick-starts/embedding-with-anonymous-access-and-qlik-cloud. Whilst there is no equivalent to Tableau Public from Qlik, setting this up on a SaaS Tenant with a Capacity licence is the best way to go.

This has allowed me to post live demos to my socials, and to the Instant Sense App landing page. There is also now a live view of Qlik Blog Posts over on the AskQV site.

Download Without Signing Up for Updates

When the Instant Sense Application was first released it was a requirement that users signed up with a valid email address and confirmed that address. I noticed that a lot of people were hitting the ISA page and then not giving their details. Fair enough, I might well do the same. For that reason, we removed the requirement to sign up to download the app. If you would like to hear from Quick Intelligence you can still sign up for our newsletter, but if you would prefer just to download the app you can do so by clicking here.

To me this application is more important as a way to aid Qlik Sense adoption and to make it much quicker and easier to get data, and insights, to users than it is as a promotional tool for Quick Intelligence.

Please take the application and do something great with it! If you do find it saves you days of effort building applications it would be great if you could like, share, comment and all those good things on our social posts. I would love to hear what you are doing with the app.

New Example Scripts and MetaData

The Instant Sense Application ships with a number of example scripts and metadata files, so applications over these data can be quickly spun up and used in your own environment. Pretty much all of these have been tweaked to make use of the new features (more on those below), some also have new expressions and dimensions. As well as this new examples are being added to the download.

Pretty much from the first release, there have been session monitors for both Sense Client Managed and for Sense SaaS, sadly the SaaS one was limited by the API that was being called and didn’t show session duration.

Since that original SaaS example script was created, Qlik have exposed a new API endpoint which shows licence consumption, allowing a much more detailed view of who is consuming which licences. By using this script Sense SaaS administrators can get a good view on who should be holding a named licence and who should be consuming capacity, based on their recent activity. The currently held licence type is denoted by the colour. The client-managed Session monitor has been expanded to better facilitate this also:

Another new script file that was added this year is for profiling libraries of QVD files, by reading the meta-data from the QVD header. It gives a count of QVDs and stats around the size and ages of those QVD files. This app can help you identify redundant files in your environment and warn when the size of files is increasing. Any one who has a QVD layer in their environment (and that should be everyone) can make use of this script.

I’m looking at adding more example scripts to the download in time, both Qlik related ones and other non-related data sources. Watch this space.

New Application Features

There have been a number of tweaks made to the application over the year, some of the more interesting and useful ones are listed here.

The first I am going to mention you have already seen above. All of the charts (except the KPI text, annoyingly) now have a colour expression on them which takes parameters that define the chart type, dimension number and measure number.

By modifying the expression in that variable dynamic colours can be applied throughout applications. These colours can be based on the dimension, expression, chart type or even be derived from values in the data, as shown here. Documentation is provided in the download showing various options for applying colours.

To further facilitate colours, and to allow the adding of bespoke functionality to your apps, there is now a Variables sheet in the metadata file. This can be used to inject variables into the app at load time. You could create all of your measures as variables, should you want.

The button functionality within Sense has improved, so there are a number of buttons to help with UI in the Instant Sense Application now. The Scatter chart has a button to toggle which expression is shown on the X and Y axis and the Top 20 trend has buttons so that the number of lines shown can be increased or decreased.

Similar buttons appear on some of the new sheets straight out of the gate. There is also a button (which is a bit work-in-progress) to link to Google Maps from the Map sheet.

The ability to use Google Spreadsheets for the meta-data has been added, so that your online application can read its settings from an online data source, taking away the step of having to upload the configuration to Cloud after every change. Using Office 365 for this also is on the to-do list.

The map sheet now hides itself if you don’t have any geographical dimensions defined in your meta data. I’ve also tweaked some of the settings on the map, so that when you have only one point selected it doesn’t zoom to a point that you can’t see any context. You might want to adjust these settings for your own purposes, but I believe the defaults are better now.

Whilst it is best practice to use Master Dimensions for everything, I have decided to strip the Master Dimension out of the Sense Profiler sheet of the application. The reason for this is that this sheet is a useful addition for any Sense application, and now it can be copied and pasted into other apps without any faff. The sheet gives you a list of all tables and fields in a data model and then shows a chart of the number of occurrences of each value in the selected field.

New Sheets And Chart Types

The most recent addition to the Instant Sense Application is the Dual Map sheet. This will appear if you have two or more location type dimensions in your dataset. The first selected dimension shows on the left and the second on the right. This can be used to show different levels of granularity within location data, as shown here, or to show to and from locations for shipping or travel analysis.

There have been a few chart types that have been absent from the application, until now, as they only work with expressions that accumulate to a total. A new column has been added to the Expressions sheet of the meta-data that defines which expressions are part-to-whole. If you have one or more of these in your data then the new sheets appear, giving you pie charts (sorry), a tree map and a Sankey chart. All of these work the same way as the existing sheets, with the ability to pick dimensions and any part-to-whole expression.

Both the Sankey and the Tree Map have the ability to toggle which is the primary and secondary dimension, which gives much flexibility in how things are shown.

Look To The Future Now, It’s Only Just Begun

Whilst extending and expanding the Instant Sense Application isn’t something I get to do as much as I would like, there are ideas I have for it. As we have deployed the application on client sites there are change requests from clients also. These often find their way back into the download file. I’m also always happy to hear suggestions from the Community on how the app could be improved.

One of the ideas I have presently is to add Dimension and Measure groups, so that these can be selected from a drop-down in one go, rather than picking them separately. For instance, with the Sense Usage apps I tend to pick a view that has Total Usage, Today’s Usage, This Weeks Usage, Rolling 30 Day Usage and Days Since Last Licence Use, all shown by User, to pick which users should be allocated or de-allocated licences. With no hide condition on Filter Panes (yet) they would leave redundant objects on the screen if not configured. Besides, bookmarks provide quite a neat way of saving groups of items – so this has remained on the back burner.

Something I have found myself doing on a number of roll-outs of the app is to add a button to each sheet. On one of the demos I have added a button to link to a web page for further information, elsewhere I have added a button to reload the app. What I am considering is adding a button on each sheet, which only appears if the destination and legend for the button is defined in the metadata spreadsheet. I think this could be a nice addition, but not one that has bubbled to the top of the list yet.

The beauty of the Instant Sense Application is that the one application can have many different data sources and meta-data files loaded into it. If I create any more, which is likely to happen, then these will be added to the download. It was also my hope that other people might contribute script and meta-data files for other data sources. If you have the Instant Sense Application pointing at a data source that might be useful for others, please get in touch to see how we might roll that out.

If you fancy creating a load script and meta-data file, perhaps over data related to a hobby or interest, then it would be great to hear from you also. That app could be made available either part of the application download or as a hosted demo.

If you are using the application, for any reason, it would be great to hear from you. You can comment on this blog post, on our social posts, or get in touch with me directly.

Thanks everyone! It has been incredible to see how many times this application has been downloaded, and I really hope it is helping you to deliver better analytics in less time.

The post Instant Qlik Sense Application Changes in 2022 appeared first on .

Why Use GMail For Sending and Receiving From Qlik?

There are a wide number of features within the Qlik ecosystem that require interaction with email. QlikView Enterprise allows sending of alerts on task failures, NPrinting is all about distributing reports and Qlik Sense SaaS has multiple features for sending mails, from task failure notifications to subscriptions to charts or sheets. Beyond the obvious use cases there are many more reasons why you may interact with a mail server, such as Sending Notifications From The Load Script or even using a mail server as a data source, such as in this example of loading router logs.

Many companies use Google Workspace for their internal systems, so using a Workspace account for sending and receiving mail makes sense, and for individuals learning Sense spinning up a GMail account for this purpose is a no cost way of achieving the same result.

Why Allow Access For Less Secure Apps?

Google, and general security best practice, recommend using 2FA for authentication. Google Accounts implement this via Google Prompts (on a signed in phone), text message, the Google Authenticator app or a physical device. Having a third party application (or, ironically, Google’s own email migrate tools) connect to your account via SMTP, IMAP or POP3 is deemed less secure as it inherently uses a username as password only to connect.

The solution for this, for the past couple of years, has been to set a switch on the Google Account that is being used in this way to say that you are comfortable with username and password access to the account in this way. Recently though, a new message has appeared on the dialog where this switch is set:

It advises that this access will be revoked from the 30th of May, and the Find Out More link gives a bit more background, but not quite enough. The long and the short is though, that if you are using GMail servers after this date, without an App Password, they will no longer function correctly.

So, How Do I Fix This?

At the bottom of that article there is a link to information on Signing In With App Passwords, this is a much more helpful document. The steps to resolve the issue are outlined here though.

Head to Account Settings and Security for the account you wish to use

Turn on 2FA for the account you wish to use

Select to create an App Password for this account

Select Other from the app list and enter Qlik Sense or similar

Copy the generated password to your clipboard

Use the App Password as you would your normal password

You can come back into the App Password settings to manage old passwords and remove them when you no longer use them. Note that when you change the password on the account any App Passwords will be removed, so you will need to recreate these and plumb them back in when this happens. Note that you can not recover an old App Password, so make sure you enter it everywhere you need to before you lose it from your clipboard (or put it in your password safe).

Once all of this has been done your Qlik apps and servers can interact with your Workspace or GMail account without issue, and they will continue to do so after the date which Google removes support for less secure apps.

I hope this post helps you avoid any nasty surprises next month.

The post Google SMTP and IMAP to break in Qlik Apps Next Month appeared first on .

API Connections in Qlik Sense Client Managed

Straight of the box with a clean install Sense has a number of REST Connections created for you to explore your environment. It’s well worth exploring what these connections offer, with regards to looking at users and usage.

For this post though we are going to be looking at the monitor_apps_REST_app connection.

Create a new app and go to the Data Load Editor. If you can see the monitor_apps_REST_app connection we are good to go. If it’s not there you will need to grant yourself rights in the QMC under the Data Connections settings, find the correct connector and select Security Rules.

From here you can grant yourself access to the one connection by your name or user id:

Or alternatively, if you are a RootAdmin on the QMC you may want to grant the RootAdmin role access to all connections on the server. This will mean that administrators can all explore all of the default connections and keep an eye on what users are creating also. Do this by replacing the GUID of the connection in the Resource Filter with an asterisk. Obviously, you can tailor the permissions to be whatever you need.

When you go back to the Data Load Editor in another tab the connections will not have appeared automatically. You will have to refresh the browser with a Shift-F5, or click the refresh button in the browser toolbar.

Loading Data From REST API Connection

Now you have access to the correct REST connection, you can use the Select Data icon next to the monitor_apps_REST_app connection to look at the data. Alternatively, you can use the script here to pull the information that you need.

Before we connect to the data though, we are going to set up some variables which describe where the apps currently reside and where we want to back them up to. The apps will be in your repository, the location of which will have been set when Sense was installed. There will be a share, most likely on the Sense server, pointing to this location. In here find the Apps folder, which contains a file for each app, with a GUID for a name and no extension. The destination is simply where you want to copy to. Here I create a dated folder for the backup:

// Set up locations of Sense Repository and a backup location
// Used in the backup script field
let vSrcFolder = 'c:\QlikSenseRepository\Apps\';
let vDestFolder = 'c:\SenseBackups\AppBackup' & date(today(), 'YYYYMMDD') & '\';

A connection then needs to be made to the REST API in order to load data from it:

// Connect to the REST connection
LIB CONNECT TO 'monitor_apps_REST_app';

With this in place we can then pull the data from the REST connection, using this code:

// Load raw JSON Data
RawRESTData:
SQL SELECT
"id" AS "id_u1",
"createdDate",
"modifiedDate",
"modifiedByUserName",
"name" AS "name_u1",
"publishTime",
"published",
"description",
"fileSize",
"lastReloadTime",
"thumbnail",
"savedInProductVersion",
"availabilityStatus",
"schemaPath",
"__KEY_root",
(SELECT
"userId",
"userDirectory",
"name",
"__FK_owner"
FROM "owner" FK "__FK_owner"),
(SELECT
"name" AS "name_u0",
"__FK_stream"
FROM "stream" FK "__FK_stream")
FROM JSON (wrap on) "root" PK "__KEY_root";

As with loading data from just about any REST connection, the data will come down in a single table, even though there are three logical tables in the data model. Each row of data will have values in columns in one of the three sets (Application Details, User Details and Stream), values in the other columns will be null. Along with these values are primary and foreign keys to join things correctly.

Using Select Data the Qlik engine will create some code which will load three separate tables into the end data model. In this case however we know there is a one-to-one relationship between an Application and its Owner and the Stream it is in (if the application is published). We can therefore create a cleaner and flatter data model with ApplyMap.

There is much more we can do also, to make it easier to select the apps we want to back up, and do further analysis on apps. The same data feed can be used to help use identify apps which are not being maintained or are bloated.

Cleaning Up The Application Table

Now we have all of the data in a temporary table we need to clean it up and augment it. This will be done with a number of RESIDENT loads and a couple of Preceding Loads.

First we are going to create some ApplyMap statements to bring in the Owner name, Owner login and Stream. These use the foreign keys in the flat table.

// Create mapping tables from parts of the REST result
Map_Owner:
MAPPING LOAD
[__FK_owner],
[name]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_OwnerLogin:
MAPPING LOAD
[__FK_owner],
[userDirectory] & '\' & [userId] as Login
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_Stream:
MAPPING LOAD
[__FK_stream],
[name_u0]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_stream]);

Then some variables which act as functions in the subsequent RESIDENT load. These all utilise the fact that a SET statement does not evaluate the code that it puts into a variable (rather it copies code verbatim) and the functionality of passing parameters into variables.

// Create parameterised variables for repeated tasks
set vTime = Date(if(Date#(left($1, 10), 'YYYY-MM-DD') < 0, null(),
Date#(left(replace($1, 'T', ' '), 19), 'YYYY-MM-DD hh:mm:ss')),
'DD MMM YYYY hh:mm:ss') as [$2 Date Time];
set vDP = Date(DayStart([$1 Date Time]), 'DD MMM YYYY') as [$1 Date],
Date(MonthStart([$1 Date Time]), 'MMM-YYYY') as [$1 Month],
Month([$1 Date Time]) as [$1 Month Name],
Year([$1 Date Time]) as [$1 Year],
WeekDay([$1 Date Time]) as [$1 Day],
today() - DayStart([$1 Date Time]) as [Days Since $1];
set vNS = if($1 = '', 'Not Specified', $1);

The first function here receives the name of a date field and checks whether it has a value, if it does it returns that converted to a date and formatted correctly. The second takes the date time field created by the previous variable (thanks to a preceding load) and creates various date parts for these dates. The final variable checks a given field name and returns Not Specified if the value is empty, otherwise it returns the value.

You will see all of these variables used in the next block of code.

We can then do our final RESIDENT load to bring through all of the original data, with the various transformations applied.

Applications:
LOAD
1 as ApplicationCount,
*,
RangeMin([Days Since Created],
[Days Since Modified],
[Days Since Published],
[Days Since Last Reload]) as [Days Since Last Touch],
// Export this column to create a script to backup your apps from the Repository
'copy $(vSrcFolder)' & GUID &
' "$(vDestFolder)' & Stream & ' -- ' &
Application & '.qvf"' as [App Backup Script]
;
LOAD
*,
// Create all required date parts
$(vDP(Created)),
$(vDP(Modified)),
$(vDP(Published)),
$(vDP(Last Reload))
;
LOAD
id_u1 as GUID,
// Convert all required dates
$(vTime(createdDate, Created)),
$(vTime(modifiedDate, Modified)),
$(vTime(lastReloadTime, Last Reload)),
$(vTime(publishTime, Published)),
[modifiedByUserName] as [Modified By],
[name_u1] as Application,
[published] as [Is Published],
$(vNS(description)) as [Application Description],
fileSize / 1000000 as [File Size],
$(vNS(thumbnail)) as Thumbnail,
[savedInProductVersion] as [Product Version],
[availabilityStatus] as Availability,
ApplyMap('Map_Owner', [__KEY_root], 'Not Known') as Owner,
ApplyMap('Map_OwnerLogin', [__KEY_root], 'Not Known') as [Owner Login],
ApplyMap('Map_Stream', [__KEY_root], 'None') as Stream
RESIDENT RawRESTData
WHERE NOT IsNull([__KEY_root]);

Then, to tidy things up and avoid any strange joins, we can remove the temporary table which was created when the data was pulled from the REST connection.

DROP TABLE RawRESTData;

And that is our load script done.

Unpacking The Application Back Up Code

There is quite a lot going on in that last step of the code. Hopefully you can work out what is going on with most, if not all, of it. Much of it is to give us some extra fields to do some analysis of the applications on our server (more on that later), but for now just to focus on the creation of the App Backup Script field.

'copy $(vSrcFolder)' & GUID & ' "$(vDestFolder)' &
Stream & ' -- ' & Application & '.qvf"' as [App Backup Script]

Here we are starting with the copy statement we wish to use (you can replace this with XCopy or add parameters if you wish), adding in the two variables we set up right at the top of the script and pilling out the GUID which is the source filename for our script. There is then the building of the output filename and the placing this in a fieldname.

The construction of the output filename and path you can also alter if you wish, I have gone for the stream name (which is set to None for items that are not published) a double hyphen (as apps and streams may have hyphens in the name already) the application name and finally the QVF extension.

This then builds code that we can use in a Command Prompt on our Qlik Sense server to move the apps to their backup location.

Creating and Using The Back Up Script

Now our script is complete, click Load Data to bring all the data in from the REST connection. Once this has finished loading click Analyse Sheet and change to Edit mode. Grab a Table object from the Charts and drop it on the sheet. Click to Add Dimension and add in the App Backup Script dimension.

Come out of Edit mode and you can then right click to export the table to Excel. Open this spreadsheet and copy and paste the code into a new Notepad (or other text editor) window and then save the script as a .bat file. This file can then be copied onto the server and executed to perform the backup.

There is a slight risk of one of the files being corrupted if it is copied whilst a write is in progress, so it is best practice to stop all Sense services before taking the backup. In practice, I have not ever hit a snag with this though and you could simply check the date stamp on the files and see if there were any that were updated at around the time you took your backup and perhaps copy them again.

So there you have a relatively quick and painless way of getting QVFs of all of your Qlik Sense Client Managed Server applications.

I hope you find the technique useful. Please let us know how you get on in the comments below.

Further Analysis of Your Application Data

As well as the the information required to filter and export the backup script there were various other data points brought in about the Applications on the Sense Server which you may want to analyse. Finding which applications have not been touched in over a year, for example.

Fortunately we have your back with this, with our Instant Sense Application. This is an application template which ships with a number of pre-written load scripts (including the one above) and metadata files so that you can re-purpose the application to use your own data sources. Out of the box you can view a number of KPIs over these application data:

It also allows you to break the various measures down over different dimensions and plot those different measures against each other, by a selected dimension:

Finally there is the user defined table, where the user can select a number of dimensions and measures and view these in a table which can then be exported as required:

Everything in the application is driven from a metadata spreadsheet, this means that you can configure the app so it can be used over whatever data source you like:

All in all there are twelve sheets of user configurable visualisations, just waiting for you to bring your own data. The best thing is that it is free to download and use.

Find out more about the application, and download it, from here:

https://www.quickintelligence.co.uk/instant-qlik-sense-app/

More technical details, including how the application is built can be found in this Instant Sense Application blog post.

If you have any questions or comments about anything above, please do drop us a comment below.

If you would like to share a load script and metadata spreadsheet for the Instant Sense Application then we would love to hear from you.

The post Back Up Qlik Sense Enterprise Client Managed Applications appeared first on .

So, Why Use IFTTT With Qlik Sense?

In my previous blog post I showed how to create an app to monitor Qlik analyzer capacity licence usage in Qlik Cloud. My favourite response to the last post was from Tom Wittig, who said he had five clients needing the app.

That got me thinking, what if you have many sites to keep an eye on in Qlik SaaS? You would want to take the information from each site, place it into a common repository and then build an app over that. But how?

There are plenty of methods for storing files out, but this would require a shared DropBox or Google Drive, between clients, or many clients’ accounts linked to by the partner. Neither of these approaches feels like a good idea. Pushing just the data that is required via a one way service that can then store the data would be ideal.

Using a combination of the REST connector, Webhooks, If This Then That and Google Sheets it is simple to achieve this.

I did a previous post on using Webhooks to start a Sonos playlist when a new customer is obtained, but this is a more practical application of the same technique.

Signing Up For An IFTTT Account

If This Then That allows you to connect all of the various services you have and get them talking to each other. It’s well worth taking a look through the various example applets that they have available. If you have an account already you will know this. If not, it is simple and quick to create one and get started.

Head along to https://ifttt.com/, here you can sign up with an email address or an Apple, Google or Facebook account. As we’re going to be using Google Sheets I’m going to use a Google account, you can take your pick though.

Creating the IFTTT Applet

Once you have signed up or signed in you will get to the main IFTTT page. You may want to take the time to take a look around, or you can just crack on and click the Create button.

As the name suggests, basic apps all consist of a trigger (If This) and an action (Then That), click the Add button to set up the If

First you need to chose the service that you will use for the trigger. Search for and select Webhooks

Many of the services will have lots of different options here, but Webhooks just have one simple action – receive a web request. Click this.

Pretty much every service will require a connection to that service, this is true of Webhooks, so click the Connect button to do this. Most services will then require credentials (as you will see when we connect to Google Sheets in a moment), but Webhooks will just create the connection for you.

You will then be prompted to create a trigger, which you need to name. This name will be used when you call the Webhook, and it will be available when we create the event caused by the Webhook. Make sure you take a note of this name.

When the name has been provided click the Create Trigger button.

With the trigger created you can now click the Add button for the action.

Now you need to search for and select Google Sheets.

From here select Add Row To Spreadsheet. Again you will need to connect to the service, do this by clicking the Connect button. A popup will appear where you can sign into your Google account and authenticate IFTTT to work with it.

There are lots of other endpoints that you can use to write to, such as Slack, Twitter and Gmail. Choose and configure the one that best suits you.

For a Google Sheet you need to provide the name of the spreadsheet (clear the default one first) enter the values to enter into each cell of the spreadsheet (defaults here are fine) and specify which folder in your Google Drive you would like the spreadsheet to reside. Click Create Action when done.

That has now created the applet and you can click Continue.

Using The New Applet

After clicking continue you will be shown the default name for the applet. This will be a description of what it does. You can swap this for a more useful name if you like. Click the Finish button when you are happy with the name.

The applet is now present and working, but the issue is that you won’t know how to call it. Click on the Webhooks icon above the to the configuration screen.

From that screen click the Documentation button at the top right of the screen. This will take you to a screen where you can test your Webhook.

Replace the text {event} with the event name that you used when setting up the trigger. Enter some values into the three value text boxes and click Test It.

You will need to copy the Trigger URL to a safe place to be able to use it later, this is the one that looks like this:

https://maker.ifttt.com/trigger/capacity_usage/with/key/ABCDE1234567890

Once the test has been fired you should be able to go and find the spreadsheet, but I have noticed it can take a few minutes to appear.

Plumbing The Webhook Into The App

To use this Webhook with the Capacity Licence checking app I made available last week you will need to download it from here. To plumb it into another app you will need to put the values you want to send into variables in your load script.

To call the URL you will need a REST connection set up, as described in the previous blog post. This can point to any URL, as we are going to change it using WITH CONNECTION, I typically use https://jsonplaceholder.typicode.com/posts for this purpose. Give the connection the name GenericGET.

You then just need to add the following code after the point where the variables are set:

// Send details to a Webhook
LIB CONNECT TO 'GenericGET';
SET errormode = 0;
SendResults:
SQL SELECT
"col_1"
FROM CSV (header off, delimiter ",", quote """") "CSV_source"
WITH CONNECTION (
URL "https://maker.ifttt.com/trigger/capacity_usage/with/key/ABCDE1234567890?value1=$(vUsed)&value2=$(vRunRate)&value3=$(vAvailable)");
SET errormode = 1;

Obviously the key part of the URL will need to change to be your own personal Webhook key that you copied above. You will also need to ensure that the trigger name matches the one you gave when creating the trigger. The URL copied in the test step above didn’t have any parameters, as these were passed in the body of the GET request. You will need to manually type the three parameters to the URL and inject variable values created earlier in the script.

Once the load script has run you can check the result from the call of the Webhook by checking the col_1 field in Analyze mode in Sense.

You should now have a new row in your spreadsheet with the values you specified in:

As you can see you can insert a row which has headers and new rows will simply be appended to the end. If you have multiple Webhooks appending to the same spreadsheet you just need to ensure that you have a value in a column which shows which numbers have come from where.

Webhooks are a great way to ping small amounts of data between different services, and it is something which is going to be more closely integrated into Sense in future with blendr.io. You can start using them straight away though, from your load scripts.

Please let me know how you get on with them in the comments below.

The post Use IFTTT and Webhooks to Send Data from Qlik Sense appeared first on .

What is Capacity Licencing?

Purchased in packs of 1,000 minutes per calendar month, a capacity licence is a subscription which can be consumed by any user for access to any app in six minute time slices. This flexibility makes the capacity licence an excellent choice for getting large numbers of infrequent users using Qlik apps.

Once you have these licences though, it is crucial that you monitor things to ensure that you are not burning through your capacity too quickly. Just looking at how many minutes have been consumed against the total is not enough, you need to be aware of how far through the month you are. Looking at current usage against a run-rate of available minutes allows this. If usage is exceeding the minutes available then someone needs to be notified, so licences can be reallocated or an additional capacity licence pack be added.

This post describes how to set up this monitoring and alerting on Qlik Cloud (SaaS). A similar approach can be taken for client managed also.

Connecting To The Qlik API

Information about your Qlik Cloud tenant is exposed through a number of API Endpoints. In order to read from these endpoints API Keys need to be enabled under Management Console Settings, a nominated user must have Developer rights in the Users section of the Management Console, and that user needs to generate an API key in the API Keys section of their Profile Settings. This key is then used in the load script of the app.

This is quite a convoluted process, but it is well described in the documentation for the App Analyzer for Qlik SaaS. The analyzer app itself is one you may want to considering in your SaaS tenant.

Once you have an API key, paste it into a safe place as you will need it later.

To implement a similar solution on Qlik Sense Enterprise for Windows (Client Managed) you can use the provided REST connection, monitor_apps_REST_license_overview.

Rather than setting up a connection to each API that is needed in your SaaS environment, it makes more sense to create just one and then modify it using WITH CONNECTION. To enable this, set up a new REST connection, pointing to a placeholder (e.g. https://jsonplaceholder.typicode.com/posts) with the type set to GET and everything else as defaults. Give this connection the name GenericGET.

Sending Alerts From The Qlik Load Script

It is possible to use Qlik Alerting in Qlik Cloud, but for this purpose I would suggest simply sending directly from the load script. The SMTP connector in Qlik Cloud allows for this.

Create a new SMTP connection, populating it with the details of the mail server you wish to send via. You may want to set up an email account, such as a free GMail account, specifically for this purpose. Give this connection the default name of SMTP.

This will allow you to send custom emails from your Qlik load script. In order to do the same in Qlik Sense Enterprise for Windows see this previous blog post on Sending Data from the Qlik Load Script.

Building The Capacity Usage App

As with any Qlik app, most of the magic happens in the load script. Here we talk through the code you need to build an alerts app.

First of all some environmental variables need to be set, specifying where your tenant can be found and what email settings you wish to use:

// Name or alias of your Qlik Sense Cloud instance
let vInstance = 'instance.region.qlikcloud.com';

// Email configuration fields
let vTo = 'recipient@domain.com';
let vFrom = 'sender@domain.com';
let vFromName = 'Sense Capacity Alert';

// The API key to use - generated under your Profile settings
let vAPIKey = 'ABC123456789';

Note that you will need to paste the API key that you obtained above into this code.

Now some options to configure. Firstly at what percentage of the time-of-month available minutes do you wish to alert, 1 is anything over 100% of usage – you can increase or decrease this as you wish. You can also specify how frequently you want to be reminded you are over threshold (if usage keeps rising) or ask to be reminded every time the app reloads:

// Decimal at which run-rate percent to send alert - 1 = 100%
let vAlertAt = 1;

// Days in which to repeat send of alert - can be a decimal fraction of a day
let vRepeatIn = 0.25;

// Binary flag indicating whether you wish to ignore increase and repeat thresholds
let vSendAll = 0;

Note that the repeat value of 0.25 is equivalent to every six hours. You can adjust as you see fit.

Now we need to connect to the REST connection. This will check your placeholder is present, but do nothing else at this point:

// Connect to the REST connector
LIB CONNECT TO 'GenericGET';

Note that if you have created your REST connection in a managed space, you will want to prefix the name with the space name, e.g. ‘MySpace:GenericGET’;.

Most of the heavy lifting of working out where we are in the month is done in this next statement. It connects to the API, gives the credentials defined above, calculates the run rate and formats the output:

// Retrieve data and calculate run-rates and percentages
CapacityUsed:
LOAD
*,
num([Used Minutes] / [Available Run Rate], '#,##0.0%') as [Percent Used Run Rate]
;
LOAD
name as [Licence Type],
usageClass as [Usage Type],
units as [Total Minutes],
unitsUsed as [Used Minutes],
num(unitsUsed / units, '#,##0.0%') as [Percent Used],
// Work out a run rate by working out how far through the month we are in days,
// divided by the total number of days in the month
num(units *
((now() - MonthStart(today())) /
Day(MonthEnd(today()))), '#,##0.0') as [Available Run Rate],
overage
WHERE name = 'analyzer_time'
;
SQL SELECT
(SELECT
"name",
"usageClass",
"units",
"unitsUsed",
"overage"
FROM "allotments" FK "__FK_allotments")
FROM JSON (wrap on) "root" PK "__KEY_root"
WITH CONNECTION (
URL "https://$(vInstance)/api/v1/licenses/overview",
HTTPHEADER "Authorization" "Bearer $(vAPIKey)"
)
;

In order to be able to check values and send them in the load script we need to peek them into variables:

// Retrieve values from the data set
let vRunRate = num(peek('Percent Used Run Rate', -1, 'CapacityUsed'), '#,##0.0%');
let vTotal = num(peek('Total Minutes', -1, 'CapacityUsed'), '#,##0');
let vUsed = num(peek('Used Minutes', -1, 'CapacityUsed'), '#,##0');
let vPercent = num(peek('Percent Used', -1, 'CapacityUsed'), '#,##0.0%');
let vAvailable = num(peek('Available Run Rate', -1, 'CapacityUsed'), '#,##0.0');

Then to give information back to the user running the script we can use these variables in a set of trace statements.

// Notify of current status to load stream
TRACE ;
TRACE $(vUsed) Minutes of $(vTotal) consumed.;
TRACE $(vPercent) of Total and $(vRunRate) of Run Rate minutes.;

Capacity subscriptions reset at the end of each calendar month. We are only interested in being notified if the previous percentage usage has been exceeded, but we need to reset that if we are in a new month. That is done here:

// Reset last run rate if we are in a new month
if Month(alt(vLastRun, 0)) < > month(today()) then
let vLastRunRate = 0;
end if

We then need to decide whether the current run-rate exceeds the value at which we want to alert, and whether other criteria such as used percentage increased and interval time elapsed. This is all in one big if statement:

// Check if we have an alert situation and whether repeat and increase criteria met
if vRunRate > vAlertAt and (vSendAll < > 0
or (vRunRate > alt(vLastRunRate, 0) and now() > alt(vLastAlert + vRepeatIn, 0))) then

Now we have to set a couple of variables, to record what level we are alerting at, and when we are doing it. We are also writing to the load script progress window:

 TRACE Sending alert to $(vTo);

// Set variables to log last time an alert sent
let vLastRunRate = vRunRate;
let vLastAlert = now();

Sending emails from the Qlik SaaS load script requires a connection to the SMTP connection to be made:

// Connect to SMTP connection
LIB CONNECT TO 'SMTP';

As with the REST connection, the connection name should be prefixed with the space name and a colon if the connection is in a managed space.

Then the sending of an email is simply a LOAD call, with the recipient, sender, subject and body of the email all included in the SELECT statement:

 // Send email and retrieve result of send
Email:
LOAD
status as [Email Status],
result as [Email Result]
;
SELECT
status,
result
FROM SendEmail
WITH PROPERTIES (
to='$(vTo)',
subject='Capacity Usage Exceeding Expected For Time Of Month ($(vRunRate))',
message='

The capacity licence usage on $(vInstance) is currently $(vUsed) minutes out of $(vTotal).
This is $(vPercent) of the total capacity, and $(vRunRate) of the minutes pro-rata for the time of month.',
html='true',
fromName='$(vFromName)',
fromEmail='$(vFrom)',
cc='',
bcc=''
);

Note that you can place HTML into the body of the message here (due to the html=’true’ flag). The code above will be missing the line break and bold tags, you can simply add them back in as required.

The response from the email server is kept in the Email table. We are not looking into that here, but you can check the values in these fields in the Analyze part of your app.

We then need to end if the IF statement we started above, and clear down the variables we have used and no longer have a need of:

end if

// Clean up after ourselves
let vInstance = ;
let vTo = ;
let vFrom = ;
let vFromName = ;
let vAPIKey = ;
let vAlertAt = ;
let vRepeatIn = ;
let vSendAll = ;

So, all being well that should be that. Your tenant’s API will be polled, capacity usage will be loaded (note other licences are in that endpoint also, just excluded by a WHERE), run rates will be calculated and emails will be sent.

Seeing the data is in a Sense app though, it would be a shame not to have a chart or two…

Building a Front End for the Application

This application will only have one row of data in it and a handful of fields that can be used for measures. Master measures can be made for these, as follows:

Total Capacity: Sum([Total Minutes])
Used Minutes: Sum([Used Minutes])
Capacity Run Rate: Sum([Available Run Rate])
Percent of Total Used: Sum([Used Minutes]) / Sum([Total Minutes])
Percent of Run Rate Used: Sum([Used Minutes]) / Sum([Available Run Rate])

You might then chose to place these onto a sheet a bit like this:

If after all of this you would rather just download an app that already has all of the script and front end in it, then you are in luck. I have uploaded the completed app to Qlik Community and you can find it here:
https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Monitor-Capacity-Licences-on-Qlik-Cloud-and/ta-p/1805633

I would be very interested to hear how you get on with this application. Please feel free to contact me via the comments if you wish to get in touch.

Expanding The Solution

If you want to take this solution further, and write the capacity usage as at each reload to an online source, you may want to read my next blog post also:

Use IFTTT and Webhooks to Send Data from Qlik Sense

The post Monitor and Notify Sense Capacity Licence Usage in Qlik Cloud appeared first on .