Why Call A Webhook?

In my previous blog post (Start a Qlik Sense Cloud Reload from a Webhook) I described how to create a webhook using Qlik Automations which can start a reload of any app in your Qlik Cloud tenant. One of the places you might want to trigger this from is from a Qlik load script. This way you can call a reload of one or more Sense apps at the end of another script, perhaps dependant on data or status.

Triggering a Qlik Automation is one example of why you might want to fire a webhook, but in this post I want to share another example – raising an alert when things are broken.

How Do You Notify When Your Notifications Are Broken?

Something we always set up when building customer environments is sufficient notifications to make us aware when anything is wrong. On Qlik Sense Client Managed this is essential, as there is very little out of the box, and on Cloud it needs to be set up space by space, so some things can be missed. Sometimes we are working on a secure site where SMTP relaying from within the site is not permitted. Other times it can be that communication between the Qlik server and the SMTP server is down due to network issues, so the route to let us know there is an issue is broken along with the rest of the network. In these cases we need a way to get a message out, when nothing else is working.

Something that is likely to work in Sense, even when all else is failing, is calling a URL to extract data. If that URL is a webhook then that call can then trigger an action. What we have is a an applet set up in IFFTT (If This Then That) which sends an email via GMail containing values from the parameters of the webhook called. I won’t go into setting up that applet here, as it will be slightly different if you want to use a different mail client or call a different service, but I have given details on this in a previous blog post on Inserting Qlik Capacity Usage Into a Spreadsheet.

Once you have a webhook set up that can send the notification you are ready to build an app that triggers that webhook when there is a problem.

Raise a Notification When Files Go Stale

The most basic measure of everything being up and working is that files that are used by your Qlik environment are all up to date. If the applications you wish to monitor do not currently create any files then it is a simple job to add a STORE statement to write a small text file that can be checked for. The application we shall build here then has an inline table (which will work even if all else is broken) with a list of file locations and how old those files can go before we want to be told about them.

Before building the load script for this app you will need to set up a REST connector that we can pass parameters in to. To do this build a new REST connector, set it to a placeholder URL (I tend to use https://jsonplaceholder.typicode.com/posts), set the type to GET, ensure that Allow WITH CONNECTION is ticked and give it a name of Generic GET.

Once that connection is in place you can build the load script.

First you will need to set up some variables for use later in the script. Obviously the webhook parameter will need to be changed to match the webhook that you want to call.

let vEnv = 'My Qlik Sense Tenant';
let vLib = 'DataSpace:';
let vGET = '$(vLib)Generic GET';
let vQVDs = 'lib://$(vLib)DataFiles/';
let vTimestamps = '$(vQVDs)Timestamp/';
let vWebhook = 'https://maker.ifttt.com/trigger/error/with/key/your-webhook-key-here';

let vLastLoad = now();

Then you need to build the table of files that you want to be checking. These could be QVD files or small files that are written as part of a load script to indicate that the load script has completed. The table requires the full path to each file to be checked, how many hours old the file can get before an alert is raised and what label is to be used in the alert when that file becomes old.

tmp:
LOAD
    File,Threshold,Name
INLINE [
File,Threshold,Name
$(vQVDs)Sales.qvd,1,Sales Application
$(vTimestamps)QVDGenerateCompleted.csv,3,QVD Generation Application
];

That inline table can then be transformed to convert the hour given in the load script into a time delta to come off the current time, which can then be taken off the current time to work out the time that the file should be newer than. The actual date of the file also needs to be looked up. These two dates can then be compared to find the delta and to flag whether the file is old.

FileAges:
LOAD
    *,
    if(FileDate < Target, 'Yes', 'No') as Old,
    num((24)*(FileDate - Target), '0.00') as Difference
    ;
LOAD
    File,
    Threshold,
    Name,
    Date(alt(ConvertToLocalTime(FileTime(File), 'London'),0), 'DD MMM YYYY hh:mm') as FileDate,
    Date((LocalTime('London') - ((1/24)*Threshold)), 'DD MMM YYYY hh:mm') as Target
RESIDENT tmp;

DROP TABLE tmp;

This table then needs to be aggregated to find out how many of the files checked are older than they should be. That value needs to be Peeked out into a variable.

Old:
LOAD sum(1) as Old RESIDENT FileAges WHERE Old = 'Yes';
let vOld = alt(Peek('Old', -1, 'Old'), 0);
DROP TABLE Old;

Now we get to the actual point of the app and the demonstration. If there are any old files, we need to do the call to the webhook endpoint along with a payload that tells us which environment is affected and how many of the files are stale.

if alt(vOld, 0) > 0 then
    SET errormode = 0;
    LIB CONNECT TO [$(vGET)];
    SQL SELECT "col_1" FROM CSV (header off, delimiter ",", quote """") "CSV_source"
    WITH CONNECTION (URL "$(vWebhook)?value1=$(vEnv)&value2=$(vOld)%20Files%20Are%20Older%20Than%20They%20Should%20Be");
    SET errormode = 1;
end if

TRACE There are $(vOld) files older than expected;

To wrap up, because we are tidy Qlik developers, we clear down the variables that we have used.

let vEnv =;
let vWebhook =;
let vLib =;
let vGET =;
let vQVDs =;
let vTimestamps =;

Exit Script;

That is it for the load script, and that is all you need to make this work. What you might want to do though, as the list of files checked is in a table in the data model along with their status, is to add a table to a sheet in the app listing the six columns in the FileAges table, with some conditional formatting to show when files are old. I would also add a button, which can then refresh the app without going into the load script.

But What If Even This App Cannot Run?

If you are running Sense on a Client Managed server it is possible (although very unlikely) that even this application will not be able to run and call the webhook, perhaps if someone has unplugged the server in order to recharge their mobile phone. In this case we need an external service to monitor what is going on. If the server is (as is most likely) behind a firewall that external service will not be able to reach in and check the status. In this case we want a service outside of the network which is expecting to hear from the server at a regular beat which can then raise an alert if it doesn’t hear anything.

This is exactly what I am going to cover in my next blog post, pulling together elements of this post and the previous post. This solution will use a webhook on a Sense Cloud tenant which calls a reload of a script that writes a to a QVD. There is also a separate application that checks the date of this QVD and notifies if that has not been updated in a specified window. The history of the webhook being called is persisted, so a history of when the calling server was up and able to reach the outside world is recorded, giving an accurate account of uptime on the server.

If you have any comments on anything you would like to see covered in that blog post, or anything related to this blog post, please use the comments box below to get in touch.

Other Uses For Calling A Webhook From The Qlik Load Script

There is literally no end to what you can achieve by calling webhooks, particularly if you use a tool such as IFTTT.

Another use you might want to consider, if you are using Qlik Sense Cloud, is kicking off a reload using a webhook. In my previous blog post I described how a Qlik Automation can be set up to be triggered from a webhook to do a reload. If at the end of the load script for one app you call the webhook with the correct parameter to kick off the next reload in a chain this reload chain will work however the reload is started, interactively or from a schedule. This gets around the limitation that Qlik Sense Cloud reload chaining from the hub presently has, where the next reload in the chain only takes place if the previous reload was started from a particular schedule. This reload chaining approach also allows for logic to be applied around whether the next reload should take place or not, perhaps the front end should only reload if there has been a change to the backend data?

I hope this post has given you plenty of inspiration of how you can link different services together within, and beyond, your Sense environment.

The post Starting External Services From Qlik Sense Load Scripts appeared first on .

The Challenge

The question was posed on Qlik Community how to overlay two 12 month measures in one chart. Using set analysis it is straight forward to have two measures which cover the correct date ranges, but when the Month dimension is used it will sort by calendar month, rather in the order required for rolling periods. You can see the question and my response in this post.

The response uses a technique I have used a number of times before but it occurred to me that I have not documented it on the blog, so here it goes.

Creating Rolling Month and Year Dimensions

This code can either sit in your master calendar code, should you use one, or you can place it directly in the load statement from source data. Typically I will go for the latter approach.

The first thing that we need to do in the load script is to create a sequential integer for each month. This then allows us to work out the number of calendar months between any two dates. This technique has many more uses than just rolling twelve month analysis and is an incredibly useful dimension to have in your data model.

If we do (Year(Date)*100)+Month(Date) we will get a sequential integer value within a year (e.g. as at now it will give 202305), but the deltas break when you go across a year (e.g. comparing 202212 to 202301). In order to fix this we can just multiply the year by 12 and add the month. So, again taking May 23 as an example you will get 24,281. This is obviously not at all useful as a human readable number (just as telling someone that the date today is 45051 is only useful if they are a machine), but it means that the numbers for Dec 22 (24,276) and Jan 23 (24,277) are now just one integer apart.

In the load script we can put this code in to create a new field called MonthNo like this:

(Year(Date) * 12) + Month(Date) as MonthNo,

Now we have that value we need to work out how many months back any given month is from today. The code that we used on the Date field can also be done on the today() function, giving a comparable figure for the current date. Using a preceding load, we can compare that to the MonthNo field to get a value for Months Back, like this:

(Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]

This new field in itself may be useful in your charts and filters within Sense, for instance it could make your set analysis for a chart showing the last twelve months much easier as you do not need to deal with date formats in the set code.

Jumping up another level of preceding load we can now use the Months Back value to create the Rolling Month and Rolling Year values.

The Month function gives us a dual data-typed value with the three letter month name and the integer order for the month from 1 to 12. The text part of that dual datatype is great for us but the number does not give us what we want.

We can use the Dual function though to build our own dual data-typed value with the correct sequential number. Using the output of the Month function as the first parameter of the Dual statement will only take the text part of that value. The correct sequential number can be calculated by working out how many months back the start of the year was and then taking away the months back for the date in question.

The whole of that code looks like this:

Dual(Month(Month), (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],

We can get the number of rolling years back from our Months Back value by taking the integer part of the number of months back divided by 12. In order to turn that into text to show as a dimension we want to suffix it with the text “Years Back”. As having “0 Years Back” looks a bit naff, we need to have anything in the past 12 months showing as “Current Year”. Having that text means that sorting on the field breaks (as it would if you have more than a decade of data), so we need to use the dual function again.

Putting all of that together gives us this statement:

Dual(if([Months Back] < 12, 'Current Year',
    Floor([Months Back]/12) & ' Years Back'),
    Floor([Months Back]/12)) as [Rolling Year]

We then have the two dimensions that we can use in a line chart:

Putting It All Together

This code generates a whole bunch of dates and then applies the date code described above in a resident load. You can copy and paste this to a new app and see the code working in isolation, or your can take parts of it to apply to your own data.

tmpDate:
LOAD
    Date(today()-RowNo(), 'DD MMM YYYY') as Date
AUTOGENERATE(2000);

Dates:
LOAD
    *,
    Dual(Month(Month),
        (Floor([Months Back]/12)*12)-[Months Back]) as [Rolling Month],
    Dual(if([Months Back] < 12, 'Current Year',
        Floor([Months Back]/12) & ' Years Back'),
        Floor([Months Back]/12)) as [Rolling Year]
    ;
LOAD
    *,
    (Year(today()) * 12) + Month(Today()) - MonthNo as [Months Back]
    ;
LOAD
    Rand() * 1000 as Value,
    Date,
    Date(MonthStart(Date), 'MMM YYYY') as Month,
    (Year(Date) * 12) + Month(Date) as MonthNo,
    Month(Date) as [Month Name]
RESIDENT tmpDate
;

DROP TABLE tmpDate;

Something else you might want to deal with when looking at rolling months is a run rate value for the current month, which takes into account how far through the month we are and what the value will be if extrapolated to the number of days in the month. I have covered this in a previous post.

Not only does this code give you dimensions that you can use in your charts, the same dimensions can be used in set analysis also. To create the variance to prior rolling twelve months gauge at the top of the chart you can now just use this code:

(Sum({<[Rolling Year]*={'1 Years Back'}>}Value) /
Sum({<[Rolling Year]*={'Current Year'}>}Value)) -1

An example QVF file including this code and some example charts can be downloaded from the Qlik Community post.

I would be very interested to hear from you on this technique and whether you have used similar or can suggest other ways it could be used. Please use the comments below if you would like to add your thoughts.

The post Compare Rolling Twelve Month Period in Qlik appeared first on .

Introducing The CSV Data

Recently I was asked to be involved with the Data On The Earth project. This is an initiative to get data that can help everyone, from policy makers to concerned individuals, make more informed decisions about environmental issues. Naturally, I was very happy to help.

The first thing I wanted to do was to present some ideas over some data that had relevance to the job in hand. The site Our World In Data seemed like a good place to start.

There I found data which related to Co2 emissions, hosted on GitHub, you can find more about these data here. The data are by Country and Year, so a bit of extra metadata on the countries would allow some demographic filtering. I found this in two further CSV files, also conveniently hosted on GitHub.

The data are not perfect, as some of the figures are not reported by all countries in every year and countries change over time. There are things that could be done to fix this up a bit (rolling forward last known figures, for instance), but I have not done that here. If you are planning to lobby a government over their emissions (or something) please check your facts and don’t rely on what is presented in these data alone!

Creating a Connection

The code in this article is for the load script of Qlik Sense. You can do similar with the REST connector in QlikView, but the syntax is a bit different, this post should give you pointers should you wish to build your own load script.

As with any load script in Sense you first need to have your connection created. Rather than create a separate connection for each data source I load I tend to create a generic connection of each type required and modify it using WITH CONNECTION. You may already have the required connection set up if you have previously followed my tutorials on Loading Sense Session Logs or Loading RSS Feeds, if not go to the Data Load Editor and click Create New Connection.

Give your connection the name GenericGET and point it to a placeholder, such as https://jsonplaceholder.typicode.com/posts, ensure the Method is set to GET and leave everything else as defaults.

Click on Test Connection, then Close and finally (if all is good) Create.

Building the Load Script

Now, onto building the load script. First we need to do some set up, to connect to our connection and set up some variables:

LIB CONNECT TO 'GenericGET';

let vCompleteYears = 'Y'; // Y/N for all years or only 1990 - 2016 where data is quite complete
let vExcludeYears = if(upper(left(vCompleteYears, 1)) = 'Y', 'AND (year >= 1990 AND year <= 2016)', '');

set vFix = if([$1] = '', '$3', [$1]) as [$2];

let v1B = 1000000000;
let v1M = 1000000;

The first variable allows us to turn on or off data for years which are not that complete, the second puts the code to implement that into a variable if switched on. Next we have a variable which includes parameters that we will use later on. If you have not used them before you should see this post on Using Parameters with Qlik Variables. Finally a couple of variables to remove the risk of typing the wrong number of zeros later on.

Now we are set up, we can do the main part of the load.

Emissions:
LOAD
iso_code & year as CountryYear,
iso_code as [Country ISO],
country as Country,
year as Year,
co2 as CO2,
consumption_co2 as [CO2 Consumption],
trade_co2 as [CO2 Trade],
co2_per_unit_energy as [CO2 Per Unit Energy],
cement_co2 as [CO2 Cement],
coal_co2 as [CO2 Coal],
flaring_co2 as [CO2 Flaring],
gas_co2 as [CO2 Gas],
oil_co2 as [CO2 Oil],
total_ghg as [Total GHG],
methane as Methane,
nitrous_oxide as [Nitrous Oxide],
population as Population,
gdp as GDP
WHERE iso_code <> '' and index(iso_code, '_') = 0 // Remove rolled up values
$(vExcludeYears)
;
SQL SELECT
"iso_code",
"country",
"year",
"co2",
"consumption_co2",
"trade_co2",
"co2_per_unit_energy",
"cement_co2",
"coal_co2",
"flaring_co2",
"gas_co2",
"oil_co2",
"total_ghg",
"methane",
"nitrous_oxide",
"population",
"gdp"
FROM CSV
(header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/owid/co2-data/master/owid-co2-data.csv")
;

As with a database load this load will use the last opened connection. The CSV on the FROM line is specifying the basic format that we are loading from this connection, the exact format of the CSV is specified in the brackets – just as it would be if you were loading the CSV from a Folder Connection. The CSV_Source relates to which data source at the connection you wish to load – with a CSV this will only ever be CSV_source, but with JSON data there may be many named sets of data for you to choose from. The WITH CONNECTION then defines the URL we are loading from.

To explore these options further create another connection, pointing to that URL and look at the Select Data dialog for that connection.

Atop of the REST load we have a Preceding Load, to tidy the fieldnames and remove data we don’t want. There are rows with data rolled up to regional levels, but we are going to do that ourselves in Qlik so need to remove the roll ups to avoid double counting. Note that when doing a WHERE in a preceding load all of the data still has to come down from the REST source first, so you may want to look if you can pass parameters to the URL you are loading from to drop some data, rather than using a WHERE statement.

So, that is the data we require from our main data source. We are going to continue though, and augment and categorise these data some more.

Bringing in more CSV Data

The next table of data is a simple table of country demographics, provided by GitHub, keyed on the ISO code for the country.

Region:
LOAD
[ISO3166-1-Alpha-3] as [Country ISO],
$(vFix(Developed / Developing Countries, Developed, Not Known)),
$(vFix(Region Name, Region, Other)),
$(vFix(Sub-region Name, Sub Region, Other))
WHERE EXISTS ([Country ISO], [ISO3166-1-Alpha-3])
;
SQL SELECT
"ISO3166-1-Alpha-3",
"Developed / Developing Countries",
"Region Name",
"Sub-region Name"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv");

This file has many more columns than we need, but we are cherry picking the few that we want. As you can see, the code for connecting to a CSV source is almost identical to above – just the URL is changing. In the Preceding load the vFix variable we created at the start is being used to replace any missing values with Not Known or Other. A WHERE EXISTS ensures we only get rows which relate to countries we have emissions data for.

Using a lookup table like this from a repository like GitHub in your Qlik apps means that your application will benefit from improvements to that dataset over time, rather than having a CSV or Excel spreadsheet languishing on your server which will never see an update.

Where In The World Is That?

As we are looking at country data it would be great to plot that on a map, and another GitHub source has just that information available for us. These data are provided and maintained by Tadas Tamošauskas in a personal repo.

Here we are going to LEFT JOIN the data onto the existing region table, bringing the latitude/longitude data into the same table as the regions. You will see why in a bit.

LEFT JOIN (Region)
LOAD
*,
GeoMakePoint(lat,long) as [Country Point]
;
LOAD
purgechar([Alpha-3 code], ' "') as [Country ISO],
purgechar([Latitude (average)], ' "') as lat,
purgechar([Longitude (average)], ' "') as long
WHERE EXISTS ([Country ISO], purgechar([Alpha-3 code], ' "'))
;
SQL SELECT
"Alpha-3 code",
"Latitude (average)",
"Longitude (average)"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://gist.githubusercontent.com/tadast/8827699/raw/f5cac3d42d16b78348610fc4ec301e9234f82821/countries_codes_and_coordinates.csv");

This now lets us plot emissions data on a map by country. Our second data source though gave us Region and Sub Region names. It would be good if we could also plot by those regions. There is probably a data source out there which has SVG data for each region, but that could take a while to find and will not necessarily map to our existing region names.

We can apply a simple hack to generating points at each of the regions though.

RegionPoint:
LOAD
Region,
GeoMakePoint(avg(lat),avg(long)) as [Region Point]
RESIDENT Region
GROUP BY Region;

SubRegionPoint:
LOAD
[Sub Region],
GeoMakePoint(avg(lat),avg(long)) as [Sub Region Point]
RESIDENT Region
GROUP BY [Sub Region];

Here, a couple of RESIDENT loads from the existing Region table can be grouped to give one row per Region and Sub Region. For each of these rows the average latitude and average longitude can be calculated. These two values can then be changed to a point that can be easily used on a map. Note that some points may appear in the ocean, but that is the joy of using averages.

Finding Latest Data and Grouping Figures

The figures from our initial dataset are great for trending over time as they are provided on an annual basis. They are not so great for categorical data and providing filters though, as a country could move from one bucket to another year to year. For these filters we only want to use the latest figures. Here we hit another minor snag though, in that the last year in which data is available differs by country.

We can deal with this by finding the latest year for each country, using a RESIDENT, GROUP BY and MAX to create a composite key.

LatestPop:
LOAD
[Country ISO] & Year as LatestPop
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE Population <> ''
GROUP BY [Country ISO];

That composite key can then be used in the WHERE EXISTS of a subsequent RESIDENT load to only bring the latest data.

Population:
LOAD
Country,
Population as [Population Latest],
dual(num(Round(Population, 100000) / $(v1M), '#,##0.0') & 'M', Round(Population, 100000)) as [Population 100K],
dual(num(Round(Population, $(v1M)) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M))) as [Population 1M],
dual(num(Round(Population, $(v1M)0) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M)0)) as [Population 10M]
RESIDENT Emissions
WHERE EXISTS (LatestPop, CountryYear);

This is a useful technique, and one I use frequently in live projects. As well as getting the exact number for the latest year for each country we can also put things into buckets. This can be done with a massive IF statement or for equal buckets an aggregation. Here I have used a Round statement, but you could use a Floor or Ceil if you prefer. The Round function in Qlik is more flexible than in most languages as, rather than just rounding to a number of decimal places, you can specify what grouping you wish to go to, whether that is more or less than one.

The same technique can then be used for GDP, then there is just a little bit of housekeeping to do at the end of our script.

LatestGDP:
LOAD
[Country ISO] & Year as LatestGDP
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE GDP <> ''
GROUP BY [Country ISO];

GDP:
LOAD
Country,
GDP as [GDP Latest],
dual(num(Round(GDP, $(v1B)), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B))) as [GDP 1B],
dual(num(Round(GDP, $(v1B)0), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)0)) as [GDP 10B],
dual(num(Round(GDP, $(v1B)00), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)00)) as [GDP 100B]
RESIDENT Emissions
WHERE EXISTS (LatestGDP, CountryYear);

DROP FIELDS lat, long, LatestPop, LatestGDP;

So, that is it. Put all those bits of script together and you should have a working load script to bring in emissions and demographic data from three separate online CSV sources.

Now you just need a quick way to analyse those data. Fortunately we have your back there.

Viewing The Data In The Instant Sense Application

To make it quick for us at Quick Intelligence to produce Proof Of Concepts for potential clients I created the Instant Sense Application. This application allows you to load any data source, and a spreadsheet of metadata about that data source, and then explore the data over twelve sheets of user configurable visualisations and tables.

We have made this application available to all, at no cost, to make it quick and easy for others to showcase what Qlik Sense is capable of.

The application can be downloaded from here:

The Quick Intelligence Instant Sense Application

Details of how the application has been built can be found here:

How To Build A User Configurable Sense Application

If you already have the application, or if you just want the script above, you can download the script and meta-data here:

Global Emissions Data By Year with Metadata

If you use the application and build more expressions that you feel will be useful for others, please share them in the comments below. As ever, I’m happy to receive questions and comments from readers, and will get back to these as soon as I am able.

The post Loading Data from Online CSV Sources into Qlik appeared first on .

Note that this article was written for Qlik Sense Enterprise Client Managed, and the solution still works on that platform. On Qlik Sense Cloud the syntax is slightly different as you have to use the SMTP connector rather than Qlik Web Connectors. Take a look at this post for details of how to do that.

Why Send Data From The Qlik Load Script

QlikView has a couple of features which are very useful which are not provided with Sense straight out of the box. The first is that the QMC sends email alerts whenever a task fails, and the other is the Alerts functionality. Using the SMTP Connector within Qlik Web Connectors allows you to replicate this functionality and do more besides. This blog post looks at sending data to users from within the load script, I’m looking to write up Task failure notifications in a future blog post. As a reminder of how you can send data from QlikView you may want to see this blog post on Sending Data In QlikView Alerts. The SMTP Connector is one of the Standard (i.e. free) connectors provided with Qlik Web Connectors, for details of these you can see this post on Free Qlik Web Connectors. Note that the SMTP Connector has been renamed from the original name of Notification Connector, which provides a bit more clarity around what it does.

What we are going to look at in this post is pushing a table of aggregated data out to an email address on each refresh of the data.

Solution Approach

Typically in a Sense app you are loading a number of details rows and then building tables and charts which provide aggregated views of the data. Using QlikView Alerts or NPrinting the Qlik engine can be used to build these aggregations for you. As we are going to be sending the data from the load script we need to perform the aggregations ourselves at this point. Both Alerts and NPrinting can also trigger mails based on criteria – whilst this example does not cover this always remember the load script is a full programming language so conditionally calling parts of the code is very simple.

In order to show the aggregated data clearly in an email we are going to be using HTML to build a table, with a touch of CSS to make it look tidy. The SMTP Connector allows you to pass the body of the email in the URL, or to link to a local file with the content. As our table may get quite large we are going to write it to a HTML file on disk and tell the connector to pull that in. The process of creating the file has been covered in this previous post on the Qlik STORE Command, but I will tell you all you need here also.

The SMTP Connector can also mail attachments and link to images (perhaps created on the fly using the Sense API), but I am not covering that in this post.

Before You Start – The Prerequisites

The solution given below assumes that your version of Sense supports the command URL IS. This was brought in in the February 2018 release of Sense. It also refers to the SMTP Connector, which was renamed at some point this year, so a up to date version of QWC is recommended. If you are using QWC without a licence (i.e. only free connectors) then you need to keep within the last couple of versions anyway, so an upgrade is always wise.

Building An App Which Sends Data

Now you are all set here are the steps you need to follow to have email sent to you directly from your Qlik load script. I’ve described connecting to some sample data (from the excellent GapMinder site), but obviously you can plumb in whatever data you like. I’ve uploaded my app to Qlik Community should you want to download it, but you should get a fully working app by following these steps.

Setting Up Libraries

This app requires two libraries to be set up.

Firstly set up a Web library called GenericWeb. This can point to any valid web page, as we will replace the URL later on in code. This is still quite a new feature in Sense, and is well worth knowing how to use to avoid loads of different Web connections.

Next set up a Folder connection called TempData to a temporary store on your Sense server or desktop. I’ve used c:\temp\. You can use a different location, just adapt the code later on. As this location will have a copy of your data in it you will need to ensure it is secure.

Creating Encoding Subroutine

A number of the variables which make up the URL which is passed to Qlik Web Connectors need to be URL Encoded. This means that characters which could be misinterpreted need to be changed to a sequence of characters instead.

Create a sub routine to do this encoding:

sub Encode(vEncodeMe, vEncoded)
let vEncoded = replace(replace(replace(replace(replace(replace(replace(vEncodeMe, ':', '%3a'), '/', '%2f'), '?', '%3f'), '=', '%3d'), '\', '%5c'), '@', '%40'), ' ', '+');
end sub

The first parameter is the value to be encoded, the second parameter is a variable to be populated with the encoded value.

Setting Constants

These three variables store some environmental values. The variable vQwcConnectionName is used when Qlik Web Connectors when it generates Standard Mode code for us, so it is worth using that name. The vConn is just to make our URL shorter later, as we can refer to the variable. Finally vEmailFile defines where our staging file will be written, this must match the folder of the library created earlier.

let vQwcConnectionName = 'lib://GenericWeb';
let vConn = 'https://localhost:5555/data?connectorID=NotificationConnector';
let vEmailFile = 'c:\temp\EMailOutput.html';

The URL of your Qlik Web Connector server may differ, so enter that in here. You may require a machine name or domain reference here, rather than localhost, if your QWC instance is on another machine.

SMTP Settings

Now we define the settings for the SMTP connection. Note that some of these variables use the Encode sub-routine we created, others we don’t need to use it for.

call Encode('notifyme@wherever.com', vMailRecipients);
call Encode('smtp.gmail.com', vSMTP);
let vPassword = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
call Encode('someaddress@gmail.com', vFromEmail);
call Encode('Sending data from Sense Load Script', vSubject);
call Encode(vEmailFile, vEmailContent);
let vUseSSL = 'True';
let vSSLMode = 'Implicit';
let vPort = '465';

The settings above are for a GMail account. You will need to plumb in the correct settings for whichever SMTP server you wish to use. As this process is for sending emails only it is a good idea to set up a new account expressly for this purpose.

The password needs to be encoded before it is entered here. This can be done using the Qlik Web Connectors UI. First go to your QWC page (https://localhost:5555/) and go to the SMTP Connector settings (this is under the Standard connectors). In here you can set up and test your SMTP settings, including entering your password. The code which is generated on successful running will include the encoded version of your password. Extract this from the URL and enter it in the right place, it will appear after Password= and before the next ampersand (&).

Note that whilst the password looks very different it needs to be passed to SMTP as the original string. This means that a two way encoding mechanism is used, so someone who has your encoded password could in theory turn it back to the original one. Hence the advice to set up a new account just for sending.

There is a function in QWC where you can send a plain text password and receive the encoded one back (the Helper connector). This means you can enter the password in plain text in your script. I wouldn’t recommend this approach though.

Note that the Subject is included in these variables. If you want to have a dynamic subject (e.g. with record count) you can move that line to later in your code.

Get Some Data

Here you need to get the source data for whatever table you want to email around. If the application you are creating is purely for sending data you may want to bring in the aggregated data that you wish to send. If you are sending from an app that is also used for analysis you probably want detailed data for that analysis, and the aggregated data can be created in the next step.

This statement uses our generic library and grabs some data.

Population:
CROSSTABLE (Year, Population) LOAD
"Total population" as Country,
[2011.0] as [2011],
[2012.0] as [2012],
[2013.0] as [2013],
[2014.0] as [2014],
[2015.0] as [2015]
FROM [$(vQwcConnectionName)]
(URL IS [https://docs.google.com/spreadsheet/pub?key=phAwcNAVuyj0XOoBL_n5tAQ&output=xlsx], ooxml, embedded labels, table is Data)
;

Aggregate That Data

When you create a table in Qlik Sense you will provide a number of dimensions and then some calculations (Measures). This will then give you one row per distinct combination of dimensions.

We need to create that same table in code. This can be done with a GROUP BY statement, like this.

AnnualData:
LOAD
Year,
num(sum(Population), '#,##0') as [World Population],
num(max(Population), '#,##0') as [Largest Population],
num(avg(Population), '#,##0') as [Country Average]
RESIDENT Population
GROUP BY Year
;

In this instance we are creating one row per Year. Additional dimensions can be added to the load list, each of these also need to be added to the GROUP BY statement, in a comma separated list.

The expressions given in a GROUP BY load are very similar to those used in the expressions in Qlik objects in the front end, but there are some differences. Try the expressions that you would use to see if they work, if not you will need to modify. Probably the most notable difference is that you do not have access to Set Analysis in the load script.

Load Header Information into a Table

We are going to write the contents of a Qlik table to a file to then send as an HTML email. So first we need to put the preamble into the table.

EMailOutput:
LOAD
[<!--EMailOutput-->]
INLINE [
<!--EMailOutput-->
<html>
<head>
<style>
tr:first-child td {font-weight: bold;background-color: #dddddd;}
h3 {Font-family: Arial;Font-size: 12pt;}
td {border-left:1px solid #555555;border-top:1px solid #555555;font-family: Arial;font-size:9pt;text-align:left;padding: 2px 10px 2px 10px;}
table {border-right:1px solid #555555;border-bottom:1px solid #555555;border-collapse:collapse;}
</style>
</head>
<body>
<h3>Population stats for past five years</h3>
<table>
<tr><td>Year</td><td>World Population</td><td>Largest Population</td><td>Country Average</td></tr>
];

Note that the field name will be written to our output file, so it needs to make sense as HTML. This is why I have put a HTML comment there.

A style sheet is given here, which means that the table which is written out with the data in can be much simpler as all styling is handled by the CSS.

Add The Data Table

We have the data we need in a table, we now just need to format it so it displays right in HTML and add it to our table.

CONCATENATE(EMailOutput)
LOAD
'<tr><td>' & Year &
'</td><td>' & replace([World Population], ',', '&#44;') &
'</td><td>' & replace([Largest Population], ',', '&#44;') &
'</td><td>' & replace([Country Average], ',', '&#44;') & '</td></tr>' as [<!--EMailOutput-->]
RESIDENT AnnualData
ORDER BY Year DESC
;

Here we are constructing a number of HTML table cells to properly contain each value in our data. Note that one row of text will be written for each row of the table.

Note that we need to do a replace on the values to URL Encode the commas in the numbers. This is not because the commas will not display correctly in HTML, rather than when the STORE statement is called later it will place double quotes around any values that include commas in them. This will break our HTML. If you don’t use commas as thousand separators then you are good to go without the replaces.

Append On The Footer Information

Just as we started with the header information we now need to concatenate on the footer information.

CONCATENATE(EMailOutput)
LOAD
[<!--EMailOutput-->]
INLINE [
<!--EMailOutput-->
</table>
</body>
</html>
];

You have probably noticed that we are simply building up our code in bite-sized chunks. In the same way we could build in a number of tables and bits of narrative in one email. We are not limited to a single table of data here.

Write The HTML to a Text File

Now we have our table constructed with all of our HTML code we need to write this to a file that the SMTP connector can use. This is simply a case of using the STORE statement that you have probably used for writing QVDs, but here we are saying that we want this written as a text file.

STORE EMailOutput INTO [lib://TempData/EmailOutput.html] (txt);

If your Library was set to point to a web server, perhaps on a UNC path or on the Sense server, then this output can be served up on a web page. This is a great way of outputting data from Sense for a wall board. A very simple Sense extension can be used to make Sense into a web server for this purpose – I will write this up in a future blog post.

For now though we are just using this output in the SMTP Connector.

Send The Email

Finally we just need to send the email. This is done by pulling data from a web page (the same as any other Qlik Web Connector call), the difference is though that this call interacts with a server to send data and not just receive it.

SendEmail:
LOAD
status as SendEmail_status,
result as SendEmail_result,
filesattached as SendEmail_filesattached
FROM [$(vQwcConnectionName)]
(URL IS [$(vConn)&table=SendEmail&SMTPServer=$(vSMTP)&useSSL=$(vUseSSL)&SSLmode=$(vSSLMode)&Port=$(vPort)&Password=$(vPassword)&to=$(vMailRecipients)&subject=$(vSubject)&message=%40file%3d$(vEmailContent)&fromName=Sense&fromEmail=$(vFromEmail)&appID=],
qvx);

Note that we have three fields coming back from the query. These will confirm to us whether the send was successful or not. If you want to bulletproof your code you should use PEEK to look at these values just to check that the process has done what you expect.

You will see in the URL all of the variables that we have set up. Crucially there is the message= parameter, which then is pointed to a file, and the location of the content we have written to disk.

Note that you can generate the bulk of this code using the Qlik Web Connectors UI, but the code here has all the various parts moved out into variables, which makes it easier to configure.

Example Application

If you have followed the instructions above you should now have an application which emails you data on each reload.

In case this hasn’t quite worked out for you, or if you would just like to check your workings, you can download my completed application from Qlik Community. You can find this here:

https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Send-Data-From-The-Load-Script/ta-p/1481383

The application also has a sheet which shows the two sets of data (aggregated and raw), the result from the SMTP send and the HTML generated as a table.

If you have any questions regarding this technique please put them in the comments below, thanks!

Community Update!

Héctor Muñoz has kindly shared this post on his blog, and not only that he has taken it to another level. With the code above it is only possible to send tables of data. What Héctor has done is provided the code needed to get data in the load script, parse it, send it to the Image Charts site to get a chart and then include that in the body of the email. This is awesome and opens up some very interesting possibilities. Many thanks Héctor for doing this.

The blog post has unfortunately now been taken down.

We always welcome people commenting on or expanding the things we do in our posts. Please used the comments field below if you have something you would like to share.

The post Send Data from the Qlik Load Script appeared first on .