Background to Using the API

The Qlik SaaS API can be used for retrieving information regarding your SaaS environment and also for making changes in your environment, such as allocating permissions. The endpoints available to end-users are the same endpoints that Qlik use to build the product. You can interact with these via REST connectors, Qlik Automations and third party tools; such as Göran Sander’s Ctrl+Q. I have blogged previously on using these APIs in the Sense load script, and describe the process of setting up and calling them in my post on Monitoring and Notifying Sense Capacity Licence Usage in Qlik Cloud. If you are not yet using APIs in your own applications then you might want to start there.

Required Changes to Fix Broken Loads

Quick Intelligence provide a free download of what we call the Instant Qlik Sense Application. This is a Sense application template, which ships with a number of example load scripts and configuration files. Two of these example scripts are for monitoring Qlik SaaS usage and sessions. Both of these scripts have been broken by the change to the API.

Fortunately the change to the API endpoint is relatively minor and the fix simple. The revised documentation for the endpoint can be found here https://qlik.dev/apis/rest/licenses.

If you wish to modify the script in a an app you already have running you simply need to find the Licences part of the script and change references to userId and UserId to subject and Subject respectively, and then remove references to name and User Name.

There is a further gotcha that I have spotted whilst changing this over though, and that is that the case of the subject field on the Users endpoint does not always match the case of the same field on the Licences endpoint. You should therefore add an upper statement around Subject in the creation of the mapping table and where it is used. If you are using the endpoints and associating the tables rather than mapping you will need to match the case in the tables themselves.

If you use these endpoints in your own scripts, or have downloaded an application that uses them, then you are likely going to have to make similar changes to that load script.

If you don’t yet use the Instant Qlik Sense Application then you can find out more about it, and download it from here:

https://www.quickintelligence.co.uk/instant-qlik-sense-app/

If you would just like the script for using the Sessions or Usage endpoints of the API these can be downloaded here:

https://www.quickintelligence.co.uk/SaaSUsage.txt
https://www.quickintelligence.co.uk/SaaSSessions.txt

With the SaaS usage script and configuration file, Sense users are coloured by their licence types and if you have a mix of both analyser and capacity licencing you can ensure that the users who use the system the most are the ones who are holding analyser licences whilst occasional users can use capacity.

If you have any questions or comments on using the API or the Instant Qlik Sense Application then please use the comments section below.

The post Change to Qlik SaaS Licences API Endpoint appeared first on .

Instant Sense Application Live Demos

It is sometimes to hard to describe what the Instant Sense Application is or how it functions. The simplest way to get your head around it is to see it working, with data that means something. In order to achieve that I set up auto-authentication on a SaaS tenant to allow anonymous usage of some ISA demos.

My thanks go to Daniel Pilla at Qlik who helped me get this working. The instructions to set this up can be found on Qlik Dev here: https://qlik.dev/embed/embed-quick-starts/embedding-with-anonymous-access-and-qlik-cloud. Whilst there is no equivalent to Tableau Public from Qlik, setting this up on a SaaS Tenant with a Capacity licence is the best way to go.

This has allowed me to post live demos to my socials, and to the Instant Sense App landing page. There is also now a live view of Qlik Blog Posts over on the AskQV site.

Download Without Signing Up for Updates

When the Instant Sense Application was first released it was a requirement that users signed up with a valid email address and confirmed that address. I noticed that a lot of people were hitting the ISA page and then not giving their details. Fair enough, I might well do the same. For that reason, we removed the requirement to sign up to download the app. If you would like to hear from Quick Intelligence you can still sign up for our newsletter, but if you would prefer just to download the app you can do so by clicking here.

To me this application is more important as a way to aid Qlik Sense adoption and to make it much quicker and easier to get data, and insights, to users than it is as a promotional tool for Quick Intelligence.

Please take the application and do something great with it! If you do find it saves you days of effort building applications it would be great if you could like, share, comment and all those good things on our social posts. I would love to hear what you are doing with the app.

New Example Scripts and MetaData

The Instant Sense Application ships with a number of example scripts and metadata files, so applications over these data can be quickly spun up and used in your own environment. Pretty much all of these have been tweaked to make use of the new features (more on those below), some also have new expressions and dimensions. As well as this new examples are being added to the download.

Pretty much from the first release, there have been session monitors for both Sense Client Managed and for Sense SaaS, sadly the SaaS one was limited by the API that was being called and didn’t show session duration.

Since that original SaaS example script was created, Qlik have exposed a new API endpoint which shows licence consumption, allowing a much more detailed view of who is consuming which licences. By using this script Sense SaaS administrators can get a good view on who should be holding a named licence and who should be consuming capacity, based on their recent activity. The currently held licence type is denoted by the colour. The client-managed Session monitor has been expanded to better facilitate this also:

Another new script file that was added this year is for profiling libraries of QVD files, by reading the meta-data from the QVD header. It gives a count of QVDs and stats around the size and ages of those QVD files. This app can help you identify redundant files in your environment and warn when the size of files is increasing. Any one who has a QVD layer in their environment (and that should be everyone) can make use of this script.

I’m looking at adding more example scripts to the download in time, both Qlik related ones and other non-related data sources. Watch this space.

New Application Features

There have been a number of tweaks made to the application over the year, some of the more interesting and useful ones are listed here.

The first I am going to mention you have already seen above. All of the charts (except the KPI text, annoyingly) now have a colour expression on them which takes parameters that define the chart type, dimension number and measure number.

By modifying the expression in that variable dynamic colours can be applied throughout applications. These colours can be based on the dimension, expression, chart type or even be derived from values in the data, as shown here. Documentation is provided in the download showing various options for applying colours.

To further facilitate colours, and to allow the adding of bespoke functionality to your apps, there is now a Variables sheet in the metadata file. This can be used to inject variables into the app at load time. You could create all of your measures as variables, should you want.

The button functionality within Sense has improved, so there are a number of buttons to help with UI in the Instant Sense Application now. The Scatter chart has a button to toggle which expression is shown on the X and Y axis and the Top 20 trend has buttons so that the number of lines shown can be increased or decreased.

Similar buttons appear on some of the new sheets straight out of the gate. There is also a button (which is a bit work-in-progress) to link to Google Maps from the Map sheet.

The ability to use Google Spreadsheets for the meta-data has been added, so that your online application can read its settings from an online data source, taking away the step of having to upload the configuration to Cloud after every change. Using Office 365 for this also is on the to-do list.

The map sheet now hides itself if you don’t have any geographical dimensions defined in your meta data. I’ve also tweaked some of the settings on the map, so that when you have only one point selected it doesn’t zoom to a point that you can’t see any context. You might want to adjust these settings for your own purposes, but I believe the defaults are better now.

Whilst it is best practice to use Master Dimensions for everything, I have decided to strip the Master Dimension out of the Sense Profiler sheet of the application. The reason for this is that this sheet is a useful addition for any Sense application, and now it can be copied and pasted into other apps without any faff. The sheet gives you a list of all tables and fields in a data model and then shows a chart of the number of occurrences of each value in the selected field.

New Sheets And Chart Types

The most recent addition to the Instant Sense Application is the Dual Map sheet. This will appear if you have two or more location type dimensions in your dataset. The first selected dimension shows on the left and the second on the right. This can be used to show different levels of granularity within location data, as shown here, or to show to and from locations for shipping or travel analysis.

There have been a few chart types that have been absent from the application, until now, as they only work with expressions that accumulate to a total. A new column has been added to the Expressions sheet of the meta-data that defines which expressions are part-to-whole. If you have one or more of these in your data then the new sheets appear, giving you pie charts (sorry), a tree map and a Sankey chart. All of these work the same way as the existing sheets, with the ability to pick dimensions and any part-to-whole expression.

Both the Sankey and the Tree Map have the ability to toggle which is the primary and secondary dimension, which gives much flexibility in how things are shown.

Look To The Future Now, It’s Only Just Begun

Whilst extending and expanding the Instant Sense Application isn’t something I get to do as much as I would like, there are ideas I have for it. As we have deployed the application on client sites there are change requests from clients also. These often find their way back into the download file. I’m also always happy to hear suggestions from the Community on how the app could be improved.

One of the ideas I have presently is to add Dimension and Measure groups, so that these can be selected from a drop-down in one go, rather than picking them separately. For instance, with the Sense Usage apps I tend to pick a view that has Total Usage, Today’s Usage, This Weeks Usage, Rolling 30 Day Usage and Days Since Last Licence Use, all shown by User, to pick which users should be allocated or de-allocated licences. With no hide condition on Filter Panes (yet) they would leave redundant objects on the screen if not configured. Besides, bookmarks provide quite a neat way of saving groups of items – so this has remained on the back burner.

Something I have found myself doing on a number of roll-outs of the app is to add a button to each sheet. On one of the demos I have added a button to link to a web page for further information, elsewhere I have added a button to reload the app. What I am considering is adding a button on each sheet, which only appears if the destination and legend for the button is defined in the metadata spreadsheet. I think this could be a nice addition, but not one that has bubbled to the top of the list yet.

The beauty of the Instant Sense Application is that the one application can have many different data sources and meta-data files loaded into it. If I create any more, which is likely to happen, then these will be added to the download. It was also my hope that other people might contribute script and meta-data files for other data sources. If you have the Instant Sense Application pointing at a data source that might be useful for others, please get in touch to see how we might roll that out.

If you fancy creating a load script and meta-data file, perhaps over data related to a hobby or interest, then it would be great to hear from you also. That app could be made available either part of the application download or as a hosted demo.

If you are using the application, for any reason, it would be great to hear from you. You can comment on this blog post, on our social posts, or get in touch with me directly.

Thanks everyone! It has been incredible to see how many times this application has been downloaded, and I really hope it is helping you to deliver better analytics in less time.

The post Instant Qlik Sense Application Changes in 2022 appeared first on .

API Connections in Qlik Sense Client Managed

Straight of the box with a clean install Sense has a number of REST Connections created for you to explore your environment. It’s well worth exploring what these connections offer, with regards to looking at users and usage.

For this post though we are going to be looking at the monitor_apps_REST_app connection.

Create a new app and go to the Data Load Editor. If you can see the monitor_apps_REST_app connection we are good to go. If it’s not there you will need to grant yourself rights in the QMC under the Data Connections settings, find the correct connector and select Security Rules.

From here you can grant yourself access to the one connection by your name or user id:

Or alternatively, if you are a RootAdmin on the QMC you may want to grant the RootAdmin role access to all connections on the server. This will mean that administrators can all explore all of the default connections and keep an eye on what users are creating also. Do this by replacing the GUID of the connection in the Resource Filter with an asterisk. Obviously, you can tailor the permissions to be whatever you need.

When you go back to the Data Load Editor in another tab the connections will not have appeared automatically. You will have to refresh the browser with a Shift-F5, or click the refresh button in the browser toolbar.

Loading Data From REST API Connection

Now you have access to the correct REST connection, you can use the Select Data icon next to the monitor_apps_REST_app connection to look at the data. Alternatively, you can use the script here to pull the information that you need.

Before we connect to the data though, we are going to set up some variables which describe where the apps currently reside and where we want to back them up to. The apps will be in your repository, the location of which will have been set when Sense was installed. There will be a share, most likely on the Sense server, pointing to this location. In here find the Apps folder, which contains a file for each app, with a GUID for a name and no extension. The destination is simply where you want to copy to. Here I create a dated folder for the backup:

// Set up locations of Sense Repository and a backup location
// Used in the backup script field
let vSrcFolder = 'c:\QlikSenseRepository\Apps\';
let vDestFolder = 'c:\SenseBackups\AppBackup' & date(today(), 'YYYYMMDD') & '\';

A connection then needs to be made to the REST API in order to load data from it:

// Connect to the REST connection
LIB CONNECT TO 'monitor_apps_REST_app';

With this in place we can then pull the data from the REST connection, using this code:

// Load raw JSON Data
RawRESTData:
SQL SELECT
"id" AS "id_u1",
"createdDate",
"modifiedDate",
"modifiedByUserName",
"name" AS "name_u1",
"publishTime",
"published",
"description",
"fileSize",
"lastReloadTime",
"thumbnail",
"savedInProductVersion",
"availabilityStatus",
"schemaPath",
"__KEY_root",
(SELECT
"userId",
"userDirectory",
"name",
"__FK_owner"
FROM "owner" FK "__FK_owner"),
(SELECT
"name" AS "name_u0",
"__FK_stream"
FROM "stream" FK "__FK_stream")
FROM JSON (wrap on) "root" PK "__KEY_root";

As with loading data from just about any REST connection, the data will come down in a single table, even though there are three logical tables in the data model. Each row of data will have values in columns in one of the three sets (Application Details, User Details and Stream), values in the other columns will be null. Along with these values are primary and foreign keys to join things correctly.

Using Select Data the Qlik engine will create some code which will load three separate tables into the end data model. In this case however we know there is a one-to-one relationship between an Application and its Owner and the Stream it is in (if the application is published). We can therefore create a cleaner and flatter data model with ApplyMap.

There is much more we can do also, to make it easier to select the apps we want to back up, and do further analysis on apps. The same data feed can be used to help use identify apps which are not being maintained or are bloated.

Cleaning Up The Application Table

Now we have all of the data in a temporary table we need to clean it up and augment it. This will be done with a number of RESIDENT loads and a couple of Preceding Loads.

First we are going to create some ApplyMap statements to bring in the Owner name, Owner login and Stream. These use the foreign keys in the flat table.

// Create mapping tables from parts of the REST result
Map_Owner:
MAPPING LOAD
[__FK_owner],
[name]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_OwnerLogin:
MAPPING LOAD
[__FK_owner],
[userDirectory] & '\' & [userId] as Login
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_owner]);

Map_Stream:
MAPPING LOAD
[__FK_stream],
[name_u0]
RESIDENT RawRESTData
WHERE NOT IsNull([__FK_stream]);

Then some variables which act as functions in the subsequent RESIDENT load. These all utilise the fact that a SET statement does not evaluate the code that it puts into a variable (rather it copies code verbatim) and the functionality of passing parameters into variables.

// Create parameterised variables for repeated tasks
set vTime = Date(if(Date#(left($1, 10), 'YYYY-MM-DD') < 0, null(),
Date#(left(replace($1, 'T', ' '), 19), 'YYYY-MM-DD hh:mm:ss')),
'DD MMM YYYY hh:mm:ss') as [$2 Date Time];
set vDP = Date(DayStart([$1 Date Time]), 'DD MMM YYYY') as [$1 Date],
Date(MonthStart([$1 Date Time]), 'MMM-YYYY') as [$1 Month],
Month([$1 Date Time]) as [$1 Month Name],
Year([$1 Date Time]) as [$1 Year],
WeekDay([$1 Date Time]) as [$1 Day],
today() - DayStart([$1 Date Time]) as [Days Since $1];
set vNS = if($1 = '', 'Not Specified', $1);

The first function here receives the name of a date field and checks whether it has a value, if it does it returns that converted to a date and formatted correctly. The second takes the date time field created by the previous variable (thanks to a preceding load) and creates various date parts for these dates. The final variable checks a given field name and returns Not Specified if the value is empty, otherwise it returns the value.

You will see all of these variables used in the next block of code.

We can then do our final RESIDENT load to bring through all of the original data, with the various transformations applied.

Applications:
LOAD
1 as ApplicationCount,
*,
RangeMin([Days Since Created],
[Days Since Modified],
[Days Since Published],
[Days Since Last Reload]) as [Days Since Last Touch],
// Export this column to create a script to backup your apps from the Repository
'copy $(vSrcFolder)' & GUID &
' "$(vDestFolder)' & Stream & ' -- ' &
Application & '.qvf"' as [App Backup Script]
;
LOAD
*,
// Create all required date parts
$(vDP(Created)),
$(vDP(Modified)),
$(vDP(Published)),
$(vDP(Last Reload))
;
LOAD
id_u1 as GUID,
// Convert all required dates
$(vTime(createdDate, Created)),
$(vTime(modifiedDate, Modified)),
$(vTime(lastReloadTime, Last Reload)),
$(vTime(publishTime, Published)),
[modifiedByUserName] as [Modified By],
[name_u1] as Application,
[published] as [Is Published],
$(vNS(description)) as [Application Description],
fileSize / 1000000 as [File Size],
$(vNS(thumbnail)) as Thumbnail,
[savedInProductVersion] as [Product Version],
[availabilityStatus] as Availability,
ApplyMap('Map_Owner', [__KEY_root], 'Not Known') as Owner,
ApplyMap('Map_OwnerLogin', [__KEY_root], 'Not Known') as [Owner Login],
ApplyMap('Map_Stream', [__KEY_root], 'None') as Stream
RESIDENT RawRESTData
WHERE NOT IsNull([__KEY_root]);

Then, to tidy things up and avoid any strange joins, we can remove the temporary table which was created when the data was pulled from the REST connection.

DROP TABLE RawRESTData;

And that is our load script done.

Unpacking The Application Back Up Code

There is quite a lot going on in that last step of the code. Hopefully you can work out what is going on with most, if not all, of it. Much of it is to give us some extra fields to do some analysis of the applications on our server (more on that later), but for now just to focus on the creation of the App Backup Script field.

'copy $(vSrcFolder)' & GUID & ' "$(vDestFolder)' &
Stream & ' -- ' & Application & '.qvf"' as [App Backup Script]

Here we are starting with the copy statement we wish to use (you can replace this with XCopy or add parameters if you wish), adding in the two variables we set up right at the top of the script and pilling out the GUID which is the source filename for our script. There is then the building of the output filename and the placing this in a fieldname.

The construction of the output filename and path you can also alter if you wish, I have gone for the stream name (which is set to None for items that are not published) a double hyphen (as apps and streams may have hyphens in the name already) the application name and finally the QVF extension.

This then builds code that we can use in a Command Prompt on our Qlik Sense server to move the apps to their backup location.

Creating and Using The Back Up Script

Now our script is complete, click Load Data to bring all the data in from the REST connection. Once this has finished loading click Analyse Sheet and change to Edit mode. Grab a Table object from the Charts and drop it on the sheet. Click to Add Dimension and add in the App Backup Script dimension.

Come out of Edit mode and you can then right click to export the table to Excel. Open this spreadsheet and copy and paste the code into a new Notepad (or other text editor) window and then save the script as a .bat file. This file can then be copied onto the server and executed to perform the backup.

There is a slight risk of one of the files being corrupted if it is copied whilst a write is in progress, so it is best practice to stop all Sense services before taking the backup. In practice, I have not ever hit a snag with this though and you could simply check the date stamp on the files and see if there were any that were updated at around the time you took your backup and perhaps copy them again.

So there you have a relatively quick and painless way of getting QVFs of all of your Qlik Sense Client Managed Server applications.

I hope you find the technique useful. Please let us know how you get on in the comments below.

Further Analysis of Your Application Data

As well as the the information required to filter and export the backup script there were various other data points brought in about the Applications on the Sense Server which you may want to analyse. Finding which applications have not been touched in over a year, for example.

Fortunately we have your back with this, with our Instant Sense Application. This is an application template which ships with a number of pre-written load scripts (including the one above) and metadata files so that you can re-purpose the application to use your own data sources. Out of the box you can view a number of KPIs over these application data:

It also allows you to break the various measures down over different dimensions and plot those different measures against each other, by a selected dimension:

Finally there is the user defined table, where the user can select a number of dimensions and measures and view these in a table which can then be exported as required:

Everything in the application is driven from a metadata spreadsheet, this means that you can configure the app so it can be used over whatever data source you like:

All in all there are twelve sheets of user configurable visualisations, just waiting for you to bring your own data. The best thing is that it is free to download and use.

Find out more about the application, and download it, from here:

https://www.quickintelligence.co.uk/instant-qlik-sense-app/

More technical details, including how the application is built can be found in this Instant Sense Application blog post.

If you have any questions or comments about anything above, please do drop us a comment below.

If you would like to share a load script and metadata spreadsheet for the Instant Sense Application then we would love to hear from you.

The post Back Up Qlik Sense Enterprise Client Managed Applications appeared first on .

Introducing The CSV Data

Recently I was asked to be involved with the Data On The Earth project. This is an initiative to get data that can help everyone, from policy makers to concerned individuals, make more informed decisions about environmental issues. Naturally, I was very happy to help.

The first thing I wanted to do was to present some ideas over some data that had relevance to the job in hand. The site Our World In Data seemed like a good place to start.

There I found data which related to Co2 emissions, hosted on GitHub, you can find more about these data here. The data are by Country and Year, so a bit of extra metadata on the countries would allow some demographic filtering. I found this in two further CSV files, also conveniently hosted on GitHub.

The data are not perfect, as some of the figures are not reported by all countries in every year and countries change over time. There are things that could be done to fix this up a bit (rolling forward last known figures, for instance), but I have not done that here. If you are planning to lobby a government over their emissions (or something) please check your facts and don’t rely on what is presented in these data alone!

Creating a Connection

The code in this article is for the load script of Qlik Sense. You can do similar with the REST connector in QlikView, but the syntax is a bit different, this post should give you pointers should you wish to build your own load script.

As with any load script in Sense you first need to have your connection created. Rather than create a separate connection for each data source I load I tend to create a generic connection of each type required and modify it using WITH CONNECTION. You may already have the required connection set up if you have previously followed my tutorials on Loading Sense Session Logs or Loading RSS Feeds, if not go to the Data Load Editor and click Create New Connection.

Give your connection the name GenericGET and point it to a placeholder, such as https://jsonplaceholder.typicode.com/posts, ensure the Method is set to GET and leave everything else as defaults.

Click on Test Connection, then Close and finally (if all is good) Create.

Building the Load Script

Now, onto building the load script. First we need to do some set up, to connect to our connection and set up some variables:

LIB CONNECT TO 'GenericGET';

let vCompleteYears = 'Y'; // Y/N for all years or only 1990 - 2016 where data is quite complete
let vExcludeYears = if(upper(left(vCompleteYears, 1)) = 'Y', 'AND (year >= 1990 AND year <= 2016)', '');

set vFix = if([$1] = '', '$3', [$1]) as [$2];

let v1B = 1000000000;
let v1M = 1000000;

The first variable allows us to turn on or off data for years which are not that complete, the second puts the code to implement that into a variable if switched on. Next we have a variable which includes parameters that we will use later on. If you have not used them before you should see this post on Using Parameters with Qlik Variables. Finally a couple of variables to remove the risk of typing the wrong number of zeros later on.

Now we are set up, we can do the main part of the load.

Emissions:
LOAD
iso_code & year as CountryYear,
iso_code as [Country ISO],
country as Country,
year as Year,
co2 as CO2,
consumption_co2 as [CO2 Consumption],
trade_co2 as [CO2 Trade],
co2_per_unit_energy as [CO2 Per Unit Energy],
cement_co2 as [CO2 Cement],
coal_co2 as [CO2 Coal],
flaring_co2 as [CO2 Flaring],
gas_co2 as [CO2 Gas],
oil_co2 as [CO2 Oil],
total_ghg as [Total GHG],
methane as Methane,
nitrous_oxide as [Nitrous Oxide],
population as Population,
gdp as GDP
WHERE iso_code <> '' and index(iso_code, '_') = 0 // Remove rolled up values
$(vExcludeYears)
;
SQL SELECT
"iso_code",
"country",
"year",
"co2",
"consumption_co2",
"trade_co2",
"co2_per_unit_energy",
"cement_co2",
"coal_co2",
"flaring_co2",
"gas_co2",
"oil_co2",
"total_ghg",
"methane",
"nitrous_oxide",
"population",
"gdp"
FROM CSV
(header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/owid/co2-data/master/owid-co2-data.csv")
;

As with a database load this load will use the last opened connection. The CSV on the FROM line is specifying the basic format that we are loading from this connection, the exact format of the CSV is specified in the brackets – just as it would be if you were loading the CSV from a Folder Connection. The CSV_Source relates to which data source at the connection you wish to load – with a CSV this will only ever be CSV_source, but with JSON data there may be many named sets of data for you to choose from. The WITH CONNECTION then defines the URL we are loading from.

To explore these options further create another connection, pointing to that URL and look at the Select Data dialog for that connection.

Atop of the REST load we have a Preceding Load, to tidy the fieldnames and remove data we don’t want. There are rows with data rolled up to regional levels, but we are going to do that ourselves in Qlik so need to remove the roll ups to avoid double counting. Note that when doing a WHERE in a preceding load all of the data still has to come down from the REST source first, so you may want to look if you can pass parameters to the URL you are loading from to drop some data, rather than using a WHERE statement.

So, that is the data we require from our main data source. We are going to continue though, and augment and categorise these data some more.

Bringing in more CSV Data

The next table of data is a simple table of country demographics, provided by GitHub, keyed on the ISO code for the country.

Region:
LOAD
[ISO3166-1-Alpha-3] as [Country ISO],
$(vFix(Developed / Developing Countries, Developed, Not Known)),
$(vFix(Region Name, Region, Other)),
$(vFix(Sub-region Name, Sub Region, Other))
WHERE EXISTS ([Country ISO], [ISO3166-1-Alpha-3])
;
SQL SELECT
"ISO3166-1-Alpha-3",
"Developed / Developing Countries",
"Region Name",
"Sub-region Name"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://raw.githubusercontent.com/datasets/country-codes/master/data/country-codes.csv");

This file has many more columns than we need, but we are cherry picking the few that we want. As you can see, the code for connecting to a CSV source is almost identical to above – just the URL is changing. In the Preceding load the vFix variable we created at the start is being used to replace any missing values with Not Known or Other. A WHERE EXISTS ensures we only get rows which relate to countries we have emissions data for.

Using a lookup table like this from a repository like GitHub in your Qlik apps means that your application will benefit from improvements to that dataset over time, rather than having a CSV or Excel spreadsheet languishing on your server which will never see an update.

Where In The World Is That?

As we are looking at country data it would be great to plot that on a map, and another GitHub source has just that information available for us. These data are provided and maintained by Tadas Tamošauskas in a personal repo.

Here we are going to LEFT JOIN the data onto the existing region table, bringing the latitude/longitude data into the same table as the regions. You will see why in a bit.

LEFT JOIN (Region)
LOAD
*,
GeoMakePoint(lat,long) as [Country Point]
;
LOAD
purgechar([Alpha-3 code], ' "') as [Country ISO],
purgechar([Latitude (average)], ' "') as lat,
purgechar([Longitude (average)], ' "') as long
WHERE EXISTS ([Country ISO], purgechar([Alpha-3 code], ' "'))
;
SQL SELECT
"Alpha-3 code",
"Latitude (average)",
"Longitude (average)"
FROM CSV (header on, delimiter ",", quote """") "CSV_source"
WITH CONNECTION
(URL "https://gist.githubusercontent.com/tadast/8827699/raw/f5cac3d42d16b78348610fc4ec301e9234f82821/countries_codes_and_coordinates.csv");

This now lets us plot emissions data on a map by country. Our second data source though gave us Region and Sub Region names. It would be good if we could also plot by those regions. There is probably a data source out there which has SVG data for each region, but that could take a while to find and will not necessarily map to our existing region names.

We can apply a simple hack to generating points at each of the regions though.

RegionPoint:
LOAD
Region,
GeoMakePoint(avg(lat),avg(long)) as [Region Point]
RESIDENT Region
GROUP BY Region;

SubRegionPoint:
LOAD
[Sub Region],
GeoMakePoint(avg(lat),avg(long)) as [Sub Region Point]
RESIDENT Region
GROUP BY [Sub Region];

Here, a couple of RESIDENT loads from the existing Region table can be grouped to give one row per Region and Sub Region. For each of these rows the average latitude and average longitude can be calculated. These two values can then be changed to a point that can be easily used on a map. Note that some points may appear in the ocean, but that is the joy of using averages.

Finding Latest Data and Grouping Figures

The figures from our initial dataset are great for trending over time as they are provided on an annual basis. They are not so great for categorical data and providing filters though, as a country could move from one bucket to another year to year. For these filters we only want to use the latest figures. Here we hit another minor snag though, in that the last year in which data is available differs by country.

We can deal with this by finding the latest year for each country, using a RESIDENT, GROUP BY and MAX to create a composite key.

LatestPop:
LOAD
[Country ISO] & Year as LatestPop
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE Population <> ''
GROUP BY [Country ISO];

That composite key can then be used in the WHERE EXISTS of a subsequent RESIDENT load to only bring the latest data.

Population:
LOAD
Country,
Population as [Population Latest],
dual(num(Round(Population, 100000) / $(v1M), '#,##0.0') & 'M', Round(Population, 100000)) as [Population 100K],
dual(num(Round(Population, $(v1M)) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M))) as [Population 1M],
dual(num(Round(Population, $(v1M)0) / $(v1M), '#,##0') & 'M', Round(Population, $(v1M)0)) as [Population 10M]
RESIDENT Emissions
WHERE EXISTS (LatestPop, CountryYear);

This is a useful technique, and one I use frequently in live projects. As well as getting the exact number for the latest year for each country we can also put things into buckets. This can be done with a massive IF statement or for equal buckets an aggregation. Here I have used a Round statement, but you could use a Floor or Ceil if you prefer. The Round function in Qlik is more flexible than in most languages as, rather than just rounding to a number of decimal places, you can specify what grouping you wish to go to, whether that is more or less than one.

The same technique can then be used for GDP, then there is just a little bit of housekeeping to do at the end of our script.

LatestGDP:
LOAD
[Country ISO] & Year as LatestGDP
;
LOAD
[Country ISO],
max(Year) as Year
RESIDENT Emissions
WHERE GDP <> ''
GROUP BY [Country ISO];

GDP:
LOAD
Country,
GDP as [GDP Latest],
dual(num(Round(GDP, $(v1B)), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B))) as [GDP 1B],
dual(num(Round(GDP, $(v1B)0), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)0)) as [GDP 10B],
dual(num(Round(GDP, $(v1B)00), '#,##0') / $(v1B) & 'B', Round(GDP, $(v1B)00)) as [GDP 100B]
RESIDENT Emissions
WHERE EXISTS (LatestGDP, CountryYear);

DROP FIELDS lat, long, LatestPop, LatestGDP;

So, that is it. Put all those bits of script together and you should have a working load script to bring in emissions and demographic data from three separate online CSV sources.

Now you just need a quick way to analyse those data. Fortunately we have your back there.

Viewing The Data In The Instant Sense Application

To make it quick for us at Quick Intelligence to produce Proof Of Concepts for potential clients I created the Instant Sense Application. This application allows you to load any data source, and a spreadsheet of metadata about that data source, and then explore the data over twelve sheets of user configurable visualisations and tables.

We have made this application available to all, at no cost, to make it quick and easy for others to showcase what Qlik Sense is capable of.

The application can be downloaded from here:

The Quick Intelligence Instant Sense Application

Details of how the application has been built can be found here:

How To Build A User Configurable Sense Application

If you already have the application, or if you just want the script above, you can download the script and meta-data here:

Global Emissions Data By Year with Metadata

If you use the application and build more expressions that you feel will be useful for others, please share them in the comments below. As ever, I’m happy to receive questions and comments from readers, and will get back to these as soon as I am able.

The post Loading Data from Online CSV Sources into Qlik appeared first on .

Logging on Qlik Sense Enterprise

With the default settings on logs in Sense Enterprise get written to two locations. One for the log file currently being written to and then an archive, where the current log is moved to periodically. By loading from both of those sources you get a good picture of what is going on with usage on your site. Connections are created to these two locations, named ServerLogFolder and ArchivedLogsFolder. There is a reasonable chance that you won’t see these connections in the Load Script Editor, so you will need to first go into the QMC and grant yourself rights – be careful not to grant more rights than you intend to when doing this!

Once you have access to these connections you can dig around and find out quite a lot about your system and how it is performing. This post, however, just focuses on the most important information. Who has done what and when.

Sense Session Log Load Script

Once you have permissions to the connections set up you can enter the Data Load Editor and start building your script.

First you need to set a couple of variables, which form part of the path to the session files. The first of these is generally the machine name, so can be left as is. The second tends to be a combination of the machine name and the local domain. You will need to click on the Select Data button underneath the connection name to confirm the path if its not what is expected.

// Set variables which identify this server and form part of the log names
// Check in the ServerLogFolder and ArchivedLogsFolder to find the values for your installation
let vServerName = ComputerName();
let vFullServerName = 'servername.domain.local'; // Usually FQDN

The next set of variables are then constructed from the connection name (which is usually the same in each installation) and the two variables you set above.

// Build out file paths and file masks based on the variables above
let vCurLog = 'lib://ServerLogFolder/';
let vLogFolder = 'lib://ArchivedLogsFolder/$(vFullServerName)/';
let vCurFile = '$(vCurLog)Engine/Trace/$(vServerName)_Session_Engine.txt';
let vArchiveFiles = '$(vLogFolder)Engine/Trace/$(vServerName)_Session_Engine_*.log';

As you would expect, there are a number of dates in the log file. Because the log is in a text file, rather than a database table, the dates are in plain text. This means they each need to be converted to dual-datatype dates so they can be used. The code for each of the conversions would need to be repeated, so here is a parameterised variable to do the job. If you have not used variables with parameters before you may want to see this previous blog post.

// Create parameterised variables for repeated tasks
set vDT = Date(Date#(left(replace($1, 'T', ' '), 15), 'YYYYMMDD hhmmss'), 'DD MMM YYYY hh:mm:ss');

The filename for the current log file has been built into a variable. Just in case we are between log files at this point we are going to check first that it exists. If it does we can bring the fields that we want from the file. If you are a completest you may want to look to the source file and bring through the fields I have dropped. This is pretty much everything that is interesting though.

Note that I am excluding sessions relating to reloads by the Sense Distribution service. If you are looking for log-jams on CPU use at specific times of day you may want to include these rows, but for most analysis they are not useful.

// Load the raw data from the current file
if alt(FileSize('$(vCurFile)'), 0) > 0 then
Temp_Sessions:
LOAD
ActiveUserId,
[App Title],
[Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser,
ActiveUserDirectory,
Timestamp,
[Exe Version],
[Server Started],
AppId,
[Doc Timestamp],
[Authenticated User],
[Secure Protocol],
[Session Duration],
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
FROM [$(vCurFile)]
(txt, utf8, embedded labels, delimiter is '\t', msq)
// Ignoring reload sessions. Remove next line if you want all sessions
WHERE ActiveUserId <> 'sa_scheduler' and ActiveUserId <> 'sa_repository'
;
end if

For the archive log files we need to enumerate around all of the files in the archive folder. Again, we already have the file-mask in a variable, so can just use this. You may be aware that a load can be done directly from a filename with an asterisk in it, to load all matching files. Personally, I would always use a for each loop, as it makes debugging and testing easier. You can also exclude files (such as files with a tilde when loading Excel files) and work with the CROSSTABLE statement, which fails when used with an asterisk in the file name.

// Concatenate onto the temporary table the raw data from all of the archive files
// Strongly suggest building an archive of QVD files for historical logs and
// concatenate these at the end of the process - this is not done in this example though
for each vFile in FileList('$(vArchiveFiles)')
Temp_Sessions:
LOAD
ActiveUserId,
[App Title],
[Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser,
ActiveUserDirectory,
Timestamp,
[Exe Version],
[Server Started],
AppId,
[Doc Timestamp],
[Authenticated User],
[Secure Protocol],
[Session Duration],
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
FROM [$(vFile)]
(txt, utf8, embedded labels, delimiter is '\t', msq)
// Ignoring reload sessions. Remove next line if you want all sessions
WHERE ActiveUserId <> 'sa_scheduler' and ActiveUserId <> 'sa_repository'
;
next

You should now have a single table with all of the log entries that are needed for analysis. There are a number of transformations and general tidying up that is required. This is done by doing a RESIDENT load from our raw-data temporary table.

// Format all raw values, calculate new values and rename fields
Sessions:
LOAD
1 as SessionCount,
*,
Date(DayStart([Session Start]), 'DD MMM YYYY') as Date,
Date(MonthStart([Session Start]), 'MMM YYYY') as Month,
Year([Session Start]) as Year,
Month([Session Start]) as [Month Name],
Hour([Session Start]) as Hour,
WeekDay([Session Start]) as Day,
Time(Round(
[Session Start] - DayStart([Session Start]),
(1/288)), 'hh:mm') as Time, // nearest 5min
Floor([Session Start] - [Server Started]) as [Days Since Server Start],
Floor(([Session End] - [Document Reloaded]) * 24) as [Hours Since Document Reload],
Round([Bytes Received] * 1024, 1) as [KB Received],
Round([Bytes Sent] * 1024, 1) as [KB Sent]
;
LOAD
ActiveUserId as User,
[App Title] as Application,
$(vDT([Session Start])) as [Session Start],
[Exit Reason],
Level,
Hostname,
ServiceUser as [Service User],
ActiveUserDirectory as [User Directory],
$(vDT(Timestamp)) as [Session End],
[Exe Version] as [Server Version],
$(vDT([Server Started])) as [Server Started],
AppId,
$(vDT([Doc Timestamp])) as [Document Reloaded],
[Authenticated User],
[Secure Protocol],
rangemin([Session Duration], 0.5) as [Session Duration], // clip as error can cause incorrect duration
[CPU Spent (s)],
[Bytes Received],
[Bytes Sent],
Calls,
Selections
RESIDENT Temp_Sessions
;

There’s quite a bit going on in the script above, so to take it one bit at a time… Field names are being renamed to make them more ‘friendly’ for end-users. The date variable created above is being applied to all date fields. I found the session duration occasionally has duff values (if a session looks to be resumed), so those are capped at 12 hours. A number of new fields are added in the preceding load, including a counter (something I always to to make things more efficient), all of the date parts and some derived values for further calculations.

As should always be done at the end of load scripts, there is then some tidying up of tables and variables to do.

// Drop the temporary table
DROP TABLE Temp_Sessions;

// Clean up all the variables we have used
let vServerName = ;
let vFullServerName = ;
let vCurLog = ;
let vLogFolder = ;
let vCurFile = ;
let vArchiveFiles = ;
let vCurFile = ;
let vFile = ;

And there you have it. Your Sense Enterprise log file, loaded, cleaned and ready to analyse.

So, How Do I Analyse My Sense Logs?

Obviously, you can build your own charts and tables over these data, but wouldn’t it be simpler if there was a pre-canned dashboard you could use? Well, we have your back on that one. Our Instant Sense Application can load any data-set and some metadata about those data and then presents it in a user-configurable dashboard. There is a short video of me presenting the application on YouTube, here: https://www.youtube.com/watch?v=jN3Vj93v9x4.

The application is driven by a spreadsheet which defines the fields to be used for dimensions (in filters, charts and tables) and also the measures to calculate over those dimensions.

If you already have the latest version of the Instant Sense Application you can simply download the zip file with the load script and metadata spreadsheet from here:

https://www.quickintelligence.co.uk/SenseSessions.zip

If you have yet to download the app you can do so for free from here:

https://www.quickintelligence.co.uk/isa

Once you have downloaded the app you will need to upload it to your Sense server (via the QMC). At this point you can rename it to Sense Session Logs or similar. Open the app in your Work area and go to the data load editor. Now find the SenseSesssions folder under DataSets in the zip file. Open the load script text file and copy and paste the script over the script in the Main Data section of the load script. The SessionLogLookups.xlsx file needs to be copied to a folder you have a connection to in Sense, and then the past to that needs to be set in the variable on the Main sheet of the load script.

Once this has been done it should just be a case of reloading and you have your Sense Session Log Analyser.

There are a number of other example scripts included with the Instant Sense Application which you might like to explore. If you are interested in how the application was built, that is all explained in one of our previous blog posts.

If you have any comments or suggestions related to this post please post in the comments below.

The post Analysing Sense Session Logs appeared first on .

Why Load Data From RSS Feeds?

The answer to that question is entirely up to you.

For me though it was purely to provide an example data set. Recently I made available my Instant Sense Application, which can load any data set and some simple metadata about those data and provide a user configurable application. Along with the application I wanted to provide some example data feeds, and loading details of 2,000+ blog posts from AskQV.com seemed like a nice example. I then set about seeing whether this would work.

If you want to know more about what the Instant Sense Application is and, how it has been built, you will want to see my previous blog post, which is on that very topic.

Setting Up A REST Connection

In Sense we can’t just type a URL straight into our load script and pull data from it. We need to create a connection which is then available as a library. Out of the box it is required that we have a different connection for each REST endpoint, and paging is a problem. That is why the WITH CONNECTION statement was introduced. This allows us to inject changes to our REST connection at run time. It’s worth creating a new connection when connecting to a new source, but you only want to have two REST connections on your production Sense server which you can then re-purpose. The reason it’s two connections is that you can not change the GET and PUT settings using WITH CONNECTION.

I always create a connection called GenericGET and point it to https://jsonplaceholder.typicode.com/posts. The type is set to GET and everything else is left as defaults.

The first part of our code is to connect to that libary:

// Connect to our generic REST connection
LIB CONNECT TO 'GenericGET';

Configuring Blog Sources

Now we have a REST connector we need to tell Sense where we want to get data from. Often I use a temporary INLINE table for this kind of purpose. In this instance though I am using the table as part of the end data model, so there are a couple more fields as part of this INLINE load. Obviously you can change the sources, and/or use a different data source such as a spreadsheet:

// Provide a list of blogs that you wish to load information from
Blogs:
LOAD
1 as BlogCount,
[Blog URL],
[Blog Name],
[Blog Title]
INLINE [
Blog URL,Blog Name,Blog Title
https://www.quickintelligence.co.uk/,Quick Intelligence,The Quick Intelligence Blog
https://qlikshow.com/,QlikShow,Patrick Tehubijuluw
https://dataonthe.rocks/,Data On The Rocks,Data On The Rocks
https://qlikfreak.wordpress.com/,Qlik Freak,Useful Qlik Resources
];

You can add any number of sources, but it is imperative that they all share the same RSS structure. As mentioned above, all these are WordPress blogs so the RSS is common.

The next thing we need to do is loop around each of these. This is a pretty standard technique for enumerating around data once you have it in a table. To get the values we need from the table we use the PEEK statement.

// Loop around each of these feeds
for iBlog = 0 to NoOfRows('Blogs') - 1

// Grab the URL of the next blog
let sWPSite = peek('Blog URL', iBlog, 'Blogs');

Getting on the Right Page

It seems that WordPress RSS feeds always serve up ten pages at a time. Some may vary, but that is what I have noted. This is then used for the control of the paging of our load. We will keep loading from the RSS feed, from newest to oldest posts, until we get fewer than ten pages returned. That will tell us we are on the last page. If the number of posts is divisible by ten that logic will not work, and though it pains me to do so I have put an ERRORMODE statement in here (rather than pulling the number of posts from the RSS header and using that).

// Set some paging variables
let iDone = 0;
let iPage = 1;

// And loop for each page - the 500 is a back-stop increase this if you need
do while iDone = 0 and iPage <= 500;

// Set the paging URL - asking for paged=1 fails
let sPage = if(iPage = 1, '', '?paged=$(iPage)');

// Cheeky error mode setting - if number of posts is divisible by 10 (the default page size) a crash can occur
set errormode = 0;

Note also here that we are setting a variable which contains nothing for page one, but has the paged=n value for all subsequent pages. The WordPress RSS does a redirect if you request page one, and that freaks out the REST connector.

Some RSS feeds may not require paging at all. For instance, the AskQV.com aggregated RSS feed (and many others I have looked at) just serve up all posts in one long feed.

Fetch the RSS

This is the part which is going to vary the most, depending on the feed that you are pulling. As already mentioned this code is good for quite a lot of blogs, but not all of them. Create a new REST connection with the specific endpoint in to have Sense generate some code for you. Make sure you only tick the fields that you are interested to try and slim that down a bit. The code that comes back from the REST connector when you use the wizard is always ugly. Make sure you tidy it up a bit after the load. In this case we are only pulling one level of data, so the keys are all irrelevant.

// Pull the raw data from the feed
RawData:
SQL SELECT
"__KEY_rss",
(SELECT
"__KEY_channel",
"__FK_channel",
(SELECT
"title" AS "title_u0",
"link" AS "link_u0",
"creator",
"pubDate",
"description",
// "encoded",
// "commentRss",
"__KEY_item",
"__FK_item"
FROM "item" PK "__KEY_item" FK "__FK_item")
FROM "channel" PK "__KEY_channel" FK "__FK_channel")
FROM XML "rss" PK "__KEY_rss"
WITH CONNECTION
(
URL "$(sWPSite)feed/$(sPage)"
);

set errormode = 1;

Note that the ERRORMODE has gone back to reporting errors was at the end of each RSS load. Other failures will notify the user and cause the load to stop.

Formatting the Data

Now we just have to tidy up the data that we have received. As it is coming to us in XML format everything is in plain text. This means that the date and time field is only a string, which needs to be converted to a dual value. We also want different date break downs. In order to have a little more to analyse some other values are derived from the data. All relatively straight forward:

// Tidy up the load of fields for the current page
BlogPosts:
LOAD
*,
Date(DayStart([Date Time]), 'DD MMM YYYY') as Date,
Date(MonthStart([Date Time]), 'MMM YYYY') as Month,
Year([Date Time]) as Year,
Month([Date Time]) as [Month Name],
WeekDay([Date Time]) as Day,
Hour([Date Time]) as Hour,
Time(
Round(([Date Time]-Floor([Date Time])),(1/24/4)),
'hh:mm') as Time, // to nearest 15 minutes
today() - DayStart([Date Time]) as [Days Ago],
len([Post Title]) as [Title Length]
;
LOAD
1 as PostCount,
'$(sWPSite)' as [Blog URL],
[title_u0] as [Post Title],
[link_u0] as [Post URL],
[creator] as [Post Author],
Date(Date#(mid(pubDate, 6, 17),
'DD MMM YYYY hh:mm'), 'DD MMM YYYY hh:mm') as [Date Time],
[description] as Description //,
// [encoded] as [Full Text],
// [commentRss] as [Comment Feed]
RESIDENT RawData
WHERE NOT IsNull([__FK_item]);

I’m sure that there are further things which can be derived, but this gave me enough for my purposes. Notice that I have commented out the load of the full text of the blog post. This would give plenty of detail for searching, but is not helpful for analysis. The commentRss value gives an endpoint for all of the comments on the given post. You can craft code to go off and get those as well, if you feel so inclined.

Finishing Up

There are a few bits of tidying up to do at the end of our script. Finishing off the loop for each of the pages and also the loop for each of the blogs. Finally, because we are good tidy coders, there is clearing out the variables we have created along the way:

// Check if we need another page and clean up
let iDone = if(alt(NoOfRows('RawData'), 0) < 10, 1, iDone);
let iPage = iPage + 1;
DROP TABLE RawData;
loop

// End loop of blogs
next

// Tidy up behind ourselves
let iBlog = ;
let sPage = ;
let iDone = ;
let iPage = ;

So, there you have it, all you need to load details of all the posts from your favourite blogs into Sense.

Visualising the Results

Once you have the data in Sense you can obviously start building your own charts and tables over those data. Another approach though is to leverage our Instant Sense Application. Included in this free download is the load script above and a metadata definition file which populates the application with all the measures and dimensions to allow you to visualise these data.

If you would like to try the application, please go to https://www.quickintelligence.co.uk/isa and fill in the form. There is also a video there which demonstrates its use.

Also included with the download are load scripts which demonstrate the use of open APIs (Kids in Data and Magic The Gathering), Qlik Cloud Connectors (Twitter) and another RSS load (AskQV). Further data sources will be added over time – if you have a suggestion for one you would like to see please use the comments below.

The post Reading RSS Feeds with Qlik REST Connector appeared first on .

Why Create an Instant Sense Application?

In my time as a Qlik developer, I have seen many attempts at a one-size-fits-all application, where you simply load different data into the one application shell and off you go. To my mind, this is never going to deliver an application that works for the people who use it. A great application can only be created when the user is carefully consulted and involved in the design.

Something that we perform frequently at Quick Intelligence is “Seeing Is Believing” type Proof of Concepts. This is where we take a set of data from a prospect, then show them an application back within a few hours. Sense makes this achievable because of how quick charts can be built. However, there is a lot of repetition and copy and paste to create an app which ultimately will (or at least, should) be superseded. This was the main driver behind building the Instant Sense Application.

What Does The Application Do?

The idea is that the template app has twelve sheets, each with different ways of viewing the data. These follow the DAR principle (Dashboard, Analysis, Reporting) with KPIs up front, breakdown of those numbers in charts, and then finally tables of data. An overview of the various sheets is shown here.

Any data can be loaded into this template by taking a copy of the app and modifying the load script. One issue, usually, with doing this is that the objects in the app will not relate to the data that has just been loaded. This app, however, also loads in metadata from a spreadsheet, which defines which fields in the data model are used as different types of dimensions (main, trend, pivot, and geographic) and then a list of expressions that can be done of the loaded data is also included in the spreadsheet, along with how each expression should be displayed.

When the application is reloaded, the filters and charts pick up the meta-data and all of the chart objects work. More than that though, the user can then select which dimensions and measures they want to see and flex the application even after reload.

I’ve produced a short video showing both how the application is used and how the metadata spreadsheet drives how the application looks and behaves.

So, How Does it Work?

If you are a regular reader of this blog, you will know that I try to share tutorials on how to build solutions. Whilst you can just download the Instant Sense App and go, I wanted to give some detail on how it has been constructed. The truth is, though, that most of the techniques used in putting together this app have already been shared on this blog. This section includes links to where you can find that information and gives a bit of context around how those techniques have been applied.

One of the key features of the application is the ability to select Dimensions and Measures from a filter pane. When Sense was released and no Cycle Group was available I put together this blog post on creating an alternative cycle group using a filter pane:

How to Build a Cycle Group in Qlik Sense

The technique involves loading a list of dimension names and then using the min function to pick the first dimension in the selection and then return that. This can then be used as a calculated dimension in a chart, table, or even filter. The min function also has the ability to pass a parameter, so the nth minimum dimension can be found. This is what is used to give charts over six different dimensions.

The expressions are a little more tricky as there are three parts to the expression: the name, the Qlik expression, and the format specifier. It is just a case of finding which the minimum expression is (exactly the same as the dimension), then each of the parts is looked up separately and then concatenated together.

You will note from the video that there is a lot of repetition in this app, the eight KPIs on the front sheet, six charts with different dimensions on the second. These could have been done with a bit of copy and paste and editing. Changes, however, would then be a nightmare. Instead, variables are used, but rather than having a separate variable for each chart or KPI, there is a variable for each type which takes a parameter, 1 through 8 or 1 through 6, depending.

This technique of passing parameters into variables is a great timesaver. This blog post describes how you can use it in your own applications:

Variables With Parameters In Qlik Load Scripts

Combinations of these two techniques drive the charts and filter panes across pretty much all of the sheets in the app.

Another feature worth mentioning, though, is the showing and hiding of fields in the straight table object. Both the dimensions and the tables are created as described above, with variables that take parameters to return the dimension or variable. There are six dimensions and six expressions by default. The user can choose any number of each, up to six. Columns are then hidden as required, using a show condition on the column. Where this is particularly useful is on the dimensions, so with six columns you will have rows for each unique combination of values. If you pick one dimension, the other columns are hidden and the table will aggregate so that only one row per distinct value in the selected dimension is shown. This is incredibly useful for custom tables.

The final sheet is the Quick Intelligence Data Profiler. This is a set of objects that can be copied and pasted into any app that allows you to pick any field in the data model (from the $Field system field), which then shows a list box and bar chart showing frequency of values within the field. This is something I have put into pretty much all of my Sense apps since the product was released, and in QlikView before that. Most Qlik consultants will have their own go-to version of this kind of sheet. A tutorial on how to build one like mine is here:

Qlik Sense Data Profiler

So, now you know a bit about how it works, why not download the app and have a go yourself?

Download the App

The application can be downloaded from https://www.quickintelligence.co.uk/isa/. We do ask for an email address before you download the app, so we can notify you of upgrades. Please don’t pass the QVF onto others, but send them to the link given here. There is no charge for the app, but please credit Quick Intelligence if you use it largely unchanged.

The data in the app is from the Magic The Gathering API, just to provide some non-contentious data so you can see it work out of the box. If you are interested in loading data from open REST APIs, then the code is all in there.

If you point this app at an open data-source that is available to all, and you are happy to share your work, please get in touch. I would like to ship meta-data files and load scripts for a number of different open data sources to show the application in action.

Please feel free to get in touch, via the comments, if you have any questions.

Enjoy the app, I hope you find it really useful.

Update!

We now have this application hosted so you can interact with a couple of demos. One looks at AskQV Blog Posts and the second Tennis Grand Slam Final Data.

The scripts for both of these apps in included in the download, should you wish to see how the data is loaded or host the script on your own Sense instance.

The post Instant Qlik Sense Application appeared first on .