Tuesday, July 18, 2017

Azure Cognitive Services - Translator Text API - Calling from Postman

Azure cognitive service suit offers a lot of different APIs. One if it is language translation. It is named as Translator Text API. The good point is that it is offered as freemium meaning there is a free tier.

Calling the Translator API

Since it is service it is offered as web service. There are so many samples to use this service from different languages such as C#, JavaScript, PHP etc... which are available in their GitHub repo. But there is tutorial how can we simply call it from Postman which is web developers best friend to consume APIs.

Steps

Unlike many other APIs, this involves 2 step process. Steps given below
  1. Get the token from Azure using STS url which is currently https://api.cognitive.microsoft.com/sts/v1.0/issueToken. This URL is present in Azure portal when we navigate to Translate service instance. This may change in future.
    1. Calling this URL has to be POST with "Ocp-Apim-Subscription-Key" in the header. The value for this header is the keys present in Azure portal.
    2. The result of this call will give us a short lived token which has to be used in next step
    3. Once the token expires, we have to get renew. 10 mins is the expiry.
  2. Now using the token we can call the real, web service located at below URL
    1. https://api.microsofttranslator.com/v2/Http.svc
    2. This method accepts some parameters a query string and some parameters via POST / request data.

Example

Below goes the screenshots. 

Getting Translator Text API Keys from Azure portal. Image credits to MSDN Blogs. This key is used in next step to get token.

The above screenshot shows how to call token service from Postman to get the short lived token used for actual API call. The Ocp-Apim-Subscription-Key should be kept secret as it will help others to use our quota. It has to be changed explicitly if it is leaked.

The above image shows the real Translator Text API call. API needs the token obtained in the previous step. The token has to be prefixed with Bearer and a space before using in this call via Authorization header.
This API accepts required parameter in a hybrid way. Some are in the query string and some can be in optional POST request.
Make sure the optional post request body is properly formatted. One such format is given below.

If the POST request body has any format mismatch the request will be rejected by the server.

Happy posting...

References & more reading

https://blogs.msdn.microsoft.com/data_otaku/2016/11/05/goofing-around-with-the-cognitive-services-translator-api/
https://dev.cognitive.microsoft.com/docs/services/57346a70b4769d2694911369/operations/57346edcb5816c23e4bf7421
https://msdn.microsoft.com/en-us/library/ff512423.aspx

Tuesday, July 4, 2017

Adding Azure VM auto shutdown via ARM template

This post is intended to show how can we use a simple Azure feature via ARM template. The feature is 'scheduled shutdown of virtual machines'. It is useful when we play with machines. Seriously, never shutdown production machines unless there is a real business scenario such as your business is strictly running during a particular time and you don't want to keep the system open for hackers to play with. eg: day trading.

Before we go ahead, this article doesn't promote the use of virtual machines. Virtual machines which is in cloud terms 'Infrastructure as a Service' is the last to consider when we design for cloud. One, it is costly. It is not truly pay as use mode. Virtual machines have to run regardless there is workload or not. We have to pay for all these idle times. Also virtual machines take more time to scale out as the boot time is more. By the time, the spike in requests/workload might be over. But there would be scenarios where we have to use VMs. When we deal with older systems without enough budget to rewrite, we have to go for VMs. Also when our corporate IT is not believing in PaaS because of security, VM is the option.

Now coming to the point. Auto shutdown can easily be done from portal as shown in the announcement. There are some sites out there which explains how to setup auto shut down via ARM. This is my experience with the same.

The schema of ARM template can be found in the below link. It is the starting point.
https://docs.microsoft.com/en-us/azure/templates/microsoft.devtestlab/schedules

Sample ARM template to shutdown

Below goes JSON to set auto shutdown to a VM
{
      "apiVersion": "2017-04-26-preview",
      "type": "Microsoft.DevTestLab/schedules",
      "name": "shutdown-computevm-nanotest",
      "location": "[resourceGroup().location]",
      "properties": {
        "status": "Disabled",
        "timeZoneId": "Pacific Standard Time",
        "taskType": "ComputeVmShutdownTask",
        "notificationSettings": {
          "status": "Disabled",
          "timeInMinutes": 15,
          "webhookUrl": "http://joymononline.in"
        },
        "targetResourceId": "[resourceId('Microsoft.Compute/virtualMachines', parameters('nanoName'))]",
        "dailyRecurrence": {
          "time": "1600"
        }
        //"hourlyRecurrence": {
        //  "minute": "1"
        //}
      }
    }
We can see webHookUrl is given to a dummy url and hourlyRecurrence is commented. Lets see the details.

Where to add this JSON fragment in ARM template

This is not a sub resource of VM mode. Rather this needs to be directly added into top level resources node in ARM template.

Can we shutdown VM hourly

If we are using an Azure subscription which credits 50$ per month, it is not good to set auto shutdown daily one. As soon as we are done with the VM, it should be shutdown. Even if we forget, it has to shutdown. But currently the feature doesn't support hourly recurrence though the schema supports. That is why it is commented.

Can we omit daily recurrence value?

Below is the error which will pop when we don't give value for daily recurrence. Notice that this error won't be displayed during validation. But throws when we execute ARM template with ARM manager in Azure.

17:48:25 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:48:25 - [ERROR]   "error": {
17:48:25 - [ERROR]     "code": "MissingRequiredProperty",
17:48:25 - [ERROR]     "message": "Missing required property DailyRecurrence."
17:48:25 - [ERROR]   }
17:48:25 - [ERROR] }'

What is big deal with the name

Below is the error receive when a pet name given to schedule. Again notice that validation don't show but execution throws. In other words its runtime error.

17:55:49 - [ERROR] New-AzureRmResourceGroupDeployment : 5:55:49 PM - Resource
17:55:49 - [ERROR] Microsoft.DevTestLab/schedules 'autoShutDownPolicy' failed with message '{
17:55:49 - [ERROR]   "error": {
17:55:49 - [ERROR]     "code": "InvalidScheduleId",
17:55:49 - [ERROR]     "message": "The schedule should be created in subscription
17:55:49 - [ERROR] 11111111-1111-1111-8888-####ffffeeee, resource group <Storage>
17:55:49 - [ERROR] and with name shutdown-computevm-nanotest."
17:55:49 - [ERROR]   }

The nanotest is the VM name.

APIVersion and region

The sample template above shows apiVersion of schedules '2017-04-26-preview'. It is region specific. Not all regions supporting all API versions. This is again an runtime error. Validation will pass.

There is a URL which tells what are available in what region. There are some programmatic techniques to find out the API version and other details.

The good part is that the error is descriptive, telling us what API version to use. That is really useful.

Rolling back ARM deployment

As mentioned above some errors happen when we deploy/execute ARM template. If the auto shutdown setup fails, does it revert the VM deployment? As software engineers we assume that it will,  same as MSI deployment or there will be some flags to deploy ARM in transaction. But it never roll backs. It is our pain to clean up what is left after a partially failed ARM deployment. Feature request below. It is there for the last one year.

https://feedback.azure.com/forums/281804-azure-resource-manager/suggestions/13701420-rollback-template-deployment-if-anything-fail-ju

The suggestion is to make our ARM template deployment idempotent. Simply saying even if we redeploy, it should not cause any harm. It should run many times by producing same result as first run. Simple if we are just setting up architecture via ARM. Gets complex if there are custom scripts running as part of deployment.

Debug ARM template

For a simple feature to get working, we may encounter these many errors. As developers we might have started thinking how to debug the ARM templates. In general the error messages are meaningful, if we are familiar with Azure terms. Debugging ARM template itself has scope for separate post. Again, it gets complicated when ARM template uses custom scripts. Below are 2 links talking about troubleshooting ARM template deployments.




Tuesday, June 27, 2017

Serverless JoymonOnline.in - Monitoring up and downtime of web sites

Side effects of Serverless

One of the side effect of Serverless is that is has so many moving parts and we need to monitor the health of all the services consumed by our application. In the case of JoymonOnline.in, there is no money involved but may be slight reputation loss when people sees the site down. How can this guy solve our problems in distributed systems if he can't make his site up all the time?

But that is not the case with real time web apps which brings revenue to the organization. Such scenarios demand the need for monitoring the health of third party services. Ideally we should design the system to switch to another alternative if possible. But at minimum we should know what is going wrong in the system.

JoymonOnline.in uses GitHub pages to host static content as it is HTML app developed using AngularJS framework. It is good to monitor the health of site as is.

Choosing monitoring service - Freemium

Freemium is a relatively new term in SaaS. When vendors sell Software as a Service, they often include a free tier to do all the basic things. For higher usage they start billing. There are many freemium services for web site/service monitoring too. One of such service is UptimeRobot. It checks the website in 5 min interval and informs us about downtime. It is very easy to setup. Only downside is it accepts only custom credentials. Not integrated with Google or Facebook logins.


There are many other services like this. Of course this is best otherwise I would not have used it :) . A list is given below.

https://github.com/lexoyo/Monitoshi - Includes the source code and a free hosted instance.
https://visualping.io/pricing - Only 2 checks /day
http://mashable.com/2010/04/09/free-uptime-monitoring/#FBROURLfQkqR
http://www.hongkiat.com/blog/monitor-website-up-downtime-30-free-web-services-and-tools/

Tuesday, June 20, 2017

C program to encode and decode string

Recently I get introduced to too many technology WhatsApp groups which talks about software development, basic electronics, robotics, AI and many more. It really helps me to know what is going in other areas instead of browsing for news. Also it is happy to be in places where people help each other.

Below is one question someone threw to a groups and I could do something after a week.

Write C program for self defined codec

1.Get a string of chars from user.
2. Convert string to array of chars (each char has value 0~255)
3.Define a lookup table (LUT) and an algorithm to encode the char values.
4.Use the LUT and algoithm from step 3 to encode the array of chars.
5.Output the encoded array of chars.
6.Decode the encoded array of chars.
7.Output the decoded array of integers to prove that your algorithm can decode.

I seriously programmed in C when I was in college and during my job seeking. After I landed in .Net world, there were less than 5 scenarios where I had to look or code in C/C++. Yes the big trouble was with pointers. Also comparing C#, I felt driving 80's model Ambassador car when programmed in C. Enough talks. Below is the code

#include < stdio.h >
char LUT[26] = "QRSTUVWXYZABCDEFGHIJKLMNOP";

int getLocationFromLUT(char c) {
  int i = 0;
  while (i < 26) {
    if (LUT[i] == c) return i;
    else i++;
  }
  return -1;
}

char * encode(char * str) {
  int i = 0;
  while (str[i] != '\0') {
    int locationInAlphabet = str[i] - 65;
    str[i++] = LUT[locationInAlphabet];
  }
  return (str);
}
char * decode(char * str) {
  int i = 0;
  while (str[i] != '\0') {
    int location = getLocationFromLUT(str[i]);
    str[i++] = location + 65;
  }
  return (str);
}
int main() {
  printf("Enter string to encode (caps & max len 64)\n");
  char word[64];
  scanf("%s", word);

  char * encoded = encode( & word[0]);
  printf("Encoded string %s\n", encoded);

  char * decoded = decode(encoded);
  printf("Decoded string %s\n", decoded);
  return 0;
}

So simple isn't it. Took me around 2 hrs to get the code completed.

It can be run in browser using tutorialspoint online environment. Technically it not in browser, browser just acts as console. I used this environment to write this code.
http://www.tutorialspoint.com/compile_c_online.php?PID=0Bw_CjBb95KQMU0tIWi1qcEJUUUk

Tuesday, June 13, 2017

Interception in Microsoft Unity framework for .Net

Introduction

This article is to demonstrate how we can combine dependency injection framework with logging og low level execution flow of application.

Prerequisites


  • Knowledge about dependency injection.
  • Knowledge about unity from Microsoft for dependency injection

Background

If we develop any application, logging is an essential thing to troubleshoot issues which comes in  later stages. Ideally there should be no bugs in production, if we test well. But it is not practical 99.99% of times. Sometimes even if there are no defects we may need to understand how the system is getting used. We call it as instrumentation, telemetry etc...Whatever we call, we have to make sure our application is capable of telling us what it did. Writing log lines inside our business logic has more efforts/costly, makes it longer and very difficult to read. 

Instrumentation technology considerations

Below are the factors to be considered when we select any logging and instrumentation technology for software.
  • Performance - Due to the logging operations our application should not be performing slow. Logging is for application not the other way.
  • Controlling behavior - We should be able to turn the logging on and off without restarting the application. Also change the level of logging without restarting.
  • Compatibility - If our system spans across different technologies better choose one which works everywhere.

Options

Traditional options include inserting log lines at the beginning and end of every methods and additional lines if the logic inside complicated. Another option is to inject the logging code at runtime. Both has their own pros and cons.

Few words on ideal composed application

If we look at SOLID principles we could see that all it tells us is to develop apps in a modular way where modules can be composed easily. In other words single responsible, closed components connected together to become an application. When we say modules, components in general they all translate to classes or functions in code.

How we compose the application by connecting the components is via dependency injection framework.

Combining DI framework and logging

Now most of the people might have got the click of idea. If the application follows SOLID principles religiously, why can't our DI framework log the method calls as it knows how application is composed. When we say how the application is composed it is same as the application flow which we need to log.

This is supported in Microsoft unity framework which provides DI capabilities for .Net. There are other DI frameworks available, one other called MEF which is native to .Net itself. This post is not to discuss what is the best DI framework but on the possibilities of combining DI with function level logging.

Steps in general

  • Have a interceptor mechanism which can tell us what is happening in the DI framework
  • Let the interceptor log the details into appropriate medium using best technology.
  • Register the interceptor into the DI framework.

Understanding sample

Coming directly to the sample which is available in the below location.
https://github.com/joymon/dotnet-demos/tree/master/patterns-practices/DI/unity/Interception

class LogCallHandler : ICallHandler

This is the interceptor we have written. It has Invoke() implemented which came via ICallHandler. Invoke() will be called by the unity DI framework when things happen there.

This is one mechanism unity supports. There are other ways to hook interceptors.

UnityConfiguration.xml

This is the configuration file which tells the unity framework about the components and how they composed. In simple words when someone request object of x interface give object of y class which implemented x. This file also has options to register interceptor.

This configuration can be specified via code too.

class Logger

This is a simple logger class which logs via .Net trace. .Net tracing is configured in the sample to write to file.Refer app.config file. We have to consider intercepting the method calls and writing the logs separate. If our log writing takes more time the performance of application may get affected. Choosing log writing mechanism is not in the scope of this article. In short, recommended mechanisms are ETW for on premise and Application insights for Azure environments.

References

https://msdn.microsoft.com/en-us/library/dn178466(v=pandp.30).aspx
http://mohtasebi.com/patterns/2013/10/24/using-unity-for-aspect-oriented-programming-and-interception-part-1.html

Tuesday, June 6, 2017

Puzzle - Sales tax

Below is famous problem which could be asked in coding interviews. I got this problem when I attended ThoughtWorks long back and could pass to the next round of pair programming.  The solution what I sent to them is uploaded to Github and URL is given at the end of this post. The mail came up when I was cleaning my mail box.

PROBLEM SALES TAXES 

Basic sales tax is applicable at a rate of 10% on all goods, except books, food, and medical products that are exempt. Import duty is an additional sales tax applicable on all imported goods at a rate of 5%, with no exemptions.

When I purchase items I receive a receipt which lists the name of all the items and their price (including tax), finishing with the total cost of the items, and the total amounts of sales taxes paid. The rounding rules for sales tax are that for a tax rate of n%, a shelf price of p contains (np/100 rounded up to the nearest 0.05) amount of sales tax.

Write an application that prints out the receipt details for these shopping baskets...
INPUT:

Input 1:
1 book at 12.49
1 music CD at 14.99
1 chocolate bar at 0.85

Input 2:
1 imported box of chocolates at 10.00
1 imported bottle of perfume at 47.50

Input 3:
1 imported bottle of perfume at 27.99
1 bottle of perfume at 18.99

1 packet of headache pills at 9.75
1 box of imported chocolates at 11.25

OUTPUT

Output 1:
1 book : 12.49
1 music CD: 16.49
1 chocolate bar: 0.85
Sales Taxes: 1.50
Total: 29.83

Output 2:
1 imported box of chocolates: 10.50
1 imported bottle of perfume: 54.65
Sales Taxes: 7.65
Total: 65.15

Output 3:
1 imported bottle of perfume: 32.19
1 bottle of perfume: 20.89
1 packet of headache pills: 9.75
1 imported box of chocolates: 11.85
Sales Taxes: 6.70
Total: 74.68

Solution

The major pattern here applicable is decorator to structure the calculation rules. Full code is present in GitHub repo.

Tuesday, May 30, 2017

Toolbox for open source developers

It is the era of SaaS and its byproduct freemium licensing model. Simply saying, so many tools and services are there for free to help software engineering. There are a lot of lists published which aggregates these free tools and services for software development. This is my version of the same which index all the software tools and services which I know, use and recommend.

There is a thought process going on about moving this to GitHub where it is easy to manage. No plans to merge with other public lists as one of the aim is to index what I used, not all those are out in the wild internet. Let google do it.

Design & Development

Project management

CheatSheets

https://dzone.com/refcardz
http://www.cheat-sheets.org/
https://www.cheatography.com/

Wireframe & diagramming

Coding

Visual Studio Code
Eclipse
Notepad++

Explore public APIs

https://www.programmableweb.com/
https://any-api.com/
https://apis.guru/browse-apis/
http://www.apiforthat.com/

Source control

Github
VisualStudio.com

Continuous integration

AppVeyor
TravisCI

Runtime

Hosting - Static site

GitHub pages

Hosting - Server side execution

Webtask.io
flaw.microsoft.com

Hosted Log management (Yet to finalize best one)

https://www.sumologic.com/pricing/
http://www.muscula.com/prices/
https://papertrailapp.com/plans
https://sentry.io/pricing/
https://www.loggly.com/plans-and-pricing/
https://rollbar.com/pricing/
https://www.bugsnag.com/pricing/
https://github.com/cheeaun/javascript-error-logging

eMail

SMS

In progress...

Sites to search for software and compare

https://www.getapp.com/
https://www.slant.co/

Make money

Algorithmia.com
https://www.chargebee.com/ - For SaaS billing.(Yet to try)

Other indexes like this

https://news.ycombinator.com/item?id=8874227
https://github.com/ripienaar/free-for-dev

The title says tools for open source developers. That's just a catchy title. Most of these tools and services are available to closed source and enterprise.

Enjoy the tools...