Components of Big Data - Hadoop System

In this blog i will explain important components which are part of Hadoop System. I will give very brief overview of these components.

Below diagram shows very high level components in Hadoop system.

• Master Node (MN)
• Name Node (NN)
• It is a daemon process runs on Master Node.
• Takes care of reading the data file to be analyzed.
• Splits the data file based on block size configured, default is 64MB and 128 MB.
• Distributes the split data file across multiple Data Node.
• Maintains the index file to keep track of where the data has been distributed. Think this as "Table of Content" in a book.
• It provides input to Job Tracker for location of the data files in Data Node.
• This is one part of HDFS system in Hadoop.

• Job Tracker (JT)
• Job tracker is also a daemon process.
• This is part of Processing Engine of Hadoop system.
• It is responsible for running the program which will analyze the data and produce results.
• Job Tracker communicates with NN to identify the location of the the data file. Once the data node locations are identified this process will move the program to those Data Node for execution.
• JT will try its best to run the Data file analysis local to the Node, so as to process it faster.
• But incase if can't assign task local to Task Tracker then next it will look for node available in same Rack.
• Once Job Tracker receives output from the multiple Task Tracker, it has to run the program again to consolidate the output and generate the final output for the analysis.
• Job Tracker important role is to monitor all the task which are running in Task Tracker, and start new job if something fails.

• Slave Node (SN)

• Data Node (DN)
• This is a Daemon process runs on Slave Node.
• There can be many data nodes as Number of Slave Node can be more than 1.
• Task of this process is to receive the data sent by Name Node.
• DN is responsible for manitaining the data received from NN in the system. It keep track of these data file.
• Together NN and DN forms and manages HDFS.

• Task Tracker(TT)
• This is Daemon process is running on Data Node.
• Program sent from JT are received by this process and stored in the Slave Node.
• After receiving the file it will initiates the program to analyze the data file.
• Once the analysis is complete, it will produce the result and share it back with JT.
• Combined together JT and TT are called MapReduce.
• Task Tracker keeps sending heartbeat signal to Job Tracker so that Job Tracker understands that process is running fine. 
• Incase if TT fails to send heartbeat to JT, then JT will re-initiate that process again in other available TT.

• HDFS
• Hadoop Distributed File System
• Two important concept in HDFS
• Block Size
• Fault Tolerance/Failure 
• Name Node and Data Node creates and manages the HDFS.
• Name Node is the master which takes care of splitting the file and distributing it across multiple data node.
• HDFS is fail safe system, which ensure that data which is stored is never lost, rather i should say chances of loosing data is very minimal.
• HDFS ensure fault tolerance by keeping copy of same data file in multiple Data Node. NN maintains 3 copies of data file so that if any one data node crahses it can consume the backup file.

• MapReduce
• Map Reduce comes under processing engine of Hadoop system.
• It is programming model which enables running process in Parallel which can be distributed across multiple nodes.
• Job Tracker and Task Tracker makes up this Processing Engine of Hadoop system.
• There are basically two pieces of MapReduce. 
• First part is Map which analyzes the data file and produces an output, which goes as an input to Second part called Reducer. Map task job is to sort and filter the data.
• Reducer takes this processed out and creates a consolidated reports.

• Secondary Name Node
• This process runs in another system in hadoop cluster.
• It kicks in when main Name Node fails or goes down. 
• It keep interacting with NN at regular interval and creates a backup of the index file in separate system. 
• Backup system can be the one where secondary name node is running or it can be other system which is in some other location or other RACK.
• Its task is to recreate new Name Node by reading the FSImage and Edit Logs.

Load Testing of Rest Webservice Using SOAP UI

SOAP UI provides easy integration with different web services like SOAP and REST along with feature of Load Testing. Below I will show how to setup SOAP UI for doing load testing.

To start with you need to have SOAP UI installed in the system. In the example I will consume REST webservice.

Web Service end point is http://localhost:9081/UserService/users

Lets start with the load testing setup now.

Start SOAP UI tool and setup the REST webservice call and do a round of testing to ensure that tool is able to connect to the service and is able to get the response. Following screens shows the same.

Setup REST Project

Create New Project

Right click on the Project and click on create REST Project.

After providing the URL in above alert box, You will get below windows where you can see the various options like

• Method
• Endpoint
• Resource
• Input parameter

Once you provide all the necessary configuration required for your webservice do test and once you see the expected response in the output window, you are good to go for next step i.e. executing load testing.




Now to setup Load Testing, start by creating "Test Suite" and "Test Case". Click on the icon highlighted in Red in above screen shot. That will prompt you to enter Test Suite name and next it will prompt you to enter Test Case name. Provide these details respectively and proceed next.



Once the above steps are completed, you can validate these test case, by launching test suite by double clicking on the left navigation as shown below. For this example I clicked on "UserTestSuit", in the window click on Green Arrow button to launch test Suite, all process should be in green. You can add Assertions to validate the response, i will cover these configuration in future blogs.




Right click on the "Load Tests" menu in above screen and add New Load Test, you will get following screens.



In the Load Test Screen, SOAP UI provides various options to launch Load test.

• Limit - Time for which you want to run the test.
• Thread - No of threads 
• Strategy - Various options, like 
• Simple - For Simple load run, good for Base-lining application performance.
• Variance - In this Strategy, load keeps varying over the specified period of Load Test. If you have given 100 as thread count and Variance is 0.5, then thread count will vary between 50 to 150 over the period load testing limit.
• Burst - This strategy sends sudden burst of user loads, by hitting the service for specified time. You need to specify Burst Delay which is gap between each burst and Burst Duration which is time each burst will last.
• Thread - In this one you can specify initial number of thread and increase it over the period of time and testing will end at the value specified in End Threads drop down.

Load test can be started by clicking on green arrow. Once the testing finishes, tool will shows you high level details in below grid.
Important headers will be TPS, Min, Max, Avg time. Error counts.

Also tool provides two graph next to Green Arrow start button. There you can see different matrix like threads, avg time, error count, tps etc. Data can be exported from this graph, which will give you entire run data, this can be achieved by clicking on export icon on the right side of window.


I hope this overview will help you to start with PT in SOAP UI, In upcoming blogs I will show how to pass parameter between multiple steps present in load testing.

thanks.

Basic Authentication Setup Java Web Application

This blog shows how to setup Basic Authentication in Web application.

I am using Spring based Web Service to demonstrate the same. To start please ensure that you have spring application configured properly. On top of that i will show what changes needs to be be made to enable Basic Authentication.

Securing application is one of the important activity which developer and designer has to keep in mind while designing. Basic authentication can be one of the basic security mechanism which can be enabled to secure web application or web service.

Basic Authentication security is where application will expect the consumer to pass User and password in request header. In case if these values are not passed then spring framework will throw back Unauthorized error code.

Following are the steps which needs to be followed.

Step1:

Add the spring security jar files in the application. Following are the jar files.

• spring-security-core.jar
• spring-security-config.jar
• spring-security-web.jar

Step2:

Add following line in "Web.xml" file to enable spring security filter. This filter is responsible for adding security to the url pattern mentioned.
Also add an entry for security xml file which we will configure in next step, this file contain basic authentication security configuration.

Code is highlighted below.

<servlet>
 <servlet-name>basicauth</servlet-name>
 <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
 <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
 <servlet-name>basicauth</servlet-name>
 <url-pattern>/</url-pattern>
</servlet-mapping>
<listener>
 <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>
<context-param>
 <param-name>contextConfigLocation</param-name>
 <param-value>  
           /WEB-INF/basicauth-servlet.xml,  
           /WEB-INF/basicauth-security.xml
        </param-value>
</context-param>
<!-- Spring Security -->
<filter>
 <filter-name>springSecurityFilterChain</filter-name>
 <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
 <filter-name>springSecurityFilterChain</filter-name>
 <url-pattern>/*</url-pattern>
</filter-mapping>

Step3:

Create spring-security.xml file and add following code to enable the security for those url to which security needs to be enabled.

<?xml version="1.0" encoding="UTF-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
 xmlns:oauth="http://www.springframework.org/schema/security/oauth2"
 xmlns:context="http://www.springframework.org/schema/context"
 xmlns:sec="http://www.springframework.org/schema/security" 
 xmlns:mvc="http://www.springframework.org/schema/mvc"
 xsi:schemaLocation="http://www.springframework.org/schema/security/oauth2 
 http://www.springframework.org/schema/security/spring-security-oauth2-2.0.xsd
 http://www.springframework.org/schema/mvc 
 http://www.springframework.org/schema/mvc/spring-mvc-3.2.xsd
 http://www.springframework.org/schema/security 
 http://www.springframework.org/schema/security/spring-security-3.2.xsd 
 http://www.springframework.org/schema/beans
 http://www.springframework.org/schema/beans/spring-beans-4.1.xsd
 http://www.springframework.org/schema/context 
 http://www.springframework.org/schema/context/spring-context-4.1.xsd ">
<http auto-config="true"  use-expressions="true" xmlns="http://www.springframework.org/schema/security">
    <intercept-url pattern="/login" access="permitAll" />
    <intercept-url pattern="/**" access="hasRole('ROLE_USER')" />
    <http-basic />
</http>
<authentication-manager alias="authenticationManager" xmlns="http://www.springframework.org/schema/security">
  <authentication-provider >
    <user-service>
      <sec:user name="apiuser" password="password" authorities="ROLE_USER"/>
    </user-service>
  </authentication-provider>
</authentication-manager> 
</beans>

cccc

Here is the explanation of the tags:

<http> - Main tag which is responsible for creating proxy for all the url which this tags intercepts.
<intercept-url> - this tag is for specifying the url's which needs to be behind access control and which doesnot need to have access control. Using attribute pattern you can specify Url pattern and access attribute specifies what access check needs to be implemented.
<http-basic> - This tag informs proxy that it needs to alert user to enter user and password when url is accessed. This adds a BasicAuthenticationFilter and BasicAuthenticationEntryPoint to the configuration.

<authentication-manager> - here type of authentication is specified, in above example i have used configuration based authentication, which means that user details are hardcoded in the same xml files.

Step4:

Create Java file for testing the following implementation. Below is the web service component which is behind basic authentication security.

package com.infoblog;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;


@Controller
@RequestMapping(value="/secure")
public class SecureController {

 @RequestMapping(value="/test", method=RequestMethod.GET)
 public @ResponseBody String secureFunction(){
  
  return "Success";
 }
 
}

Step5:

Start the server and test the functionality. Browser will prompt you to enter user details as shown below.

GIT vs SVN

GIT vs SVN

In the DevOps world, selection of right SCM tool is a significant decision. Since these tools are kind of driving force behind the process, a wrong SCM selection may lead to inefficiencies throughout your delivery pipeline. I personally have used CVS and SVN in most part of my career. Those projects were not really Agile projects. While we had some kind of CI and CD, there were no end to end DevOps process in place. So we were able live with it.

Recently I started using GIT - both private GIT as well as public GITHUB and BIT BUCKET.  Also in the recent assignments, we don't just go with what is provided by customer. We demanded the right SCM. We study the customer needs and recommend centralized or distributed SCM. 

In this blog, I will  compare the capabilities of GIT and SVN at very basic level. This blog does not cover the detailed scenarios like when GIT performs better over SVN and vice versa or how GIT's way of working on the content instead of file helps in some large assignments.

S. No	GIT	SVN
1	Distributed Source Control - Developers can work offline on their local repository and keep committing so that they can do back to the earlier version if something goes wrong. Once they are online, they can push the changes to the master repository. So consistent code in Master repository. If any repository is lost due to system failure only the changes which were unique to that repository are lost. If users frequently push and fetch changes with each other this tends to be a small amount of loss, if any. GIT supports single repository too.	Centralized Repository - You cannot have offline repository in your laptop. So the developers has to be connected to SVN to commit and working offline is not easy. Loss of central repository needs recovery from backup and changes since that last backup are likely to be lost (Many man days of effort) Advantage of Single repository is that  there is little doubt about where something is stored.
2	Performance - Extremely fast  since all operations (except for push and fetch) are local there is no network latency involved	Performance - Slow Compared to GIT as all the file history and version info is present centrally
3	Storage/Space Requirements - Git repositories are much smaller in file size than Subversion repositories	Storage/Space Requirements - May need more space than GIT
3	Branching and merging support is a lot better. Git provides better auditing of branch and merge events	Branching and merging is good for small projects
4	Usability - Bit complex than SVN For newbies, need some additional learning as it is different from SVN. Also more concepts and more commands. The developers need to understand when to use checkout vs clone; commit vs push	Usability - Simple to use because Very limited concepts and commands Every revision is numbered sequentially and it is easy to walk through Better client GUI tools
5	Access Control - Due to being distributed, you inherently do not have to give commit access to other people in order for them to use the versioning features. Instead, you decide when to merge what from whom.	Access Control - Because the repository is centralized and every developers needs commit access, it is really difficult to have repository owner.

Mockito setup in Spring application

Mockito is used widely for application code Unit Testing. In this blog I will show the dependencies which are required to configure Mockito in Spring Web application.

In the existing Spring based application, following dependent jar files needs to be added.

• asm-1.0.2.jar
• hamcrest-all-1.3.jar
• json-path-1.2.0.jar
• json-smart-2.1.0.jar
• junit-4.11.jar
• mockito-core-1.9.5.jar
• objenesis-2.1.jar

after adding these jar file you are all set to start writing the Mockito based JUnit test cases.

Few of the useful plugin for Eclipse IDE that will help in analyzing code coverage can be installed. The plugin which I used for code coverage is "EclEmma".

This can be installed from Eclipse market place which gives report on percentage of code line which has been covered by Unit Testing. 

In subsequent blogs i will show some sample Mockito Unit Test cases classes.

Use Swagger UI for showing interactive API documentation

Following Swagger post I showed how to configure Swagger with Spring project. In this blog I will demonstrate using Swagger UI for showing interactive documentation.

Swagger UI is part of the Swagger project and it allows you to generate, visualize and consume API. It is package which basically consist of html, css and javascript, and parses the json output generated from application code scanning. This scanning and configuration was explained in following post.

Swagger UI Setup can be downloaded from "here".

Copy the files to a Dynamic Web project in eclipse and paste the content from "dist" folder to WebContent as shown below.




Here I am using tomcat server to host this UI application. By doing so it will be easy to share the API application with other user. It shows following UI. 

Please note that by default swagger will load the petstore Api details. I will cover customizing them in next point. 




Swagger UI presents very nice interactive tool, using which users can read the documentation, can see the exposed API, detailed input and output parameter, also they can test the API real time. This tool present by parsing the json output which was generated while scanning the API project. 

Above UI shows all the available API exposed in petstore application, method type supported and some basic details at top level. 

You can see entire API details after expanding the service  by clicking on the link. Below "/store/inventory" API has been expanded and you can see description of this API, what is response code returned when successful, at the last of the section you can see "Try it Out!" button, using which you can test the API. 

Response from the API will be shown in the same section below the button.

 

By default Swagger UI shows details pet store API, now if you want your custom build API to be listed, then few changes needs to be done in the above setup. Below I am going to explain the steps which needs to be done to customize the output.

• Ensure that application is completed with the swagger setup and json output is accessible when you access following url, "http://localhost:9081/SpringSwaggerDemo/v2/api-docs", this is the url for the sample api application  which i created, you need to change it as per your setup, but last part of URL remain same i.e. "/v2/api-docs".
• To load api documentation in the Swagger UI enter the above url in test field provided in top header section and press on "Explore" button.
• Application will parse json and it will display the content in below section. This temporary loads you custom API details.
• This url change will not be persisted in the UI code if you refresh the page it will be lost, so to persist the url change you need to modify the script code as shown below.
• Locate index.html file, open with editor do following changes. modify the url value to you API url as shown below. After this change launch the UI again, you will see by default application loads the API details.

  var url = window.location.search.match(/url=([^&]+)/);
  if (url && url.length > 1) {
     url = decodeURIComponent(url[1]);
  } else {
      //url = "http://petstore.swagger.io/v2/swagger.json"; // change this value to your API json url. 
      url="http://localhost:9081/SpringSwaggerDemo/v2/api-docs";
  }

• Check below UI where it shows the custom API details.

• Next I will explain some more customization's related to language and UI.

• Different Language support.
• Swagger support showing UI menu in different languages. To enable this change index.html file needs to be modified and you need to add following script files. These files comes with the package which you downloaded. Locate "lang" folder inside the project, you will find different language files which by default comes with package. If you need to enable French language, so following code needs to be added in the index.html file. 

  <script src="lang/translator.js" type="text/javascript"></script>
  <script src="lang/fr.js" type="text/javascript"></script>

• By default basic menu and other controls of Swagger UI will be converted to french language. Incase you have any other language for which file is not present then you need to create that file and place it inside this folder and include it in the html file.
• UI Customization
• Swagger UI can be customized and configured as per the organization and individual needs and requirements.
• Entire source code is provided, so any modification can be easily done. Generally following are the section in UI which needs to be modified like Header, Logo, Color theme, etc.
• To change the logo change the images inside "img" folder.
• To apply new css, modify the css or override by putting new css file which extends the style provided by swagger.
• To modify parsing logic and other logic modify js file inside lib folder.
• Below is one sample changes to UI I have done. You can see the difference between above UI and the modified UI.

I hope this will be helpful post for people looking to use swagger as API documentation tool using Swagger UI. This post will help you kick starting the process.

Share your feedback.

Reading log file path from environment Variable in logback

In this short blog i will be showing how to read the log folder location from environment variable.

In general log path for an application is one time configuration. So why to keep multiple property files for each environments. Better approach could be using environment variable on server, which can be configured once.

Here i am using Tomcat 7.x as a server. For Logback using following jars.

• logback-classic-1.1.3.jar
• logback-core-1.1.3.jar

First lets start by creating environment variable in tomcat server. For this you need to open context.xml file located at TOMCAT_HOME/conf folder. Open with text editor and add following line in the file.

  <Environment name="LOG_PATH" type="java.lang.String" value="/logs"/>

you are done with creation of environment variable which will be accessible via jndi lookup.

next will be open logback.xml file which contains application logging configurations.

Use JNDI lookup to retrieve the environment variable. Logback provides a tag "insertFromJNDI" which has attributes like "env-entry-name" and "as".

"env-entry-name" - in this attribute, jndi lookup path needs to be mentioned. e.g.

env-entry-name="java:comp/env/LOG_PATH"

"as" - in this attribute,variable will be created where the value returned from lookup will be stored.

<insertFromJNDI env-entry-name="java:comp/env/LOG_PATH" as="LOG_PATH" />

 Assign this variable to Context and then access it via "${CONTEXT_NAME}". shown below.

<contextName>${LOG_PATH}</contextName>

Complete configuration code is given below.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xml>
<configuration>
 <insertFromJNDI env-entry-name="java:comp/env/LOG_PATH" as="LOG_PATH" />
   <contextName>${LOG_PATH}</contextName>
 <appender name="SAMPLE-DEBUG"
  class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${CONTEXT_NAME}/debug.log</file>
  <filter class="ch.qos.logback.classic.filter.LevelFilter">
   <level>debug</level>
   <onMatch>ACCEPT</onMatch>
   <onMismatch>DENY</onMismatch>
  </filter>
  <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
   <Pattern>
    %d{yyyy-dd-MM HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
   </Pattern>
  </encoder>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
   <fileNamePattern>${CONTEXT_NAME}/archived/debug.%d{yyyy-dd-MM}.%i.log</fileNamePattern>
   <timeBasedFileNamingAndTriggeringPolicy
    class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
    <maxFileSize>2MB</maxFileSize>
   </timeBasedFileNamingAndTriggeringPolicy>
  </rollingPolicy>
 </appender>
 <root level="DEBUG">
  <appender-ref ref="SAMPLE-DEBUG"  />
 </root>
</configuration>

share your feedback. 

Swagger API doc for Spring Based Webservices

When any API is created one of the major challenge is to create documentation, also once documentation is created then updating it over the period of time for any changes which are done in API.

In this blog I will be demonstrating how to create API documentation for REST webservices using Swagger tool.

Swagger is very mature API doc creation tools. It provides interactive documentation with very good ecosystem like Swagger UI, Swagger Editor.
Swagger is intelligent enough to scan through the code automatically and it generates am output which are easily readable by human and even system.

In this blog i will be showing swagger integration with Spring based rest webservices(bottom up).

Prerequisite for this blog is that you have Spring API project created and working.

• To start first download following jar files and add them in your project.

1. aopalliance-1.0.jar
2. aspectjrt-1.8.6.jar
3. aspectjweaver-1.8.6.jar
4. classmate-1.2.0.jar
5. guava-18.0.jar
6. mapstruct-1.0.0.CR1.jar
7. spring-hateoas-0.18.0.RELEASE.jar
8. spring-plugin-core-1.2.0.RELEASE.jar
9. spring-plugin-metadata-1.2.0.RELEASE.jar
10. springfox-core-2.2.2.jar
11. springfox-schema-2.2.2.jar
12. springfox-spi-2.2.2.jar
13. springfox-spring-web-2.2.2.jar
14. springfox-swagger-common-2.2.2.jar
15. springfox-swagger2-2.2.2.jar
16. swagger-annotations-1.5.3.jar
17. swagger-models-1.5.3.jar

• Next will be to add Swagger Config java file, this is the file which is responsible for loading swagger config, specifying the url which needs to be scanned, basic API details.

Here is the sample code.

package com.swagger.apidoc;

import static com.google.common.base.Predicates.or;
import static springfox.documentation.builders.PathSelectors.regex;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.google.common.base.Predicate;

import springfox.documentation.builders.RequestHandlerSelectors;
import springfox.documentation.service.ApiInfo;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;

@Configuration
@EnableSwagger2
public class SwaggerConfig {
 @Bean
 public Docket api() {
  return new Docket(DocumentationType.SWAGGER_2).select()
    .apis(RequestHandlerSelectors.any())
    .paths(paths())
    .build()
    .apiInfo(getApiInfo());
 }
 private ApiInfo getApiInfo() {
  ApiInfo apiInfo = new ApiInfo("Swagger Demo API details",
    "These are API is demonstration of Swagger integration with spring","1.0", "TOS", 
           "abcd@test.com","@Swagger Demo", "http://java-treat.blogspot.com");
  return apiInfo;
 }
 private Predicate paths() {
  Predicate filter  = or(regex("/api.*")); // apiurl which starts with /api will be scanned by swagger.
  return filter;
 }
}

• Above code will scan through following controller class present in the project.

package com.swagger.controller;

import java.io.IOException;

import org.apache.http.client.ClientProtocolException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.servlet.ModelAndView;

import com.swagger.bean.Response;
import com.swagger.exception.CustomTwitterException;
import com.swagger.service.PostService;

import oauth.signpost.exception.OAuthCommunicationException;
import oauth.signpost.exception.OAuthExpectationFailedException;
import oauth.signpost.exception.OAuthMessageSignerException;

@RequestMapping(value="/api")
@Controller
public class ApiController {

 private static final Logger LOGGER = LoggerFactory.getLogger(ApiController.class);

 @Autowired
 PostService demoService;

 @RequestMapping(value = "/demo/{clientId}", method = RequestMethod.GET)
 public @ResponseBody
 ResponseEntity demoMethod(@PathVariable String clientId) throws OAuthMessageSignerException,
   OAuthExpectationFailedException, OAuthCommunicationException, ClientProtocolException, IOException,
   CustomTwitterException {
  LOGGER.debug("START: ----- demoMethod [GET] method");
  ResponseEntity entity = null;
  Response resp = null;
  if (!StringUtils.isEmpty(clientId)) {
   resp = demoService.postMessageSelf(clientId);
   entity = new ResponseEntity(resp, resp.getStatus());
  } else {
   resp = new Response();
   resp.setCode("ERR404");
   resp.setMessage("System Error occurred: Invalid parameter passed, correct 
                                         the input parameter and retry");
   resp.setDeveloperMessage("System Error occurred: Invalid parameter passed, 
                                         correct the input parameter and retry");
   resp.setStatus(HttpStatus.UNAUTHORIZED);
   resp.setProcessedCount("0");
   resp.setReceivedCount("0");
   entity = new ResponseEntity(resp, resp.getStatus());
   LOGGER.debug(
   "postMessage method: Invalid parameter is passed during posting message operation,
                         throwing back error to caller {} ",
    entity.toString());
  }
  LOGGER.debug("END: ----- demoMethod [GET] method");
  return entity;
 }
 @RequestMapping(value = "/test", method = RequestMethod.GET)
 public ModelAndView demoMethod1(@PathVariable String clientId) {
  ModelAndView mdv=new ModelAndView();
  mdv.setViewName("ppTesting");
  return mdv;
 }
}

• I have custom response class, sample for same is given below.

package com.swagger.bean;

import java.io.Serializable;

import javax.xml.bind.annotation.XmlRootElement;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
@XmlRootElement
@JsonIgnoreProperties(value={"throwable"})
public class Response implements Serializable{
 String message;
 String code;
 String receivedCount;
 String processedCount;
 public Response(){
  this.code = null;
  this.message = null;
  this.receivedCount=null;
  this.processedCount=null;
 }
 public String getMessage() {
  return message;
 }
 public void setMessage(String message) {
  this.message = message;
 }
 public String getCode() {
  return code;
 }
 public void setCode(String code) {
  this.code = code;
 }
 public String getReceivedCount() {
  return receivedCount;
 }
 public void setReceivedCount(String receivedCount) {
  this.receivedCount = receivedCount;
 }
 public String getProcessedCount() {
  return processedCount;
 }
 public void setProcessedCount(String processedCount) {
  this.processedCount = processedCount;
 }
}

• Once above step has been completed, next start the server. Swagger API will scan the packages and it will generate the necessary json documentation. 
• This json documentation can be parsed and documentation can be generated. In next blog i will show how to use Swagger provided UI, which does the parsing of the json output and displays in nice look and feel UI.
• To access the generated json file, access following url, and you should see something as shown below.
• URL: http://<hostname>:<port>/SpringSwaggerDemo/v2/api-docs
• For my demo context root of the application was "SpringSwaggerDemo", 
• Below is the sample json output file.

{ "swagger": "2.0", "info": { "description": "These are API is demonstration of Swagger integration in Spring project using spring fox", "version": "1.0", "title": "Swagger Demo API details", "termsOfService": "TOS", "contact": { "name": "abcd@test.com" }, "license": { "name": "@Swagger Demo", "url": "http://java-treat.blogspot.com" } }, "host": "localhost:9081", "basePath": "/SpringSwaggerDemo", "tags": [{ "name": "api-controller", "description": "Api Controller" }], "paths": { "/api/demo/{clientId}": { "get": { "tags": ["api-controller"], "summary": "demoMethod", "operationId": "demoMethodUsingGET", "consumes": ["application/json"], "produces": ["*/*"], "parameters": [{ "name": "clientId", "in": "path", "description": "clientId", "required": true, "type": "string" }], "responses": { "200": { "description": "OK", "schema": { "$ref": "#/definitions/Response" } }, "401": { "description": "Unauthorized" }, "403": { "description": "Forbidden" }, "404": { "description": "Not Found" } } } } }, "definitions": { "Response": { "properties": { "code": { "type": "string" }, "message": { "type": "string" }, "processedCount": { "type": "string" }, "receivedCount": { "type": "string" } } } } }

In future blogs I will show different annotations which are available which can be used for more customized documentation and also configure Swagger with Jersey based api projects, using and configuring Swagger UI.

Setup Ubuntu Virtual Machine on Windows

In this blog i will be showing how to configure Virtual machine in windows machine. In this blog i will show the demo using Virtual Box which is an open source tool, and it can be freely downloadable from Virtual Box .

• Download the "VirtualBox-5.1.4-110228-Win" installer from above link.
• Download "Ubuntu" OS.
• Ensure that you have sufficient disk space and decent RAM, as VM needs use from the available resources.
• Double click on the installer , and follow the wizard steps. Below is combined screenshot of installation steps.

Virtual box installation steps screenshot

• After installation is complete, start virtual box application. 
• Before we go further ensure that you downloaded Ubuntu OS from the link mentioned above.
• Ubuntu setup will be required while creating Virtual Machine. There are various OS supported, you can see that while setting up.
• Start the Virtual Box application, click on New menu, and select following in the popup

                     1. Name: Give name for your VM
                     2. Type: Linux
                     3. Version: Ubuntu (64 bit), for 64 bit version of Ubuntu, otherwise 32.

• In next two steps you will specify the RAM for the VM, higher is the better, but based on your system capacity provide the value. Also select Create a Virtual Hard Disk now option in the last screen. As shown below. Then click on Create button.

• Specify dis related information as shown below in sequence of wizard popup.

• This is how it looks when VM creation is done.

 

• Next step will be setup ubuntu OS. Right click on the Ubuntu VM in above screen and click on Settings. Go to "Storage" in the popup and on the right pan select "Empty" and click on Disk icon and select the drive where your Ubuntu installer is kept. Select the installer in file selection popup and Click Ok on below popup.

 

• Right click on the VM Ubuntu, and click on start. This will initiate Installation of Ubuntu OS in the VM.

• Ubuntu installation will start and following screens will be shown, follow the installation wizard. and complete the installation.

•  Now the OS in VM installation is complete. and it can be used for any setup which you have plan to use

SSO on Windows using Waffle - Java Web Application

Waffle is open source API which helps in windows based authentication. If project requirements is to auto login a user with Windows login credentials then Waffle provides one option to achieve the same.

Waffle supports Negotiate, NTLM and Kerberos. In this blog I will create a Java Web Project to demonstrate how to get the windows logged in credentials using Waffle.

Prerequisites:

1. Tomcat
2. Eclipse

• Create a new Dynamic Web Project in eclipse. 

Create Dynamic Web Project

• Add following jar files related to Waffle setup.

1. commons-logging-1.1.1.jar
2. guava-r07.jar
3. jna.jar
4. platform.jar
5. waffle-jna.jar

• Next step will be to add the filter classes to handle SSO with windows. 

Add filter class "waffle.servlet.NegotiateSecurityFilter", this class takes care of doing negotiation with windows system by invoking necessary classes with in the waffle jars. 

• Sample code for the same is given below for web,xml file.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns="http://java.sun.com/xml/ns/javaee"
 xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
 id="WebApp_ID" version="3.0">
 <display-name>WaffleDemo</display-name>
 <filter>
  <filter-name>SecurityFilter</filter-name>
  <filter-class>waffle.servlet.NegotiateSecurityFilter</filter-class>
 </filter>
 <filter-mapping>
  <filter-name>SecurityFilter</filter-name>
  <url-pattern>/*</url-pattern>
 </filter-mapping>
</web-app>

• This filter class takes several init-params, which can be added to customize the default behavior of the filter class. I will explain few of them below.

1. allowGuestLogin - if this flag is set to true, Waffle will allow any guest user will also be authenticated and waffle will return true. This will allow all the user to access the application. This flag will be useful if you have requirement to allow guest user as well to login to app with some minimal access.
2. waffle.servlet.spi.NegotiateSecurityFilterProvider/protocols -  Here list of supported security protocol can be passed. like Negotiate, NTLM.
3. principalFormat: Specifies the name format for the principal.
4. roleFormat: Specifies the name format for the role.

• Next step will to add code in servlet to retrieve the user details.

request.getRemoteUser() // for getting the user name.
session.getId();// will print the user session id.
request.getUserPrincipal().getName()  // this to get user name.

• now deploy the application on server and access the url, you should be able to see the user details.
• Now next step will be based on project need, if any local authorization needs to be implemented then you need custom implementation. Otherwise you are good to go.

Handling Cross site Scripting

Typically in web application cross site scripting issue is one of the most occurring issues. Cross site scripting vulnerability occurs when hackers are able to execute script code in your application. This can happen due exploiting weakness in the application code.
Like

1. Trusting data which comes from any of the system.
2. Lack of data filter for data cleansing before data goes inside the system.
3. Use of Scriptlet  <%=%> for printing data in JSP without validating the data.

CSS can broadly be categorized in two subsections.

• If hackers are able to persist their malicious code in application persistence layer (DB), and whenever any application user visits the web page, these malicious code gets executed and hacker will be able to exploit user. This type is persistent and are more dangerous.

         Hacker can add dynamic redirection to some of his malicious site, and capture confidential data. As this redirection happened from the parent site user also will not doubt and can provide their confidential data.

• Runtime execution of malicious code where Hacker can execute these script code by passing them from UI on the runtime. These are non-persistent types.

Generally if this vulnerability is present in the application, application developer or lead should not target specific fixes where these issues are identified.They need to fix the issue application wide a individual fixes will not make entire application free from CSS. 

Following are the points which needs to be taken care.

• Sanitize data from client
• Use Filters for XSS
• SSL - How to enable SSL in Tomcat

The data needs to be sanitized at the entry point only so that application never receives data which is malicious. 

In Java this can be handled by adding Filter which sanitizes all the request which is coming from Web. 

Note: This only sanitizes the data which is coming from Web, if you have different source of data like Feed file, Materialized views, direct DB updates, reading data from Webservice calls. They need to be also taken care of. These are out of scope for this blog.

Creating filter, The main purpose of this filter class is to encode/escape the value which has tags in it. Plain string and other value will be kept as is. By escaping these characters will make them dumb and browser will not execute them as script tags or predefined html tags.

 first step is to Create a Request Wrapper class which is responsible for encoding the values. This class will extend "javax.servlet.http.HttpServletRequestWrapper" class which provides method such as "getParemeter", "getParemeterValues", in these methods only escaping logic needs to be added.

Below is the sample code

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;

import org.apache.commons.lang.StringEscapeUtils;

public class XssReqWrapper extends HttpServletRequestWrapper {

 public XssReqWrapper(HttpServletRequest request) {
  super(request);
 }

 @Override
 public String getParameter(String name) {
  if (name != null) {
   String values = super.getParameter(name);
   String newValues = StringEscapeUtils.escapeHtml(values);
   return newValues;
  }
  return super.getParameter(name);
 }

 @Override
 public String[] getParameterValues(String name) {
  if (name != null) {
   String[] values = super.getParameterValues(name);
   if (values != null) {
    String[] encodedValues = new String[values.length];
    for (int i = 0; i < values.length; ++i) {
     encodedValues[i] = StringEscapeUtils.escapeHtml(values[i]);
    }
    return encodedValues;
   }
  }
  return null;
 }
}

Here i am using class from "org.apache.commons.lang.StringEscapeUtils" to escape the characters. For this you need add "commons-lang-2.6.jar" file.

Create a Filter Class which implements "javax.servlet.Filter", and override the methods doFilter, destroy, init. In this example I will using only doFilter method, as that is the method which will have escaping logic for tags. Below is the sample code

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;

public class XssFilter implements Filter {
 @Override
 public void destroy() {
  //left intentionally empty
 }

 @Override
 public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
   throws IOException, ServletException {
  HttpServletRequest httpReq=(HttpServletRequest) req;
  chain.doFilter(new XssReqWrapper(httpReq), res);
 }

 @Override
 public void init(FilterConfig config) throws ServletException {
  //left intentionally empty
 }
}

Once above classes are created, Open Web.xml file. and add following line of code. This will enable the filter in the web application. And for each and every request this filter will be called.

<filter>
 <filter-name>xssFilter</filter-name>
 <filter-class>com.xss.sample.XssFilter</filter-class>
</filter>
<filter-mapping>
 <filter-name>xssFilter</filter-name>
 <url-pattern>/*</url-pattern>
</filter-mapping>

Below is demo where i will show behavior of application without filter and with filter.

Application without XssFilter.

Below is login page where i will submit page with "<script> alert('Hello')</script>" as a user name.



Once this page is submitted to server it does simple forward to another page where i am printing the user name. See below what happens. Here browser interprets the script tag as a valid tags and executed it while loading the page. This is where hackers can redirect user to some malicious page and capture some information from them.

Also see the console output of the value.

    

Application with XssFilter.


I will pass exact same value as shown in the above login page. But in the subsequent page, now browser does not show any alert. It treats it as normal value and display it in the UI.





Also see the changes which filter makes with the submitted value. Check console logs.



Here Filter class encodes the "<" and ">" values to "&lt;" and "&gt;", which browser treats as normal string and display the value.

That is all from Cross Site scripting, i will cover some other Security Vulnerabilities in upcoming post.

SQL Injection security Vulnerability

This article i will be covering SQL Injection security vulnerability.

What is SQL Injection?
SQL Injection means, modifying application query in such a way that it tweaks the actual query results. With this issue present in application Hacker can easily manipulate the query and can gain access to application data and functionality.

How to address this issue?
There are many things which application developer can take care during the development phase which can avoid these issues.

1. Usage of PreparedStatement while doing database operations.
2. Usage of CallableStatement which is used to call database Procedures. 
3. Input validations and Data Encoding.
4. Database related constraints like, usage of proper grant, privilege, views etc.   

Below is simple example on how the attacker can use this vulnerability to exploit system.

Lets assume that in the web application there is login screen which prompts user to enter user credentials. 

In the application to validate the user following query is used.

SELECT USERNAME FROM USER WHERE PASSWORD='ABCD' AND USERNAME='ABCD' ;

and in java code this query is prepared using string concatenation operation something like shown below.

public String validateUser(String userName, String password){
 Statement stmt=null;
 String query="SELECT USERNAME FROM USER WHERE PASSWORD='"+password+"' AND USERNAME='"+userName+"'";
 stmt=con.createStatement();
 ResultSet res= stmt.executeQuery(query);

}

Now to exploit this limitation in the code,hacker can easily manipulate the query by passing additional query while login from the application login page and he can access the application without having valid user account. This can be done as shown below.

Application will receive the information from UI and it will prepare following query based on above java code.

SELECT USERNAME FROM USER WHERE PASSWORD='ABCD' AND USERNAME='abcd' OR 'A'='A' ;

Once above query is executed by database, even if the hacker enters the wrong user name and password, he is able to get into the system, as the condition "OR 'A"='A"" always returns true. This value hacker passed from UI and application blindly accepts this value and prepares the query.

This is one of the kind of exploitation which hacker can do if application is vulnerable to SQL Injection.

to address this issue one of the simple solution is to migrate to PreparedStatement. But there are situations where this migration is not very straight forward when application team doesnot want to modify legacy codes. Then other approaches can be taken up like escaping/encoding the user input and append those value in sql queries.

Prepared Statement Approach:
Below is the sample code which will remove the SQL injection issue. There are ORM framework available which can be used in the application and these framework takes care of all these standards. like, MyBatis, Hibernate, Spring DAO etc.

public String validateUser(String userName, String password){
 String query="SELECT USERNAME FROM USER WHERE PASSWORD=? AND USERNAME=?";
 PreparedStatement stmt=connection.prepareStatement(query);
 stmt.setString(1,password);
 stmt.setString(2,userName);
 ResultSet res= stmt.executeQuery();
}

I hope this explanation helps to understand the SQL injection issue.

Parallel AJAX calls

Consider a scenarios where you have multiple source of data and that needs to be consolidated in the UI and presented to the users. There can be different ways to implement this. One option can be using AJAX approach. JQuery provides a way to initiate parallel AJAX calls. In this blog i will be showing a sample code to achieve same.

Using Jquery provided "$.when" function, which provides a way of executing asynchronous events. $.when takes asynchronous ajax events as a parameter, and the output of these AJAX events will return in ".then" in the same order.

$.when(ajax1, ajax2,....).then(resp1,resp2,....);

Following is the sample code to make parallel ajax call.

var url1="http://localhost:8080/url1";
var url2="http://localhost:8080/url2";
var param="s=abcd";

$.when(
 $.ajax({timeout:10000,error:function(){handleErrorScenario();},type: "get",
   url:url1, async : true, data:param}),
     $.ajax({timeout:10000,error:function(){handleErrorScenario();},type: "get",url:url2,
   cache: true,async : true, data:param})
 )
 .then(function(url1Resp, url2Resp){
    
 if(!url1Resp || !url2Resp){
  handleErrorScenario();//define this function.
 }else{
  handleSuccessScenario(url1Resp, url2Resp); //define this function.
 }
}

In the above sample there are two AJAX function which are passed as parameter. url1Resp and url2Resp will hold the output from the call. Code inside ".then" will be only executed when all the deferred AJAX response comes back. Some of the function needs to be defined in the your code. I am using them as dummy reference.

Using Apache CXF initiate SOAP call

IN this blog i will be showing how to consume SOAP webservice in java by generating Client code using Apache CXF.

Prerequisites for this blog are as follows.

1. Eclipse Mars
2. Java
3. Apache CXF
4. WSDL file for any SOAP webservice.

Install CXF

Apache CXF can be downloaded from "CXF". After downloading setup, unzip it in your system. 

Configure cxf in eclipse. 

Go to Windows - Preferences - Webservice - CXF 2.0 Preferences

Then select CXF Runtime - click on Add - Specify CSF installed location - Click on Finish.

I have highlighted the steps with blocks in below screen shot.

Setting Up CXF in Eclipse

Create Web Project or Paste WSDL in existing project.

After configuring you need create Project in eclipse, or if you are using it in existing project then paste the WSDL file which you have received from service provider.

Next step will be to generate the client code

Right click on wsdl file, Go to - Webservice - Generate Client

Webservice Client popup will come, next specify the Configuration by clicking on the "Webservice runtime" link. In the Client configuration popup select "Apache CXF 2.x". Click OK.

Click next button, specify the package incase it needs to be changed. Otherwise keep everything default and finish the steps.

Steps to create Client Code

Once the client code is generate. Following are the important note, which will help you in quick startup.

•  Locate file which has name with *_Client.java. This is sample client code which is generated by CXF. This shows invocation steps.
• Locate file which extends "javax.xml.ws.Service", this is the main service class which will be utilized in invoking the service.

  
  GlobalWeather ss = new GlobalWeather();
  GlobalWeatherSoap port = ss.getGlobalWeatherSoap();  
  java.lang.String _getCitiesByCountry__return = port.getCitiesByCountry("Country Name");
  System.out.println("getCitiesByCountry.result=" + _getCitiesByCountry__return);

Incase you need to override the end point url, following code can be used to override the end point url.

  GlobalWeather ss = new GlobalWeather();
  GlobalWeatherSoap port = ss.getGlobalWeatherSoap();  
 
//code to override the end point
  BindingProvider bp = (BindingProvider) port;
  bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
    "NEW URL");

  java.lang.String _getCitiesByCountry__return = port.getCitiesByCountry("Country Name");
  System.out.println("getCitiesByCountry.result=" + _getCitiesByCountry__return);

Cross domain Calls in AJAX with Jsonp

Web browser doesnot allow initiating cross domain call from javascript. There are multiple ways to initiate cross domain calls, I will be showing example on how to make cross domain call from javascript using jsonp.


Using Jquery following is the way to initiate Cross Domain call.

in $.ajax method, "dataType" parameter should be set to "jsonp", here jsonp means Json with padding, with jsonp a javascript code
is injected in client browser, which enables code to make cross domain call.

function initiateCrossDomainCall(url) {
 $.ajax({
     dataType: 'jsonp', // json with padding
     type:"GET",
            url : url,
            success: function ( data) {
             parseResp(data);
            },
     error: function ( data, status, error) {
  parseErrorResp(data, status, error)
     },
            timeout: 2000
  });
}
function parseResp(data){
 //add code to parse responsedata.
}
function parseErrorResp(data, status, error){
 // parse errr response.
}

Basically when above javascript call is initiated from client, and dataType is mentioned as JSONP, then jquery by default adds a parameter to the url. parameter name will be "callback" and name of the method will dynamically generated by jquery. Server response should be wrapped inside this method.. e.g. in the below url call, you can see the callback=jQuery11230083279708298031_1472922945432, "jQuery11230083279708298031_1472922945432" this is the function name in which response should be wrapped and sent back from the server side code.

http://localhost:8080/location?callback=jQuery11230083279708298031_1472922945432&_=1472922945433

Incase if custom callback method is defined in $.ajax method then jquery will send that method name in the "jsonpCallback" parameter. something like this. jsonpCallback:'handleResp' in the url call you can see the callback=handleResp is passed.

http://localhost:8080/location?callback=handleResp&_=1472922945433

To make the server side code support jsonp approach, following should be the logic on server side code. Sample code in java with spring. but this can be done in any framework with similar output.

@RequestMapping(value="/location", method=RequestMethod.GET)
public @ResponseBody String returnLocation(HttpServletRequest request){
 //reading callback paarameter
 String callback = request.getParameter("callback");
 StringBuilder sb=new StringBuilder();
//preparing response based on if callback parameter is present or not.
 if(callback!=null){
  //appending callback method name and adding json response inside that method.
 sb.append(callback).append("(").append("{\"location\":").append("BL").append("})");
 }else{
 sb.append("{\"location\":").append("BL").append("}");
 }
 return sb.toString();
}

following is the response from the server to the jsnop call.

  jQuery11230083279708298031_1472922945432({"location":"BL"})

 once the client browser receives the above response it calls back the success block of the $.ajax function and executes either "parseResp" or "parseErrorResp" method based on success or error.

that is all from jsonp.