SigmaWay Blog

According to Dermot O'Kelly (Senior Vice President for Oracle UK, Ireland and Israel region) for the past few years the stage is being set for an analytics based future in which large volumes of valuable information can be collected and used to take informed decisions more efficiently. The rise of big data has been rapid. But very little emphasis has been placed on trying to understand exactly how big data can impact and improve our lives. Big data tools help in linking all information together and assess it from various standpoints at once in the hope of detecting new insights, to approach difficult questions from a fresh angle, or to process huge data sets very quickly. They also make the collection and processing of this data exponentially faster than before. Read more at: http://www.in.techradar.com/news/world-of-tech/Big-data-in-the-real-world/articleshow/42649169.cms

There is no privacy in the era of big data. Personal data is collected and traded and there are few ways to control it. "We're being monetized in essence. We are being mobilized as products with inducement of the services of we use such as Facebook and Twitter" says Rob Livingstone, a fellow of the University of Technology and the Head of a Business Advisory Firm. However, major problem that regulators are facing is - how they can regulate the collection, storage and trading of personal data on the internet, when all of these activities, and corporations, operate across multiple continents and jurisdictions. Read more at: http://analytics.theiegroup.com/article/53a4371c3723a8398400014e/Little-Privacy-In-The-Age-Of-Big-Data

Information available about consumers on the web is enough to scare marketers. There are large rewards for those who confront and harness big data. Big data can create problem for brands. Information collected from large, complex data sets weblogs, social media, smartphone analytics and even medical records is difficult for brands to manage and process within traditional database systems. According to a report from the McKinsey Global Institute, Big Data is the next big thing for innovation, competitive advantage and productivity. But, some companies are missing this opportunity because of a lack of data management expertise. Retail is one industry that has a great potential for big data. Customer transactions (on & offline), conversations and intentions can all be brought together so that brands can get ideas how to reach potential consumers. Read more at: http://analytics.theiegroup.com/article/53a2f6693723a807f3000029/Big-Data-Embracing-The-Elephant-In-The-Room

According to author Shreya Sharma, with Big Data becoming an essential part of an organization, it is getting harder to balance it with data privacy. Marketers should be clear about the data they are collecting and make sure that data storage and security infrastructure are safeguarded, otherwise it can have serious effects on the company as most consumers expect a company to protect their personal data, and their consuming behavior depends on this. It is therefore crucial to bear a high level of awareness with regards to the significance of data privacy and protection. The responsibility should be seen as norm and not just be restricted to people who are directly involved in mining big data. To know more, please follow: http://analyticsindiamag.com/in-action-how-companies-should-follow-a-data-centric-security-approach/

Big data enables understanding of complex customer sentiments and intent which improves service management. Today it is easier to process customer sentiments and intents through social media and develop social customer relationship management or social CRM. To make this effective it is important to capture the natural language knowledge and rapidly and securely making those insights available to tye best pay off channels. 

This podcasts bring together customer analytics services provider Attensity, with its natural language processing technology, and HP Vertica with Big data analytics capabilities to explain how effectively listen to social Web and rapidly gaining valuable insights and actionable intelligence.

Some participants are Howard Lau, chairman and CEO of Attensity, and Chris Selland, vice president of marketing and business development at HP Vertica. The discussion is moderated by Dana Gardener, principal analyst at Interarbor Solutions.

more at: http://www.crmbuyer.com/story/80826.html

Big Data is about making something out of enormous and high speed structured data as well as unstructured data, but most organizations do not have a business case for the use of Big Data and invariably, a not so conducive culture to leverage Big Data technologies. The main area of focus in India seems to be-Real Time Reporting on Transaction Systems. Through the partitioning/ clustering, distributed computing and in-memory processing methods, Big Data can run high quality real-time reports, with the ability to drive additional queries. But it will be not be the best use case for Big Data. If we focus on this use case, Information Technology will just end up being a cost center. Big Data investments should address either the fraud prevention story that helps the bottom-line, or the customer offer story that helps the top line of a company. The immediate need however is to get the data fixed to achieve the goal. Data quality continues to remain a huge challenge across more than 80 percent of the mid and large organizations. Read more at:

http://www.informationweek.in/informationweek/perspective/297428/companies-artists

Ever since the video gaming industry has entered the online space, this fast-paced industry is almost bursting. Bringing in about $20 billion of revenue a year in the U.S. market alone, this industry is now scaling new heights with vast data out-pours and advances in big data analytics. In the world of online and offline video games, joystick movements and every interactive step is a source of valuable data-points, which can deliver fascinating solutions for enhancing gamer experience and increasing revenue streams. The world of games is like any other form of entertainment-it requires addictive rewards to make the gamers come back again and again. According to reliable market reports, a large, video-game manufacturer has the potential to generate around 50 terabytes of data each day. In US, the gaming industry is certainly bigger than the movie industry in terms of revenue generation. Now that this prolific industry is rapidly embracing big data technologies, the industry is expected to make waves in novel methods of customer engagement, optimized and targeted advertising, and enhance the end-user experience.

Read more at: http://www.mapr.com/blog/big-data-revolutionizing-video-game-industry#.U-d_4vmSxvM

Malware and cyber-attacks have become increasing concerns among companies. Many of them struggle to predict and mitigate threats, which can spring and evolve quickly. Companies also have some concerns about the physical security of their infrastructure. The result is that they are in search of more comprehensive resources to address these challenges. Big data analytics, particularly when they are coupled with machine learning, represent a logical solution because it allows companies to consider multiple threat scenarios and determine the best response. 

In today’s complex network environments, Advanced Persistent Threats (APTs) and other cyber threats eradication may be accomplished by getting intelligence from data providers. 

For these cyber threats, appliances should be monitoring threat feeds from trusted providers for indicators of compromise (IOCs), including big data feeds like domain name systems (DNS) feeds, command and control (C2) feeds, and black/white lists, in order to correlate and hunt threats in a data set and recommends six steps to combat the potential threats.

Read the article at: http://www.darkreading.com/analytics/6-tips-for-using-big-data-to-hunt-cyberthreats/a/d-id/1278970

Big data has been big news in the travel industry for a few years now. It offers the chance for travel companies to increase the number of sales and improve the travel experience of customers. While big data is a vast, complex challenge for many organizations, it is one of the key factors driving the evolution of the travel industry today and for the foreseeable future. Airlines, hotels, cruise companies, travel management, railways and travel agencies have an opportunity to improve their business and the customer experience by effectively handling the big data at their fingertips. That's not to say it is easy to collect, identify and analyze all of the bits of disparate data types that comprise big data. Travel companies can make better decisions based on the data aggregated from their customers to personalize services like travel booking to make them easier to navigate. For many years, CRM systems have allowed companies to create relevant, targeted marketing campaigns. Big data, however, uses customer information to create a truly personalized service. It allows travel websites to recommend a specific hotel to a specific user based on their previous holidays, requirements and preferences.

Read more at: http://www.forbes.com/sites/centurylink/2014/04/30/how-big-data-is-transforming-the-travel-industry/

Proficiency in Big Data provides a competitive advantage to banks. Banks too often depended on traditional technologies such as aggregation and normalization of data which resulted in several weaknesses like lack of flexibility in responding to upstream and downstream data changes. Data lineage may be lost after aggregation and summarization and data governance is likely weakened when several constituents retain responsibility for an extended, multi-stage data flow. These weaknesses are detrimental to the success of big data initiatives. So a new approach is required.  Big data represents a new way that banks can interact with and leverage their data. As a result, banks need to shift the paradigm for designing, developing, deploying, and maintaining big data solutions with new approaches to data storage (e.g., NoSQL databases)  and maturity of distributed-computation software frameworks (e.g., Hadoop). The approach to Big Data implementation also needs to change through rapid, iterative, and incremental deployment of solutions in a way that aligns well to the speed at which the underlying data are measured, understood, and parsed. This will take banks to an acceptable level of competency and capability. Read more at:

http://www.informationweek.in/informationweek/news-analysis/297426/mean-banks

Huge amounts of data are being created with every passing day.The UN has called for a 'data revolution' to come up with new development goals - so that sustainable development can be better practiced and can be integrated into making decisions and also provide transparency. But finding out trends hidden within data takes skill - and so the question is whether jury is out as to whether nations and development organizations have the capacity to interpret the so-called 'big data' by themselves. The question is how best to make up the ground. The problems faced in implementing Big Data involve lack of expertise to train staff to use big data, and also awareness of the technology needed for analysis. Lack of money is another concern, particularly in developing countries. If governments do wish to see big data used for development, their best bet could be to allow the private sector to do the groundwork. In many developing countries, problems of patchy internet connections, intermittent power supplies and poor reach of high capacity cables are being faced. But the digital divide challenges may be circumvent by other sources of data, for instance, call data records (CDRs).The barriers facing big data becoming a useful tool for development are numerous and great. But, they are not insurmountable. There is reason to believe that big data can fulfil its promise in the years to come. Read more at:

http://www.scidev.net/global/data/feature/obstacles-big-data-development.html

Big data analytics is increasingly playing a role in the fight against crime. Publicly shared information combined with data from local authorities, social services and intelligence gathered by beat officers is helping police forces around the world spot trouble before it starts. It helps the police be much less reactive, and slowly starts to reveal the real trouble spots and troublemakers in a neighborhood, estate or street. When information like that becomes clear, the police can do something about it long before anyone dials 999. And that counts for people as much as it does for pubs or clubs. Law enforcement is finding new ways to use technology and big data against crime. CCTV cameras are no longer impotent. They are commonly used in police cars and carried by officers to create a permanent digital record of everything going on around them. 

 This will make it harder for criminals to commit crimes. In recent years we have witnessed criminality moving off the streets with a huge increase in the amount of credit card and online identity fraud. But even there new big data algorithms are being developed to detect fraudulent behaviors in real time.

Read more at: http://smartdatacollective.com/bernardmarr/199521/big-data-analytics-and-criminals

This is the age of Big Data and the amount of data surrounding us is actually huge. The rate at which new data is created almost doubles every month. Some examples to show the trend of data driven decisions in almost every sphere are as follows. The big business of sports has led the charge. We're using our smart phones, watches, and other wearable devices to gather data about ourselves to better understand fitness, nutrition, health, and behavioral tendencies. Local and national governments are contributing too with significant movements towards transparent publication of data on websites. The approaching Internet of things -- as governed by new devices such as the Nest Thermostat, Quirky devices, or even the Waze service that uses consumers' GPS-enabled smart phones to gather information --have such companies as GE and Google making substantial investments based on their potential to both generate and find value in big data. Though there exists so much of data some companies still face problem in dealing with it due to existence of some challenges. Read more at: http://tdwi.org/articles/2014/07/08/turning-big-data-into-smart-data-1.aspx

Results show that over 75 percent of the organizations have gained big payoffs with the application of big data and analytics in their organization. Also the Return on Investment (ROI) has increased within six months of application. Certainly executive support as well as their involvement in analytics is vital to value creation since in organizations with low levels of executive support, analytics implementations are hampered by lack of funding, resources and follow through. Besides, strong governance and security are important in instilling confidence in the data, and trust is necessary. Also the direct factor which has implication on organization's value is the trust between people within an organization. This is not trust in the quality of the data but the old fashioned trust that is earned by getting to know someone's character and what they are capable of delivering. The level of trust - a belief that others will do a competent job, deliver on promises and support the organization's best interest - among executives, analysts and data managers significantly impacts the willingness to share data, rely on insights and work together seamlessly to deliver value. Read more at: http://www.informationweek.in/informationweek/perspective/286293/roi-about-trust

Big data is a key infrastructure in the Internet of Things (IoT), but it's far from the only piece of the fabric. In the coming global order, every element of the natural world, and even every physical person can conceivably be networked. Everything will be capable of being instrumented. If you think that the world of driverless cars, robots carrying out maintenance in hazardous locations like oilrigs, or advertising that reads and responds to individuals' unique facial expressions sound like science fiction. As these trends come to fruition, each of us will evolve into a walking, talking, living beneficiary of the Internet of Things. These are all developments happening today and they're prompting a new exciting phase in analytics that needs to be addressed now. Those that embrace data will be more likely to be surfing on top of the wave of creative destruction, instead of having it crash down on top of them.

Read more at: http://blogs.computerworld.com/business-intelligenceanalytics/23447/internet-things-what-it-and-what-does-it-mean-analytics

Data is more than just power. Organisations across all industry verticals are upgrading their data management systems, investing in new resources, and using their rich databases to streamline the practices of their departments. Indeed the message from experts is clear: organisations that fail to adapt and evolve to meet the emergence of big data, face the prospect of falling behind. As with any phenomenon, however, there are lessons to be learnt. 

The way data is being used in sports is a poignant example. Moneyball, the popular book inspired by Oakland Athletics manager Billy Beane, explained the core philosophy of the manager’s vision for the baseball team: using statistical analysis to maximise player acquisition and performance with a low budget. 

The Moneyball philosophy had huge ramifications for the sporting world. People started adopting variants of it in all sports – from soccer to basketball to football. Arguably the most noticeable application of Beane’s philosophy was by Andy Flower, the former England cricket coach. Flowers was known for his admiration of Beane’s work, and he too would use statistical analysis to not only determine who would be on the field but also what decisions players should make once they were selected and enjoyed notable victories also. Both of them have stood by data analytics and the benefits it can bring. Yet, what is often untold is that data was both a virtue and a vice for both men.

His 5-0 defeat at the Ashes last year was one of England’s most disappointing performances to date. As commentators suggested, it was a classic case of overreliance on data, replacing intuition with numbers, and allowing data to dictate rather than inform. Flower ultimately got the balance between trusting people and numbers wrong. He was in good company, those who thrive will not be those who use data most—but those who use it most smartly. But data is emphatically not a substitute for intuition and flair - either in the office or on the cricket field.

These instances of sports analytics are particularly relevant for organisations looking to add big data analytics to their existing operations. The example of Beane and Flower show how data does not have all the answers, and relying too heavily on it can have devastating effects.

Read more at: http://www.espncricinfo.com/magazine/content/story/724435.html

Big Data Analytics is all about turning volume and variety of data into meaningful insights. When data is refined and combined, new patterns and ideas emerge, and one can take better decisions using these insights. Big Data is being used across almost all the sectors these days. In the Online Apparel Industry where success of next season's collection hinges on selecting the accurate designs, colors, fabrics, shapes, and sizes, Big Data can be a big game changer. Online apparel industry is mostly influenced by predictions that are based on identifying the most popular/liked parametric values (colors, fabric, style and many more) of the apparels. If you predict it right, it may bring a profitable season for you, else it may lead to heaps of discarded inventories. For many years, analysts and fashion reporters have tried to control these drifts. It is a great advantage to recognize customer preferences that will lead into high prospect ratio.

One of the ways to understand customers’ emotions behind the interactions made on social media sites and other forums is sentiment analysis. Sentiment analysis scans tweets, comments, likes, etc., for evidence of positive, negative, or even indifferent impressions to identify the overall trend of sentiments towards any entity. For example, possible positive expression on a personal level would ideally be like – “I like to wear plain cotton clothes in summers…” to an opinion projected in general - “I am looking for some cool blue apparels for my next vacations as it feels comfortable.”

Similarly, negative expressions may go like – “I am fed up of seeing bright yellow apparels all around in summers.” Or an opinion in negative tone may be like - “People look damn horrible in yellow.” A range of tools and methods are available to help determine customers’ sentiments; one of the ways to track is using the Twitter Sentiment APIs.

Big data is also extremely useful in a marketing capacity, using information like customer demographics and spending habits, in terms of how much they spend, on what and where. In addition to these habits, companies that invest in cloud computing studies can monitor how their existing marketing strategies are working - eye scanning data can be analyzed to see the effectiveness of billboards and other visual advertising. Every aspect of the business will change, from what color will be in next season to how to make clothing that fits different body types and how to optimize  supply chains.

Read more at: http://www.gogrid.com/news/2014/07/23/cloud-computing-public-cloud-big-data-how-big-data-will-change-fashion-industry

Big data is the basis for business intelligence, which is about taking all that information and turning it into knowledge to drive better business decisions. Whether its data about retail consumers or homebuyers, it's all the same game.  The business intelligence industry has been analyzing large data sets in corporations for years — decades, really. It’s only now coming to the real estate industry. The amount of data used in the real estate industry isn’t that large. A single major retailer will generate more sales data in a year than the entire real estate industry will in a decade. However, it’s all relative, and the real estate industry is still trying to figure out what data it has, let alone how to use it.

The point is that big data in real estate is about presenting a “whole consumer” picture. It’s about using data to find out who buys what, when, where, why and how. It’s about finding out who will sell a house — when, where, why and how. 

All that data can be used to create tangible insights into consumer behavior using forecasting and modelling software. It’s the analysis that makes the magic happen, that is identifying customers or providing them better services. Analytics is where raw data and the algorithms that crunch it come together. Mining census information, the results of consumer surveys, listings of homes for sale and rent, geographic information systems data and more combine what they draw from numerous databanks with their own proprietary user-generated content. The tools can deliver to consumer’s information about their property's potential value and help them understand home-value trends within a particular milieu, such as a neighborhood or a ZIP code. 

Beyond the consumer and industry-facing aspects of big data, institutions such as banks can plug into big data resources to determine whether a foreclosure or short sale is really worth what a buyer or investor might be offering.

For now, the analysis of big data is likely to stay with those who gather it and companies willing to pay for access, such as the lead generation companies. What real estate agents need to know now is that the data is there and it’s available, in some form or another, to those who are willing to use the right tools.  Read more at: http://mashable.com/2014/07/09/big-data-real-estate/

Big data has the automobile in its sights and the results will be good for both the vehicle and its owner. In the coming years we can expect to see both safer vehicles and car-to-car communications. You'll be advised of a needed repair before a problem and recall notices will be delivered through the car. Ford gathers data from over four million cars with in-car sensors and remote application management software. All data is analyzed in real-time giving engineers valuable information to notice and solve issues in real-time, know how the car responds in different road and weather conditions and any other forces that could affect the car. Ford is also installing numerous sensors in their cars to monitor behavior. They install over 74 sensors in cars including sonar, cameras, radar, accelerometers, temperature sensors and rain sensors. As a result, their Energi line of plug-in hybrid cars generate over 25 gigabytes of data every hour. This data is returned back to the factory for real-time analysis and returned to the driver via a mobile app. The cars in its testing facility even generate up to 250 gigabytes of data per hour from smart cameras and sensors. 

Big data is also used to find out how people wanted their cars to be improved. Nowadays, Ford listens carefully to what their customers are saying online, on social networks or in the blogosphere, and performs sentiment analysis on all sort of content online and uses Google Trends to predict future sales.

Internally, Ford uses big data to optimize its supply chain and to increase its operational efficiency. From the parts before they reach the Ford factory, to the car waiting in the dealer for a customer, big data has infiltrated every part of the supply chain, creating large amounts of data. With so many different parts coming from so many different suppliers, it is vital for Ford to get a complete and detailed overview of all parts within the supply chain at any moment in time. To read more visit: http://www.bigdata-startups.com/BigData-startup/ford-drives-direction-big-data/

In the era of Big Data, the fight for protection has as of now been battled and lost. The personal data is routinely gathered and exchanged and there are few powerful controls over how it is utilized or secured. Data scientists and analysts are now saying that now is the right time for enactment to recover some of that protection and guarantee that any information that is gathered remains secure.

We have become the product and are being productised and sold to anyone. We’re being monetised and mobilized as products with inducement of the services of we use such as Facebook and Twitter. The dilemma that the regulators are facing is how they can regulate the collection, storage and trading of personal data on the on the internet, when all of these activities, and the corporations themselves, operate across multiple continents and jurisdictions.

The task of reclaiming some semblance of privacy is all the more urgent because the rate at which personal data is being collected is accelerating. The buzz around big data is attracting millions of dollars of from investors and brands hoping to turn a profit, while intelligence agencies are also furiously collecting information about our online activities for much different purposes.

And alongside these, there’s also the black market operators that make millions of dollars a year out of things like identity theft and matching disparate data sets across the web to help identify people who might be suitable targets for a scam. 

New privacy principles were recently passed into law which required all businesses earning more than $3m annually to disclose to customers how their information was being stored and used, however the new legislation stopped short of mandating compulsory data breach notifications for businesses who fall victim to security violations.

A bill that would make it illegal to hide security problems was set to pass into law last year, however it failed to make it through both houses of the Senate before the election. And since the Coalition took power, the legislation has stalled. 

Still, there are many privacy challenges ahead, and the problems have by no means been solved. Most methods of anonymizing do not scale well as p or n get large. Either they add so much noise that new analyses become nearly impossible or they weaken the privacy guarantee. Network-like data pose a special challenge for privacy because so much of the information has to do with relationships between individuals. In summary, there appears to be “no free lunch” in the trade-off between privacy and information. To read more: http://www.theguardian.com/technology/2014/jun/20/little-privacy-in-the-age-of-big-data