Archives

5G: A giant step in the evolution of the telecommunication

The unfolding dimensions in technology had made Data the new oil of the 21st century. Today the world order is determined by data. Therefore, technology has ushered a new competition across the globe with “DATA” in the center of it. This race started with the onset of the 21st century that saw a prodigious technology development. Increased internet speed and bandwidth are other contributing factors.

A short story on the long evolution of the Internet

It all started when first-generation(1G) analogy phones came with  the relatively simple picture. With the evolution of this prototype, the forthcoming generations 2G, 3G, and 4G were developed over decades of years.

The 4G technology is widely used, allowing to share live messages, video calls, and bring in virtual reality. According to Ericsson, global mobile data will increase 39% by 2023 with 107 Exabytes of data generated per month. At this rate, the number of connections will rise over 29 billion in 2022.

Their successor, 5G has today created a great buzz in the world.

What is 5G?

The term 5G refers to the fifth-generation technology prototype that uses high-frequency radio waves to transfer data. It provides low latency and high connection density.

5G will reduce the data sharing time to only 1ms that was 40ms-60ms in the 4th generation.

It claims to be the bridge to the future. It addresses the need for more bandwidth and speed. 5G ultra-wideband will eventually allow the development of more proficient technology that will bring a massive change. In 2024, 5g will host 25% of mobile data traffic that would be 30% more traffic than that of today.

Roadmap of 5G revolution

5G revolution will open doors for numerous advancements such as self-driven cars communicating with each other or remote robotic surgeries. We must remember 5G revolution won’t happen overnight. The transition takes time to reach the masses. Currently, all major service providers are rolling out 5G services. Verizone claimed the availability of its 5G network in more than 60 US cities till the end of 2020.

Every year, there is an exponential increase in 5G service providers. We can expect 5G network to reach beyond large cities.

The cause of delay is the infrastructure.  5G has multiple frequency bands. The infrastructure of lower frequency can be mapped to a 4G network. Other bands will need new infrastructure. Also, 5G works on shorter wavelengths that require more antennas.

IHS Markits says 5G’s full impact will be seen near 2035 with about $13.2 trillion business from goods and services. 

Use cases of 5G

Increased speed, low latency, and more connection density give limitless possibilities. Initial infra costs may be tremendous. It must be accommodated with the launch of a new set of services for customers.

Driverless automobiles that are already on their way by companies like Tesla will rely on real-time data with no or less latency. Next one will be VR and AR applications. The computing speed required will be easily catered to by 5G.

Cloud computing will be more efficient. Due to low latency, PC hosting cloud app may be shifted to devices with less processing power.

As of now, there are 14.2 billion IoT devices in use. It will increase to more than 55 billion by 2025. Industrial IoT will also leverage the benefits of 5G networks. Consequently,an entire supply chain can be reshaped.

Conclusion

5G will help businesses in stepping up their game. And customers will get exposure to new methods in their life.

Have you used it yet? Tell us more about your 5G experience at contact.us@virtuetechinc.com and let us assist you.

Read More
Dataroma Case Study

The Challenge

GoDaddy is world’s largest domain name registrar company, with more than 16 million domain name registrations and 13 million customers. GoDaddy collects all the marketing related information for its customers to help them along in their customer’s lifecycle.

This marketing data is highly messy as it consists of data from Google Adwords, yahoo, bing etc. and thus it was a huge challenge for the GoDaddy team to get insights from this data.  To collect data from these multiple sources and

Why Amazon Web Services

GoDaddy stores information on Amazon Simple Storage Service (Amazon S3), and processes data in parallel with Amazon Elastic MapReduce (Amazon EMR). EMR decouples compute and storage, giving us the ability to scale each independently and take advantage of the tiered storage of Amazon S3. With EMR, we can provision one, hundreds, or thousands of compute instances or containers to process data at any scale. The number of instances can be increased or decreased automatically using Auto Scaling (which manages cluster sizes based on utilization) and we only pay for what we use.

Running Critical Applications on AWS

Salesforce data is extracted in Amazon Simple Storgae Service (S3) in CSV format through API. Amazon EMR runs a Pyspark framework, which has automated the entire process of transformation, cleanup and loading data into S3 ADS Layer. Amazon Redshift table is created on top of this S3 ADS Layer. Tableau is connected to this redshift table via the Redshift connector, which allows analysts to analyze data and generate useful business insights.

The Benefits

The primary benefit to GoDaddy of moving to AWS is that enough resources are available to provide the services to customers of all sizes and onboard those customers in days. To generate deep insights with a more effective reporting process, GoDaddy turned to Virtue Tech to help deliver greater value for its client base by shifting time away from manual data wrangling and towards better analysis and insights.

Read More
Clean Layer Case Study

GoDaddy, the domain registration and management giant, is now migrating their majority of their infrastructure and data warehouse to AWS. GoDaddy group decided to go with AWS due to its deep experience in delivering a highly reliable global infrastructure, as well as an unmatched track record of technology innovation, to support their rapidly expanding business.

The Challenge

GoDaddy is the world’s largest web host by market share, with over 62 million registered domains. This global growth has led to extremely large amount of data being generated across various projects and teams. Multiple teams of GoDaddy wanted to streamline their process of cleaning data before loading it into Amazon S3. This called for a generic configuration based framework to reduce the development efforts, which can be used across the different teams of GoDaddy.

Why Amazon Web Services

Running a cleaning process on more than 40 tables with huge amounts of data requires a scalable and cost-effective infrastructure, which is why Virtue Tech recommended GoDaddy to go with AWS as the capacity provided by AWS is a perfect fit. It gives the broadest and deepest portfolio of purpose-built analytics services optimized for unique analytics use cases. These services are all designed to be the best in class, which means we never have to compromise on performance, scale, or cost when using them. Spark on Amazon EMR runs 3x faster than standard Apache Spark 3.0 and we can run Petabyte-scale analysis at less than half of the cost of traditional on-premises solutions.

Running Critical Applications on AWS

GoDaddy provisions Amazon Elastic Map Reduce (Amazon EMR) cluster to run generic configuaration based Pyspark framework, which has automated the entire process of loading the clean data to AWS Glue. This framework is developed in such a way that user is given option of handle incremental as well historical data for both partitioned and non-partitioned tables. In addition to that, it supports txt, csv, json and parquet input file formats. It also handles slowly changing dimensions (SCD), depending upon the usecase. Column mapping feature of this framework helps the user to map the raw column name from source to new column names in AWS Glue tables, which enables all the stakeholders from GoDaddy to run their SQL like queries and analyze the same on AWS glue and Amazon Athena. Not only is the process of querying now simpler, but the queries themselves also take much less time to complete, since the queries now runs on Amazon Athena, and thus reports takes no time to generate, which earlier used to take lots of time.  

GoDaddy Clean Layer System Configuration Diagram

The Benefits

GoDaddy’s decision to move to a new AWS-based architecture to streamline the process of loading clean data across multiple teams helped the company save on time and money. With an on-premise solution, developing the framework for each team, used to take more than two weeks of time; however, after migrating to AWS and streamlining the process, the process takes hardly a day or two, as only system configuration file needs to be created.  Running framework on Amazon EMR not only made the process 3x times faster but also saved on more than 50-80% of total costs.

Read More
JIRA Case Study

GoDaddy, the domain registration and management giant, is now migrating their majority of their infrastructure and data warehouse to AWS. GoDaddy group decided to go with AWS due to its deep experience in delivering a highly reliable global infrastructure, as well as an unmatched track record of technology innovation, to support their rapidly expanding business.

The Challenge

Jira Software is a project management tool that supports any agile methodology, featured with agile boards, backlogs, roadmaps, reports. This Jira data is gathered over the period as various teams logs and tracks their projects work and issues using Jira Software. Python API calls pulls the data from Jira in the form of nested JSON files into Amazon S3. To analyze this semi-structured data so as to get the business insights form it, was a huge challenge for the GoDaddy analysts

Why Amazon Web Services

Amazon Simple Storage Service (Amazon S3) provides secure, durable, and highly scalable storage for structured and semi structured data and is the preferred storage service of choice for the data lake. Amazon Redshift can also efficiently query and retrieve data from json files in S3 without having to load the data into Amazon Redshift native tables. We can create Amazon Redshift external tables by defining the structure for files and registering them as tables in the AWS Glue Data Catalog. The advantage of this approach is that it is very simple and we are using only native AWS services, which are all closely integrated.

 Running Critical Applications on AWS

Virtue Tech implemented Extract-Load-Transform (ELT) approach to deliver quick results. Python data extraction framework calls the Jira API to extract the data from JIRA and loads the data into Amazon S3 in the form of JSON files without performing any transformations. This raw extraction process is scheduled to run daily through Apache Airflow.  This data is cleaned through the PySpark framework running on Amazon Elastic Map Reduce (Amazon EMR), which cleans the data, converts it into a readable format, and stores the data in parquet format. Once data is cleaned and transformed, we copy the data into redshift for easy reporting and analytics. In case of success or failure, an Email is sent to the respective teams using Amazon Simple Email Service (SES).

GoDaddy Jira Generic System Configuration Diagram

The Benefits

Semi-structured JSON data from Jira was loaded into Amazon Redshift for easy reporting. Using Redshift and Amazon EMR, GoDaddy has greatly improved the ability of internal teams to quickly access and analyze data, allowing for easy scaling as the business grows. With AWS, not only scaling is much more effective, but also query’s performance has improved from hours to mere seconds. This results in richer, real-time analytics that is benefitting all of GoDaddy’s business teams

Read More
GoPay Case Study

GoDaddy, the leading cloud platform dedicated to small, and independent ventures. Helping take millions of ideas into the digital space and providing an all-in-one place to make your digital presence. The company’s cloud platform provides 20+ million customers and entrepreneurs to make their mark on the digital landscape by providing various hosting, website building and ecommerce support.

The Challenge

As E-commerce evolves, so does the needs that new entrepreneurs have. For them to compete with the largest of stores, they too require a payment gateway that would allow convenient online subscription payments. For this, GoDaddy looked to acquire GoPay. It matched the current needs of the intricate web store infrastructure and could provide it to their customers. The challenge was to merge the newly acquired company into an application for GoDaddy’s customers. Re-engineering the existing infrastructure for actionable insights and strategic development.

Key Challenges:  

  • The data was not ready for Business Intelligence & Analytics.
  • The information systems between the client and the company were not compatible forming Data Silos.
  • Adoption of the standard practices of authorization was a requirement set by client.
  • GoPay was hosted on a PCI AWS account.

Why AWS?

Both GoDaddy and GoPay had been using AWS for certain pieces of infrastructures, as it was the best option for those services. Building on top of that Virtue Tech suggested GoDaddy to leverage the large amount of transactional data being created for GoPay and create a bespoke scripting tool for extraction and migration of the data into a newly created data lake. With its unlimited storage capable of storing the terabytes of daily transactional data, the Amazon Simple Storage Service (S3) was used to implement the data lake. S3 is able to provide a secure, durable, and scalable storage for large data collections, and integrates with Amazon’s EMR. EMR is a fully managed extract, transform and load (ETL) service, which is used with PySpark and Amazon Athena to provide data ingestion and processing.  

Benefits of AWS?

  • Swiftly migrated, updating 120 tables in just 2 months.
  • Standard practices of authorization were introduced, reinforcing enterprise grade security throughout the GoDaddy infrastructure.
  • Enabled rapid testing of new tools and technologies, with the increased availability of data.
  • Removal of Data Silos, led to removal of redundant values, providing significant savings in data center costs.
  • Reduced costs for infrastructure maintenance by 50%, while also cutting licensing costs by integrating open-source tools.
Read More
PyDeque Case Study

GoDaddy, the domain registration and management giant, is now migrating their majority of their infrastructure and data warehouse to AWS. GoDaddy group decided to go with AWS due to its deep experience in delivering a highly reliable global infrastructure, as well as an unmatched track record of technology innovation, to support their rapidly expanding business.

The Challenge

GoDaddy, with its web hosting and world’s largest domain name registration business manages over 57 million domains worldwide. It ingests on a daily basis  over 20 terabytes of new, uncompressed data everything,  from website traffic and usage metrics to server management and ecommerce statistics.  All that data is used to configure products and provide client services to its 14.7 million customers which ranges from major corporations to small businesses. This data comes from many known and unknown sources with highly varied formats and disparate meanings and uses. There are conflicts and inconsistent or contradictory phenomena among data from different sources.  In the case of small data volume, the data can be checked by a manual search or programming, even by ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform). However, these methods are useless when processing such huge data volume. This is a great challenge to the existing techniques of data processing quality.

Why Amazon Web Services

Building a data lake in the cloud eliminates the costs and hassle of managing the necessary infrastructure required in an on-premises data center. That’s why VirtueTech recommended GoDaddy to go with AWS Cloud, as it offers even more benefits by virtue of their broad portfolio of services that offer options for building a data lake as well as maintaining the quality of data. That includes Amazon Simple Storage Service (Amazon S3) for storing data in any format, securely, and at massive scale. Deequ, which is used internally at amazon, can be used to define and verify data quality constraints, and be informed about the changes in data distribution.

Running Critical Applications on AWS

GoDaddy is evaluating Data Quality using PyDeequ, an open-source Python wrapper over Deequ (an open-source tool developed and used at Amazon) to make sure that the data used ultimately is relevant and trustworthy. PyDeequ uses Spark to read from sources such as Amazon Simple Storage Service (Amazon S3), and computes data quality metrics such as completeness, maximum, or correlation through an optimized set of aggregation queries. In addition to that, it also checks for constraints that are to be verified. It generates a data quality report, which contains the result of the constraint verification. PyDeequ can also analyze the whole dataset or only its part, and suggest you the validation constraints from there. In case of data quality failure, the downstream process is stopped and an email is sent to the producer team using Amazon Simple Email Service (SES). This report consists of an attachment notifying the team, what has failed, and why it failed.

GoDaddy PyDeque System Configuration Diagram

The Benefits

Improved data quality has led to better decision-making across the organization. Users can easily discover high-quality data in optimized formats, and teams are reporting reduced latency for their analytics results. Incomplete or inconsistent data takes significant amounts of time fixing that data to make it useable. This takes time away from other activities and means it takes longer to implement the insights the data uncovered. Quality data is also helping to keep your company’s various departments on the same page so that they can work together more effectively. This also led to increased profitability of the company, as business insights are now much more reliable and efficient.

Read More
Registry Case Study

GoDaddy, the domain registration and management giant, is now migrating their majority of their infrastructure and data warehouse to AWS. GoDaddy group decided to go with AWS due to its deep experience in delivering a highly reliable global infrastructure, as well as an unmatched track record of technology innovation, to support their rapidly expanding business.

The Challenge

GoDaddy completed its $218 million acquisition of Neustar’s domain name registry business in the year 2018, marking a big entry into the wholesale part of the domain business.

Neustar registry was hosted in Greenplum Data warehouse server and was processing almost two TBs of data every day. The existing database had performance issues and the cost of ownership was too high. Hence, GoDaddy needed a new database solution, which is scalable, easier to maintain and is able to process the huge amount of data.

Considering how critical and busy this data warehouse is, migrating the data warehouse to a new environment seamlessly without any downtime was a challenge for the GoDadddy team.

Why Amazon Web Services

After evaluating a number of solutions, including other cloud service providers, VirtueTech suggested Amazon Web Services (AWS) because of AWS’ highly scalable, fully managed and cost-effective storage and compute services, as well as its ability to accommodate open-source software and hybrid architecture.

Amazon Redshift delivers fast query performance on gigabytes to petabytes of data, and from a few users to thousands. The ability to connect directly from Tableau to a Redshift cluster means letting anyone easily access your cloud data warehouse and quickly perform visual analysis via Tableau’s intuitive drag and drop interface.

Running Critical Applications on AWS

While there were plenty of ways to migrate the workloads to cloud, the cheapest and easiest method was be the lift-and-shift migration. It enabled GoDaddy to enjoy the full benefits of AWS, such as enhanced speed and lower cost of operation

GoDaddy is using Amazon Simple Storage Service (Amazon S3) to store the cloud-based portion of its data lake, AWS Redshift to handle data warehouse operations and Apache Airflow to programmatically author and schedule their workflows and monitor them via the built-in Airflow user interface. End user is notified of failure via slack with the help of AWS managed airflow. Tableau is connected to the AWS Redshift tables through Redshift connector, which is used for report visualization.

The Benefits

By replacing its existing on-premises database with the Amazon Redshift cloud data warehouse, GoDaddy now has a scalable, powerful data solution that is easier to maintain than its previous solution.

Using the AWS Redshift data warehouse has dramatically reduced the time required to obtain registry analysis reports. Analysis performance has also improved since shifting to Amazon Redshift, with faster response and better usability for heavy users.

The AWS cloud has also met GoDaddy’s requirement to maintain operating costs at the same level as with the Greenplum data center.

Read More
How to remove hidden obstacles on your smooth cloud journey?

You kick off your cloud migration journey. You lay out a plan, choose the best suitable strategy, and keep track of vital parameters like the number of applications migrated, performance, etc. However, your core business committee is skeptical about the benefits that the cloud brings. When your stakeholders seem unimpressed with the results that take immense effort and investment, it hints towards unprecedented issues.

Value Gap: the most overlooked Obstacle

Cloud migration is more than just technological advancement. It requires you to figure out a long list of technology topics like account hierarchies, permissions, supplying written scripts, configuring virtual machines.

Initially, migration will quickly yield benefits. It makes applications more robust, reliable failed server recovery, reduces costs due to optimized resource usage, better monitoring, and security. After initial noticeable business value addition, the track of technical progress and business value would diverge. Despite technical migration, the forecasted and actual value will level off. It is a value gap. The Value gap is when your investment and progress in the technological area are not in proportion with generated business value.

Cause of the value gap

Determining the value gap is much easier than turning it into real-life advice. The most common reason is organizational change. Though, everyone is aware of it. However, it fails to meet the pace of technical change.

It includes redefining internal processes, hierarchies, creating new job roles, and utilizing existing silos. Such changes take time which is one of the major causes of the value gap. It is a topic of utmost concern nowadays. Without it, you cannot leverage the maximum benefit of the cloud. For example, suppose your organization is willing to buy a large on-premise server that seems cheaper than a cloud instance. What they forgot to consider is the additional cost that they have to incur over years to come. If your purchasing team is making the comparison without considering scaling, hardware refresh, backup, and operation costs, they are undermining the abilities of the cloud.

So, organizations are rushing towards agile mode, holding digital workshops, and even instating autonomous teams to take up decisions. It appears to be helping though it isn’t. If you cannot define the business value of cloud migration, it is hard to benefit from these. Also, without a seamless software delivery pipeline, you might not achieve the value that you desired at the start.

Three actions to overcome the value gap –

A.   Intertwined org-tech cycle

Better value assessing metrics increases transparency, allows you to make better decisions, and optimize value generation. In turn, it will increase return on investment, maybe a better software delivery pipeline. Lower resistance in software delivery will increase transparency. Such organizational and technical alignment is a must for diminishing the value gap. 

A.    Focusing more on business metrics

Most companies thoroughly access the environment but keeping only technical parameters in mind. Such metrics are good. But they are far away from real success in terms of business. These IT metrics are of use to the IT department only as they lack business oriented-parameters. The executives will be interested in decreased operational costs, better performance in case of a surge in demand, customer satisfaction, and security.

It is good to have technical metrics at the start. Switching to business metrics with time will help in overcoming the value gap.

B.    Practice Quality over quantity.

While starting the migration, you lay down a basic foundation aiming to move a large number of applications to the cloud. Benefits will be visible in terms of cost reduction, increased bandwidth, and better transparency. However, you will not look at automation of application deployment, improving software delivery. Rather than moving application with low business value, migrate applications with more business value potential.    

  Conclusion:

Almost every cloud journey will hit the value gap. Perceive it as a sign of realignment between an organization and IT. Chain of technical and organizational changes is the solution to smoothen your cloud journey and boost your transformation.

Reach out to us at contact.us@virtuetechinc.com to know more about we can be of help in smoothening your Cloud journey.

Read More
RTF – Case Study

GoDaddy, the domain registration and management giant, is now migrating their majority of their infrastructure and data warehouse to AWS. GoDaddy group decided to go with AWS due to its deep experience in delivering a highly reliable global infrastructure, as well as an unmatched track record of technology innovation, to support their rapidly expanding business. 

The Challenge

GoDaddy is committed to continuous innovation, technology, and platform improvements to create a great experience for its customers. Real Time Finance (RTF) team of GoDaddy, wanted to streamline their process of generating reports, which used to happen earlier using json files. While the reports did give this company specific things they needed, but it was taking a good amount of time to generate the reports and also was not much analyst friendly. Thus, the company decided to shift to more cloud-native solution for generating reports.

Why Amazon Web Services

AWS provides a superior global footprint and set of cloud capabilities, which is why GoDaddy selected them to meet their needs today, and into the future. It also enables GoDaddy to accelerate the delivery of its products and services, and easily deploy them globally in minutes, to its customers worldwide.  In addition to that, AWS will enable GoDaddy to leverage emerging technologies like machine learning, quickly test ideas, and deliver new tools and solutions to their customers with greater frequency.

Running Critical Applications on AWS

GoDaddy provisions Amazon Elastic Map Reduce (Amazon EMR) cluster to run Pyspark framework, which has automated the entire RTF process. Column mapping feature of this framework helps the user to map the raw column name from source to new column names in AWS Glue tables. By adding this component to the architecture, company not only preserved their reporting system, but also enabled more user-friendly report generation. This framework is developed in such a way that an incremental data is loaded in the final glue table on a daily basis and thus it gives only the latest records from the system, enabling all the stakeholders from GoDaddy to run their SQL like queries and analyze the same on AWS glue and Amazon Athena. Not only is the process of querying now simpler, but the queries themselves also take much less time to complete, since the queries runs only on delta data, and thus reports takes no time to generate, which earlier used to take lots of time. 

GoDaddy RTF System Configuration Diagram 

The Benefits

VirtueTech advised GoDaddy to switch to a new AWS-based architecture to generate reports for its RTF team. With an on premise solution, the process used to take a large amount of time to generate reports; however, after migrating to AWS, reports not only takes less time but also saves on costs. Analysts also wanted to query their data from SQL like tool, so it will be easy for them to generate reports. Therefore, as a solution, they used Glue/Athena combination for faster and quicker analysis. There are now significantly more opportunities to make data-driven decisions within the organization.

Read More
Advertising Channel Data Mart Migration Case Study

The Challenge

Advertising is important for every aspect of a business, as it lets the business gain more customers, thereby increasing business turnaround. It can be achieved using various media like television, social media, websites etc. 

GoDaddy being an online business, used to collect Data from these sources and store into Apache hive, which was then loaded into GCP Big Query for the transformations and Amazon S3, Amazon Redshift for generating reports. 

Too many hops in this data pipeline was creating dependencies on other platforms like Hadoop and GCP, and also this approach lacked the ability to manage the upscaling of the application. Thus, Virtue Tech suggested a way to decommission the GCP, Hadoop and manage AWS for both transforming and analytical purposes.

Why Amazon Web Services

Lack of capability to use columnar storage for storing metric data in GCP was increasing the storage space and the costs compared to using S3 for storage by and EMR for processing the data.  Amazon EMR makes it easy to set up, operate, and scale your big data environments by automating time-consuming tasks like provisioning capacity and tuning clusters.  We also saved 50-80% on the cost of the instances by selecting Amazon EC2 Spot for transient workloads and Reserved Instances for long-running workloads. Three R5.4 nodes used within EMR cluster costs 3 x $0.25/hour equates to less than a dollar for both tasks during a batch run

Running Critical Applications on AWS

Data from multiple sources such as Google Analytics, Salesforce etc. is pulled into Amazon Simple Storgae Service (S3) in CSV format through APIs. Amazon EMR runs a generic configuaration based Pyspark framework, which has automated the entire process of transforming, cleaning and loading the data into S3 ADS Layer.  Amazon Redshift table is created on top of this S3 ADS Layer. Tableau is connected to this redshift table through Redshift connector, which lets the analysts analyze the data and generate useful business insights

The Benefits

Using AWS, GoDaddy now has access to the scale it requires to deliver a reliable and insightful service to its customers. With its earlier setup comprising of GCP and on-premise, the availability of its system ran to 98 percent, but on its AWS cloud infrastructure, this has risen to 99.965 percent. Running the pipelines on AWS infrastructure has also resulted in saving time as well as money. While, the previous system took almost more than 50 minutes to run the end-to-end flow, however on AWS it hardly takes 20 minutes to complete. Company is also saving almost 3000 USD per month since previous infrastructure included extra storage cost of google cloud storage and on-premise hadoop cluster.  

Read More