This repository hosts a state-of-the-art Online Advertising Platform—a comprehensive big data project with components for efficient ad campaign management, seamless ad delivery, feedback collection, and detailed billing reports.
- Problem Statement
- Idea / Solution
- Dependencies / Limitations
- Future Scope
- Setting up a local environment
- Usage
- Technology Stack
- Contributing
- Authors
It is useful to design and follow a specific format when writing a problem statement. While there are several options for doing this, the following is a simple and straightforward template often used in Business Analysis to maintain focus on defining the problem.
- IDEAL: This section is used to describe the desired or “to be” state of the process or product. At large, this section should illustrate what the expected environment would look like once the solution is implemented.
- REALITY: This section is used to describe the current or “as is” state of the process or product.
- CONSEQUENCES: This section is used to describe the impacts on the business if the problem is not fixed or improved upon. This includes costs associated with loss of money, time, productivity, competitive advantage, and so forth.
Following this format will result in a workable document that can be used to understand the problem and elicit requirements that will lead to a winning solution.
Make sure that we are configuring the EMR cluster properly.
We will need to make sure that we have Hadoop, Sqoop, Hive and Spark installed on wer cluster with Hue as an optional service. Also as an added step, make sure that in the Hardware configuration step for the EMR cluster generation, scroll down to the EBS Root Volume configuration and type the Root device EBS volume size as 20 GB.
As part of the project, broadly we are required to perform the following tasks:
Task 1: Setup MySQL Environment. we need to create "users", "ads" and "served_ads" table in MySQL using the given schema.
Task 2: Writing Ad Manager using PyKafka. Here we need to read the data from Kafka then add the additional attributes followed by MySQL queries and finally printing the Campaign Id, Action & Status on Console.
Task 3: Writing Ad Slot Budget manager using MySQL Connector. Here we need to write a python script for slot budget calculation and updation. Here we also need to set up a proper cron job.
Task 4: Writing Ad Server using Flask & MySQL Connector. Here we need to perform the following task:
- Write the API code using Flask.
- Performing the ad auction.
Task 5: Writing Feedback Handler using Flask & MySQL Connector. Here we need to perform the following major tasks:
- Write the API code using Flask.
- Retrieve the entry from "served_ads" table.
- Add the additional attributes with the user feedback data.
Task 6: Writing User Feedback Writer using PySpark.
Task 7: Backing up Ad data from MySQL to Hive using Sqoop.
Task 8: Report Generator wherein we have to use Hive create table query for user feedback. Also, we need to generate reports based on the following parameters:
- Top 10 under-utilised Ad Campaign
- Top 10 spending Ad Campaign
- Total expenditure and CTR of the Ad Campaigns
- Top 5 Interactive(based on the CTR)
- Top 10 spending Ad Category
- Top Auction price differences.
- What are the dependencies of your project?
- Describe each limitation in detailed but concise terms
- Explain why each limitation exists
- Provide the reasons why each limitation could not be overcome using the method(s) chosen to acquire.
- Assess the impact of each limitation in relation to the overall findings and conclusions of your project, and if appropriate, describe how these limitations could point to the need for further research.
Write about what you could not develop during the course of the Hackathon; and about what your project can achieve in the future.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
What things you need to install the software and how to install them.
Give examples
A step by step series of examples that tell you how to get a development env running.
Say what the step will be
Give the example
And repeat
until finished
Add notes about how to use the system.
- @shinde-chandrakant - Idea & Initial work
See also the list of contributors who participated in this project.