Assignment Task:
Build a Big Data Processing Pipeline using Databricks and Azure.
You are a data analytics professional supporting a local photography competition. During the competition photographs submitted will be uploaded to a cloud storage container in the cloud, you are tasked with making an automated data pipeline which will periodically scan all images uploaded using AI based cloud services and extract meta data about the photographs submitted to a database which can be viewed by the judges and the organisers.
Build a cloud-based pipeline using components from Microsoft Azure and Databricks with a complementing Visual Dashboard in Power BI.
a) Provide a narrative with screenshots to highlight your proposed solution the components involved and how it works. Also note any other considerations that you have in terms of the pros and cons of your approach.
b) Discuss 5 insights that you can derive from the dashboard, use with appropriate visuals to reinforce your insights.