Aveek Das

Author
Recent Posts

Aveek is an experienced Data and Analytics Engineer, currently working in Dublin, Ireland. His main areas of technical interest include SQL Server, SSIS/ETL, SSAS, Python, Big Data tools like Apache Spark, Kafka, and cloud technologies such as AWS/Amazon and Azure.

He is a prolific author, with over 100 articles published on various technical blogs, including his own blog, and a frequent contributor to different technical forums.

In his leisure time, he enjoys amateur photography mostly street imagery and still life. Some glimpses of his work can be found on Instagram. You can also find him on LinkedIn

View all posts by Aveek Das

Triggering Function Apps from Azure Queue Storage

September 23, 2021 by Aveek Das

In this article, we are going to learn how to trigger Function Apps from Queue Storage in Azure. Function Apps has been one of the most popular cloud services of Microsoft Azure. Function Apps allow users to write code in any language and then execute the code in the cloud. There is no infrastructure to be managed and hence is very flexible for writing and building applications on the go. Every Function App can be triggered in multiple ways, for example, by calling the function URL using an HTTP endpoint or from some other functions in Azure. In this article, we are going to trigger the Function App from Queue Storage in Azure and see how to pass a message from the queue to the Function App.

Getting started with the Azure Queue Storage using Python

September 14, 2021 by Aveek Das

In this article, we are going to talk about the Azure Queue Storage service using python as a programming language. As it goes by the name, Azure Queue Storage is a simple queue and messaging service in Azure that can be used to build queues and process messages in an asynchronous fashion. Queues help to decouple multiple systems and provide the middle layer in a distributed architecture. The Queue Storage service in Azure allows users to connect to and store messages up to 64 KB in size from anywhere in the world. It is exposed by using HTTP or HTTPS endpoints and can contain millions of messages that need to be processed.

Working with Python in Power BI

August 26, 2021 by Aveek Das

In this article, we are going to learn about working with Python in Power BI. Power BI has been adding a lot of features to it recently and one of the most essential was adding Python functionality to it. Users are now able to run Python scripts directly from within Power BI. Python is a great programming language, especially when it comes to the field of data analysis. Most of the machine learning frameworks and data science libraries are written in Python and there is a huge community of open-source programmers that maintain and promote the use of Python along with these libraries.

Implementing Security in Power BI Workspaces using User Roles

August 19, 2021 by Aveek Das

In this article, we are going to learn about implementing and managing security in Power BI Workspaces. Power BI is one of the leading data visualization tools that is used by a plethora of companies around the globe. Power BI has risen to prominence in recent years due to the ease of use and familiarity of concepts with Microsoft Excel, which helped a lot of analysts tutor themselves and learn to work in Power BI. Power BI is now available in both, desktop and the cloud and users can choose to start with anyone. However, the desktop version of Power BI, known as the Power BI Desktop is used mainly for creating simple and complex data models within the Power BI environment and then published to the cloud, known as the Power BI Service.

Embedding Power BI reports in Jupyter notebooks

August 13, 2021 by Aveek Das

In this article, we are going to explore how to embed Power BI reports in Jupyter notebooks. Power BI is one of the most popular data visualization tools in the market that is being heavily used by Data Analysts to visualize existing data and to discover patterns lying in the data. Power BI provides a desktop application to author the reports and it can be deployed to the Power BI Service and consumed by users using a web browser.

An overview of Analytic Functions in SQL Server

August 2, 2021 by Aveek Das

In this article, we are going to talk about the various analytic functions that are supported by the SQL Server Database Engine. As it goes by the name, these are some special functions using which we can execute analytic queries on the dataset and obtain useful results. In comparison to standard SQL queries, sometimes it becomes necessary for the Data Analysts to deep dive into the data more and obtain insights from analytic perspectives. This article will discuss all the analytic functions supported by SQL Server with some details.

An overview of Azure Cognitive Services

July 26, 2021 by Aveek Das

Microsoft Azure has been a leading cloud service provider over the past few years. In this article, we are going to look into an overview of various cognitive services offered by Azure. In simple English, the word ‘Cognitive’ means to involve in intellectual activities, such as thinking or reasoning. Similarly, Azure Cognitive Services are also related to intelligence related to the computer and the way humans behave with these computers. For example, these services allow humans to interact with the machines as if they have the intellectual ability to talk, think and respond to events accordingly.

Learn NoSQL in Azure: Getting started with DocumentDB SQL API

July 7, 2021 by Aveek Das

This article is a part of the series Learn NoSQL in Azure where we will learn about the DocumentDB SQL API in detail. Azure Cosmos DB is a serverless NoSQL database service provided by Azure that is highly available, globally distributed, and responds within the minimum latency. Due to these features, it is a perfect fit for building modern-day applications using the DocumentDB or SQL API of Azure Cosmos DB.

Learn NoSQL in Azure: Diving Deeper into Azure Cosmos DB

June 25, 2021 by Aveek Das

This article is a part of the series – Learn NoSQL in Azure where we explore Azure Cosmos DB as a part of the non-relational database system used widely for a variety of applications. Azure Cosmos DB is a part of Microsoft’s serverless databases on Azure which is highly scalable and distributed across all locations that run on Azure. It is offered as a platform as a service (PAAS) from Azure and you can develop databases that have a very high throughput and very low latency. Using Azure Cosmos DB, customers can replicate their data across multiple locations across the globe and also across multiple locations within the same region. This makes Cosmos DB a highly available database service with almost 99.999% availability for reads and writes for multi-region modes and almost 99.99% availability for single-region modes.

Learn NoSQL in Azure: An overview of Azure Cosmos DB

June 16, 2021 by Aveek Das

In this article, we are going to learn Azure Cosmos DB. This article is a part of the series Learn NoSQL in Azure, where we will explore all the different types of non-relational databases that are supported in Azure at the moment. Azure is one of the most popular public cloud platforms that has a big market share all over the world. Cosmos DB is a part of the Databases section in Azure that allows customers to create and use NoSQL or non-relational databases and consume these at scale. You can leverage Cosmos DB to build highly scalable and robust cloud-based applications that support modern big data workloads. Let us understand more about what a NoSQL database is all about and how it is different from a relational database. Although this article focuses on the NoSQL related to Azure, it is to be known that other open-source projects support NoSQL databases like Apache Cassandra, etc. However, these topics are out of the scope of this article and we will focus on Azure mostly.

An overview of PGAdmin – PostgreSQL Management Tool

June 10, 2021 by Aveek Das

In this article, we are going to learn about PGAdmin, a PostgreSQL management tool. As you are aware SQL Server Management Studio (SSMS) and the MySQL Workbench are the GUI management tools for SQL Server and MySQL respectively. Similarly, in order to manage the Postgres database and its services, PGAdmin is used. PGAdmin is a web-based GUI tool used to interact with the Postgres database sessions, both locally and remote servers as well. You can use PGAdmin to perform any sort of database administration required for a Postgres database.

Install and upgrade PostgreSQL to support Spatial Data

May 26, 2021 by Aveek Das

Spatial data deals with data related to geography. In this article, we are going to understand the various concepts related to geographic or spatial data and how PostgreSQL can be leveraged as a database to store such geographic information. As you might be already aware, PostgreSQL is a popular and widely used open-source relational database management system that can handle production workloads very easily. With the availability of the cloud, you can quickly spin up instances of Postgres on major public cloud providers like AWS, Azure, GCP, etc.

Working with Regular Expressions in PostgreSQL

May 14, 2021 by Aveek Das

In this article, I am going to talk about using regular expressions in a Postgres database. Regular Expressions, also known as RegEx are pattern matching criteria that can filter data based on the pattern. It is heavily used to match string values to a specific pattern and then filter the results based on the condition. From a beginner’s perspective, these regular expressions can seem to be quite complex in the first, however, as you will start using these on a daily basis, you will come to the underlying logic, and then you can start writing your own RegEx statements.

An introduction to Power BI Dataflows

May 7, 2021 by Aveek Das

In this article, we are going to understand what the Power BI Dataflows is all about and how we can get started by building dataflows in Power BI Service. A dataflow can simply be considered as an extract transform and load pipeline that can be used to connect to source data, transform the data by applying business rules, and then finally preparing the data to be available to visualize. In a general data architecture, a dedicated ETL tool is used to prepare and transform the data, which is then loaded into a data warehouse. Power BI was used to connect to this data warehouse and visualize the data from here on.

How to set up and run SQL Server Docker image

April 28, 2021 by Aveek Das

In this article, I am going to discuss in-depth setting up docker and running a SQL Server Docker image. SQL Server, as you might already be aware that it is one of the most popular relational database technologies in today’s world. SQL Server is widely used in various applications that support transactional and analytical workloads. Docker, on the other hand, is a containerization technology using which you can bundle your applications within a container and distribute them. Docker helps users to build applications independent of the underlying operating system. We will learn more about Docker and using the SQL Server Docker image in this article.

Setting up a PostgreSQL Database on Mac

April 23, 2021 by Aveek Das

In this article, I am going to discuss different ways in which you can install and setup Postgres Database on a Mac. Postgres is an open-source relational database system that can be used to develop a wide variety of data-based applications. Postgres has been popular for analytical workloads as well since it has support for column-store index and in-memory storage as well. Postgres is also available on all the major public cloud services like AWS, Azure, and GCP. In order to use those services, it is recommended that you should also have them installed on your local machine before deploying your databases to the cloud directly.

Spinning up MySQL instances on RDS using CloudFormation Templates

April 20, 2021 by Aveek Das

In this article, we are going to discuss how to set up a MySQL instance on AWS RDS using Cloud Formation templates. In my previous article, How to configure an Amazon RDS environment for MySQL, I have provided a detailed walkthrough of how to set up a MySQL instance on Amazon. You can use the AWS console to provide all the information required for setting up the instance and then use it. However, in this article, we will discuss an automated way of achieving the same functionalities using Cloud Formation templates.

Introduction to Apache Spark

April 12, 2021 by Aveek Das

In this article, I am going to discuss Apache Spark and how to create robust ETL pipelines for transforming big data. I will start from the very basics of Spark and then provide details on how to install Spark and start building the pipelines. In the later part of the article, I will also discuss how to leverage the Spark APIs to do transformations and obtain data into Spark data frames and SQL to continue with the data analysis.

Getting started with Amazon Athena and S3

April 7, 2021 by Aveek Das

In this article, I am going to discuss Amazon Athena and how we can analyze data stored in S3 using Athena. As you might know, Amazon’s AWS has a lot of services in the field compute, databases, analytics, machine learning, and robotics, one of the most important and popular services is Amazon Athena. By the official definition, “Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.”

Understanding args and kwargs arguments in Python

April 2, 2021 by Aveek Das

In this article, I am going to talk in detail about the functions and arguments in Python. Python is one of the most popular and in-demand programming languages. Recently, a lot of programmers are gaining interest to work with python and as such, there is a huge community around it that is constantly evolving. Python is also considered to be one of the most flexible languages as it can be used to develop web-based applications, REST APIs as well as can also be used significantly in the scientific computation world to deal with data analysis and machine learning.

Working with JSON data in Python

March 30, 2021 by Aveek Das

In this article, I am going to write about the various ways we can work with JSON data in Python. JSON stands for Java Script Object Notation and has become one of the most important data formats to store and transfer data across various systems. This is due to its easy-to-understand structure and also because it is very lightweight. You can easily write simple and nested data structures using JSON and it can be read by programs as well. In my opinion, JSON is much more human-readable as compared to XML, although both are used to store and transfer data. In modern web applications, by default JSON is being used to transfer information.

Setting up a Machine Learning environment using R and RStudio

March 23, 2021 by Aveek Das

In this article, I am going to introduce a few concepts of how to set up and get started with R and RStudio to perform machine learning workloads. It has always been the heat of the discussion on whether to choose Python or R for performing Machine Learning analysis. In my opinion, both the languages excel in their own space and there is no point-to-point comparison between the two directly. Mathematicians and statisticians like to work within the R environment, while programmers choose to work with Python.

An introduction to AWS Athena

March 19, 2021 by Aveek Das

In this article, I am going to introduce AWS Athena, a service offered by Amazon which allows users to query data from S3 using standard SQL syntax. AWS is considered to be a leader in the cloud computing world. Almost more than a hundred services are being offered by Amazon which offers competitive performance and cost-effective solutions to run workloads as compared to on-premise architectures. The services offered by Amazon range widely from compute, storage, databases, analytics, IoT, security, and a lot more. One of the popular areas of these services in the Analytics domain. This allows the customer to build architectures that answer key questions to their business decisions.

Create REST APIs in Python using Flask

March 12, 2021 by Aveek Das

In this article, I am going to explain what a REST API is all about and how to get started with creating APIs in Python using Flask. In the recent software world, REST APIs play a major role as a communication channel between different services. It has become the de facto standard of passing information across multiple systems in the JSON format. This is because it has a uniform interface to share messages across two different systems. Let us learn more about REST APIs in this article.

SQLShack