Importing, exporting

Sifiso W. Ndlovu
Error at OLE DB Destination component during data migration using SSIS's Data Flow Task

Dynamic column mapping in SSIS: SqlBulkCopy class vs Data Flow

February 14, 2020 by

The Data Flow Task is an essential component in SQL Server Integration Services (SSIS) as it provides SSIS ETL developers with an ability to conveniently extract data from various data sources; perform basic, fuzzy to advance data transformations; and migrate data into all kinds of data repository systems. Yet, with all its popularity and convenience, there are instances whereby the Data Flow Task is simply not good enough and recently, I got to experience such inefficiencies. To demonstrate some of the limitations of SSIS’s Data Flow Task, I have put together a random list of Premier League’s leading goal scorers for the 2019-2020 season.

Read more »
Rajendra Gupta
Importing compressed data into SQL Server: Add an OLE DB destination

Importing Data into SQL Server from Compressed Files

March 5, 2019 by

Introduction

I have seen many organizations receive data from various sources and import into SQL Server. You might receive data in various formats and want to import into SQL Server. We can prepare a ETL (Extract-Transform-Load) process to import data into the SQL Server. In doing so, might receive data in a compressed file, which helps to send data over the network using a ZIP file format because it reduces the file size significantly. If we are receiving a ZIP file to import into SQL Server, we need to unzip it and then only we can import data. We might need to create a ZIP file as well from the existing files.

Read more »
Rajendra Gupta
Execcute and monitor status of SQL Server Python SSIS package

Using Python SQL scripts for Importing Data from Compressed files

March 4, 2019 by

Using Python SQL scripts is a powerful technical combination to help developers and database administrators to do data analytics activities. Python provides many useful modules to perform data computation and processing of data efficiently. We can run Python scripts starting from SQL Server 2017. We can create the ETL solutions to extract data from various sources and insert into SQL Server.

Read more »
Rajendra Gupta

Export SQL Server FILESTREAM Objects with PowerShell and SSIS

February 14, 2019 by

In this series of articles on SQL Server FILESTREAM (see TOC at bottom), we explored various ways to store unstructured data in the file system with the metadata in SQL Server tables. If we have a large number of objects in the file system, it is advisable to use the fast disk for storage purpose. It is faster and provides better IO in comparison with the traditional file system.

Rajendra Gupta

Importing SQL Server FILESTREAM data with SSIS packages

February 1, 2019 by

Initial configuration

We have been exploring the SQL Server FILESTREAM feature in this ongoing series of articles. In this previous article, Managing data with SQL Server FILESTREAM tables, we wrote about inserting FILESTREAM data into a FILESTREAM table and performing DML activities on it. Suppose we have created the FILESTREAM database in our instance and now we want to insert a large number of files into a FILESTREAM container. It is easy to write out the insert queries for a small number of files, but if the numbers of files were in huge quantity, it would be difficult to write out the code and insert data into it. It is difficult to manage such kind of requests regularly in the environment.

Read more »
Ben Richardson

Importing and Working with CSV Files in SQL Server

January 28, 2019 by

Introduction

CSV (comma separated values) is one of the most popular formats for datasets used in machine learning and data science. MS Excel can be used for basic manipulation of data in CSV format. We often need to execute complex SQL queries on CSV files, which is not possible with MS Excel. See this article for what is possible with Power BI.

However, before we can execute complex SQL queries on CSV files, we need to convert CSV files to data tables.

Read more »
Timothy Smith

SQL Bulk Insert Concurrency and Performance Considerations

January 18, 2019 by

One of the challenges we face when using SQL bulk insert from files flat can be concurrency and performance challenges, especially if the load involves a multi-step data flow, where we can’t execute a latter step until we finish with an early step. We also see these optimization challenges with constraints as well, as fewer steps to complete a data flow results in saved time, but possibly less accurate data.

Read more »
Esat Erkec

How to use JSON data in SSRS

November 6, 2018 by

In this article, we will explore the concept of using JSON data in SQL Server Reporting Services (SSRS). This usage concept will include a different approach than the usual methodologies because we will take advantage of SQL Server R service support.

What is JSON?

JSON is an abbreviation for JavaScriptObjectNotation. The main purpose of JSON data was to provide data transfer between server and web applications for JavaScript but today’s JSON is the most popular data interchange format. JSON offers two advantages to us. These are; light-weight text based format easily readable by humans. Currently, JSON is not only used in JavaScript applications, it is also used in all popular software programing languages like JavaScript, C++, C#, Perl, Java, Python, Php etc.

Read more »
Prashanth Jayaram

The BCP (Bulk Copy Program) command in action

August 13, 2018 by

There are various methods available for bulk data operations.

  1. BCP utility
  2. BULK INSERT
  3. Using OPENROWSET
  4. Import/Export wizard

The BCP (Bulk Copy Program) utility is a command line that program that bulk-copies data between a SQL instance and a data file using a special format file. The BCP utility can be used to import large numbers of rows into SQL Server or export SQL Server data into files. The BCP data files don’t include any schema details or format information. Hence, it is recommended to create a format file to record the data format so in case of any failures, you can refer to the format file and better understand the data format to determine what may have gone wrong..

We’ve been using the BCP tool for a long time, the reason being that it has a very low overhead, and works great for bulk exporting and importing of data. It is one of the most efficient ways to handle bulk import and export of data.

Read more »