Deb won the Sexiest Female costume, you can see why in the pictures .  It was fun, and we will give it another try next year!

There are also pictures from Brian & Shelley’s house for trick or treat with Avery.  God she is cute.

[Show as slideshow]

My little sister has a non-HD flip video recorder.  These flips record their movies in .AVI format.  I have a mac and use iMovie to create movies for the family put together off of all their individual movies.  I have them set up so their movies can be transmitted to me over the internet to my home server because I am aways away from them.  .AVI files pose a problem to me as iMove will not even recognize that format.  HandBreak however is a wonderful video converter that has a command line interface and can convert the .AVI files to something I can use .m4v files.  So here is the problem.

Problem: I need to processes a large amount of files in a directory.  I need all the .AVI files and I want to process them through a program and have the output file be the same name with a different extension.  I then want the original file removed from the disk.

While this would be a simple problem to solve in any number of programming languages, the end result would be the same.  The program would need to generate a list of .AVI files.  Parse those names, and generate the output names, and then create a command to execute against the shell and run it.  You are introducing a middle man into the equation that does not need to be there.

Bash can very quickly and simply do these steps for us.  And here is how.

First we need to collect all of the files.  In bash we can do this, while generating a loop at the same time.

for input in "\Users\axe\convert_movies\*.AVI"
do
  #
  #processing code for each file will go here
  #
done

This for loop takes all the .AVI files, if any, from the \Users\Axe\convert_movies\ directory and, one at a time, assigns their full file name to the input variable. The “do” and “done” are the start and end of the for loop.

Now we have the input name of the file that we want to convert and it is stored in $input.  We want to take this name and do a little text processing on it to change that name to have  a .m4v extension on it.

  #
  # Generate the Output File name
  #
  output=${input%.AVI}.m4v

This is a simple parameter substitution. We are using the input value of $input and removing the “.AVI” from the end of that variable. Then concatenating on the “.m4v” to the end of the result. This result is placed in the $output variable. Note that we do not use the $ prefix when we are creating a variable in bash, we only use it when we need to access it. There is a great deal that you can do within scripts using parameter substitution, and they are way out of scope for this post so for now I am just going to stick with that simple explanation.

We now have two variables in our script $input contains the name of the input file we want to run the conversion on, and $output which is the name that we want the name converted to.  This is the command for HandBreak, which is what I needed these files run through, but you can substitute any command that you can run from the command line here (ie. cp, rm, mv, zip …).

  #
  # Execute the movie transcode
  #
  /Users/axe/HandBrakeCLI -i $input -o $output

This part is very simple, this is the fully qualified path to the HandBreak command line executable (it is not in my path so I need to give it the fully qualified path name) the -i flag is for the input file and the -o flag is for the output file. If there where any other flags or options that needed to be specified for the HandBreak command they would go in this line, but for me I am using the default behavior here and only need the input and output filenames.

The final part of the script is to delete the original file. I am going to be a little more careful here than just blindly deleting the file, but in honestly I am not being as careful as I really should be. So what I am going to do is check to make sure that the output file name that I constructed actually exists in the file system before I go an delete the original, just in case there was some error in the conversion step.

  #
  # If the output file exists, delete the input file
  #
  if [ -e $output ] ; then
    rm $input
  fi

First off, be mindful of the spaces in this. THEY ARE IMPORTANT. What we are doing here is checking, using the -e command which checks to see if the filename to the right of it exists on the file system and returns true if it does. If it is true, then the inside rm, remove, command is executed on the $input file.

Here is the completed script and the last part of the explanation.

#!/bin/bash

#
# Loop Through all the AVI files in the Source Directory
#
for input in "/Users/axe/convert_movies/*.AVI"
do
	#
	# Generate the Output File name
	#
	output=${f%.AVI}.m4v

	#
	# Execute the movie transcode
	#
	/Users/axe/HandBrakeCLI -i $input -o $output

	#
	# If the output file exists, delete the input file
	#
	if [ -e $output ] ; then
		rm $input
	fi
done

Notice the first line of the script, this line is important for the script to execute. This is line is special and tells the shell to execute all of the following contents in the file using the /bin/bash executable.

This line can be any executable that can be executed in the shel, for example if you made a php script, you could make it executable and add this line, #!/bin/php, to the start of the file and it could be executed by itself.  This is assuming that the php executable file is located in the /bin directory.

I am going to add in a little bonus piece here.  I want to be able to pass in any path that I want to this script and have it execute on it.  So I am going to show you a modification to the above script that allows for an argument to be passed to the script and used to make it a little more flexible.

#!/bin/bash

#
# Set the Source Directory
#
Source=$1'*.AVI'

#
# Loop Through all the AVI files in the Source Directory
#
for input in $Source
do
	#
	# Generate the Output File name
	#
	output=${input%.AVI}.m4v

	#
	# Execute the movie transcode
	#
	/Users/axe/HandBrakeCLI -i $input -o $output

	#
	# If the output file exists, delete the input file
	#
	if [ -e $output ] ; then
		rm $input
	fi
done

The $1 is a special bash variable. This is the first argument after the script. So an example of running this.
[ axe@macbook ~] $ ./converter.sh /Users/axe/convert_movies/

This will run the script, I named it “converter.sh” and made is executable, and it will have the path “/Users/axe/convert_movies/” stored in the $1 variable.

So I hope that this helps, and if there is anything that is not clear, leave a comment and I will try to answer it for you.

Remember : SET BRAIN ON;

This is the most common usage of a join, specifically an inner join.  The results only include the entries that match up from each table.  There are many other situations that you can incorporate a JOIN to solve.  Particularly here we are going to talk about using a JOIN to suppress information from one table by using another.

For this example we are going to use two simple tables.  The first is a KeyTable containing integer keys.  The second is a SuppressionTable containing integer keys as well.  Here are the sample tables with some sample information inserted.

The goal that we have is to get a list of Keys, from the KeyTable that are NOT in the SuppressionTable.  A simple way to accomplish this is using the following SQL.

SELECT KeyTable.Key
FROM KeyTable
WHERE Key NOT IN (SELECT SuppressionTable.Key FROM SuppressionTable)

Here we incorporate a sub query to suppress the Keys.  This method is perfectly acceptable for small amount of information and very simple to understand.  It is not however, an efficient method when you have millions of records in both tables.  In most databases the code around JOIN is very well optimised and if the tables are constructed properly join results can be generated very quickly.  So lets work on an alternative using JOIN.

We can utilise a full outer join to get all the results from each table using the following code

SELECT KeyTable.Key, SuppressionTable.Key
FROM KeyTable FULL OUTER JOIN SuppressionTable
ON KeyTable.Key = SuppressionTable.Key

The table below is the result of the FULL OUTER JOIN.  As you can see from this result set that there are NULL values filled in when there is an entry in one of the join tables that does not have a join value in the other table.

We can use this side effect as a filter to remove the instances where there is no join value.  Adding a condition in the WHERE clause to suppress anything from the SuppressionTable that has a value or a match we come up with the following query

SELECT KeyTable.Key
FROM KeyTable FULL OUTER JOIN SuppressionTable
ON KeyTable.Key = SuppressionTable.Key
WHERE SuppresionTable.Key IS NULL

This query will generate the results that we are looking for.  All of the KeyTable.Key records that do not have an entry in the SuppressionTable.  This is a much more efficient method for suppressing data from large data sets.

An alternative method that you can use is to encorporate a LEFT or RIGHT OUTER join as you only need all the values from one of the tables involved.  Depending on the database that you are using you will need to look up the syntax to use for LEFT or RIGHT outer joins.  So the following query will result in the same output and is more efficient.

SELECT KeyTable.Key
FROM KeyTable LEFT OUTER JOIN SuppressionTable
ON KeyTable.Key = SuppressionTable.Key
WHERE SuppressionTable.Key IS NULL

This query will not pull back the records from the SuppressionTable that we do not need and depending on the overlap can help the overall efficiency of the query.

Since we are using a join and can have multiple conditions in the ON clause we have more flexibility using this method than we would using a sub query.  Well, there we have it, an efficient and quick method for suppressing a large amount of records from a large amount of records.

*All the code to this demo is on the next page along with sample tables and sample data*

The official definition from MSDN is:

PIVOT rotates a table-valued expression by turning the unique values from one column in the expression into multiple columns in the output, and performs aggregations where they are required on any remaining column values that are wanted in the final output.

Lets do a simple example. *Source Code Available on next page* So we have  a table with the following schema.  And for whatever reason it made more sense to load/store in a tall schema.  Maybe there is a lot of necessary meta data that you have to store with the values, whatever the reason was for the decision, this is what you got to work with.

Lets say that for a single company [Company_ID] = 1 we have the following information available in the table.

Tall Schema Table Example

The following is the SQL that will transform the tall schema from above

SELECT * FROM
 (
   SELECT
     Company_ID,
     Company_Column,
     Company_Column_Value
   FROM
     Company
   WHERE
     Company_ID = 1
 ) AS DataSource
PIVOT
 (
   MAX(Company_Column_Value) FOR Company_Column IN
   (CompanyName, Address1, Address2, City, State, Postal, Employees)
 ) AS PVT

and return the wide schema results seen here.

Transformed Wide Schema Result

I will try and explain the parts of the query, I will be going from the inner queries and work my way out.  Starting with the inner top select statement

SELECT
  Company_ID,
  Company_Column,
  Company_Column_Value
FROM
  Company
WHERE
  Company_ID = 1

This gets the dataset that is shown in the tall schema data sample show first in the post.  This is our source data to execute the PIVOT function on.

PIVOT
 (
   MAX(Company_Column_Value) FOR Company_Column IN
   (CompanyName, Address1, Address2, City, State, Postal, Employees)
 ) AS PVT

PIVOT needs to use an aggregate function, I personally recommend MAX, there is only a single value that will be going through the aggregate function so the choice should not matter.  Inside the aggregate is the column that contains the data to be placed in the new columns created.  The FOR is the source for the new column names to be created by PIVOT.  The IN has the actual columns to create.

There may be more available values in the [Company_Column] column than are listed out inside the IN.  You only have to list the columns that you want back in the output.  In this example I have put all the column names that are available, but I could have only selected CompanyName and Address1 and it would not have been a problem.

You can also put values in here that there are no values for.  I could have put a “SomeName” inside the IN clause and a NULL valued column named “SomeName” would have come out with the results.

The top SELECT simply gets all the results from the operation.

So that is PIVOT in a nut shell.  We can get the same results without using PIVOT, this is benificial to understand in case you do NOT have MSSql, or an older version that does not support the PIVOT function.  Below will yield the same results as the PIVOT statement did above.  In fact, this is essentially what is happening behind the scenes when we use the PIVOT function.

SELECT
	Company_ID,
	MAX(CASE WHEN Company_Column = 'CompanyName' THEN Company_Column_Value ELSE NULL END) AS CompanyName,
	MAX(CASE WHEN Company_Column = 'Address1' THEN Company_Column_Value ELSE NULL END) AS Address1,
	MAX(CASE WHEN Company_Column = 'Address2' THEN Company_Column_Value ELSE NULL END) AS Address2,
	MAX(CASE WHEN Company_Column = 'City' THEN Company_Column_Value ELSE NULL END) AS City,
	MAX(CASE WHEN Company_Column = 'State' THEN Company_Column_Value ELSE NULL END) AS State,
	MAX(CASE WHEN Company_Column = 'Postal' THEN Company_Column_Value ELSE NULL END) AS Postal,
	MAX(CASE WHEN Company_Column = 'Employees' THEN Company_Column_Value ELSE NULL END) AS Employees
FROM
	Company
WHERE
	Company_ID = 1
GROUP BY
	Company_ID

So, why use PIVOT when we can accomplish the same thing using a more standardised SQL command set?  For a small example like this, there is no reason to use it.  It is much simpler to write and understand what is happening in this instance by using the CASE statements.  But what if you are performing this operation on 200 columns?  I will have another post up some time in the future on some more advanced uses of PIVOT for data transforms as well as a post on using UNPIVOT.

Well I hope this helped, and I hope that you find it as useful as I have for manipulating your result sets.

* Source code is on the next page *