Posts Tagged ‘archive’

Setting up an automated ad monitoring service for TV

Friday, October 26th, 2012

So you want to set up your own automated advertisement monitoring for some TV channels?  And you probably have an idea how to sell the reports from the whole system?  Let me try to explain one of the possible ways of doing it.

Overview

Advertisement monitoring system isn’t so complicated, but it isn’t simple either.  You’ll need computers, people, and some kind of service to automatically track advertisements that are spotted once.

Recording

For starters, you have to be able to record all your needed TV channels.  Depending on the TV system used in your country, you’ll have several options for it.  From our shop, we can solve recording for analog tv, DVB-T, DVB-S, IPTV.  In any case, if you can get composite video signal from your set-top box, you will be able to record it with VideoPhill Recorder.

Storing and archiving

Recorded broadcast should go to some storage, depending on the number of days that you want your broadcast archive to be available.  To calculate how much storage space you will need for it, you can use this on-line calculator.

Clipping and tagging

So now we have recordings of the TV broadcast.  Next step is to form a team of people who will find and tag the first occurrence of an advertisement.  Number of people and workstations required for the job depends on many factors:

  • number of channels monitored
  • channel ‘difficulty’ (how easy is to find commercials on the channel)
  • number of shifts that people will do

In short, you’ll need some way of accessing the archive and clipping the portions of it in order to have clips of advertisements extracted and prepared for automated archive search.

One possible way of doing the job is by using VideoPhill Player application.  To see it in action, please see video below…

Automated search

Almost there…  Now, you have your archived broadcast, and you have your clip library.  To find all of the occurrences of all clips on all your channels, you’ll simply pass whole archive and clip library to a PlayKontrol Service and get your results.  Results can be in any format that you require, such as text, excel, PDF, XML, and so on.

Producing reports for your customers

Really final component of the system (apart from selling the reports) is a team of people who will use raw data that PlayKontrol will provide and produce nice reports for your customers.  People on this job should be able to understand the needs of the media buyers and planners, and generate the reports that would be useful for them.

Creating a small (hopefully usable) utility: File Deleter

Wednesday, January 11th, 2012

When you are in media monitoring, you have TONS of files.  For example, look at this:

Multitude of files, StreamSink archive

Another bunch of files, created by PlayKontrol

Every recorder and logger and input produces a number of files on your system.  Of course, each application such as VideoPhill Recorder or StreamSink have the option for deleting a files after they expire (one month for example), but what if you have other way of gathering information (media, metadata, something) that won’t go away by itself?  I have several such data sources, so I opted to create a MultiPurposeHighlyVersatile FileDeleter Application.  I’ll probably find a better name later, for now lets call it ‘deleter’ for short.

The Beginning

The application to delete files must be such a triviality, I can surely open Visual Studio and start to code immediately.  Well, not really.  In my head, that app will do every kind of deleting, so let’s not get hasty, and let’s do it by the numbers.

First, a short paragraph of text that will describe the vision, the problem that we try to solve with the app, in few simple words.  That is the root of our development, and we’ll revisit it several times during the course of the development.

Vision:

‘Deleter’  should able to free the hard drive of staled files (files older than some period) and keep the level of hard drive space at some pre-determined minimum.

Here, it’s simple enough that I can remember it, and I’ll be able to descend down from it and create the next step.

The Next Step

For me, the next step (let’s say in this particular case) would be to try and see what ‘features’ does the app have.  The only way it works for me is to create a mock of the application UI and write down the things that aren’t visible from the UI itself.  Since this UI won’t do anything but gather some kind of parameters that will define behavior of the app, it will be a simple one, and it will be possible to fit it nicely on one screen.

For the sketch I’ll use Visual Studio, because I’m most comfortable with it.  If it wasn’t my everyday tool, I’ll probably use some application such as MockupScreens, which is completely trivialized app sketching gadget with powerful analyst features.

The process of defining the UI and writing down requirements took some time, I repeatedly added something to UI, then to the list below, until I had much clearer picture what I’m actually trying to do.

Features:

  • it should have ability to delete only certain files (defined by ‘mask’ such as *.mp3)
  • it should be able to delete file by their age
  • it should be flexible in determining the AGE of the file:
    • various dates in file properties: created, modified, accessed
    • by parsing the file name of the file
  • it should be able to delete from a multiple directories
  • it should be able to either scan directory as a flat or dig into subdirectories
  • it should be able to delete files by criteria other than age
    • files should be deleted if their total size exceeds some defined size
      • in that case, other files should be taken into account, again by mask
    • files should be deleted if minimum free drive space is less then some defined size
    • file size
  • when deleting by criteria other than file age, specify which files should be first to go
  • should be able to support multiple parameter sets at one time
  • should run periodically in predetermined intervals
  • should be able to load and save profiles
    • profiles should have names
  • should disappear to tray when minimized
  • should have lowest possible process priority
And here’s the screen:

Mock of the Deleter UI used to define and refine the requirements

As you look at the UI mock, you’ll see some mnemonic tricks that I use to display various options, for example:

  • I filled the textboxes to provide even more context to the developer (myself with another cap, in this case)
  • I added vertical scrollbars even if text in multi-line textboxes isn’t overflowing, to suggest that there might be more entries
  • for multiple choice options I deliberately didn’t use combobox (pull down menu) – I used radio button to again provide visual clues to various options without need for interaction with the mock

From Here…

I’ll let it rest for now, and tomorrow I’ll try to see if I can further nail down the requirements for the app.  From there, when I get a good feeling that this is something I’m comfortable with, I’ll create a object interface  that will contain all the options from the screen above.  While doing that, I’ll probably update requirements and the UI itself, maybe even revisit The Mighty Vision above.

BTW, it took me about 2 hours to do both the article and the work.  I excluded my wandering around time, of course :)

Testing 3rd party stream capture application

Monday, January 2nd, 2012

This is a response to a question from one of my prospects, and it can be summarized as:

Why should I buy StreamSink at $10.000
when there is Replay A/V that can do same
thing for $100 (if I buy 2 licences for 
2 computers)?

I can make several objections to the idea of having a consumer product in use for business purpose, but instead of that, I’ll try to focus on functionality (at least for this posting).

Purchasing and installing

I quickly purchased Replay A/V for $50, and went on to installing it.  Upon installation, it offered to install WinPopcap (to provide stream discovery) and some other utility for conversion of the saved material.  I declined.

Entering stations

Once installed, I will try to copy my stream list into it and have it record it continuously.

After some investigation, I found out that there is no way to insert the list of the stations at once, so I’m going to enter them one by one.

OK, I entered Antena Zagreb with its stream URL, and went on to fiding the start button for it.  I found it under context menu for the item that was on the list (right click, start-recording, …).

I remembered that I went through the options for a channel and found that you have to explicitly have to enter the option for splitting the file into segments, so I went on and did that.

I’ll leave it run now and will move on to enter the rest of the stations.

I was about 1/4 way down the list, then I got to WMA stream, and was really curious whether it will be accepted, since there is nowhere a option to pick a stream type.  It was, and for now, it seems that it’s captured normally.

When I am entering the data into the software, and it does its file splitting at every 5 minute intervals, whole GUI freezes and becomes unresponsive for 2-3 seconds.  What I am interested in is whether there will be a gap in the recording of the station that is cut.  BTW, the computer I am doing the analysis at isn’t so weak…

Also, it seems that I entered a stream that doesn’t exists.  Application is persistent in trying to connect to it, but while doing so, it freezes again for few seconds.  However, it’s nothing to be alarmed about.

I also found out that in order for the app to be persistent about recurrent connecting, it has to be additionally configured, as it is not the default mode of the operation.

OK, so I finally entered all the stations.  It gets rather annoying after few minutes, because on the 5 minute chunk interval, app gets its freezing moments rather frequently, and despite the fact it doesn’t pose a problem AFTER everything is entered, it really is annoying.  Here is the filled up application:

Testing the recorded stuff

To do that, I will first share the folder with recordings so I would be able to see it from another (this) machine.

As expected, every channel is saved in its appropriate folder:

Now, let’s examine the contents of some folders that are recorded here…

First folder I have is Antena Zagreb, and here it is:

I won’t comment file naming now, but will tell you what happened when I double-clicked .m3u file that should have the list of mp3 files that are recorder. Winamp loaded it and CRASHED my machine completely. I don’t say it will crash yours, but my Winamp, when faced with certain media files that it can’t recognize, goes berserk. The problem here lies in the fact that Antena Zagreb has AACPLUS stream, and it was interpreted erroneously, creating mp3 files that crashed the Winamp. Here is one file for you to try, use it on your own risk.

Antena Zagreb Jan 02_05

Media Player crashed as well, but I could END it, with Winamp I had to restart the whole machine.

Last test I want to do in this post is to see if the subsequent files are saved so there is no gap between.  For that, I have to find a mp3 file that won’t actually break my player.

Found it, and had no luck.  Even with pure mp3 files, Winamp gives up and puts its legs in the air.  Tested the same with Media Player, and it seems that recordings overlap by few seconds, so that checks out.

Before conclusion, let’s just take a look at resource usage of the application:

Conclusion

You might be able to use Replay A/V for your media monitoring purposes, and save great deal of money.  However, please note that:

  • I didn’t find any option for error reporting (which will enable you to see that the stream is off-line for extended time)
  • if all the channels would cut the file at the same time, it would create unresponsive app for at least 2*number_of_channels seconds
  • CPU usage profile is minimal, however I just found out that memory usage rises LINEARLY over time, and that would lead to immanent application death after some time (you do the math)
  • I didn’t use scheduler to create persistent connections, if I would, and am having bad connection with lots of breaks, app would be nearly impossible to use due to freezing upon connection
  • there is no (or I wasn’t able to find it) option for renaming the files so they would use some time-stamped names
  • it doesn’t provide support for VideoPhill Player, which is a archive exploration tool created just for Media Monitors

Additional info…

After several hours (around 6) this is the memory usage that is taken using Procexp.

For those that can’t read memory usage graph, this means that the application has a memory leak, and by this rate, it would exhaust its memory in less then 24 hours, since it is x86 process.  Quick remedy for that would be to raise the interval for the file cutting, because I suspect that memory leak occurs at that time.

And then, I got in… (story of data visualization)

Thursday, December 15th, 2011

How to see the data?

If the data is numeric, and it represents some series, it will be mostly represented with a graph of some sort.  There are hundredths types of graphs available, and they all have some purpose, otherwise they would not exist.

However, for some special occasions, you have to see different kind of data.

The problem (this particular instance)

Since I am developing a internet media streaming CAPTURE and ARCHIVE application (StreamSink) I am also continuously testing it on one of my servers.  I am adding channels, removing them, stopping the server, sometimes something goes wrong and the whole thing freezes or crashes, so the archive I have is rather heterogeneous in quality.

Let me go through the operational view – the mere GUI of the StreamSink, so I can present some problems and solutions so far.

StreamSink

Several things were important to the operator of the software that had to be present on the main (status) screen.  For example:

  • whole list of channels should be visible
  • channel status should be visible at first glance
  • I am interested what happened to the system recently
  • I need to know the status of my connection
  • it would be good to know how many disk space is available

I could dwell on it but the main point of this post is something else.

The problem here is that I had to create PlayKontrol report for a demonstration purpose (for them: http://ihg.hr/), that would scan 7 days of the archive (multiple channels, of course), and produce the reports (playlists) for 300 songs.

So the problem is: to

find, in the archive that is damaged in various ways, 7 days of continuous archive that spans multiple channels.

The solution (prelude)

Since I am kind of explorer by nature, I wasn’t inclined to use a solution that would present raw data as an answer, but was into thinking about seeing the data and determining the period and channels ‘visually’.

StreamSink has a integrated feature that is called ‘archive report’, that has data similar to what I need, but with it I would only get limited information.  You can see the report here:

StreamSink Archive Report

Most useful info on the report in this particular situation would be the graph on the right side of the report.  Let me explain…

For each day StreamSink is able to record up to 24 hours of media.  Due to network situations, it sometimes is less then 24 hours, and I decided that I would present that number in the form of percentage that archive is covered for the day.  As you can see from the report, that percentage is shown for the whole archive lifetime, for last month, last week and last 24 hours.

Also, it is shown in the form of graph, where on the leftmost part of the graph is the current day, and as we go to the right, we sink onto the past, having divider lines at each 7 days.  Nice, eh? :)

But, as nice as that report is, I can’t read what 7 days and what channels are to be scanned – I have to find another way in.

 Solution (at last)

For this one, I picked something that I learned from the above mentioned report.  That was:

  • I will have a channel list
  • I will have some sort of calendar
  • I have to see how much is covered for the archive for each day

Also I decided to show each day as a cell in a table-style matrix, where rows would be occupied by channels, and columns will be days.  Time flow was inverted here, so left is past, and right is the present.

Whole thing looks like this:

Archive Digger

Same thing little zoomed in:

Archive Digger Detail

Note: green is the color for the days that have 90% or more archive covered.

At last, you can see from the both pictures that much of the data is revealed at the first glance. For example, 0 means that there were no archive that day at all. Numbers below 90 suggest that either it was some problem with the channel that day, or StreamSink was either started or stopped in the middle of the day.

I could even color-code that information on the chart – but the utility will be expanded further only if there’ll be demand for it, since I know what I needed to know, from it.

BTW, I don’t want to brag here, but to code that utility it took 2-3 hours of thinking and coding, and almost no debugging.  It’s most probably due to fact that I’m doing that stuff over and over again for some years :)