Posts Tagged ‘video logging’

Finally – an Osprey Alternative

Wednesday, October 1st, 2014

For years, video capture, at least for media monitoring companies, was dependent on Osprey capture cards.  They are the best there are in the field, and once you try it, you don’t look for anything else anywhere else.  You just pay the price and are satisfied with it.  The card has excellent drivers with tons of options, SimulStream as an (paying) option, …  real real beauty.

However, as we said above, it is pricey.  For Osprey 460e, you need to hand out about $1200 USD.  That’s $300 per channel.

Now, click here:

http://www.vd-shop.de/simultaneously-capture-d130fpsinput-interface-rcabnc-inter-p-591.html

YES!  6 channels for 320 Euro ($400 USD).  I won’t calculate per channel price here, since it is already obvious that Osprey is beaten, at least as price is concerned.

In fact, lets see, on a setup of say 24 channels, how much do you save using new cards:

Osprey: 6 cards, $7200
VCAE: 4 cards, $1600

So only on capture hardware, you could save $5600.  Add to that lower cost of hardware (servers) since you can pack everything in lesser amount of PCs.

So, to follow up on the excitement of finding that this card exists, I immediately ordered a sample to try it with our capture software.  It came in few days, and we went on and installed it…

And the story has to end here, since card works as it should out of the box, enabling media monitoring installations to be even cheaper now.  Not only that, a card has an interesting form factor and low consumption, and will prove ideal in multiple channel scenarios.

I will update the article with 24/7 testing in real world, as soon as we make an installation that has such properties.

Creating a small (hopefully usable) utility: File Deleter

Wednesday, January 11th, 2012

When you are in media monitoring, you have TONS of files.  For example, look at this:

Multitude of files, StreamSink archive

Another bunch of files, created by PlayKontrol

Every recorder and logger and input produces a number of files on your system.  Of course, each application such as VideoPhill Recorder or StreamSink have the option for deleting a files after they expire (one month for example), but what if you have other way of gathering information (media, metadata, something) that won’t go away by itself?  I have several such data sources, so I opted to create a MultiPurposeHighlyVersatile FileDeleter Application.  I’ll probably find a better name later, for now lets call it ‘deleter’ for short.

The Beginning

The application to delete files must be such a triviality, I can surely open Visual Studio and start to code immediately.  Well, not really.  In my head, that app will do every kind of deleting, so let’s not get hasty, and let’s do it by the numbers.

First, a short paragraph of text that will describe the vision, the problem that we try to solve with the app, in few simple words.  That is the root of our development, and we’ll revisit it several times during the course of the development.

Vision:

‘Deleter’  should able to free the hard drive of staled files (files older than some period) and keep the level of hard drive space at some pre-determined minimum.

Here, it’s simple enough that I can remember it, and I’ll be able to descend down from it and create the next step.

The Next Step

For me, the next step (let’s say in this particular case) would be to try and see what ‘features’ does the app have.  The only way it works for me is to create a mock of the application UI and write down the things that aren’t visible from the UI itself.  Since this UI won’t do anything but gather some kind of parameters that will define behavior of the app, it will be a simple one, and it will be possible to fit it nicely on one screen.

For the sketch I’ll use Visual Studio, because I’m most comfortable with it.  If it wasn’t my everyday tool, I’ll probably use some application such as MockupScreens, which is completely trivialized app sketching gadget with powerful analyst features.

The process of defining the UI and writing down requirements took some time, I repeatedly added something to UI, then to the list below, until I had much clearer picture what I’m actually trying to do.

Features:

  • it should have ability to delete only certain files (defined by ‘mask’ such as *.mp3)
  • it should be able to delete file by their age
  • it should be flexible in determining the AGE of the file:
    • various dates in file properties: created, modified, accessed
    • by parsing the file name of the file
  • it should be able to delete from a multiple directories
  • it should be able to either scan directory as a flat or dig into subdirectories
  • it should be able to delete files by criteria other than age
    • files should be deleted if their total size exceeds some defined size
      • in that case, other files should be taken into account, again by mask
    • files should be deleted if minimum free drive space is less then some defined size
    • file size
  • when deleting by criteria other than file age, specify which files should be first to go
  • should be able to support multiple parameter sets at one time
  • should run periodically in predetermined intervals
  • should be able to load and save profiles
    • profiles should have names
  • should disappear to tray when minimized
  • should have lowest possible process priority
And here’s the screen:

Mock of the Deleter UI used to define and refine the requirements

As you look at the UI mock, you’ll see some mnemonic tricks that I use to display various options, for example:

  • I filled the textboxes to provide even more context to the developer (myself with another cap, in this case)
  • I added vertical scrollbars even if text in multi-line textboxes isn’t overflowing, to suggest that there might be more entries
  • for multiple choice options I deliberately didn’t use combobox (pull down menu) – I used radio button to again provide visual clues to various options without need for interaction with the mock

From Here…

I’ll let it rest for now, and tomorrow I’ll try to see if I can further nail down the requirements for the app.  From there, when I get a good feeling that this is something I’m comfortable with, I’ll create a object interface  that will contain all the options from the screen above.  While doing that, I’ll probably update requirements and the UI itself, maybe even revisit The Mighty Vision above.

BTW, it took me about 2 hours to do both the article and the work.  I excluded my wandering around time, of course :)

Having NAS is great (or is it?)

Saturday, December 17th, 2011

During the years I had many deliberations over the fact if either NAS would be used or it wouldn’t be used for the video archives created by video logger system such as VideoPhill Recorder.  At first, I was firm believer in one methodology, then completely turned my side to the other, and now, …  Well, read on, and I’ll take you through it.

Stakes

On the one side of the stake set we have actual requirements, and on the other there are considerations.  Actual requirement are sometimes hard to pinpoint at first, but they always come out sooner or later.

So, let me list possible requirements that might be in effect here…

Common low level requirements

Storage for video recording (for logging purposes) needs to have following abilities:

  • low but constant and sequential write rate – data rate for 4 channels are as low as 5mbit per second (500kbytes/sec) but is CONSTANT and SEQUENTIAL – there won’t be much stress for the hard drive because of constant seeking
  • high durability over time – what gets written once, has to be there.  It should survive single drive failure
  • reading isn’t common, but when done, it has to be sustainable, but again at low data-rate and great predictability (it usually is sequential)
From everything above, I can guess that any data storage expert would read RAID 5 and won’t allow you to create anything else for the video archive storage.

Archive duration scalability

The archive duration is directly proportional with the hard drive space that is available.  To determine what kind of hard drive space you need for your first installation, you can use on-line hard drive size estimator calculator that I created right for this blog.

If you plan extend the duration of your archive one day, you have this requirement, and you have to plan for it.  Having the storage at one place can simplify the adding of the drive space, but can also completely block it.

Let’s say that you have the archive of 40 channels that span 92 days (3 months).  And let’s say that you decided to use 1mbit video with 128 kbit audio for the archive.  By using the calculator above, you’ll find out that you have 42 terabytes of storage already in place.  Even at this date, that kind of storage set in one place is kind-of-a challenge to build.

If you have already invested in 42 TB storage system, and have foresight to plan for an upgrade to say its double size for it, you are in luck.  But, say that after a few more months (just 3) your management decides to expand the requirement to 12 months of storage.  Wow.  Now, you have to have 127 TB total.  If the current system will hold that much drive space, again you are in luck, however – say it doesn’t.  Your options are:

  • add one more to the chain
  • create a bigger unit, copy everything to it, scrape the current one
I’ll stop my train of thought here, and leave you only with few things to think about before I go on with other requirements: who needs used 82 TB system (if you want to sell it), do you know how much it is to COPY 82 TB even at extreme network speeds, adding one more will break the ‘all in one place’ requirement, …

Having it all in one place (i.e. for web publishing)

If you need everything in one place for the publishing, then this is a solid requirement.  Web server will have the content on its local hard drives, and it will publish it smoothly.

But, is that really a requirement?  I must admit that I didn’t see web server that properly served files from the network locations (despite the thing that there are option to do that), but I’m sure that IIS and Windows Server gurus will be able to shut me down and say that this is normal thing that is done routinely.  So, if we know that each channel recorder has its own directory ANYWAY, what’s the use of having

c:\archive\channel1
c:\archive\channel2
c:\archive\channel3

instead of

\\rec1\channel1
\\rec1\channel2
\\rec2\channel3

Reducing single point of failure

This requirement is very common, and having NAS system as a ‘point’ it brings us that having the NAS leads us to having single point of failure.  I understand that there are multiple redundancies that could be installed into the system, such as RAID 5, or obscene configurations such as RAID 1+5.  Note: for later, it seems that the article author has a same opinion on it as me:

Recommended Uses: Critical applications requiring very high fault tolerance. In my opinion, if you get to the point of needing this much fault tolerance this badly, you should be looking beyond RAID to remote mirroring, clustering or other redundant server setups; RAID 10 provides most of the benefits with better performance and lower cost. Not widely implemented.

In my words: if you need such system, it’s better to have recording drives distributed on each recording machine, have RAID 5 there, and additionally have NAS (or some other form of storage) to DUPLICATE everything.

Bandwidth issues

At a configuration with 4 channels recorded at one machine, and with above mentioned data rate for video and audio, each machine will produce 5 megabit of content every second.  Roughly, that is .5 megabyte.  Even ZIP drive could almost handle that.  However, if you have 10 times that (for 10 recorders) and have central storage for the whole bunch of channels, that is 5 megabytes of data at a constant rate that never stops.

Consider that central storage in question is NAS has hard drives configured in RAID 5.  That means that it will have to receive, calculate parity for, move the drive heads, write to drive, …  It will be very busy NAS, and with everything else in mind, it won’t have a second of a break.  Add to that occasional reading of the content from the archive diggers, and you’ll soon figure out that the NAS will have to take it all itself.

On the other hand, archive access applications such as VideoPhill Player doesn’t have anything against having the channels on different recorder machines.

Conclusion (for bandwidth issues) – having each recorder machine handle both encoding and storage for 4 channels will reduce single point of stress for both recording and the archive access.

Overall…

Having dumped my intuition in this few paragraphs, I hope that I presented case that is strong enough against having NAS for video logger/archive storage.  Again, everything said is from my experience on the subject, and I’m no storage expert who will talk petabytes, just a simple consultant trying to get my clients best bang for the buck.  Please, I won’t mind objections to the text, on the contrary…