Archive for the ‘tech’ Category

Newer Entries »

How to reduce hard drive fragmentation

Tuesday, May 17th, 2011

The topic of drive fragmentation might be a little out in this days, but since I spent great deal of my youth watching PC Tools defragment my drive in a graphically pleasing fashion, I am inclined to think that drive fragmentation (when excessive) can severely reduce both computer performance and hard drive life.

As this might be true for the common day-to-day user, it is particularly true for corporate/enterprises that do need their data to be:

  • accessible,
  • quickly accessible,
  • accessible for a long time

In a common computer use scenario, most of the files are there for computer to read an use, either as software that has to be loaded into memory, or documents that have to be shown to the user. Writing to the hard drive is uncommon operation (when you put it against the number of reads) and thus the drive fragmentation however present is in fact easily ignored.

Continuous stream recording, enter…

In my business (my clients businesses’ to be exact) the hard drives are working in opposite. They WRITE all the time, and read only on occasions. And the problem that will surely lead to fragmentation is that in most situations they need to write MULTIPLE long files continuously. adidas 2017 pas cher Let me try to explain what, first from the aspect of why, then move to what…

When either running VideoPhill Recorder for recording video, or using StreamSink to record internet media streams, in most cases user has MULTIPLE channels recorded on one computer. Files that are created by that recording are commonly created at one time (all of them) and are grown continuously until closed. Since Windows is, as it is now, an operating system that can’t reserve drive space in advance (maybe it can, but software doesn’t know how long the files would be) the space for them will be allocated as the time goes by. If we have 4 files that are written slowly but concurrently (and are grown at the same time), we’ll certainly have the following situation on the hard drive (I’m talking ONLY about the data that is stored here, and am simplifying physical hard drive storage as a continuous slate):

file1_block1
file2_block1
file3_block1
file4_block1
file1_block2
file2_block2
file3_block2
file4_block2
.
.
.
file1_blockN
file2_blockN
file3_blockN
file4_blockN

That means fragmentation. fjallraven kanken soldes File isn’t in continuous blocks, but is scattered in evenly and can’t be read sequentially from the hard drive. fjallraven kanken rugzakken You might be lucky and your blocks could be scattered in a way that sectors on the drive will be adjacent and this won’t pose a problem, but what are the chances? :)

And when file1 gets deleted, what remains on the hard drive? A blocks filled with nothing, left there for other files to fill them. New files will try to fill them, and the drive will soon be completely jumbled. It will all be hidden from you by the OS, but still, OS will have to deal with it.

And that is the story of 4 channels. What about situation when you have 60 channels recorded on one machine (I’m talking about internet stream recording, of course). Such an archive could be found here: http://access.streamsink.com/archive/

If you aren’t convinced that this really IS a problem, you can stop reading now.

Rescue #1 – Drive Partitioning

It is feasible in situations where there is low number of channels that needs to be recorded. If you have 4 channels, you’ll create 4 partitions, and each partition will have nice continuous files written to it. Done.

However, you can’t have 50 partitions on one drive and get away with it.

Rescue #2 – Queued File Moving

Other solution for large number of channels presents itself in a form of a temporary partition for initial file recording, and then moving out the files to their permanent location later, but ONE FILE at a time, in a queue.

Queued Moving of Files in StreamSink

This is implemented in StreamSink, and it even has an ability to throttle data rate when moving the files to another drive. adidas schoenen Only thing that is of a problem here is wasting of a temporary hard drive, because it gets beaten by fragmentation.

Rescue #3 – Using RAM Drive on Method #2

While I was writing the article about NAS, thought flashed across my mind – can we avoid writing to the temporary drive and reduce the load ONCE more?

Yes, we can. Mochilas Kanken No.2 I know that RAM Drives are also out of fashion, but here one will come handy. It’s the shame that support for it isn’t included in the system already, so with little googling I found this: http://www.ltr-data.se/opencode.html/#ImDisk

I installed it on the testing server, re-configured the application to use new temporary folder, and from now on, it runs so smooth I can’t hear it anymore :)

Some technical stuff:

  • in this instance, I am currently recording 62 channels and cumulative rate for it is around 5 megabit/second
  • my files have duration of 5 minutes, which means that recorded chunks are closed and moved to permanent storage every 5 minutes
  • during those 5 minutes, each file will grow so much that the whole content for those 5 minutes won’t get over 200megabytes
  • I created 512 megabyte ram drive, just to be safe

Conclusion

Take care of your hard drive, and don’t dismiss old-techs such as RAM Drives just yet.

If I was about to implement this on an application level, I would have to spend a great deal of time, and some media types won’t even be possible to implement – Windows Media for example, writes to disk or to other places if you employ magic…

Tags: , , ,
Posted in tech | 8 Comments »

Having NAS is great (or is it?)

Tuesday, May 17th, 2011

During the years I had many deliberations over the fact if either NAS would be used or it wouldn’t be used for the video archives created by video logger system such as VideoPhill Recorder. air max pas cher At first, I was firm believer in one methodology, then completely turned my side to the other, and now, … Well, read on, and I’ll take you through it.

Stakes

On the one side of the stake set we have actual requirements, and on the other there are considerations. Actual requirement are sometimes hard to pinpoint at first, but they always come out sooner or later.

So, let me list possible requirements that might be in effect here…

Common low level requirements

Storage for video recording (for logging purposes) needs to have following abilities:

  • low but constant and sequential write rate – data rate for 4 channels are as low as 5mbit per second (500kbytes/sec) but is CONSTANT and SEQUENTIAL – there won’t be much stress for the hard drive because of constant seeking
  • high durability over time – what gets written once, has to be there. Nike Air Max TN Homme It should survive single drive failure
  • reading isn’t common, but when done, it has to be sustainable, but again at low data-rate and great predictability (it usually is sequential)
From everything above, I can guess that any data storage expert would read RAID 5 and won’t allow you to create anything else for the video archive storage.

Archive duration scalability

The archive duration is directly proportional with the hard drive space that is available. To determine what kind of hard drive space you need for your first installation, you can use on-line hard drive size estimator calculator that I created right for this blog.

If you plan extend the duration of your archive one day, you have this requirement, and you have to plan for it. Having the storage at one place can simplify the adding of the drive space, but can also completely block it.

Let’s say that you have the archive of 40 channels that span 92 days (3 months). And let’s say that you decided to use 1mbit video with 128 kbit audio for the archive. By using the calculator above, you’ll find out that you have 42 terabytes of storage already in place. Even at this date, that kind of storage set in one place is kind-of-a challenge to build.

If you have already invested in 42 TB storage system, and have foresight to plan for an upgrade to say its double size for it, you are in luck. But, say that after a few more months (just 3) your management decides to expand the requirement to 12 months of storage. Wow. nike air max 90 pas cher Now, you have to have 127 TB total. If the current system will hold that much drive space, again you are in luck, however – say it doesn’t. Nike Air Max 2016 Heren blauw Your options are:

  • add one more to the chain
  • create a bigger unit, copy everything to it, scrape the current one
I’ll stop my train of thought here, and leave you only with few things to think about before I go on with other requirements: who needs used 82 TB system (if you want to sell it), do you know how much it is to COPY 82 TB even at extreme network speeds, adding one more will break the ‘all in one place’ requirement, …

Having it all in one place (i.e. for web publishing)

If you need everything in one place for the publishing, then this is a solid requirement. Scarpe Adidas Web server will have the content on its local hard drives, and it will publish it smoothly.

But, is that really a requirement? I must admit that I didn’t see web server that properly served files from the network locations (despite the thing that there are option to do that), but I’m sure that IIS and Windows Server gurus will be able to shut me down and say that this is normal thing that is done routinely. So, if we know that each channel recorder has its own directory ANYWAY, what’s the use of having

c:\archive\channel1
c:\archive\channel2
c:\archive\channel3

instead of

\\rec1\channel1
\\rec1\channel2
\\rec2\channel3

Reducing single point of failure

This requirement is very common, and having NAS system as a ‘point’ it brings us that having the NAS leads us to having single point of failure. I understand that there are multiple redundancies that could be installed into the system, such as RAID 5, or obscene configurations such as RAID 1+5. Note: for later, it seems that the article author has a same opinion on it as me:

Recommended Uses: Critical applications requiring very high fault tolerance. In my opinion, if you get to the point of needing this much fault tolerance this badly, you should be looking beyond RAID to remote mirroring, clustering or other redundant server setups; RAID 10 provides most of the benefits with better performance and lower cost. Not widely implemented.

In my words: if you need such system, it’s better to have recording drives distributed on each recording machine, have RAID 5 there, and additionally have NAS (or some other form of storage) to DUPLICATE everything.

Bandwidth issues

At a configuration with 4 channels recorded at one machine, and with above mentioned data rate for video and audio, each machine will produce 5 megabit of content every second. Roughly, that is .5 megabyte. Even ZIP drive could almost handle that. However, if you have 10 times that (for 10 recorders) and have central storage for the whole bunch of channels, that is 5 megabytes of data at a constant rate that never stops.

Consider that central storage in question is NAS has hard drives configured in RAID 5. That means that it will have to receive, calculate parity for, move the drive heads, write to drive, … It will be very busy NAS, and with everything else in mind, it won’t have a second of a break. Add to that occasional reading of the content from the archive diggers, and you’ll soon figure out that the NAS will have to take it all itself.

On the other hand, archive access applications such as VideoPhill Player doesn’t have anything against having the channels on different recorder machines.

Conclusion (for bandwidth issues) – having each recorder machine handle both encoding and storage for 4 channels will reduce single point of stress for both recording and the archive access.

Overall…

Having dumped my intuition in this few paragraphs, I hope that I presented case that is strong enough against having NAS for video logger/archive storage. Nike Air Max Goedkoop Again, everything said is from my experience on the subject, and I’m no storage expert who will talk petabytes, just a simple consultant trying to get my clients best bang for the buck.

Tags: , , , ,
Posted in tech | 1 Comment »

And then, I got in… (story of data visualization)

Sunday, May 15th, 2011

How to see the data?

If the data is numeric, and it represents some series, it will be mostly represented with a graph of some sort. There are hundredths types of graphs available, and they all have some purpose, otherwise they would not exist.

However, for some special occasions, you have to see different kind of data.

The problem (this particular instance)

Since I am developing a internet media streaming CAPTURE and ARCHIVE application (StreamSink) I am also continuously testing it on one of my servers. I am adding channels, removing them, stopping the server, sometimes something goes wrong and the whole thing freezes or crashes, so the archive I have is rather heterogeneous in quality.

Let me go through the operational view – the mere GUI of the StreamSink, so I can present some problems and solutions so far.

StreamSink

Several things were important to the operator of the software that had to be present on the main (status) screen. For example:

  • whole list of channels should be visible
  • channel status should be visible at first glance
  • I am interested what happened to the system recently
  • I need to know the status of my connection
  • it would be good to know how many disk space is available

I could dwell on it but the main point of this post is something else.

The problem here is that I had to create PlayKontrol report for a demonstration purpose (for them: http://ihg.hr/), that would scan 7 days of the archive (multiple channels, of course), and produce the reports (playlists) for 300 songs.

So the problem is: to

find, in the archive that is damaged in various ways, 7 days of continuous archive that spans multiple channels.

The solution (prelude)

Since I am kind of explorer by nature, I wasn’t inclined to use a solution that would present raw data as an answer, but was into thinking about seeing the data and determining the period and channels ‘visually’.

StreamSink has a integrated feature that is called ‘archive report’, that has data similar to what I need, but with it I would only get limited information. You can see the report here:

StreamSink Archive Report

Most useful info on the report in this particular situation would be the graph on the right side of the report. Nike Air Max 2016 Heren wit Let me explain…

For each day StreamSink is able to record up to 24 hours of media. Fjallraven d’Occasion Due to network situations, it sometimes is less then 24 hours, and I decided that I would present that number in the form of percentage that archive is covered for the day. As you can see from the report, that percentage is shown for the whole archive lifetime, for last month, last week and last 24 hours.

Also, it is shown in the form of graph, where on the leftmost part of the graph is the current day, and as we go to the right, we sink onto the past, having divider lines at each 7 days. air max 1 pas cher Nice, eh? :)

But, as nice as that report is, I can’t read what 7 days and what channels are to be scanned – I have to find another way in.

Solution (at last)

For this one, I picked something that I learned from the above mentioned report. Cheap Fjallraven Kanken Outlet That was:

  • I will have a channel list
  • I will have some sort of calendar
  • I have to see how much is covered for the archive for each day

Also I decided to show each day as a cell in a table-style matrix, where rows would be occupied by channels, and columns will be days. Time flow was inverted here, so left is past, and right is the present.

Whole thing looks like this:

Archive Digger

Same thing little zoomed in:

Archive Digger Detail

Note: green is the color for the days that have 90% or more archive covered.

At last, you can see from the both pictures that much of the data is revealed at the first glance. For example, 0 means that there were no archive that day at all. Nike Max Shoes UK Numbers below 90 suggest that either it was some problem with the channel that day, or StreamSink was either started or stopped in the middle of the day.

I could even color-code that information on the chart – but the utility will be expanded further only if there’ll be demand for it, since I know what I needed to know, from it.

BTW, I don’t want to brag here, but to code that utility it took 2-3 hours of thinking and coding, and almost no debugging.

Tags: , , , ,
Posted in tech | No Comments »

Windows Media Encoder – loss of conection

Saturday, May 23rd, 2009

When using WME (look at the title) common problem is that out-of the box application doesn’t handle connection problems when using PUSH method. air max pas cher We adressed that problem with VideoPhill Recorder and made so it reconnects automatically every time connection is lost. ray ban femme pas cher So you won’t lose any effort for reconnecting to the server mannually if your connection is shaky.

So, Asics Pas Cher if you have VideoPhill Recorder, nike goedkoop and want to do streaming from the same computer,

Tags:
Posted in tech | No Comments »

Bitrate Calculator

Friday, May 22nd, 2009

In contact with customers, or prospective customers, hard drive capacity questions are most common ones. new balance Chaussures To that effect, I prepared two tools: one being PDF file with various bit rates, days, and hard drive spaces, and other being this little calculator that you can use on-line to help yourself see what hard drive you need.

One comment though: manufacturers of hard drives usualy have 1000 where should be 1024. new balance femme blanche 996 That means that 1G is 1.000.000.000 bytes for the manufacturer, and in reality it is 1.073.741.824 bytes. Kopen Nike Air Max 2017 Goedkoop So, after making a calculation with tool below, please add 7.3% to the hard drive space calculated.

Please note: you should enter all values EXCEPT the one marked with radio button. nike air max That values will be calculated when you press ‘calculate’ button.

Here is the table: hard-disk-bitrate

Calculator below can be used to calculate all combinations of values – for example, if you have fixed hard drive space, and fixed days that you need to record, you can use it to estimate bitrate you should use. Under Armour CurryPas Cher Just click on the radio button left of bitrate field, and fill all of the remaining fields. air max soldes On the other hand, if you need to fix your bitrate, and you have disks of fixed capacity, and want to see how many days of recording that will come to, just click on radio button left of days field, fill in rest of the stuff and click calculate.

When experimenting with various drives/bitrates/days etc. you can make yourself a table of values for comparation.

Tags:
Posted in tech | 2 Comments »

Windows Services to Kill

Thursday, May 21st, 2009

VideoPhill Recorder is a dedicated software, and it should have it’s own machine. Nike Air Max 90 Pas Cher Pour Homme So, bare Windows XP, all the updates, all the right drivers, and that’s it. baskets ASICS By default, if you open Services in Computer Management, you’ll see that many of them are started, and they also have start-up type set to Automatic. Nike Air Max 1 Heren Which means they start with Windows, regardless of their usage. They eat memory, and some of them eat CPU and other resources. lunette de soleil ray ban So I went further and compiled a list of services that VideoPhill users could and should kill, thus freeing about 400MB memory when system is running. cheap adidas uk Other Windows XP user could do the same, but maybe to lesser extent.

Here’s the list:

  • Automatic Updates
  • Cryptographic Services
  • Distributed Link Tracking Client
  • Error Reporting Service
  • IPSEC Services
  • Net Logon
  • Print Spooler
  • Protected Storage
  • Remote Registry
  • Secondary Logon
  • Security Center
  • System Restore Service
  • Task Scheduler (if not used)
  • Themes
  • WebClient
  • Windows Firewall/Internet Connection Sharing (ICS)
  • Wireless Zero Configuration (if you don’t use WLAN)

I also plan to make a script, batch file that will make all the services have start-up type = manual.

Tags:
Posted in tech | No Comments »

VolumGuard feedback

Friday, May 15th, 2009

I recently released VolumGuard application prototype to few trusted colleagues that I know that will give me proper feedback on it. VolumGuard is an application that is used in radio broadcast environments and which monitors live audio input (vocals from speaker/talent) and when it goes above certain trigger level drops the output level on selected wave output devices in the PC machine.

Here is the screenshot, but please, I know it’s ugly, and I’m trying to concentrate on functionality first…

VolumGuard Screenshot

VolumGuard Screenshot

You might already notice – application function is trivial, and user interface looks complicated. new balance 574 homme gris I know that is horrible way to go. Fjallraven Kanken Baratas It was supposed to be other way around. Nike Air Max Norge So, some phone calls were concerned about it…

One thing I can do here is to remove all of the sliders from the ‘main screen’ of the application and make another main screen that will be very simple and will show that application is running fine. From that screen, user could select ‘options’ and be given this beautiful and scary window.

Whole application is made in MFC, which is something I got used to many years ago, and for small apps I find it great. Nike Air Max 2016 Heren grijs I made graphing control myself, and it’s rather flexible, is able to scroll to the left and draw various real-time data. kopen nike air max 2017 It also supports various markers, both horizontal and vertical.

What can be done here to improve all this? Here are some user suggestions:

  • to be able to turn the volume monitoring on and off
  • to be able to switch profiles with the hot-key
  • to be able to switch volume monitoring with the hot-key
  • to run program in the system tray – pulling it out with the hot-key

For that, I’ll need some kind of global hot-key hook. Cheap Nike Air Max Shoes UK I don’t know how dangerous are they to the system, but since there are many programs that use them, I guess that it will be ok for me to use them as well.

Tags:
Posted in tech | No Comments »

Medianet visited

Wednesday, May 13th, 2009

In our continuing mission to find new customers and enable them to remove their VHS bases video archives, we found medianet.hr company. Goedkoop Adidas Schoenen They do their press-clipping and media-clipping work and they really need video, audio, and other types of archives. So, we engaged to provide video services for them.

I can say that guys there were really interested in our work. You can’t really know how motivating that is. Their main concern is their enormous archive that goes several years in the past, and is stacked onto the shelves of their office.

One repeating issue was brought up on the meeting. That is our renown DVD export for VideoPhill player. nike tn noir femme So, with so many interested parties, it seems like we are obliged to make it.

On to the Internet we go… nike air max pas cher to find something that will enable us to do so. I can take two roads here: GPL road and fully commercial road. One leads to ffmpeg with dvdflick (or parts of it), and other leads to MainConcept.

Also, something that has troubled me for many times, is that they pay massive fees annually to the company that provides software for video clip recognition. Since that feature lies exactly on my interest path, I guess that I will try to hit that road also. I am thinking that some basic clip recognition could be build, but to fully test it we’ll need massive archive. So another curious task, to build one. My flat/office already is starting to look like a hi-tech warehouse, who know where it will go from there.

And at the end – it’s now very clear that stream logger project that I started as a pet project needs to be produced commercially. There really is a strong need for such

3 ways to check out in 6 days buy instagram followers that like three ways to check out in eight days

a product, or a service that we could provide to them. nike air max 2016 wit Again, to test it and develop it massive storage space should be provided. nike air max 2017 pas cher I found out that Addonics has very interesting gadgets to that effect.

Tags: , , ,
Posted in partners, tech | No Comments »

Hard Disk Drive : Safer Reliable Databank

Tuesday, May 12th, 2009

Which Media???

Hard Disk Drive(HDD)images1

  • A hard disk drive (HDD), commonly referred to as a hard drive, hard disk, or fixed disk drive, is a non-volatile storage device which stores digitally encoded data on rapidly rotating platters with magnetic surfaces.HDD, itself, was built with multiple read-write, should come as no surprise that this format’s biggest advantage, millions rewrite times.

Digital Versatile Disc(DVD)dvd

  • DVD, also known as “Digital Versatile Disc” or “Digital Video Disc,” is a popular optical disc storage media format.The wavelength used by standard DVD lasers is 650 nm, and thus the light has a red color.Since DVD ROM stands for Digital Versatile Disk – Read Only Memory, it should come as no surprise that this format’s biggest disadvantage is that it can’t be used more than once.

How much Storage???

HDD: Large storage capacity

  • A typical desktop HDD can store between 120 GB and 2 TB of data (on Current US market data)

DVD: Medium storage capacity

  • 4.7 GB (single-sided single layer)
    8.54 GB (single-sided double layer)
    17.08 GB (double-sided double layer)
  • Lower Data Transfer/access speed

We all need speed/fast access???

HDD: Transfer/Access Speed

  • HDD Rotate at 5,400 to 10,000rpm and have a media transfer rate of 1Gbit/s or higher (1GB = 109 B; 1Gbit/s = 109 bit/s) Stores and retrieves data much faster than DVD/CD.
  • As of 2008, a typical 7200rpm desktop hard drive has a sustained “disk-to-buffer” data transfer rate of about 70 megabytes persec.
  • Latest, 3.0 Gbit/s SATA, which can send about 300 megabyte/s. from the buffer to the computer, and thus is still comfortably far ahead of today’s DVD-to-buffer transfer rates.
  • Most external hard-disk-drive cases with FireWire or USB interfaces. eSATA, standardized in 2004, provides a variant of SATA meant for external connectivity. Full SATA speed for external disks (115MB/s)
  • eSATA attracts the enterprise and server market, because of its hot plug (and online) capability and low price.
  • Access speed does not reduce, with increase in capacity.
  • Cheap on a cost per megabyte compared to other storage media.(DVD/CD).Recent market survey shows 1TB HDD costs starts from $60 above
  • Hard disks can be replaced and upgraded as necessary Can have two hard disks in a machine, one can act as a mirror of the other and create a back up copy.

DVD: Transfer/Access Speed123

  • Access speed reduces tremendously with increase DVD space.
  • Very Costly, to get same 1TB of space ($200), moreover not available on 1 disk, multiple DVD Disc’s needs to be used.
  • DVD Duplication/data copy requires burning a blank DVD, which is software & DVD-writer speed dependent process, tedious time consuming process, has compatibility issues with uses on different drives

Longer media life???

HDD: Longevity/Degradation

  • HDD’s have a longer shelf life, Hard disks and cartridges will last longer because the disks are rigid, Compact, Enclosed. More than 15 years is Life expectancy.
  • Because of casing, compactness, external factors have least effect, on internal mechanism.

DVD: Longevity/Degradation

  • DVDs and CDs have a shorter shelf life.DVD/CD technology uses a dye that can fade, especially if exposed to UV light. You should expect life for 5 years
  • Environmental forces will degrade the data layer much faster than the polycarbonate substrate layer (the clear plastic that makes up most of the disc).

Where is Web accessibility???

HDD: Online Storage

  • RAID combines two or more physical hard disks into a single logical unit by using either special hardware or software. Access rate, transfer rate is best for network storage.
  • Cost per online Gigabyte: $0.55
  • Share HDD online: Most simple way is to install a free FTP server (software) like FileZilla, give certain people access to your files through an FTP client or web browser.
  • With portable USB HDD, you can carry your large data (GB’s), and get online anytime. Adidas NMD Dames Only USB port needed. asics gel lyte 3 pas cher (Available on 9 out of 10 machines)

DVD: Offline Storage only

  • Used for Offline storage; very sluggish access over Local Area Network.
  • Sharing across web (online) is not possible.
  • For mobility, of Carrying large data(GB’s), one needs multiple DVD’s, with availability of DVD-Rom on that machine

More for Less: Data storage density???

HDD: Data Storagenew-picture

  • Data is stored in a very orderly pattern on each platter. Bits of data are arranged in concentric, circular paths called tracks. Nike Air Max 2016 Heren Each track is broken up into smaller areas called sectors. Part of the hard drive stores a map of sectors that have already been used up and others that are still free.
  • When the computer wants to store new information, it takes a look at the map to find some free sectors
  • Typically, Data up to 100 GB’s can be store on single platter
  • With so much information stored in such a tiny amount of space, a hard drive is a remarkable piece of engineering. nike air max 95 pas cher That brings benefits

DVD: Data Storage12344

  • A DVD is composed of several layers of plastic, polycarbonate base, totaling about 1.2 millimeters thick. Chaussures Nike Pas Cher Writing data to the DVD is done by a red laser beam modulated by the serial data stream.

Posted in tech | 1 Comment »

Newer Entries »