[cma-l] Chronicle ROT Logger Beta
Cal McLean
mail at callum-mclean.co.uk
Mon Mar 13 15:15:31 GMT 2017
Hi all,
Apologies for the delay in reply - I've been working loooong days on
Crufts (for my day job - I'm a TV vision mixer/engineer!)
As a couple of people have pointed out, using CBR MP3 does remove the
uncertainty about how much space a file will take up, and is then easily
calculable. At 320kbps (unnecessarily high, but equates roughly to using
-q:a 0 when using FFMPEG to convert to MP3), a 1-hour file will take up
~140MB. 42 days' worth is 137GB. Using WAV (44.1kHz, 16-bit stereo), you
will need 595GB of space.
With regard to sensible naming strategies ('semantic naming'?) to allow
for straightforward backup to Glacier/S3/Nearline/etc., I've just pushed
an update through that allows one to create directories automatically,
given a specific date/time format - e.g. 'chronicle -f
"%Y/%m/%d/%H_%M_%S" ' would record to the file '2017/03/13/15_00_00.wav'
when started on 13th March 2017 at 15:00:00. This is a common, standard
way of achieving date-time substitutions, often referred to as
'STRFTIME-compatible'. Separating out files into a sensible directory
structure not only makes archiving easier, but avoids long lists of files.
Thanks to everyone who's mentioned Chronicle on social media and the
like, and to James for including it in his newsletter!
I'm still very much welcoming ideas and thoughts, either on here or on
GitHub - http://github.com/calmcl1/chronicle
As always, more info is always available at
http://calmcl1.github.io/chronicle
Cal
On 06/03/2017 12:25, tlr at gairloch.co.uk wrote:
>
> We use a similar sort of scheme. We log hourly segments of all output
> at 128kbps MP3, plus wav files of all live output hours. We do that so
> that we have the legal logs of all output, plus high-quality
> recordings of all live material in case something goes out that we
> afterwards want to re-use or save for future re-use (eg a spontaneous
> interview). Of course we would get slightly better quality using AAC
> but MP3 is more convenient for various reasons.
>
> The logs are streamed to a 64GB USB key, and archived as MP3 files to
> a mirrored pair of drives in our main file server. We archive each
> show just after it has been recorded, and old files are deleted
> automatically, maintaining a minimum buffer of free space on the
> memory key. That means we can generally go back to live show wave
> files for around 6 months, or lower quality MP3 logs forever.
>
> A 4TB drive can hold about 15 years of 24x365 output at 64kbps, so
> there's a good chance a drive will fail before becoming full! For
> cloud storage a 1TB Dropbox account can similarly hold over 3 years
> worth, with the bonus that if desired it can provide online streaming
> 'listen again' playback for anyone who wants.
>
> In parallel with that, we run a background job on another PC maintains
> 7 weeks of 64k MP3 logs of all output just as a backup in case the
> main log fails or becomes inaccessible for any reason. Overkill maybe,
> but comforting!
>
> Alex
>
>> On 04 March 2017 at 23:13 James Cridland <james at cridland.net> wrote:
>>
>> I think my only point about file size is to answer the question "how
>> large a hard drive should I have?" If you wanted the answer - a 750GB
>> hard drive should be enough.
>>
>> Since this software is there to run a legally-required service, the
>> last thing you want it to do is to silently fail due to a lack of
>> disc space.
>>
>> The "never delete" archive comment - I can't see any requirement to
>> stream directly to S3 (and as you say, it's virtually impossible
>> anyway). I'd run an hourly, daily or weekly cron job to upload the
>> files - which you might want to ensure are in sensible folder
>> structure (/2017/03/12/0400-0500.wav) so you don't end up with an
>> unweildy file list. Using a separate cron job is less likely to hurt
>> the recording process, I'm assuming.
>>
>> Transferral to Amazon Glacier, incidentally, can be set to
>> automatically happen after a period of days with AWS. (For those
>> reading along - Glacier is an even cheaper place to dump your files,
>> with the drawback that if you want them, you need to request them and
>> they take about four hours to become available).
>>
>> This is good stuff - I'm sure that logging has been re-invented many
>> times, but anything that is free and simple to use (and, most
>> importantly, open for others to improve) is a good thing in my book!
>>
>> //j
>>
>>
>>
>>
>> On Sat, 4 Mar 2017, 19:43 Alex Gray <tlr at gairloch.co.uk
>> <mailto:tlr at gairloch.co.uk>> wrote:
>>
>> If you select fixed rate (CBR) MP3 coding, then file sizes are
>> pretty constant. We play out most material at 256kbps, and an
>> hour's programme is about 112MB, and the player's time to run
>> display is accurate.
>>
>> But if instead you select variable rate (VBR) encoding the rate
>> is adaptive to the material, and file length becomes variable
>> according to content. Also a player's time to run display becomes
>> only approximate, which is a big disadvantage for studio use.
>>
>> Alex
>>
>> But some people use
>>
>> On 4 Mar 2017 08:46, Tony Bailey <ravensound at pilgrimsound.co.uk
>> <mailto:ravensound at pilgrimsound.co.uk>> wrote:
>>
>> Correction: should read "mp3 files are always the same size"
>>
>> Hello all,
>>
>> I've no idea what this thread is all about - just one item
>> caught my eye: file lengths - I've been playing around with
>> an audio chunk upload using arecord with fixed file lengths
>> and the mp3 files are always the same length, the ogg files
>> vary according to content however.
>>
>> Tony Bailey
>>
>>
>> On 03/03/17 19:58, mail wrote:
>> Hi James,
>>
>> Thank you! That's really kind of you :D
>>
>> With regard to #1:
>> Being able to show a projected file size for a file of a
>> given length, channel count, bitrate, etc is fairly
>> straightforward for uncompressed files (such as WAV). The
>> default settings (16-bit, 44.1kHz, stereo) recording for an
>> hour will always produce a file of ~605MB. Compressed
>> formats, however, are harder to predict given that the
>> compression ratio overall depends on the content of the
>> audio. Still, it shouldn't vary too much and a rough average
>> figure should be sufficient - just not *perfectly* accurate.
>> See https://github.com/calmcl1/chronicle/issues/22.
>>
>> #2
>> Not deleting archives automatically is fine (see
>> https://github.com/calmcl1/chronicle/issues/21) if you want
>> to manage your archives directly, but a direct upload to S3
>> will take a little work, as one can't *stream* directly to S3
>> and must instead upload short chunks. Not impossible, though :)
>>
>> Unfortunately, a direct Glacier upload isn't possible as one
>> must know the *exact* file size before starting - you'd have
>> to wait for it to complete and then upload.
>>
>> Cold-storage/nearline archiving like this would be better
>> implemented by logging to a given directory and then running
>> an automated hourly/overnight backup operation to Glacier.
>> I've actually got some software for that, too, called Cupo
>> (github.com/calmcl1/cupo-backup
>> <http://github.com/calmcl1/cupo-backup>), but that's still
>> quite beta (though we do use it for our Myriad database
>> backups at HCR).
>>
>> Does this help, or have I missed the point entirely?
>>
>> Cal
>>
>> m: 07426 437449 e:mail at callum-mclean.co.uk <mailto:mail at callum-mclean.co.uk> w:callum-mclean.co.uk <http://callum-mclean.co.uk/>
>>
>>
>>
>> ---- On Fri, 03 Mar 2017 08:08:00 +0000 *James Cridland
>> <james at cridland.net> <mailto:james at cridland.net>* wrote ----
>>
>> This is nice - have tweeted it and it'll be in my newsletter.
>>
>> Suggestion #1 is to make it clear how much space this
>> will take on a local drive.
>>
>> Suggestion #2 is to archive to Amazon S3 or similar (or
>> point to a cron job that will do this for you). Please
>> don't delete archives - instead, Amazon Glacier will be a
>> really cheap place to store every single second of your
>> radio station that was ever broadcast. Imagine how
>> awesome that would be.
>>
>> J
>>
>>
>> On Thu., 2 Mar. 2017, 19:19 Canalside's The Thread,
>> <office at thethread.org.uk
>> <mailto:office at thethread.org.uk>> wrote:
>>
>> --
>>
>>
>> http://james.crid.land - get my weekly newsletter
>> https://media.info - the media information website
>>
>>
>> Tel: +61 447 692 743 | UK: 020 7100 1811 ('till
>> midday) | @jamescridland
>>
>> _______________________________________________
>>
>> Reply - cma-l at commedia.org.uk <mailto:cma-l at commedia.org.uk>
>>
>> The cma-l mailing list is a members' service provided by
>> the Community Media Association - http://www.commedia.org.uk
>> Twitter: http://twitter.com/community_media
>> http://www.facebook.com/CommunityMediaAssociation
>> Canstream Internet Radio & Video:
>> http://www.canstream.co.uk/
>> _______________________________________________
>>
>> Mailing list guidelines:
>> http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
>>
>> _______________________________________________
>>
>> To unsubscribe or manage your CMA-L mailing list
>> subscription please visit:
>> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
>>
>> I’ll get our tekky guys Simon and Blackers onto this
>> Cal …… might be worth a dabble and a play. I think
>> they’ve got bored of our Train set now and need a new
>> Toy to tinker with J LOL
>>
>>
>> Nick H Dumpty / Head Bottle Washer
>>
>> Canalside Calamity Radio
>>
>>
>> *From:* cma-l-bounces at mailman.commedia.org.uk
>> <mailto:cma-l-bounces at mailman.commedia.org.uk>
>> [mailto:cma-l-bounces at mailman.commedia.org.uk
>> <mailto:cma-l-bounces at mailman.commedia.org.uk>] *On
>> Behalf Of *Cal McLean
>> *Sent:* 01 March 2017 22:55
>>
>>
>> *To:* The Community Media Association Discussion List
>> <cma-l at mailman.commedia.org.uk
>> <mailto:cma-l at mailman.commedia.org.uk>>
>> *Subject:* [cma-l] Chronicle ROT Logger Beta
>>
>>
>>
>> Hi all,
>>
>> Forgive the shameless-self-promotion post - I was
>> wondering if anyone in the CMA might be willing to
>> try out a bit of software that's currently in beta
>> (though stable enough for production use)?
>>
>> As part of Halton Community Radio's upgrade, I ended
>> up writing an ROT logger, since none of the options
>> available to use particularly seemed to be good value
>> for what we needed. It's now running on a hand-built
>> computer in our production environment!
>>
>> However, I always feel with the bits of software that
>> I write that if they can be useful to me, perhaps
>> they can be useful to other people. If anybody might
>> want to play-test a bit of logging software, it's
>> available at:
>>
>> https://calmcl1.github.io/projects/chronicle
>>
>> The link above has much more information about the
>> state of the software, how it came about and what I'm
>> intending to do with it. It's all open source on
>> GitHub, so developer types are free to modify and
>> play as they wish :)
>>
>>
>> Cal McLean
>>
>> *Head Technician
>> *Halton Community Radio
>>
>> _______________________________________________
>>
>> Reply - cma-l at commedia.org.uk
>> <mailto:cma-l at commedia.org.uk>
>>
>> The cma-l mailing list is a members' service provided
>> by the Community Media Association -
>> http://www.commedia.org.uk
>> Twitter: http://twitter.com/community_media
>> http://www.facebook.com/CommunityMediaAssociation
>> Canstream Internet Radio & Video:
>> http://www.canstream.co.uk/
>> _______________________________________________
>>
>> Mailing list guidelines:
>> http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
>> _______________________________________________
>>
>> To unsubscribe or manage your CMA-L mailing list
>> subscription please visit:
>> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
>>
>>
>>
>>
>> _______________________________________________
>>
>> Reply -cma-l at commedia.org.uk <mailto:cma-l at commedia.org.uk>
>>
>> The cma-l mailing list is a members' service provided by the Community Media Association -http://www.commedia.org.uk
>> Twitter:http://twitter.com/community_media
>> http://www.facebook.com/CommunityMediaAssociation
>> Canstream Internet Radio & Video:http://www.canstream.co.uk/
>> _______________________________________________
>>
>> Mailing list guidelines:http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
>> _______________________________________________
>>
>> To unsubscribe or manage your CMA-L mailing list subscription please visit:
>> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
>>
>>
>>
>> --
>> Local Reports athttp://www.ravensound.pilgrimsound.co.uk
>>
>>
>> _______________________________________________
>>
>> Reply - cma-l at commedia.org.uk <mailto:cma-l at commedia.org.uk>
>>
>> The cma-l mailing list is a members' service provided by the
>> Community Media Association - http://www.commedia.org.uk
>> Twitter: http://twitter.com/community_media
>> http://www.facebook.com/CommunityMediaAssociation
>> Canstream Internet Radio & Video: http://www.canstream.co.uk/
>> _______________________________________________
>>
>> Mailing list guidelines:
>> http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
>> _______________________________________________
>>
>> To unsubscribe or manage your CMA-L mailing list subscription
>> please visit:
>> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
>>
>> --
>>
>> http://james.crid.land - get my weekly newsletter
>> https://media.info - the media information website
>>
>> Tel: +61 447 692 743 | UK: 020 7100 1811 ('till midday)|@jamescridland
>>
>> _______________________________________________
>>
>> Reply - cma-l at commedia.org.uk
>>
>> The cma-l mailing list is a members' service provided by the
>> Community Media Association - http://www.commedia.org.uk
>> Twitter: http://twitter.com/community_media
>> http://www.facebook.com/CommunityMediaAssociation
>> Canstream Internet Radio & Video: http://www.canstream.co.uk/
>> _______________________________________________
>>
>> Mailing list guidelines:
>> http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
>> _______________________________________________
>>
>> To unsubscribe or manage your CMA-L mailing list subscription please
>> visit:
>> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
>
>
> _______________________________________________
>
> Reply - cma-l at commedia.org.uk
>
> The cma-l mailing list is a members' service provided by the Community Media Association - http://www.commedia.org.uk
> Twitter: http://twitter.com/community_media
> http://www.facebook.com/CommunityMediaAssociation
> Canstream Internet Radio & Video: http://www.canstream.co.uk/
> _______________________________________________
>
> Mailing list guidelines: http://www.commedia.org.uk/about/cma-email-lists/email-list-guidelines/
> _______________________________________________
>
> To unsubscribe or manage your CMA-L mailing list subscription please visit:
> http://mailman.commedia.org.uk/mailman/listinfo/cma-l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.commedia.org.uk/pipermail/cma-l/attachments/20170313/b4e4891e/attachment.html>
More information about the cma-l
mailing list